Tuesday, February 05, 2008

armenzg: Sess12: Automated tests running and getting results

As you know from last post I have set up buildbot on my laptop to build Firefox and run automated tests every 50 mins.
I started to build properly around midnight when BUILD #5 started which was completed successfully.

Let me talk about the results of BUILD #6(which has already build once on BUILD#5), #7 & #8:

BUILD #6
BUILD #7
BUILD #9
checkout and compile2-3 mins2-3 mins2-3 mins
MAKE CHECK1hr 8mins1hr 20mins42 mins
REFTEST10mins20mins20mins
CRASHTEST4mins1min1min
MOCHITEST22mins31mins30mins
MOCHICHROME6mins11mins12mins
BROWSER CHROME2mins2mins2mins
TOTAL
~2hr05mins
~2hr28mins
~1hr50mins

I have marked in red those values that are over 20mins
We can easily notice that testing takes A LOT OF TIME:
  • Estimated time for all 6 tests: 2 hours
  • DEP BUILD takes 2-3mins
  • a CLEAN CHECKOUT & CLEAN BUILD takes 41mins (value comes from BUILD# 5)

COMMENTS

  • The buildbot's waterfall view is really interesting and allows you to view how the build is going, but the summaries of the tests are really poor and were meant to output on Tinderbox, therefore the "
    " that are shown when looking the summaries of the tests

PROBLEMS

  • I had FF2 open to see the results of the build and when it reached the mochitests it required to open FF3, therefore, there was an alert saying that Firefox was open bla bla bla.
  • At that point, I thought that it waits for me to press "OK" and then run the test but if so, how did I got the results during of builds 6, 7 & 8 if I was sleeping?
  • I remember that during the night time I woke up and I had to select "OK" and it seems that only after that it run
  • I have tried to modify the "perl runtest.pl" step that runs the mochitest by adding "--setenv=MOZ_NO_REMOTE=1" or "--appname='../../../bin/dist/firefox --no-remote" but this did not make any change (BTW I restarted to be able to do apply the changes) and I have also tried to run it manually and I would still get that alert that would wait for a user "OK"

QUESTIONS

  • shall I try to set up a "try server" style set up on my computer? Adam suggested not too, because he will take care of doing it for the Hera cluster
  • would it be interesting to run tests in parallel? Let see what the timings are in the Hera cluster, I bet is better than in my laptop. I believe it could be done if a slave would pass to other computers the BUILD OBJDIR over the net and give them the steps to run the MAKE CHECK (which takes the longest out of all tests) and get the results back
  • which others things shall I add to my steps? Bonsai-poller? Allow clobbering?
  • we should set up a way for the developer to get in their inbox the results of the build/test

In this week's iteration I do the same but with Cairo and see what Adam does with the Hera cluster and try to put my stuff there

No comments:

Post a Comment