I will describe the situation and some of the problems that we have currently:
- l10n nightlies happen in old dedicated machines instead of running in generic buildslaves. This means if the machine fails, there are no nightlies at all. (Read more about inconveniences of not using generic machines in John O'Duinn's blog)
- l10n nightlies are not been kept in our servers (unless I am mistaken); for each locale the only one build available is the "latest" one (latest-trunk-l10n, latest-Mozilla1.8-l10n) which gets overwritten every night. By being "kept" means that we don't keep the l10n build of a specific day, even though in the tinderbox's log you always get a URL, to grab the file (log, e.g. nightly/2008-05-17-09-trunk-l10n)
- The previous might be a desired effect, since l10n repackages are not computational costly but storing them it is, all you have to do to get an l10n build is ... ... ... ... ... (I am sorry, it is not that straight forward). I have written this past Friday a little script that does some parts of that process for an specific locale, in this case, "af" (--->create locale script<---). The mozconfig used from the source code has an l10n tag (There are 4 tags: l10n, l10n_release, MOZILLA_1_8_l10n, MOZILLA_1_8_BRANCH_l10n_release). From running the previous script, I obtain the "af" .dmg file (inside of browser/dist) and the af.xpi (browser/dist/install/mac-xpi), which are the files that I want for a mac
- We want to make l10n repacks to happen in such way that can be scalable. Currently it works like this (which is not scalable):
- We do a "make configure"
- Download all locales
- Download from nightly/latest-trunk the latest en-US archive file (Isn't it dangerous to download something that can be overwritten while downloading it??)
- Do the repack for all locales (~1min per locale)
- After that, we upload all locales
Solutions:
- A better way would be to break this down by 1) creating a locale 2) upload it and 3) announce it. This would allow to do l10n repackaging in parallel which would be a great improvement
- This would also allows to generate "just" one locale if needed. Right now, we have to generate all of them even if we just want one
I don't want to prolong this by explaining how the l10n repackaging works currently but I will do so when we file this week the bug that will cover this project and will allow more discussion
NOTE = Just FYI we do l10n repackages when we have releases in the automation system instead of using the old tinder client machines and is just slightly different by using Bootstrap/Step/Repack.pm
Hi Armen, here's some comments on your post.
ReplyDelete"l10n nightlies happen in old dedicated machines instead of running in generic buildslaves". It's true that we use dedicated machines for Firefox 3.0 l10n nightlies (eg l10n-win32-tbox) but they are the same reference platform as the en-US builders (eg fx-win32-tbox, fx-win32-1.9-slave2). The machines on Mozilla1.8 are certainly all old and crufty, and not all the same.
The link that tinderbox gives to the non-existent l10n nightlies is a bit misleading but I suspect most localizers don't look at that page. Instead they tend to look at a locale specific one that never has a D link. We could fix it here if we're sticking with tinderbox.
You're right that we don't keep every day's l10n builds because they would use a lot of space, but also because it's not a very efficient use of space. I mean that it can be very useful to go back to find which build broke bookmarks or rendering, but not so much when a localization change took place. Bonsai is better for that.
"Isn't it dangerous to download something that can be overwritten while downloading it??" Maybe a little bit. The nightly for en-US is the first build to start after 4am PDT. On a normal day those will be done by 7am, but it could be later if the tree is burning. The l10n nightly is the first build to start after 10am so there is a good safety margin there. Only a nightly (aka clobber) build writes into latest-trunk, so it's only risky if a 2nd nightly is triggered and happens to finish when the l10n box comes along. Pretty low odds really.