The roads I take...

I've been talking about it in weekly status updates for a while, now it's in a state where I can publicly link it: The Build System Porting Tool I've set up to track changes in the Mozilla build system that (may) need to be ported to the comm-central build system files.

The tool's database knows about over 2200 "changes" (one change per Mercurial revision and file, i.e. one revision touching 3 tracked files create 3 "changes"), and people with edit access to the tool can flag any such change as ignored or ported as well as add a bug report and revision for the port. Currently only site admins have edit access, i.e. that's only me, but the code should be fairly easy to adjust to give other people such access as well (this solution was just the fastest to get the tool working well enough at all).
So, the really large part of the work was to narrow down those >2200 changes and eliminate all that can be ignored (e.g. not needed to be ported, before comm-central creation point, listed on 1.9.1 but before branch point) and mark all that have already been ported, of course with the correct information. Coming down to about 350 changes was easy by ignoring changes by time (branch point, etc.) but then it needed me to take a look on every single change to determine if it needs work. I did this over the last few weeks, and now we're down to just under 190 changes we still need to take a look at.

Serge has been working on a number of those we need to be on par with 1.9.1 recently, to get up to speed with mozilla-central we'll need a good pile of more work - but now we have a way to track all this.

As discussed and decided in the last SeaMonkey Status Meeting, I've started to move SeaMonkey to the new buildbot configurations that have been in the testing phase for a while now and will open the possibilities for doing builds based on mozilla-central soon.

The following changes are made that you should be aware of:

Phase I (done yesterday, first nightlies from today):

• The nightlies in latest-comm-1.9.1 are the official ones now.
• FTP has latest-comm-central and latest-trunk symlinked to latest-comm-1.9.1 for the first part of the transition.
• Those are static builds with reduced build size and hopefully better performance.
• Dependent builds ("build" columns, "hourlies") continue to be shared builds.
• Linux has a "leak test build" column that runs debug builds.
• Nightlies and dependent builds are being run on x86_64 Linux, even though this should still not be considered a tier-1 platform and has no L10n builds.
• Localized nightlies are in latest-comm-1.9.1-l10n and built with the "L10n merge" tooling that injects en-US for missing strings, eliminating most cases of the "yellow screen of death".
• The SeaMonkey tinderbox waterfall page has those buildbots reporting as columns containing "comm-1.9.1" in their names.
• Due to ongoing slight instabilities in the 10.5 VMs, the old "comm-central" 10.4 unit test machine stays up there as well.
• All other comm-central builds disappear from the scene.
• For localizers, the comm-1.9.1 builds on their Mozilla-l10n-* waterfall pages are the relevant reference.
• All VMs on the same platform run in generic pools, so that any of the machines can do any dependent build, nightly, L10n repackaging or unit test run, the buildbot master hands those jobs on a "first come, first serve" basis to the available slaves.

Phase II (once bugs 502031 and 502033 are solved):

• Builds with mozilla-central are set up in addition, uploading nightlies to a latest-comm-central-trunk directory.
• A version number of "2.1a1pre" is used there for now, there's no definitive decision on what version the next SeaMonkey will be yet though, this is the lowest possible version to use atm.
• FTP symlinks for latest-comm-central and latest-trunk are removed.
• The comm-1.9.1 builds are switched to report to a new SeaMonkey2.0 tinderbox waterfall page (as well as the 10.4 tests).
• The comm-central-trunk builds report to the SeaMonkey page.
• No unit test builds will be activated for comm-central-trunk while we don't have all the machines we need in the pools (due to Parallels network problems).
• L10n builds for comm-central-trunk use l10n-central work and also report to the Mozilla-l10n-* tinderbox waterfalls, dashboard for this might only come up some time later.

Phase III (once bug 493321 is solved):

• Unit tests for comm-central-trunk are turned on.
• A few other incremental improvements can be thought of when Phase II is completed, such as running packaged tests, AUS for L10n, etc. but all those can be done individually, step by step, then.
• Once 10.5 Mac VMs are completely stable and not intermittently crashing because of Parallels issues, we'll turn off the 10.4 unit test machine and hand it back to Mozilla IT.

I hope this whole transition works well for everybody. It was a bumpy ride and a good amount of work to get this all up, but it largely reduced the amount of maintenance needed and makes us share more buildbot code with Firefox and Thunderbird already now and even more once this all is completed.

If anyone of you felt that I was unusually unresponsive in the last few days, this had a good reason - I was deeply buried in intense work:

After Mozilla IT has purchased and basically set up a Parallels XServe, they handed over 8 VMs with base systems to SeaMonkey (we're playing guinea pig for Parallels and virtualized Macs within Mozilla) - that is, I got the build machine names and passwords. Of course, that doesn't mean that everything we need is suddenly working.

I basically spent almost all my time Wednesday through Friday bringing up 3 old Linux reference platform images, 2 pretty current Windows reference platform images, 2 fresh Mac OS X 10.5 installs and a fresh CentOS 5.0 x86_64 install up to really current Mozilla reference platforms and to having everything up to serve as buildbot slaves and all my time today to get a buildbot master up on one of them, do several small fixups to the machine configurations, improving my work on abstracted paths in buildbot factories and basing new SeaMonkey buildbot configs on those, using generic slave pools per platform to do the job. After a number of fixes to get the master to start correctly, I started the slaves and hooked them up - of course, just to notice more errors in the configurations that needed to be corrected.

In any case, the master now has columns for normal ("hourly") builds, and nightlies for all of Linux, Linux64, Mac OS X, and Windows, unittest builders on Linux, Mac and Windows, and debug/leaktest builds on Linux as well as all the columns it uses for release automation. The Linux and Windows slaves as well as one OS X slave are up and trying to satisfy all build requests that come in for them from the master. One OS X machine had strange problems with Terminal.app and then hung up during reboot (even though both have the same reference setup), but eventually it did and just joined the pool; and I'll only care about the 64bit stuff when the rest is working well. I currently have builds reporting to SeaMonkey-Ports on tinderbox, as that waterfall had been unused for a longer period of time, even though I'm currently building standard 1.9.1-based SeaMonkey, no actual "port" of any kind.

So far, only the unit test builds are actually working reasonably, though all builds do compile SeaMonkey fine, we're running into either buildbot config issues (abstracted paths need more fixups for leak test builds) or missing SeaMonkey build features (no "make upload" target yet) right now.

Still, this is major progress and we should now basically have the infrastructure to do both trunk and branch builds in the future, as well as doing releases based on the same automation harness used by Firefox. I only need to get the configurations going for all that, and we need to figure out a few things like what to do about with the old 10.4 Macs vs. the new 10.5 ones (probably can't just go into the same slave pool) and some disk space issues on our old slaves.

In any case, we're one large step into an improved future for SeaMonkey building and releasing architecture. Thanks to the Mozilla Community Giving program for getting us those machines, Mozilla IT for operating the hardware, setting things up and giving us technical support for doing so, and the Build and Release team for creating so nice buildbot configs we could rip off and generic config code to share as well as support in setting things up for our needs. Without all that great help, we wouldn't be able to go anywhere with that.