Now I Understand The Scotch Story...

Yesterday, I finally could upload SeaMonkey 2.0 Alpha 1 candidate build 1 for (smoke)testing by our community QA people.

For me, this has been a complete new experience. All previous (1.x) SeaMonkey releases have been done on my private tinderbox machines, were cvs- and xpfe-based, had no crash reporting or automatic update support - things that all changed now. 2.0 Alpha 1 was created on Mozilla-hosted build machines, is hg- and toolkit-based, has a working open-source crash reporting system that needs build symbols uploaded, has has support for automated updates to later releases. That's the bright side.

The dark side is that I have a half-automated process for doing 1.x release, but neither Mozilla nor I have any working automated process for hg-based releases yet.
For 1.x, I have a script that gives me list of commands for version updates and tagging that I only need to copy and watch the output for doing stuff properly, and I just could copy the tinderbox configs, modify them to run only once and do a clobber and full rebuild, pull the release tag and upload to the candidate directory (the latter two are the only needed changes for every release), and then run those tinderbox client processes on the three machines (Linux, Mac, Windows). I even stuffed an additional script into the Linux config to create source tarballs and put them in the right place for being uploaded by tinderbox' upload step.
For this release though, I had to do everything - yes everything manually. And I learned painfully what that means (even though I had FF 3.1a2 build notes as a base):

• I pulled comm-central locally into a new directory, performed a client.py checkout and had to manually enter the commands for tagging cvs in mozilla/extensions/typeaheadfind, mozilla/extensions/wallet, mozilla/extensions/irc and mozilla/extensions/venkman and tagging hg repos in mozilla/extensions/inspector and mozilla/ before adding the last patch, update client.py for the tags to pull and tag comm-central itself. I didn't know how to correctly merge this back to the last tip checkin so I reverted and pushed - before I realized I forgot to change the version from 2.0a1pre to 2.0a1 for the release and Callek went in and corrected this and moved the comm-central tags for that (thanks again).
• On all the build slave machines that create SeaMonkey 2 nightlies, I created a builds/release directory and pulled comm-central from the tag there and ran client.py checkout, copied the .mozconfig from the nightlies with replacing the "nightly" update channel to "beta", did set the MOZ_OBJDIR environment var and ran the client.mk build - and repulled and restarted the builds after the tag update for the version messup. I went to sleep , logged in the next day, and realized that Windows hadn't built palmsync, so I tried a number of times until I had figured out how I needed to set the env vars to get a rebuild with that extension.
• I set a bunch of env vars and ran the symbol build and upload steps - only to discover that both Mac and Linux issued a number of strange-looking warning during that, including errors for 6 libraries on Mac. None of the Mozilla build people could really help me with those warnings, so after getting them again in re-runs of those steps, I decided to accept them and go with the results we got out. For the 6 libs on mac, this means that we actually have incomplete symbols for Intel Mac, which is a known failure of symbol dumping on OS X 10.4 - I hope it can be worked out for future releases.
• Following that, I ran the build steps to create packages - only to realize I couldn't run that in either a ssh login or a screen console on Mac but had to run it a third time from a graphical terminal window via vnc so that hdiutil would succeed and we'd get a dmg finalized.
• Fortunately, the .mozconfig contained the configs for update-packaging from the beginning so I could run the buildstep to create MAR files for full updates, which we'll need to create partial update files if we want to offer automated updates to the next release.
• Directories on the FTP staging server had to be created manually and scp run manually on the machines to upload the packages.
• For a source tarball, I pulled the tagged tree again locally, removed all .hg directories (so the packages won't be too oversized), tarred that up and manually uploaded it.

This was about 8 hours of actual work time for me, spread over two days, and with a large number of "WTF?" experiences, making me end up half-ill and very tired. See my 2.0a1 build notes for commands and detailed data.

I hope we will not need another candidate build for this release (even though I learned a number of things I hopefully wouldn't run into a second time), and I hope the automated release process based on hg will be far enough so we can use it for Alpha 2 - but I finally understand the stories about Scotch whiskey being the best friend of release engineers...