The roads I take...

At the SeaMonkey and Thunderbird meetings this week, I started advocating for the SeaMonkey 2.0 schedule to be decoupled from Thunderbird 3.0, specifically, to have SeaMonkey 2.0 Beta 2 and final be released before their Thunderbird "counterparts".

For some fun around this, let's view a proposal of a schedule I sent to the SeaMonkey Council in mid-December 2008:
We actually did release Alpha 3 on March 3 and Beta 1 on July 21, and added a second beta that is still to be released, apart from the fact that Thunderbird plans changed significantly as well. I don't really know if I should laugh or cry when reading that proposal from 7-8 months ago.

So, what went wrong or not as expected? For one thing, there is this old rule that every project takes longer than expected, even if you plan with that circumstance in mind. For the other, we decided to follow Thunderbird schedules so that freezes for the shared mailnews code code are aligned and we cause each other as few problems as possible. And that turned out to be the wrong thing to do for the SeaMonkey project, because the objectives the two projects have for the upcoming release series are quite different and therefore impose different scheduling.

Thunderbird has a new team that took over after a period where development stalled, and they try to show off that things are moving as well as make the product better fit for the mass-market.
SeaMonkey has the team that has been in charge for some years now, where it did continuous development, and we try to get our existing userbase to a new level while making the product more compelling to the same target audience as before

Thunderbird's current stable release is built on top of the "new toolkit" platform also used by Firefox, including good add-on management, mail and news standards haven't seen significant new features, and their security updates are driven by MoCo on well-managed, redundant release infrastructure.
SeaMonkey's current stable release is built on top of the "old xpfe" platform inherited from Mozilla Suite, with quite bad add-on management, the web sees lots of new developments in standards, and I need to manage our security updates myself, building them on my private machines that definitely show their age and have zero redundancy.

On top of that, Thunderbird has multiple people working full-time on the project, so that they are able to hand off management of releases where needed and have some organizational redundancy while in SeaMonkey land I'm the only one working full-time on the project and doing all release management and build engineering work without real redundancy.

And then, I have booked the flight for a three-week vacation in November, believing this to be far from any release. There I need to get off my thoughts from day-to-day work, have enough breathing room to really have some fun and appreciate the beauty down in Dixie so that I can "recharge my batteries", figuratively speaking. The last thing on my mind there is a major release that still need to be completed even though it should have been out for half a year.
I don't think I'd manage to really relax with an unreleased 2.0 on my mind, and that would probably leave lasting damage to my mental health, which I would like to prevent.

Given all that, I convinced the team at the SeaMonkey Status Meeting to target a freeze for Beta 2 on September 1 and assure that we will release final before November. We will re-evaluate the target freeze date in the next meeting on August 11 and make firm schedules there - but remember that a code freeze for that last beta also means a feature freeze for the whole 2.0 series!

This might mean that we possibly won't get everything into 2.0 that we would have liked to have there, but we are in no way short of total improvements in this series, and it's time we get something better than 1.1.x out to our regular users. There will be further releases, hopefully with smaller steps, following the "release early, release often" methaphor that's proven to be successful in open source projects overall.

I would have liked to sync up with Thunderbird as freezing shared code at different times can be somewhat painful at times, but given the different objectives, they seem to be less pressed by time and more pressed by features, while we already have a big load of features to show off in this release, but are overdue to deliver something.

I hope things work out fine that way and I'll blog again when we know even more on concrete schedules.

Here's a summary of SeaMonkey/Mozilla-related work I've done in week 30/2009 (July 20 - 26, 2009):

• Release Process:
  I spent quite some time the SeaMonkey 2.0 Beta 1 release this week again, the good news is that it was about setting the finishing touches and making the release public on Tuesday (missing the moon landing anniversary by just a day). Builds for Windows, Linux and Mac in 17 languages as well as partial and complete updates (for US English, we didn't have any localized versions before) could be delivered right from the first minute and what I wrote up in the announcement seems to be reflected in some media coverage we get.
• Build System and Release Harness:
  The version bump problem I found in Beta 1 work got reviews and could land once I had time after the release was public. With that and the beta release coming off it, I could mark the release harness as fixed.
  Build with mozilla-central now requite wireless-tools on Linux, I fixed the x86_64 box by installing those. Also mainly for the mozilla-central trees, I installed Java 6 on the Windows machines, this also fixed a leak that only had happened on a single machine but introduced a known leak that is fixed on mozilla-central but not 1.9.1 yet.
  As a side note, I should have the SeaMonkey 1.1 build machines up and running again, I'm still investigating an issue with the Linux box and its X server.
• SeaMonkey Project Website:
  Landed the the reworked 2.0 Beta 1 download page with the locale matrix and the download button figuring out the fitting language as well as OS.
• Statistics on Dev Site:
  From time to time, I like to play with my SeaMonkey development website and get resources up there that I think are helpful for our project. Last weekend, I created more elaborate weekly bug and release radar statistics to get a better view of current work in historic perspective and to get a better overview for release planning.
• SeaMonkey L10n:
  14 locales plus US English shipped as official builds in 2.0 Beta 1, two shipped as experimental/unofficial with a note that they didn't fully meet quality and formal requirements for official builds at this time, but I think both should be up to making it for Beta 2. Overall, this beta was a good experience in terms of L10n, and I hope we'll get more locales joining in for the second beta and the final release.
• German L10n:
  Synched up with geolocation and mailnews string changes, discussed de community IRC meetings with Topal.
• Various Discussions:
  Add-on blocklist, landing dlmgr search fix, tabmail, mozilla-central build problems, Gecko/platform 1.9.2 planning and SeaMonkey/Thunderbird, security updates for vulnerabilities disclosed at BlackHat, etc.

The SeaMonkey 2.0 Beta 1 release seems to go well so far. We have received reports of a few minor regressions, which we are working on, but almost all of the feedback is quite good. So far, we have over 13,000 tracked downloads for this beta and the numbers are growing (somewhat more slowly than at other times because of the holiday season). The media reports I saw are fairly OK, though the note comes through that we're not up to feature parity with Firefox 3.5 and Thunderbird 3 on all accounts, which we know, but we also should be doing better there than SeaMonkey 1.1.x in comparison to Firefox 2.0 and Thunderbird 2.0, so we are on the right track.
Actually, there are only two major features we miss that we want to ship in SeaMonkey 2.0 that are not present in this beta yet, and that is tabbed mail and a reworked Mac theme. Both are being worked on right now, in their final touches and review phase, and hopefully landing soon. Other than that, all that's left is making the experience rounder and fixing some annoyances and bugs, everything else is topping on the cake.
SeaMonkey 2.0 Beta 1 already works and feel so much better than 1.1.x in so many ways that it's really time for us to push for releasing as soon as we can, even if that means pushing some nice-to-have things to the next version. We don't have to be concerned about not doing everything in 2.0, as this version already more than warrants its major version jump. There will be a next release after it that can improve many more things, and the solid add-ons platform even makes 2.0 able to be improved a lot by itself.
I hope everyone of you will support and help us to get a great final as soon as reasonably possible for us!

As a side note, thanks to Matthias "matti" Versen for caring about out add-ons blocklist and trying it keep it up to date in the future!

Last night I came around to do more work on my SeaMonkey development website and now can expose a lot of additional numbers there:

Weekly bug statistics on the SeaMonkey product (i.e. of bugs that are there nowadays) are now available for all of 1998 to now (the last three weeks get automatically updated, the rest needs manual triggering from me if wanted). See the Weekly Bug Stats pages for all the data.

Also, the site is now displaying numbers to the release radar queries, so even on the front page you can see how many blocker requests or open blockers etc. there are for currently watched releases, a radar overview page displays them for all releases in the database, once again updates are only done for a subset of those releases as configured in the DB, all others need to be triggered manually.

I hope those numbers are not just interesting for myself, but I mainly did this out of my personal interest (or for my own release driving in case of the radars).

This is a random collection of thoughts going around in my head - probably all of them would warrant a full blog post, but I don't come around to doing them, so I'll put down the short summaries.

• Geolocation: Why do we have to rely on Google magically knowing where someone might be? Wouldn't it be better to have a community-driven database we all could contribute to, which could give you croudsourced location data?
  For example, I know where I am, I can give precise location of both my wifi IDs as well as a range of IPs up to the building, but Google Location Service (what Firefox uses for the internal geolocation module) just tells me I'm somewhere in Vienna. They don't have a way for me telling them my info and improving that information, but an open, community-based service could. And OpenStreetMap would even know full address data for this location. How dull that all our modern technology just tells me I'm somewhere in a multi-kilometer radius around the Vienna City Center.
• SeaMonkey Meetup: The SeaMonkey project has some amount of donations stacked up at Mozilla Foundation, and I think it would be cool to use that to finance a SeaMonkey meetup, paying for accommodation and travel of major contributors as well as the (if needed) the place to do the meetings. Would this be a good idea? Who would come, maybe even if we can't pay for him, in what city should we do this event?
• Contribution Statistics: I have thought a few times about doing a script that parses our Mercurial pushlog at least between releases, and gathers data on how many changesets and +/- codelines people have created and/or reviewed, to get a view of which people are how active in the community. The same could be done for bug triage. The result would be something like Jonathan Corbet's Linux Kernel Developer Statistics.
• SeaMonkey QA: We're really missing someone to lead and coordinate SeaMonkey QA work - Andrew Schultz is quite busy with some strange thing called "real life" nowadays and can't really do that work right now. I'm sure we'd all be quite happy if he could pass the torch and we'd support anyone who tries to do it with all help we can provide.
• Web-based Help and Support Resources: We have some weekness in SeaMonkey help and support resources on the web. While our in-product help is fine, it would be good if a Google search would turn up something helpful and if we could point people to URLs. One partial solution would be to have a script that periodically converts our inline help to usable web pages, and better solution would be to set up a copy of SUMO for SeaMonkey, with a knowledge base and possibly even a web forum - but someone needs to drive that. Any volunteers?
• SeaMonkey Marketing: Even though I'm theoretically responsible for marketing right now, I badly fail on getting anything done, starting from putting a page with a collection of logos and web buttons up, and moving on with all other possibilities of fostering community marketing. This is another area where I'd be happy to have someone come on board and drive this. I'm happy to support any efforts from a technical and project organization POV, but we probably need someone else to lead those efforts.
• Mozmill: Thunderbird is starting to automate tests on Mozmill now, Firefox QA people start using it for smoketests, could someone get it to run for SeaMonkey so we can do those things as well?
• SMILE, Weave, Jetpack: There are more things out there that probably need help: the equivalent to FUEL and STEEL, which we call SMILE, getting Weave to work for SeaMonkey, and last not least, getting Jetpack to work (which probably needs SMILE).
• Parallels: Why does it need to be so painful to run OSX in VMs? And does nobody else run a larger number of VMs, including OSX machines, on Parallels? I don't understand how we can have basic problem like not being able to run more than 8 VMs on one host and OSX VMs being unstable esp. if they have access to more then 1 CPU core and both thing not getting much traction from Parallels devs. It can't be that we are the only customer who see those problems.
• Statistics: I would love to have a lot more statistics on users, downloads, etc. for SeaMonkey and esp. SeaMonkey 2 but it seems to be hard to get the data and tooling that exists inside Mozilla systems out in a way we can use it. I guess Mozilla Corporation is not as open as it could and should be in some areas.
• openSUSE: With the inclusion of SeaMonkey 2.0 Alpha 3 (soon to be Beta 1, they already have Thunderbird 3.0 Beta 3), in the current openSUSE Factory, it looks very much like openSUSE 11.2 will be the first distribution to have SeaMonkey 2 in their official package repositories. Thanks to their package manager Wolfgang Rosenauer for making this possible!
• Moving: I'll finally be moving from a student dorm to my own flat in August, lots of stuff to think about there. Also, the machines building SeaMonkey 1.x nightlies and releases are about to be moved to a different location, Linux and Windows being unavailable there recently is connected to this, I had to clear up how to do this, and some missing responsiveness on the side of my provider contributed a lot to finally deciding to switch providers in that process.
• Vacation: I have already booked the flight for my vacation this year, I'll be away for three weeks in November, traveling through the US gulf region, circling from Houston via New Orleans, Pensacola, Atlanta, Nashville, Memphis, Dallas back to Houston. It will be quite a distance to travel, but it should be manageable and be a good distraction from my usual work, and lots of things to see and experience. I easily get excited when talking about this.
• Music: Sometimes I'd love to be a signer in a local Blues/Rock/Country band, but I hardly find the time to practice the guitar any bit or type the lyrics of my recently written songs into the web database I have for them. At least I come around to some Karaoke singing every week.
• Space: How come that the great things NASA does is not worth more to the public than the half cent of every US tax dollar it actually gets? How come that it isn't worth more to other countries as well? Isn't exploration of new frontiers, world-wide cooperation to do amazing things not because they are easy, but because they are hard, aren't all those things one of the main drivers of what makes humanity great? Are we losing focus in that we are only caring about our own small biotopes and internal affairs and forgetting to expand our knowledge and horizons?
• Test Coverage: It would be so nice to increase coverage of SeaMonkey code with automated tests, but it's proving even hard to require tests for new things added, as we also don't want to slow down progress - esp. when we are already behind the schedules we hoped to follow a few months ago.
• Mozilla 1.9.2 and SeaMonkey/Thunderbird: Mozilla platform maintainers decided to do a 1.9.2 branch very soon now and base Fennec 1.0 on it as well as a Firefox 3.6, not featuring lots of application changes, but some good platform improvements. Some of those changes in the platform would be good to have for SeaMonkey and Thunderbird, but we also know of some problems we'll have there due to doing our experimental builds with mozilla-central all along. Moving over to the branch now would probably delay our stable releases for a few weeks more, but we are already running behind the schedules we wanted to have, so we think it's better to stick with 1.9.1 for now and get SM 2.0 and TB 3.0 out before even thinking of what to do about 1.9.2 - we could either ignore it completely or do smaller-step 2.1 and 3.1 releases on top of it just like FF does with 3.6, but we're not sure what's best. For now, we'll watch it but not actually do anything about it.
• MailNews API refactorings: It would be really nice if we could port the JS-driven folder pane and the various refactorings done for gloda search from Thunderbird to SeaMonkey UI, as those would sync our APIs with theirs and make life easier for add-ons, next to making work with folder and thread panes easier internally as well. Once again, what we're missing is someone to do the work - we are a volunteer open source project after all, and people here tend to work on those things that are fun for them, and of course only in the little free time they have.
• Local Communities: Every face-to-face meeting i had locally with open source developers around here in Vienna was very rewarding, and I should engage much more with those communities. Also, my recent talk for IT businessmen on "project management in open source" was a very exciting and successful thing, I believe I could, with the help of a few fellow open source community people, dampen a lot of FUD that arises with people used to traditional IT business but who are still interested in how thing work "on our side" - which I hope to have proven to not actually be that much different as they often think. By the way, and I got got comments like "Oh, so the suite is still being developed? Nice, I need to try SeaMonkey then!"

You see, lots and lots of things are going around in my head all the time - this is still only an excerpt, but I thought a should get a few words on most of them out to you some time, so here's those bits. More thoughts will come in the future, I'm sure - i'm a thinker, you know

After somewhat more time than we had expected earlier, we finally have a great SeaMonkey 2.0 Beta 1 release available for you. This release really brings the suite up to modern standards, supporting feeds as well as HTML5 audio/video, managing add-ons in a comfortable way (as known from Firefox 3.5), having customizable toolbars in both browser and mail windows, and much more - and all that in more than a dozen languages right from the beginning.

In many ways this beta is much better than our stable 1.1.x releases, and it's probably not even less stable from most points of view, but it still has some corners that need smoothing and some features we want to add before calling it a final 2.0 release.

Pushing it out to the public today was one small step for a release engineer, but one giant leap for the project. One after after celebrating the 40th anniversary of the first human setting foot on a different celestial body, i.e. the Apollo 11 mission to the moon, this is our mission of exploration, boldly moving the well-known Internet suite where it has never been before - into a state where it fully fit for the challenges the Internet and web sets for us today and laying the base of where it can be tomorrow.

To echo what Buzz Aldrin said two days ago when asked if we can manage the challenges to go the way even further and step further out - in the case of space exploration, that's bringing humans to the Mars and beyond, in the case of SeaMonkey it's a 2.0 release and beyond - I fully agree with him paraphrasing a nowadays almost overused sentence:

Yes, we can.

(And you can join us in that task by testing this beta, reporting bugs, and if you're really bold, exploring possible improvements and writing patches. Yes, you can.)

Here's a summary of SeaMonkey/Mozilla-related work I've done in week 29/2009 (July 13 - 19, 2009):

• Build System and Release Harness:
  After some poking of release drivers, I could land the Mac unpacking fix on 1.9.1 so that L10n repackaging of Mac comm-1.9.1 nightlies and 2.0b1 builds could work. The botched checkin of the equivalent for release tools was fixed as well, so now we should be able to reliably unpack DMGs everywhere (would have been easier if Apple wouldn't create commandline tools that act asynchronously).
  We found a good solution for automatically removing debug UI from beta and final releases and I landed that also in time for Beta 1.
  When starting to run release automation I found out about a version bump problem with the release tools and created a patch to handle our special case.
• Release Process:
  The really big thing this week was working on the SeaMonkey 2.0 Beta 1 release itself. The tracking bug and the Build Notes have a good amount of the story of how it went. I did run into a few small issues, including one obscure error that looks like a machine issue with one VM, but the release automation work done in the last weeks turned out to have been almost perfect, as that part performed like a charm. We have builds for Windows, Linux and Mac in 17 languages now, as well as partial and complete updates (for US English, we didn't have any localized versions before) and we are all set for release from that point of view.
• SeaMonkey Project Website:
  I needed to come up with some new things for the 2.0 Beta 1 download page, as this is the first time we ship localized builds, and while I was at that, I included some small style improvements to the overall site, like highlighting headers when they are direct targets in the called URL.
• SeaMonkey L10n:
  Some significant work went into the opt-in process for the beta, where I took the time to do a fast productization review of all opted in revisions, which proved to be interesting as I also got to see all the websites of those locales that use custom homepages or bookmarks. We ended up with 15 locales opting in, one of which (hu) has over 450 obsolete strings which I disliked for releasing as official, but the others looked just good. One (pt-PT) had a complete and looking-good localization but failed to opt in. Because of that, we'll release the beta in 15 official languages (including en-US) and offer hu and pt-PT as "unofficial" one with a note that they didn't fully meet quality and formal requirements for official builds at this time.
• Various Discussions:
  Various version changes on websites and tools to support 2.0b2pre and 2.1a1pre nightlies, landing support for various shortcut keys in download manager, progress window accessibility, tabmail, getting old "-tbox" slaves into new buildbot pools, mac theme work, real and unreal FF 3.5 (Gecko 1.9.1) exploit reports, Gecko/platform 1.9.2 planning and SeaMonkey/Thunderbird, etc.

Apart from one strange issue that appears to be related to a single VM in our build machine pool, creation of the SeaMonkey 2.0 Beta 1 release went really smoothly, and I'm really excited to see this version hit the public in 15+2 languages (see above), probably today or tonight. Even thought there are 20 language packs available to be used for 1.1.17, we never had any release that had official localized builds, and never had any version that had fully builds available in such a number of languages right from the start. We really can be proud of starting the beta phase of SeaMonkey 2.0 with that achievement, even though the collection of languages shows pretty much how Europe-centric our project is - I hope more parts of the world will join in for the next beta and for the stable releases.
In addition to that localization story, the first Beta offers the same Gecko and platform as Firefox 3.5.1, the completely reworked download manager, feed preview including better feed subscription, customizable mail toolbar, mail archiving and many more features and fixes in addition to all the good things we already had in the alphas - about 130 fixes were done just in SeaMonkey-specific code since Alpha 3, many more come from mail/news code shared with Thunderbird 3 Beta 3 and the Gecko/platform code we additionally share with Firefox 3.5.1.

I hope this beta release will be remarkable not just for our team, but also for all of you out there!

I ran into strange problems with the Mac builds of SeaMonkey 2.0 Beta 1 (a step in the process that works flawlessly everywhere else hangs with very strange errors), but that shouldn't stop us from starting tests on the other two platforms that have builds available now.

So, please help us testing the available Windows installers and Linux packages, both available in 17 languages including US English!

I hope to sort out the Mac problems ASAP, we'll have disk images when those are solved and will get updates ready on the testing channels once all platforms have builds ready.

Update: Mac disk images are now available for all languages as well.

Update #2: Updates are available on the betatest channel.

If you have the SeaMonkey 2.0 Beta 1 Schedule in mind, you know that by now, all the code and strings are fully frozen for this release, all we are waiting for is for additional L10n opt-ins until their deadline of midnight Pacific time today.

10 locales have opted in so far, 8 of those (be, de, fr, gl, pl, ru, sk, tr) are good to go, the other two (ca and hu) are not green on dashboard due to obsolete strings, they should just remove those and are good to go as well.

13 locales are green on dashboard, so 5 of those haven't opted in yet (cs, es-AR, lt, nb-NO, pt-PT) - if yours is one of them, please tell us which revision to use for release in the opt-in thread on the mozilla.dev.l10n group/list - 2 more locales (es-ES and nl) have just a handful of missing strings and also could make this release easily, but any opt-in needs to be there until midnight Pacific time, remember that. (The remaining 7 locales on dashboard, ja/ja-JP-mac, ka, pt-BR, ro, si, sv-SE, need a good amount of work to get ready, any locale not on dashboard yet please follow the process outlined in our wiki to get on for future releases.)

For me, this release will be special not only because we finally can release official localized builds and do that in sync with the en-US builds, but also because we'll be using release automation for the first time, the same hg- and buildbot-based tools Firefox is using for their releases. I just checked in a fix for repackaging localized builds on Mac, so all that tooling we need should be there and working now, but it's the first live run, so far I only did test runs of those tools. They will provide us with partial updates as well (for en-US only, as that's the only language for which we have an earlier release), so we have a number of firsts in this release from a build and release management point of view.

Of course, that only complements the firsts we have in the source, we have the same platform and web functionality as Firefox 3.5 (including the 3.5.1 security updates), the same mail and newsgroups backend as Thunderbird 3.0 Beta 3 and about 130 SeaMonkey-specific fixes upcoming in this release.

I will start release automation early in the morning tomorrow, so localizers, remember to opt in today, and testers (including L10n testers!) be ready for candidate builds being available later in the day tomorrow (including updates on the betatest channel)!

Here's a summary of SeaMonkey/Mozilla-related work I've done in week 28/2009 (July 6 - 12, 2009):

• SeaMonkey Build/Release Harness:
  A series of patches for buildbot tooling concerning L10n repackaging and release automation for comm-central applications got reviews and landed during a Mozilla maintenance window (to ensure that Firefox machines would be undisturbed). I also landed configuration files for SeaMonkey release automation, so that we are mostly ready for using that harness for 2.0 Beta 1.
  The Mac unpackaging fix needs to land on 1.9.1 (I could land it on mozilla-central already and it performs well there) and the same for release tools needs to land correctly before we are fully ready.
  I also could do Phase II of switching new buildbot configs to production, now the SeaMonkey waterfall has builds based on mozilla-central, while the default builds based on 1.9.1 are on the SeaMonkey2.0 page.
  The fix for trunk Mac builds landed before doing that, as did the fix by Neil that also made the nightly update breakage go away, so things looked quite good for the trunk builds from the beginning, though I needed to switch off tests for Windows nightlies and enlarge the timeout for making build symbols to get the nightlies for Windows going there.
  In addition to all that, I did a patch for automatically removing debug UI from beta and final releases, which should land soon and be in place for Beta 1.
• Release Process:
  Continued uploading a few remaining contributed builds for SeaMonkey 1.1.17.
• SeaMonkey L10n:
  We could add Turkish to the SeaMonkey trunk locales, so that we now have 25 languages listed there, including US English. It will be interesting to see how many of them make the first automated simultaneous localized release in SeaMonkey history (2.0 Beta 1).
• German L10n:
  I kept up with the late-l10n change of the download manager prefs and mailnews search so that de should be good for beta 1.
• Various Discussions:
  Adding 2.1a1pre to AMO, DEL and other keys in download manager, Download manager pref panel, tabmail, getting Beta 1 into shape, etc.

It looks like Beta 1 might finally just fall into place nicely, even thought we are still tracking an issue with history import from 1.x builds, we now have a clue on how it could be fixed in toolkit code (mozilla-central appears to be fixed), but it might just not make the 2.0b1 build cutoff, unfortunately. We'll deal with that in this week's SeaMonkey Status Meeting and we'll re-evaluate if we can ship with that bug and just relnote it or if we need to have it fixed (we did ship Alpha 3 with this bug, apparently). For everything else, we look good for freezing Beta 1 this Tuesday and start builds for it later this week, making them available for testing. I hope you all will help to do that testing and hopefully clear the builds for an important milestone release, which will be the first time that we'll offer official localized release builds for all major platforms.

Here's a summary of SeaMonkey/Mozilla-related work I've done in week 27/2009 (June 29 - July 5, 2009):

• SeaMonkey Build/Release Harness:
  Phase I of switching new buildbot configs to production has been done, official nightlies are now coming from those.
  Automated updates stopped working, probably because of the first step for Phase II, using a different version number for mozilla-central builds, for which I apparently made a somewhat incomplete patch checked in.
  I also fixed up making repack independent of ChatZilla or venkman being enabled just before that large switch.
  The small patch for installer dealing with "pretty names" finally got into the tree, and I did put up a patch for a small fix for trunk Mac builds Mark pointed me to.
• Release Process:
  Continued uploading contributed builds for SeaMonkey 1.1.17.
• German L10n:
  Fixed some mailnews strings to keep the de locale green.
• Various Discussions:
  Talk on open source project management at an event of the local IT branch of the chamber of commerce, DEL and other keys in download manager, Download manager pref panel, tabmail, Firefox 3.5 release, etc.

We finally have a schedule for SeaMonkey 2.0 Beta 1, freezes are already happening and things are progressing nicely towards making the goal of shipping this beta within the current month. We had an unfortunately row of delays, not all of them self-inflicted, since the last alpha, but this major step to a fully revamped SeaMonkey is now coming near.

Wenn ihr euch wundert, warum mein Blog, SeaMonkey deutsch, Find Your NetFriends, andere Communities auf unserem System, www.kairo.at, CBSM oder jegliche andere der Webseiten auf unserem Server für den Großteil des vergangenen Tages nicht verfügbar waren, hier die eine kurze Geschichte dieses toxischen Vorfalls:

• Der Server hat quasi den Kontakt zu einer Festplatte verloren (sie ist einfach nirgends mehr aufgeschienen),
• die Software-RAID-5-Treiber unseres System-"Laufwerks" kannten sich nicht damit aus,
• ich hab ein Kommando erzwungen, um das wieder in Schwung zu bekommen und lief in einen I/O-Fehler (autsch, das heißt meist nichts Gutes),
• wir schalteten den Server ab, nahmen die Platte raus,
• gaben sie wieder rein, und sie funktionierte wieder,
• wir haben das RAID-Array neu erstellt und stellten fest, dass es sein Dateisystem nicht mehr erkannte,
• das Dateisystem-Reparations-Tool (fsck.ext3) spuckte Millionen von Fehlermeldungen über ungültige Journale und Inodes aus,
• wir haben es neu formatiert und das ganze System neu installiert,
• sowie alle wichtigen Daten von Backups wiederhergestellt.

Wie ihr euch wahrscheinlich denken könnt, war im den Tränen nahe, als mir klar wurde, dass unsere geliebte System-Installation zur Gänze flöten gegangen war und ich alles neu aufsetzen konnte. (Nein, unter Linux und besonders bei Servern ist das nicht üblich, das ist KEIN Windows-Heimcomputer.)

Gott sei Dank war ~5-6 Stunden vor dem anfänglichen Systemausfall ein Backup gelaufen, sodass nicht zu viel verloren ging, aber ich brauchte 10-12 Stunden bis zum jetzigen Stand, an dem alles wieder halbwegs in Ordnung zu sein scheint. (Sicherlich werden in den nächsten Tagen noch ein paar Kleinigkeiten auftauchen, aber die hautsächlichen Webseiten scheinen alle OK zu sein, die Mails fließen wieder, usw.)

Wenn ihr mir zwischen 3 und 9 Uhr in der Früh am 5. Juli E-Mails geschickt habt, sind die ziemlich sicher verloren, in anderen Fällen sollten sie angekommen sein oder reinkommen, während das SMTP-Netzwerk mitbekommt, dass unser Server wieder verfügbar ist.

Ich hoffe, die Zeit, die ich dadurch beim Lernen verloren habe, ist nicht zu toxisch für die Prüfung morgen Nachmittag, deren Thema ganz "zufällig" - Toxikologie ist.

As discussed and decided in the last SeaMonkey Status Meeting, I've started to move SeaMonkey to the new buildbot configurations that have been in the testing phase for a while now and will open the possibilities for doing builds based on mozilla-central soon.

The following changes are made that you should be aware of:

Phase I (done yesterday, first nightlies from today):

• The nightlies in latest-comm-1.9.1 are the official ones now.
• FTP has latest-comm-central and latest-trunk symlinked to latest-comm-1.9.1 for the first part of the transition.
• Those are static builds with reduced build size and hopefully better performance.
• Dependent builds ("build" columns, "hourlies") continue to be shared builds.
• Linux has a "leak test build" column that runs debug builds.
• Nightlies and dependent builds are being run on x86_64 Linux, even though this should still not be considered a tier-1 platform and has no L10n builds.
• Localized nightlies are in latest-comm-1.9.1-l10n and built with the "L10n merge" tooling that injects en-US for missing strings, eliminating most cases of the "yellow screen of death".
• The SeaMonkey tinderbox waterfall page has those buildbots reporting as columns containing "comm-1.9.1" in their names.
• Due to ongoing slight instabilities in the 10.5 VMs, the old "comm-central" 10.4 unit test machine stays up there as well.
• All other comm-central builds disappear from the scene.
• For localizers, the comm-1.9.1 builds on their Mozilla-l10n-* waterfall pages are the relevant reference.
• All VMs on the same platform run in generic pools, so that any of the machines can do any dependent build, nightly, L10n repackaging or unit test run, the buildbot master hands those jobs on a "first come, first serve" basis to the available slaves.

Phase II (once bugs 502031 and 502033 are solved):

• Builds with mozilla-central are set up in addition, uploading nightlies to a latest-comm-central-trunk directory.
• A version number of "2.1a1pre" is used there for now, there's no definitive decision on what version the next SeaMonkey will be yet though, this is the lowest possible version to use atm.
• FTP symlinks for latest-comm-central and latest-trunk are removed.
• The comm-1.9.1 builds are switched to report to a new SeaMonkey2.0 tinderbox waterfall page (as well as the 10.4 tests).
• The comm-central-trunk builds report to the SeaMonkey page.
• No unit test builds will be activated for comm-central-trunk while we don't have all the machines we need in the pools (due to Parallels network problems).
• L10n builds for comm-central-trunk use l10n-central work and also report to the Mozilla-l10n-* tinderbox waterfalls, dashboard for this might only come up some time later.

Phase III (once bug 493321 is solved):

• Unit tests for comm-central-trunk are turned on.
• A few other incremental improvements can be thought of when Phase II is completed, such as running packaged tests, AUS for L10n, etc. but all those can be done individually, step by step, then.
• Once 10.5 Mac VMs are completely stable and not intermittently crashing because of Parallels issues, we'll turn off the 10.4 unit test machine and hand it back to Mozilla IT.

I hope this whole transition works well for everybody. It was a bumpy ride and a good amount of work to get this all up, but it largely reduced the amount of maintenance needed and makes us share more buildbot code with Firefox and Thunderbird already now and even more once this all is completed.

Thunderbird came around with setting up a schedule just in time, so at yesterday's SeaMonkey Status Meeting, we agreed on adopting Thunderbird 3.0b3 dates as the SeaMonkey 2.0 Beta 1 schedule as well (times of 23:59 US/Pacific, as usual):

• Slushy String freeze date: 2009-07-02 (Thur)
• Slushy Code Freeze date: 2009-07-07 (Tues)
• Firm String / Code freeze date: 2009-07-14 (Tues)
• l10n-mozilla-1.9.1 freeze date: 2009-07-16 (Thur)?*
• Target Ship date: 2009-07-21 (Tues)

This is the first time we have string freezes for SeaMonkey, so let me give you some explanation:

After the slushy string freeze, string changes should be avoided where possible, and those needed must get approval-seamonkey2.0b1+ before being checked in.

After the slushy code freeze, the same is true for any code changes, though blocking-seamonkey2.b1+ bugs without string changes can go in without further approval (blocking+ serves as approval).

The firm freeze should be the cutoff for any changes at all, unless there are blockers we still need to fix.

* I'm not completely sure about the freeze for L10n, we might not need that at all, as we'll likely do the same opt-in process as Firefox did recently and so might be able to just take any L10n changes up to the time when we start the builds. This is the first time we're doing that, so please excuse roughness in the process.