The roads I take...

The culmination of a long stream of work has just happened: The switch to the toolkit download manager has just landed on comm-central, along with the reworked tree-based download manager UI and the rewritten per-download progress windows (which formally aren't "dialogs" any more).



This is a rather large change code-wise: hg diff -r qparent -r qtip | diffstat before doing the actual qfinish and push told me that all those patches result in:
53 files changed, 5766 insertions(+), 620 deletions(-).

Most of the new code is the manager and progress UIs as well as the tests, the actual switch patch removes more than it adds due to obsoleting pref-applications-edit.xul, which the old file handling dialog needed to save the "remember this decision" checkbox info (yes, an evil hack).
Once we come around to remove the code we actually obsoleted in the mozilla-{central,1.9.1} tree, the deletions of that work will probably outweigh the current insertions of code lines overall - some of the obsoleted code might even still be packaged but not used by toolkit (and therefore, even Firefox).

Thanks to Justin (Callek) for figuring out how to do this all and taking on the work to make the actual backend switch (really work), thanks to Frank (mcsmurf) for fixing all the review comments to the actual switch patch while Justin is mostly absent due to being busy with work and family and not having a useful Internet connection around - and thanks to Neil for all the reviews.
Thanks for MDC and MXR and whoever wrote up useful in-code description of the tree interfaces that are not documented usefully on MDC yet, thanks to those who made it able that dropping in a custom UI to work with the toolkit download manager works at all.
And thanks for all the people who have been patiently waiting for this to happen - including the localizers who couldn't have fully working localized builds - up to now.

And here's what this means for you:

• If you are a SeaMonkey user, you will see a reworked download manager appearing in any 2.x builds from now on, starting with tomorrow's (!) nightlies, and 2.0 Beta 1. This implementation can resume downloads (even across sessions), search in your performed or ongoing downloads, clear the list of completed downloads through "Clear Private Data" and store more download history without making SeaMonkey startup time longer, among other things.
• If you are a localizer, you will see the localized builds working fully (and can advertise them to your testing community), including the download manager, also starting with those nightlies and (pre)release.
• If you are a Mozilla or extension developer, you can count on SeaMonkey supporting/using the full toolkit download manager backend from now on.

There surely are a number of things to improve in the UI still, a number of which are filed as dependent bugs of the download manager and progress UI bugs linked above. We are in a state though where we feel we are ready for putting what we have into the upcoming SeaMonkey 2.0 Beta 1.
Let us know how well it works for you and what we still can improve, both the backend and the UI code we have now (hey, even I understand the latter!) are a much better base to build those developments upon than the old xpfe code we had been using so far - you might even be able to work on it yourself!

Working towards SeaMonkey 2, we're improving Add-Ons support tremendously, as you might know. This also includes more and better integration with Mozilla's SeaMonkey Add-Ons site.

I was just reminded by a few events that I need to get in closer contact with the "AMO" (addons.mozilla.org) team for making the experience of that service better for our users.

While I'm writing a mail about this, I thought it would be a good idea to become aware of any additional issues that I should bring into that discussion at a later stage (note that not everything might get resolved right away, but AMO folks are willing to help us make our experience better).

So if you know of problems with the SeaMonkey add-ons website that are about its interaction with or application to SeaMonkey specifically, please let me know in comments to this post, I'll forward the problems (ideally you can also provide an already-filed bug about this for tracking).

Here's a summary of SeaMonkey/Mozilla-related work I've done in week 21/2009 (May 18 - 24, 2009):

• SeaMonkey Build Machines:
  Lots of work still ongoing with the new buildbots with the ultimate goal of getting L10n builds from them, which I was able to resolvein the end with somewhat more work than anticipated. This involved fixing up locales Makefiles, fixing the fix to them, making our applications provide the info printed by part of those fixes and be nicer with including the Mac README in SeaMonkey.
  As a result, I could post a patch for L10 repack factory abstraction that I had proven to work for us and hopefully can land in the shared buildbotcustom repo.
  In other news, I found that the Linux x86_64 build problem was actually being out of disk space because I forgot to add the extra disk on the VM. As soon as I had added it, builds started to work fine.
  I also slightly fixed up the slave name TinderboxPrint functionality thankfully introduced by gozer into the buildbotcustom factories.
• Geolocation:
  Network Geolocation Provider could land on SeaMonkey with some more ported-from-Firefox improvements but without the actual URL for a provider on the network. This has been split off into its own bug, we need to get permission from Google to use their service before actually adding it.
  I filed dependent bugs on web page and help updates - help wanted (literally)!
• Build System:
  The automatic update of Windows file versions has landed on Mozilla 1.9.1 and comm-central for the SeaMonkey part. No worry about forgetting something and the Windows .exe reporting a different versions as the app itself any more (both for Firefox and SeaMonkey, by the way)!
  Release automation was updated to work with that change, some unused code in its tools can be cleaned up later as well.
• German L10n:
  A number of new strings to keep up with mostly mailnews development, also fixing 1.9.2 toolkit for German.
• Various Discussions:
  SeaMonkey tabmail work, mozilla.org planning, Mozilla meeting times, starting thoughts on new stable security release, etc.

We are fighting Parallels problems for our new buildbots still even though we're avoiding the more-than-8-VMs networking issue for the moment. I'll keep the SeaMonkey-Ports waterfall page updated with the open issues.
That said, the machines are producing and uploading nightly builds for Linux i686, Mac OS X, Windows and the somewhat unofficial Linux x86_64, as well as localized builds for all languages that build successfully on the three official platforms.

In other news, the download manager switch is getting pretty hot, as of today I have positive reviews even on the progress dialogs/windows, and the main switch patch is shaping up nicely now that Frank ("mcsmurf") took over driving it from Justin ("Callek") who is temporarily unavailable. We'll likely land all the reviewed changes in one push, which is now likely to contain 5 patches from 3 bugs and a combined diff set of 63 files changed, 5850 insertions and 660 deletions, according to diffstat output on the not-yet-completely-finished patch set I have in my Mercurial queue right now. I hope this will happen soon, it surely will be a significant change that brings us one giant leap nearer to a SeaMonkey 2 release.

I posted an idea in a triage-related newsgroup post a month ago and lacking any reaction, I brought it up in the recent SeaMonkey Status Meeting, where it was decided to get feedback on it before deciding on it, so please tell me what you think about this:

SeaMonkey inherited a large legacy of of bugs from the Mozilla suite, most of which have an unconfirmed value nowadays, four years after founding the new project. The Bugzilla database reflects those as current bugs though, often bringing them up in queries as if they were current, even though we can't confirm nowadays that they still apply to our current development code or even releases.

We have almost 2800 bugs that haven't seen a comment since the project was started but are still in the NEW state, see the bug query if you don't believe it.

Ideally, we'd go and actively triage every single one of those bugs manually to try to find out if they are still valuable. If that takes only 5 minutes in average per bug, that's 29 8-hour workdays or almost 6 full work weeks for one person, and even if we would have someone with a capacity of so much time to work on SeaMonkey alone, I think we'd have higher priority stuff to spend the time on than finding out if bugs that weren't touch for more than 4 years still have any value.

When I thought about this and about how just mass-resolving bugs without warning is quite rude (we already had such incidents), I realized that we already have an identifier in Bugzilla that tells that "this bug is not confirmed to be valid" that also doesn't mean the bug is resolved for good in any way: UNCONFIRMED.
(That even includes enhancement requests for which we can't tell if they still apply to current code or project plans, by the way.)

So, what I'm proposing is that we mass-move those bugs in the SeaMonkey component that had no comment since 2005-03-10 and are still NEW to the UNCONFIRMED state.

This leaves them open but shows everyone that we are unsure that they are still valid within the SeaMonkey project nowadays without just kicking them off into nowhere land. Everyone who get bugmail about this and considers his pet bug in there to be still valid, can comment on that and/or move the bug back to NEW, which is easy - but instead of making one person go though a pile of rotten stuff to find the good stuff, we push the work to more people and to those who actually care about the specific bugs.

In a few months, when that has likely been sorted out and SeaMonkey 2.0 is out the door as a stable release, we can think about what to do with those bugs that still remain UNCONFIRMED - we might leave them as such or ultimately mass-resolve parts or all of them, but we have months before even thinking about this. For now, the proposal is just to make the state reflect reality and make them UNCONFIRMED.

What do you think about this idea to clean up our bug queries to find relevant stuff more easily?

Here's a summary of SeaMonkey/Mozilla-related work I've done in week 20/2009 (May 11 - 17, 2009):

• SeaMonkey Build Machines:
  I once again spent almost all my time on the build machines, trying to get things running more stable, evaluating what a Parallels server upgrade helped us, setting up the 64bit box and getting it imaged, making automatic start of buildbot on slave reboots work everywhere (though a windows slave has problems apparently caused by that), getting our improvements into buildbotcustom, and closing step 1 of that new buildbot setup.
  We now have that setup produce nightlies that are put into latest-comm-1.9.1 for testing and should work identically to the still official nightlies from the old configuration.
  A number of things still have to be resolved until the new configuration can move to production (sucking the old buildslave machines into the new generic pools) and beyond, for all of which I have filed bugs now: localized builds, Mac VM crashes/reboots, x86_64 build/linking problems, getting graphs up, finally getting the release harness to work.
• Various Discussions:
  SeaMonkey tabmail work, mozilla.org planning, Mozilla meeting times, starting thoughts on new stable security release, etc.

I hope we can figure out the Parallels VM network losses and the Mac VM instability cited above, as once I have figured out the L10n build stuff, the new configuration should be on par with the old and able to take over production, which will make maintenance of our build and test boxes easier and ultimately enable us to do builds with Mozilla trunk as well as move towards release automation, which again will be a huge leap forward.
All that currently needs almost my full work time although I should take care of other stuff as well, but once this all is finished, the build and release processes should be easier to take care of, so ultimately I should have more time for other things. Let's hope it actually will work out that way!

Here's a summary of SeaMonkey/Mozilla-related work I've done in week 19/2009 (May 4 - 10, 2009):

• Download Manager:
  The patch for making toolkit UI tests not fail with our new UI has now landed, and so has a first build system part of the backend work so that a correct rebuild will be triggered when we change app-config.mk with the main switch.
• Build System:
  A patch to make version changes apply to the Windows .exe automatically could also go into mozilla-central this week, I hope to get it into 1.9.1 so it actually helps the SeaMonkey release process.
• Geolocation:
  My geolocation patch has review now, but we need to seek permission from Google to actually use their service by adding that URL to our default prefs, and they don't have official policies up for that as it is a new service. I'm in talk with them and hope we find a solution soon enough to be able to ship final, possibly even beta with it.
• SeaMonkey Build Machines:
  The vast amount of time this week ran into the new SeaMonkey build machines. I did set up all the slaves and got them to a build configuaration that does most of what we need, even though a few bugs are left and l10n repackaging isn't on yet (needs more work).
  With having that to test, I could do an actual patch for mozilla dir abstraction in buildbot factories and for adding comm-central build classes to the shared buildbotcustom repo.
  There was a lot of fallout in bugs I filed during that work a some patches attached for the buildbotcustom stuff, also some more requests to IT.
  The plan is now to suck the suck the "old" Windows and Linux VMs into the generic slave pools of the new configuration and replace the old Tiger Macs with additional Leopard VMs and make the new buildbot master drive both trunk and branch with those pools then.
  We still have some distance to go with that, but things start to look quite good for the future of the SeaMonkey build infrastructure now. Thanks to everyone involved to make this possible.
• German L10n:
  Updates for string changes in mailnews so that German goes green again.
• Various Discussions:
  Error console, xpcshell-tests target and test directories, multi-process plans for platform, test reporting on tinderbox, mozilla.org planning, redundant master password prompts, cleaning up personal bugmail folders, polish bugs, etc.

The bug for them was filed half a year ago and it took some poking of people and some time, but the great thing has happened and we now have 14 virtual machines for building SeaMonkey instead of the 6 we had up to now, and we get to have our all-virtual Macs running on Leopard instead of the physical Tiger machines that are being obsoleted with the new configuration going into production. We also have the ability of all machines for one platform being able to run any cycles, so that we don't end up with build machines being idle all the time and unit tests not getting run enough or branch machines being idle and trunk being clogged (once we have both trunk and branch running, which is the primary purpose of all this). And we even will be able to run automated release building off those machines - once I have fully configured and tested that. And we will be sharing a lot more of the custom buildbot class code with Thunderbird and even Firefox so we all can profit from each other's work.
It has been long in coming and there's still some work left to do, but this is really great and should help us immensely.

Once again, thanks to everyone involved, from Community Giving via IT to Build & Release and others who care and help!

If anyone of you felt that I was unusually unresponsive in the last few days, this had a good reason - I was deeply buried in intense work:

After Mozilla IT has purchased and basically set up a Parallels XServe, they handed over 8 VMs with base systems to SeaMonkey (we're playing guinea pig for Parallels and virtualized Macs within Mozilla) - that is, I got the build machine names and passwords. Of course, that doesn't mean that everything we need is suddenly working.

I basically spent almost all my time Wednesday through Friday bringing up 3 old Linux reference platform images, 2 pretty current Windows reference platform images, 2 fresh Mac OS X 10.5 installs and a fresh CentOS 5.0 x86_64 install up to really current Mozilla reference platforms and to having everything up to serve as buildbot slaves and all my time today to get a buildbot master up on one of them, do several small fixups to the machine configurations, improving my work on abstracted paths in buildbot factories and basing new SeaMonkey buildbot configs on those, using generic slave pools per platform to do the job. After a number of fixes to get the master to start correctly, I started the slaves and hooked them up - of course, just to notice more errors in the configurations that needed to be corrected.

In any case, the master now has columns for normal ("hourly") builds, and nightlies for all of Linux, Linux64, Mac OS X, and Windows, unittest builders on Linux, Mac and Windows, and debug/leaktest builds on Linux as well as all the columns it uses for release automation. The Linux and Windows slaves as well as one OS X slave are up and trying to satisfy all build requests that come in for them from the master. One OS X machine had strange problems with Terminal.app and then hung up during reboot (even though both have the same reference setup), but eventually it did and just joined the pool; and I'll only care about the 64bit stuff when the rest is working well. I currently have builds reporting to SeaMonkey-Ports on tinderbox, as that waterfall had been unused for a longer period of time, even though I'm currently building standard 1.9.1-based SeaMonkey, no actual "port" of any kind.

So far, only the unit test builds are actually working reasonably, though all builds do compile SeaMonkey fine, we're running into either buildbot config issues (abstracted paths need more fixups for leak test builds) or missing SeaMonkey build features (no "make upload" target yet) right now.

Still, this is major progress and we should now basically have the infrastructure to do both trunk and branch builds in the future, as well as doing releases based on the same automation harness used by Firefox. I only need to get the configurations going for all that, and we need to figure out a few things like what to do about with the old 10.4 Macs vs. the new 10.5 ones (probably can't just go into the same slave pool) and some disk space issues on our old slaves.

In any case, we're one large step into an improved future for SeaMonkey building and releasing architecture. Thanks to the Mozilla Community Giving program for getting us those machines, Mozilla IT for operating the hardware, setting things up and giving us technical support for doing so, and the Build and Release team for creating so nice buildbot configs we could rip off and generic config code to share as well as support in setting things up for our needs. Without all that great help, we wouldn't be able to go anywhere with that.

Here's a summary of SeaMonkey/Mozilla-related work I've done in week 18/2009 (April 27 - May 3, 2009):

• Download Manager:
  I updated the tests for the new UI for the review comments, everything is ready for checkin there now, only waiting for the backend.
  I also worked out another approach to making toolkit UI test not fail when our UI patch is in, and reacted to review comments on progress dialogs - I need some more clarifications there before continuing work on them.
• Automated Tests:
  The random browser test orange we had in SeaMonkey should be fixed now - the problem was us relying too fast on focus() being successful, the solution to use setTimeout() to just let the focus settle. Yes, this is one of the few cases where setTimeout() makes the test more reliable when usually it's the other way round.
  When a new places test broke in SeaMonkey, I found out the problem and fixed it. I'm starting to get tired of the tests stuff though, doing it is too little fun - and that's a bad sign as tests are quite helpful overall.
• Build Infrastructure:
  Slowly, I hear that the new SeaMonkey buildbot machines are becoming a reality, unfortunately the Linux refplatform image doesn't convert as nicely to the new host infrastructure as the Window image does.
  Without those machines, it's not that easy for me to test the WIP for mozilla dir abstraction in buildbot factories, but I started this to get the ball rolling - maybe we have a chance to get as far as being able to try release automation with Beta 1.
  The patch for automatically inserting the Windows version in our .exe files should also be nearing checkin, which should ease release generation no matter what way it's being done.
• Geolocation:
  Recently, the Firefox geolocation support got a big overhaul and a network-based geolocation provider was integrated into toolkit. I hacked up a first patch for getting this all to work in SeaMonkey, but it needs a bit more work from how I read the review comments (though I don't fully understand them yet).
• SeaMonkey L10n:
  After stalling some time for review, the patch for making profiles easier to localize could make it into the tree, and though the landing was somewhat bumpy, we now should be at a state where localizers have less work to get the default profile files localized and we have more of a guarantee of what is actually in the default profile in all localizations.
• German L10n:
  I fixed a long-standing password management bug in the German localization of toolkit and mostly kept up with German SeaMonkey localization (except for stuff landed over the weekend).
• Various Discussions:
  Solaris and aus2-community, error console, EV cert UI, test failures, mail account autoconfig work, SeaMonkey statistics, Mac theme rework, etc.

Theoretically, we would already have or be about to freeze for SeaMonkey 2.0 Beta 1 right now. While this is being pushed out a bit due to our friends from Thunderbird not being ready to freeze for the next beta themselves, and us still missing the two larger features of download manager and tabmail, things start to feel more and more like being in beta phase. The feed integration suite is feature complete with having detection, preview and an internal reader for feeds now. Some minor polish might still be in order before final, but that part is surely beta-worthy now. I hope we get the last missing features as well as the better Mac theme in very soon so we can officially designate SeaMonkey 2.0 to be in beta and feature-complete, which will be a quite important milestone. Stay tuned here for more news on that and where where we're heading after that step!