The roads I take...

After my recent post about SeaMonkey and Thunderbird on Mercurial, discussions started about possible solutions in both our projects as well as the Calendar project. I worked on the preliminary solution I had a bit further, and this week, we had a meeting of our projects with hg and build system experts, concluding that we'll create a new Mercurial repository shared between all three projects, and we'll already start off with more or less our own build system in place.
At the beginning, we'll import static snapshots of SeaMonkey and Thunderbird code, calendar will be pulled from cvs for now and added as another static snapshot once the calendar team is ready to switch to trunk for development. ("Static snapshot" in this case means current state of code without history, cvs/bonsai can be consulted for the history and work is ongoing to integrate the annotate view of hgweb with a database of older history to make it more feature-complete.)

I volunteered to create a test repository and work on getting things to build on local computers, eventually even work on test/build machines so that we have a good migration path.
For this purpose, I rewrote the SeaMonkey:hg-based_build page on the wiki to describe how to build with that test repository, and as of now, the instructions there (should) lead to fully working builds of SeaMonkey and hopefully also Thunderbird!

Please let me know of any problems when trying to build this or run the resulting binaries.

The next steps I'll work on are getting Lighting to build with Thunderbird, make tests run correctly, drive the two needed patches into mozilla-central and get my changes (build system, mainly) reviewed.

It looks like we can soon end up with a working code repository for our projects - what we still need is a good name for it. My first temporary name was "calemaisu" (calendar-mail-suite), internally, I'm currently using "momomo" (Mozilla Messaging, i.e. "MoMo" + SeaMonkey), which was a proposal in #maildev. There's still some need to discuss this topic between our projects, I guess.

It might seem that Mozilla people suddenly turned into chemists and are embracing the only liquid-under-normal-conditions metal, a toxic substance known as Hg or Mercury. Or they may just have learned that distributed version control systems (DVCS) have advantages over the old, file-based, centralized cvs system, and turn to a tool called "Mercurial", or "hg" for short, as a reasonable compromise between feature sets, well-working, well-maintained software and being able to run it decently on all our development platforms (including, even very prominently, a proprietary OS from a small company based in the town of Redmond, WA, USA).

The "trunk" of Mozilla development has moved to this tool at least, to a repository we call "mozilla-central" (people are working on getting this "hgweb" display nearer to feature-parity with the bonsai tool we used with cvs, including bug links and such stuff), the code in which is also web-browsable via MXR. With cvs HEAD now being basically the "1.9.0 branch" of Mozilla development and mozilla-central not including code outside of the Mozilla platform and Firefox, projects like SeaMonkey and Thunderbird need some changes if they still want to follow trunk development. As even Mozilla 1.9.1 will come off mozilla-central and the two mentioned projects may want to release their next big releases (Thunderbird 3 and SeaMonkey 2) off that (no firm decisions have been made yet), this topic gets even more pressing that originally imagined, when the move to hg was supposed to be for "Mozilla 2" only.

Over the last two weeks, I have intensely been investigating how we could do such a move, based on "option A" of a repository options doc that Benjamin Smedberg has thankfully written up for us, as most SeaMonkey and Thunderbird developers I talked to seemed to agree with Benjamin that this "option A" would be the best solution to go with, if it's feasible. Because of that, I worked on a proof-of-concept of this option - and amazingly, with a lot of help by Ted Mielczarek, I could figure that out more easily than I would have imagined.

And here's what I did:

Let's start with what this "option A" actually means. The basic building stone of this new build structure is that the mailnews/ directory with shared mail/news backend code as well as the Thunderbird-specific mail/ and SeaMonkey-specific suite/ directories can live in a shared hg repository, with the easy possibility of single changesets affecting all three (which we have frequently when a mailnews/ backend change needs e.g. locale string changes, which are done in both mail/ and suite/). While we can nest hg repos in a way that one subdirectory inside one repo is actually another repo that is being pulled in, we can't have a selection of subdirectories being in a different repo than another selection on the same level. What this means is that mail/, mailnews/ and suite/ can't live in the toplevel like in (the shared) cvs.

So, in this option, the new repository, which I call "mailsuite" for now, contains the said directories and pulls in mozilla-central into a mozilla/ subdirectory in parallel to them (we can automate that with a client.py script, of which I have a working version on my disk). As Ted proposed, we can still use the Mozilla build system to build our applications with one central tweak: In your mozconfig (or ./configure call), set --enable-application=../suite (or ../mail). This means that the top-level source directory for the Mozilla build system is actually the mozilla/ subdirectory of our mailsuite repository! Because of that, the specified objdir also needs to end in "/mozilla" - both those tweaks can be hidden with client.mk/configure scripts in the mailsuite repository in the future.

Starting with that, we need a list of rather boring changes to our "mailsuite" files to deal with the situation that they now live below $(topsrcdir)/../ instead of $(topsrcdir)/ from the view of the Mozilla build system, as well as some tweaks to set our own build variables ourselves, when they were set by e.g. Mozilla's configure script previously. Benjamin did a very fast patch to introduce app-config.mk/app-rules.mk which we can use to do the latter.

A full list of how one can get SeaMonkey or Thunderbird building with that structure is available in a wiki doc I wrote up, most of this will be done by e.g. client.* in our own mailsuite repository (including pulling in external extensions like ChatZilla or venkman) in the future or just checked into files in that repository.
Still, there are a few things left I still need to figure out:

• LDAP - directory/ needs to be either in mozilla-central or reworked to live in and build off mailsuite
• Dealing with SeaMonkey's built-in extensions, i.e. making them optional at build time but not require symlinking them into mozilla/extensions/
• Get make-makefile to work, Ted has promised to look into this
• Building Calendar (Lightning) from the same source structure, as a built-in extension to Thunderbird/SeaMonkey
• L10n: localized builds, L10n repackaging

The main thing here is though that I have working builds of SeaMonkey and Thunderbird built off that structure on my computer - I actually even could successfully run unit tests on SeaMonkey!

I personally think we can and should move to this structure (and to 1.9.1) for Thunderbird 3 and SeaMonkey 2, esp. as I'm told that there's the target to keep the 1.9.1 trunk in a shippable state at all times. We probably need to do some reconfiguration of our build and testing machines, but I think that's manageable.

The biggest open question (apart from the mostly technical issues listed above) is how we do the transition the this new repository. I could import a static snapshot of our cvs directories right now for testing and apply my local changes as a first changeset, but I'd rather have firm decisions if Thunderbird and SeaMonkey really want to go with that solution in the future and import the whole history into the final repo right away, as well as putting that transition changeset of mine right there.
For the time where we have both repositories (cvs and hg) in action, we still need to figure out if people should manually commit to both or if we use a script that merges in the changes from cvs (which was done for mozilla-central in for some time). And, of course, we need to figure out timeframes for all the needed steps.

So, all in all, we have still some lose ends to tie up, but we have a sketched out good way to go towards hosting SeaMonkey and Thunderbird in Mercurial - and I hope and am pretty sure it turns out less toxic than what the name "hg" suggests to me as an actual chemist.