The roads I take...

A few minutes ago, I landed the calendar code on comm-central for bug 455727.

This means that Lightning and Sunbird are moving to hg trunk development soon and that next to SeaMonkey and Thunderbird, Sunbird is the third application to be built from this community and communication repository.

Let's give the calendar people a warm welcome - we're looking forward to a good time of joint forces here!

As reported a few times in the last week, comm-central has been created as the new repository to host Thunderbird and SeaMonkey (and later Sunbird/Lightning) code on Mercurial and build it with the Mozilla 1.9.1 platform, on which SeaMonkey 2 and Thunderbird 3 will be based.

We populated the repository with code this Tuesday and since worked on getting our build and test boxes in shape so they build and test the new development trees.

The Thunderbird boxes are all green now and have nightlies, the SeaMonkey boxes build fine, but the testers are trying to do a lot more tests than the Thunderbird ones and so are running into a small number of problems still, mainly leaks in running mochitests, but also a unit test failure and a reftest failure on Mac, which we are still investigating.

Despite that, we are confident that development can be made somewhat reliably on top of this and so we opened comm-central for development for the first time now!

Please read MDC documentation on comm-central and the related documents linked there before doing active work with this.

But remember, cvs is dead in terms of active Thunderbird or SeaMonkey development!

There's a new Mercurial repository in town!

Reed just created http://hg.mozilla.org/comm-central/ - which is where Thunderbird and SeaMonkey, and later also calendar will live in the future.

The repository is empty for now, and will stay this way at least until shortly after Thunderbird/Shredder 3.0a2, code for which should be frozen on Tuesday. Calendar code will join even later than Thunderbird and SeaMonkey, we'll wait until after Sunbird/Lighting 0.9 for that, which will be the last release that supports 1.8 branch code. In the future, all our project will be focused on comm-central and based on Mozilla 1.9.1, we'll skip 1.9.0 completely from a release Point of view.

Until we switch, tracked by bug 437643, there are still some bugs to fix - we need some slight changes on mozilla-central, which are waiting for review, calendar code needs a few small changes so we can pull it in from CVS and build Lighting with Thunderbird (also waiting for review), and our build code changes need review, I'll reach out for that soon.
Of course, we still can need all the testing we can get on building the test repository, please report any problems to me, I'll try to fix them. I'm also already trying to get build machines up for SeaMonkey so we'll still have nightlies. And I have not yet tried all test suites, I only know we run unit tests as well as before.

All in all, a numbers of things are still to be done, but we are on track for switching to Mercurial soon!

After my recent post about SeaMonkey and Thunderbird on Mercurial, discussions started about possible solutions in both our projects as well as the Calendar project. I worked on the preliminary solution I had a bit further, and this week, we had a meeting of our projects with hg and build system experts, concluding that we'll create a new Mercurial repository shared between all three projects, and we'll already start off with more or less our own build system in place.
At the beginning, we'll import static snapshots of SeaMonkey and Thunderbird code, calendar will be pulled from cvs for now and added as another static snapshot once the calendar team is ready to switch to trunk for development. ("Static snapshot" in this case means current state of code without history, cvs/bonsai can be consulted for the history and work is ongoing to integrate the annotate view of hgweb with a database of older history to make it more feature-complete.)

I volunteered to create a test repository and work on getting things to build on local computers, eventually even work on test/build machines so that we have a good migration path.
For this purpose, I rewrote the SeaMonkey:hg-based_build page on the wiki to describe how to build with that test repository, and as of now, the instructions there (should) lead to fully working builds of SeaMonkey and hopefully also Thunderbird!

Please let me know of any problems when trying to build this or run the resulting binaries.

The next steps I'll work on are getting Lighting to build with Thunderbird, make tests run correctly, drive the two needed patches into mozilla-central and get my changes (build system, mainly) reviewed.

It looks like we can soon end up with a working code repository for our projects - what we still need is a good name for it. My first temporary name was "calemaisu" (calendar-mail-suite), internally, I'm currently using "momomo" (Mozilla Messaging, i.e. "MoMo" + SeaMonkey), which was a proposal in #maildev. There's still some need to discuss this topic between our projects, I guess.

It might seem that Mozilla people suddenly turned into chemists and are embracing the only liquid-under-normal-conditions metal, a toxic substance known as Hg or Mercury. Or they may just have learned that distributed version control systems (DVCS) have advantages over the old, file-based, centralized cvs system, and turn to a tool called "Mercurial", or "hg" for short, as a reasonable compromise between feature sets, well-working, well-maintained software and being able to run it decently on all our development platforms (including, even very prominently, a proprietary OS from a small company based in the town of Redmond, WA, USA).

The "trunk" of Mozilla development has moved to this tool at least, to a repository we call "mozilla-central" (people are working on getting this "hgweb" display nearer to feature-parity with the bonsai tool we used with cvs, including bug links and such stuff), the code in which is also web-browsable via MXR. With cvs HEAD now being basically the "1.9.0 branch" of Mozilla development and mozilla-central not including code outside of the Mozilla platform and Firefox, projects like SeaMonkey and Thunderbird need some changes if they still want to follow trunk development. As even Mozilla 1.9.1 will come off mozilla-central and the two mentioned projects may want to release their next big releases (Thunderbird 3 and SeaMonkey 2) off that (no firm decisions have been made yet), this topic gets even more pressing that originally imagined, when the move to hg was supposed to be for "Mozilla 2" only.

Over the last two weeks, I have intensely been investigating how we could do such a move, based on "option A" of a repository options doc that Benjamin Smedberg has thankfully written up for us, as most SeaMonkey and Thunderbird developers I talked to seemed to agree with Benjamin that this "option A" would be the best solution to go with, if it's feasible. Because of that, I worked on a proof-of-concept of this option - and amazingly, with a lot of help by Ted Mielczarek, I could figure that out more easily than I would have imagined.

And here's what I did:

Let's start with what this "option A" actually means. The basic building stone of this new build structure is that the mailnews/ directory with shared mail/news backend code as well as the Thunderbird-specific mail/ and SeaMonkey-specific suite/ directories can live in a shared hg repository, with the easy possibility of single changesets affecting all three (which we have frequently when a mailnews/ backend change needs e.g. locale string changes, which are done in both mail/ and suite/). While we can nest hg repos in a way that one subdirectory inside one repo is actually another repo that is being pulled in, we can't have a selection of subdirectories being in a different repo than another selection on the same level. What this means is that mail/, mailnews/ and suite/ can't live in the toplevel like in (the shared) cvs.

So, in this option, the new repository, which I call "mailsuite" for now, contains the said directories and pulls in mozilla-central into a mozilla/ subdirectory in parallel to them (we can automate that with a client.py script, of which I have a working version on my disk). As Ted proposed, we can still use the Mozilla build system to build our applications with one central tweak: In your mozconfig (or ./configure call), set --enable-application=../suite (or ../mail). This means that the top-level source directory for the Mozilla build system is actually the mozilla/ subdirectory of our mailsuite repository! Because of that, the specified objdir also needs to end in "/mozilla" - both those tweaks can be hidden with client.mk/configure scripts in the mailsuite repository in the future.

Starting with that, we need a list of rather boring changes to our "mailsuite" files to deal with the situation that they now live below $(topsrcdir)/../ instead of $(topsrcdir)/ from the view of the Mozilla build system, as well as some tweaks to set our own build variables ourselves, when they were set by e.g. Mozilla's configure script previously. Benjamin did a very fast patch to introduce app-config.mk/app-rules.mk which we can use to do the latter.

A full list of how one can get SeaMonkey or Thunderbird building with that structure is available in a wiki doc I wrote up, most of this will be done by e.g. client.* in our own mailsuite repository (including pulling in external extensions like ChatZilla or venkman) in the future or just checked into files in that repository.
Still, there are a few things left I still need to figure out:

• LDAP - directory/ needs to be either in mozilla-central or reworked to live in and build off mailsuite
• Dealing with SeaMonkey's built-in extensions, i.e. making them optional at build time but not require symlinking them into mozilla/extensions/
• Get make-makefile to work, Ted has promised to look into this
• Building Calendar (Lightning) from the same source structure, as a built-in extension to Thunderbird/SeaMonkey
• L10n: localized builds, L10n repackaging

The main thing here is though that I have working builds of SeaMonkey and Thunderbird built off that structure on my computer - I actually even could successfully run unit tests on SeaMonkey!

I personally think we can and should move to this structure (and to 1.9.1) for Thunderbird 3 and SeaMonkey 2, esp. as I'm told that there's the target to keep the 1.9.1 trunk in a shippable state at all times. We probably need to do some reconfiguration of our build and testing machines, but I think that's manageable.

The biggest open question (apart from the mostly technical issues listed above) is how we do the transition the this new repository. I could import a static snapshot of our cvs directories right now for testing and apply my local changes as a first changeset, but I'd rather have firm decisions if Thunderbird and SeaMonkey really want to go with that solution in the future and import the whole history into the final repo right away, as well as putting that transition changeset of mine right there.
For the time where we have both repositories (cvs and hg) in action, we still need to figure out if people should manually commit to both or if we use a script that merges in the changes from cvs (which was done for mozilla-central in for some time). And, of course, we need to figure out timeframes for all the needed steps.

So, all in all, we have still some lose ends to tie up, but we have a sketched out good way to go towards hosting SeaMonkey and Thunderbird in Mercurial - and I hope and am pretty sure it turns out less toxic than what the name "hg" suggests to me as an actual chemist.

I've been hearing about git for a long time as I already was following LWN.net closely when all those BitKeeper vs. Linux kernel problems came up and Linus started to work on a replacement. Meanwhile, over just two years, that technology has matured enough that it's being used by a wide rage of projects nowadays.

While the Mozilla project will use Mercurial ("hg") in the future, I suggested trying git a few times and probably annoyed decision makers a bit with that, but our big problem is that we need decent support on Windows, and git is not there yet (even though current development looks promising for the future). Mercurial is probably a good choice as well, interestingly it has been started about at the same time as git for practically the same reasons - and it's probably compatible in multiple ways: Not only are both distributed version control systems (DVCSes), they're both using SHA-1 hashes to identify revisions, and even screen shots of their graphical tools look similar.

In any case, I've personally never used anything else than CVS, and that aged, centralized, per-file version control system has served me well for my private projects. I understand its usage as well as its repository quite well and I hardly ever run into its real problems like file moving or such.

That said, I always found git's concept of tracking content instead of files interesting (that's why file moves are more or less irrelevant to it), and tracking changesets instead of single-file-changes could also prove useful in my personal projects. Oh, and then there's one thing that really attracts me about DCVS solutions: I'm doing lots of work when I'm on a train without a net connection, history can prove useful to have there, as well as tracking several different changes to a single file - and then I'm merging this to my other computer and two production systems (or actually, checking in to a central repository and check out on those systems, in the CVS model). And then, learning new technologies is always a good thing.

So, I had enough reasons to try out git for my personal projects, and I spent about 6 hours today figuring out how to work with it, a set of directories I wanted to import, and how to import those from CVS. So far, it looks like everything has worked. Oh, and the PHP code generating this website and blog (the CBSM system) is now running off a git checkout.

After some time of working with this, I'll see if I want to git even more code into such repositories - for now, let's see how this works out