The roads I take...

As reported a few times in the last week, comm-central has been created as the new repository to host Thunderbird and SeaMonkey (and later Sunbird/Lightning) code on Mercurial and build it with the Mozilla 1.9.1 platform, on which SeaMonkey 2 and Thunderbird 3 will be based.

We populated the repository with code this Tuesday and since worked on getting our build and test boxes in shape so they build and test the new development trees.

The Thunderbird boxes are all green now and have nightlies, the SeaMonkey boxes build fine, but the testers are trying to do a lot more tests than the Thunderbird ones and so are running into a small number of problems still, mainly leaks in running mochitests, but also a unit test failure and a reftest failure on Mac, which we are still investigating.

Despite that, we are confident that development can be made somewhat reliably on top of this and so we opened comm-central for development for the first time now!

Please read MDC documentation on comm-central and the related documents linked there before doing active work with this.

But remember, cvs is dead in terms of active Thunderbird or SeaMonkey development!

There's a new Mercurial repository in town!

Reed just created http://hg.mozilla.org/comm-central/ - which is where Thunderbird and SeaMonkey, and later also calendar will live in the future.

The repository is empty for now, and will stay this way at least until shortly after Thunderbird/Shredder 3.0a2, code for which should be frozen on Tuesday. Calendar code will join even later than Thunderbird and SeaMonkey, we'll wait until after Sunbird/Lighting 0.9 for that, which will be the last release that supports 1.8 branch code. In the future, all our project will be focused on comm-central and based on Mozilla 1.9.1, we'll skip 1.9.0 completely from a release Point of view.

Until we switch, tracked by bug 437643, there are still some bugs to fix - we need some slight changes on mozilla-central, which are waiting for review, calendar code needs a few small changes so we can pull it in from CVS and build Lighting with Thunderbird (also waiting for review), and our build code changes need review, I'll reach out for that soon.
Of course, we still can need all the testing we can get on building the test repository, please report any problems to me, I'll try to fix them. I'm also already trying to get build machines up for SeaMonkey so we'll still have nightlies. And I have not yet tried all test suites, I only know we run unit tests as well as before.

All in all, a numbers of things are still to be done, but we are on track for switching to Mercurial soon!

After my recent post about SeaMonkey and Thunderbird on Mercurial, discussions started about possible solutions in both our projects as well as the Calendar project. I worked on the preliminary solution I had a bit further, and this week, we had a meeting of our projects with hg and build system experts, concluding that we'll create a new Mercurial repository shared between all three projects, and we'll already start off with more or less our own build system in place.
At the beginning, we'll import static snapshots of SeaMonkey and Thunderbird code, calendar will be pulled from cvs for now and added as another static snapshot once the calendar team is ready to switch to trunk for development. ("Static snapshot" in this case means current state of code without history, cvs/bonsai can be consulted for the history and work is ongoing to integrate the annotate view of hgweb with a database of older history to make it more feature-complete.)

I volunteered to create a test repository and work on getting things to build on local computers, eventually even work on test/build machines so that we have a good migration path.
For this purpose, I rewrote the SeaMonkey:hg-based_build page on the wiki to describe how to build with that test repository, and as of now, the instructions there (should) lead to fully working builds of SeaMonkey and hopefully also Thunderbird!

Please let me know of any problems when trying to build this or run the resulting binaries.

The next steps I'll work on are getting Lighting to build with Thunderbird, make tests run correctly, drive the two needed patches into mozilla-central and get my changes (build system, mainly) reviewed.

It looks like we can soon end up with a working code repository for our projects - what we still need is a good name for it. My first temporary name was "calemaisu" (calendar-mail-suite), internally, I'm currently using "momomo" (Mozilla Messaging, i.e. "MoMo" + SeaMonkey), which was a proposal in #maildev. There's still some need to discuss this topic between our projects, I guess.

It might seem that Mozilla people suddenly turned into chemists and are embracing the only liquid-under-normal-conditions metal, a toxic substance known as Hg or Mercury. Or they may just have learned that distributed version control systems (DVCS) have advantages over the old, file-based, centralized cvs system, and turn to a tool called "Mercurial", or "hg" for short, as a reasonable compromise between feature sets, well-working, well-maintained software and being able to run it decently on all our development platforms (including, even very prominently, a proprietary OS from a small company based in the town of Redmond, WA, USA).

The "trunk" of Mozilla development has moved to this tool at least, to a repository we call "mozilla-central" (people are working on getting this "hgweb" display nearer to feature-parity with the bonsai tool we used with cvs, including bug links and such stuff), the code in which is also web-browsable via MXR. With cvs HEAD now being basically the "1.9.0 branch" of Mozilla development and mozilla-central not including code outside of the Mozilla platform and Firefox, projects like SeaMonkey and Thunderbird need some changes if they still want to follow trunk development. As even Mozilla 1.9.1 will come off mozilla-central and the two mentioned projects may want to release their next big releases (Thunderbird 3 and SeaMonkey 2) off that (no firm decisions have been made yet), this topic gets even more pressing that originally imagined, when the move to hg was supposed to be for "Mozilla 2" only.

Over the last two weeks, I have intensely been investigating how we could do such a move, based on "option A" of a repository options doc that Benjamin Smedberg has thankfully written up for us, as most SeaMonkey and Thunderbird developers I talked to seemed to agree with Benjamin that this "option A" would be the best solution to go with, if it's feasible. Because of that, I worked on a proof-of-concept of this option - and amazingly, with a lot of help by Ted Mielczarek, I could figure that out more easily than I would have imagined.

And here's what I did:

Let's start with what this "option A" actually means. The basic building stone of this new build structure is that the mailnews/ directory with shared mail/news backend code as well as the Thunderbird-specific mail/ and SeaMonkey-specific suite/ directories can live in a shared hg repository, with the easy possibility of single changesets affecting all three (which we have frequently when a mailnews/ backend change needs e.g. locale string changes, which are done in both mail/ and suite/). While we can nest hg repos in a way that one subdirectory inside one repo is actually another repo that is being pulled in, we can't have a selection of subdirectories being in a different repo than another selection on the same level. What this means is that mail/, mailnews/ and suite/ can't live in the toplevel like in (the shared) cvs.

So, in this option, the new repository, which I call "mailsuite" for now, contains the said directories and pulls in mozilla-central into a mozilla/ subdirectory in parallel to them (we can automate that with a client.py script, of which I have a working version on my disk). As Ted proposed, we can still use the Mozilla build system to build our applications with one central tweak: In your mozconfig (or ./configure call), set --enable-application=../suite (or ../mail). This means that the top-level source directory for the Mozilla build system is actually the mozilla/ subdirectory of our mailsuite repository! Because of that, the specified objdir also needs to end in "/mozilla" - both those tweaks can be hidden with client.mk/configure scripts in the mailsuite repository in the future.

Starting with that, we need a list of rather boring changes to our "mailsuite" files to deal with the situation that they now live below $(topsrcdir)/../ instead of $(topsrcdir)/ from the view of the Mozilla build system, as well as some tweaks to set our own build variables ourselves, when they were set by e.g. Mozilla's configure script previously. Benjamin did a very fast patch to introduce app-config.mk/app-rules.mk which we can use to do the latter.

A full list of how one can get SeaMonkey or Thunderbird building with that structure is available in a wiki doc I wrote up, most of this will be done by e.g. client.* in our own mailsuite repository (including pulling in external extensions like ChatZilla or venkman) in the future or just checked into files in that repository.
Still, there are a few things left I still need to figure out:

• LDAP - directory/ needs to be either in mozilla-central or reworked to live in and build off mailsuite
• Dealing with SeaMonkey's built-in extensions, i.e. making them optional at build time but not require symlinking them into mozilla/extensions/
• Get make-makefile to work, Ted has promised to look into this
• Building Calendar (Lightning) from the same source structure, as a built-in extension to Thunderbird/SeaMonkey
• L10n: localized builds, L10n repackaging

The main thing here is though that I have working builds of SeaMonkey and Thunderbird built off that structure on my computer - I actually even could successfully run unit tests on SeaMonkey!

I personally think we can and should move to this structure (and to 1.9.1) for Thunderbird 3 and SeaMonkey 2, esp. as I'm told that there's the target to keep the 1.9.1 trunk in a shippable state at all times. We probably need to do some reconfiguration of our build and testing machines, but I think that's manageable.

The biggest open question (apart from the mostly technical issues listed above) is how we do the transition the this new repository. I could import a static snapshot of our cvs directories right now for testing and apply my local changes as a first changeset, but I'd rather have firm decisions if Thunderbird and SeaMonkey really want to go with that solution in the future and import the whole history into the final repo right away, as well as putting that transition changeset of mine right there.
For the time where we have both repositories (cvs and hg) in action, we still need to figure out if people should manually commit to both or if we use a script that merges in the changes from cvs (which was done for mozilla-central in for some time). And, of course, we need to figure out timeframes for all the needed steps.

So, all in all, we have still some lose ends to tie up, but we have a sketched out good way to go towards hosting SeaMonkey and Thunderbird in Mercurial - and I hope and am pretty sure it turns out less toxic than what the name "hg" suggests to me as an actual chemist.

(First of all, my apologies, I know I'm late for my weekly status update, I promise to deliver it, but I've been working on my talk for FOSDEM this weekend.)

The big news of this week is Mozilla Messaging being launched as the effort that should parallel Mozilla Corporation's Firefox effort in the email and messaging area by further developing Thunderbird.

The question this might raise with a few people around the Mozilla community is what this means to the SeaMonkey project, which shares a substantial amount of code with Thunderbird in the mail and news area.

My opinion on that is that it's cool that we now have a partner in mail and news development who takes messaging much more serious than Mozilla Corporation ever could. Mozilla Messaging does not emphasize a browser over all other products, it actually doesn't even develop a browser. This makes us sure we have an additional ally in the area of Mozilla(-based) applications that are looking beyond the browser. Even more, putting full-time developers, new ideas and visions behind the Thunderbird project might gives the mail/news code a new future and puts more development force behind that than what the SeaMonkey project and some assorted other volunteers could provide so far.

As the SeaMonkey vision is - in my eyes - to have "the Internet" as the center for browsing, messaging and more on your desktop (in our case as a single, integrated application), I'm really happy about Mozilla Messaging emphasizing that Mozilla can actually provide all that - and not "only" a damn good browser.

I (and hopefully all of my colleagues on both sides) will try to get as much cooperation between Mozilla Messaging and the SeaMonkey project as possible, so that we can drive forward both our products, which both have their respective place on their markets, as much as possible.

I'm convinced we both can profit from each other and the Mozilla mission of choice on the Internet has always shown that our community is really good at providing nothing less than excellence in fulfilling that mission.

I've just read an interesting blog post about rewrites in MailNews, which would improve both Thunderbird and SeaMonkey by replacing very long-lived, historically grown, patched and hacked interfaces in the mail and news code by interfaces that cleanly apply to present architecture and are again open for relatively clean additions in the future. This doesn't mean the current code was that bad at the time it was created, it was made for the state of mailnews in those times, and those times being almost 10 years ago (early Mozilla code) or even before (code we inherited from Netscape) clearly explains why things like message tags or feed account types require some sort of hacks to apply cleanly to the old code. Also message meta info being stored at folder (newsgroup) level instead of account level means that it's e.g. hard to mark cross-posted message read in multiple newsgroups. A side effect of those reworks is also that old storage formats like Mork databases and RDF datasources can be replaced by more modern backends like the SQLite-backed mozStorage mechanism.

Joshua Cranmer, who wrote the blog post I cited above, is currently trying to get a picture of where we want to end up with such rewrites, how we can get rid of Mork ("kill-mork" work) and many RDF usages ("kill-RDF" work), and how to design interfaces in a way that they are fit for future enhancements.
When the picture gets clearer, he intends to work on kill-mork tasks in address book first, from what I read.

I am looking forward to seeing this work evolve, I think it could improve some of the code areas that many people are unwilling to touch because of their historically grown clumsiness - and in open source, it's always bad if people refrain to touch code because they don't fully understand it.

There has been some rumbling today about the future of Thunderbird, starting with Mitchell's call to action. Thunderbird has been a step-child of the Mozilla Corporation for quite some time while it spearheaded the rediscovery of the open web by spreading Firefox. This is an important job, but a company of perhaps 150 people or so working on Firefox and two working on Thunderbird just doesn't work out. I guess that time has come when MoCo has to split off the step-child mail client and become the Firefox Corporation that it has been acting as for a while now. In some way, mail client and browser get separated once again, some years after doing that on the technical application level.

This is a possibility for Thunderbird to step out of the shadow of its overwhelming browser brother if it's done right. Almost everyone, including Scott as the head of Thunderbird development, is thinking the best option is to form Thunderbird into a community project like SeaMonkey, and Sunbird (or even Camino-like-Firefox).
I also think that a Thunderbird community project is a good solution for stepping out of Firefox' shadow, but I hope the organizational overhead can be kept as low as possible. We don't have any official organization for SeaMonkey, being backed "only" by the Mozilla Foundation, and that structure works well (the SeaMonkey development community might be bigger than Thunderbird's at the moment, despite probably the opposite is true for user numbers). Of course, Scott and David should still be paid for Thunderbird development, so there has to be some form of organization. Maybe we should even think about an organization that backs multiple Mozilla-based projects, or this can even be done via the Foundation itself? Let's see how this turns out.

In any case, the SeaMonkey project will happily team up with the Thunderbird project where useful, it's in the interest of both our projects to have a healthy and well-developed email and newsgroup backend, so we both can deliver top-quality software to our respective target audiences.
I wish the Thunderbird good luck on this interesting journey and hope that we can continue working well together for a good future of Mozilla-based Mail and News clients.