<< Weekly Status Report, W17/2008 (+W15/W16) | The roads I take... | Weekly Status Report, W18/2008 >>

Automated SeaMonkey Testing

As many of you know, one of the new addition to Firefox development between the Gecko-1.8.1-based Firefox 2 and the Gecko-1.9-based Firefox 3 releases was performing automated tests on all kinds of functionality in this software. This greatly improves software quality by having constant checking of a good part of this complex system and being able to early spot problems introduced in code and revert/correct those.

Starting this week, we also have machines that run a broad range of automated tests on SeaMonkey trunk (the Gecko-1.9-based code that will end up in SeaMonkey 2). Currently, we have Windows and Linux boxes (bug for a Mac box is filed - unfortunately, Apple doesn't support those being run as VMs), reporting to the SeaMonkey-Ports tinderbox waterfall page - and failing a number of tests. :(

The test failures we see with those machines have different causes:
The machines will be switched to report to the main SeaMonkey tinderbox waterfall page and be required not to break in future development, once we have fixed those problems and all tests pass.

We have run the unit tests for Linux in our main tinderbox for a while now, but without detailed reporting of numbers, and without coverage of reftests, crashtests, mochitests, chrome or browser tests or other platforms. We now have all those test suites running on two platforms, hopefully adding Mac "soon".

Even a number of about 51500 passing tests doesn't mean that a lot of SeaMonkey-specific code is tested, though. What we're currently are running, are mainly platform tests we share with Firefox - we only have a total of 1 suite and 14 mailnews unit tests being run, and the sanitizer tests account for 31 passed browser-chrome tests (with my patch applies so they actually pass). All other SeaMonkey application code is still not covered by such tests.

What this means is that even though we have machines running the tests, we badly need tests for SeaMonkey code (including mailnews [hint for Thunderbird people reading this]) to be checked in so they get run. Still, we know that the platform still works fine in the configuration used by SeaMonkey, which is good to know and already a major improvement to earlier times.

I'm convinced that amount of automated testing will even improve the jewel of product that SeaMonkey 2 will be. That said, automated testing can only augment manual testing by our community, and never replace it, so please keep on testing our nightly builds - those tests will help to make us not have bad breakage in those nightlies though (at least not for long - it's always a good idea to check for green tinderboxes before downloading and using a nightly, as I hope you know).

Entry written by KaiRo and posted on May 1st, 2008 15:41 | Tags: Mozilla, SeaMonkey, tests | 2 comments | TrackBack



Smokey Ardisson

Thanks for finding all that Firefox-only functionality stuck in core tests! I noticed a bunch of failures on odd things when I was working on getting Mochitest running for Camino a few months ago but didn't have time to investigate. Hopefully your work here means fewer of those failures when I get back to work on that project.
2008-05-01 17:54



Hmm, the main thing I did there was the work to get chrome-chrome tests pass on SeaMonkey, which is probably not a suite that Camino wants to run at all (I guess that suite requires a XUL UI).
Andrew's work for the mochitest focus stuff might be helpful to even Firefox, which sees those problems randomly as well. If the mochitest suite basically works for Camino, that work might possibly help you as well, I guess.
2008-05-04 00:51

Add comment