Automated SeaMonkey Testing

As many of you know, one of the new addition to Firefox development between the Gecko-1.8.1-based Firefox 2 and the Gecko-1.9-based Firefox 3 releases was performing automated tests on all kinds of functionality in this software. This greatly improves software quality by having constant checking of a good part of this complex system and being able to early spot problems introduced in code and revert/correct those.

Starting this week, we also have machines that run a broad range of automated tests on SeaMonkey trunk (the Gecko-1.9-based code that will end up in SeaMonkey 2). Currently, we have Windows and Linux boxes (bug for a Mac box is filed - unfortunately, Apple doesn't support those being run as VMs), reporting to the SeaMonkey-Ports tinderbox waterfall page - and failing a number of tests.

The test failures we see with those machines have different causes:

• The mochitest framework has some focus problems, and generating a keypress event doesn't have the intended effect when the wrong window has focus. Applying the patch from bug 431464 helped with the vast amount of those problems (thanks, Andrew, for figuring that out).
• The remaining mochitests failures on Windows might be because not the whole iframe is in focus and some generated mouse events don't point into it, maybe due to a too small window size. We need to look into improving that.
• Failures in browser chrome tests due to multiple problems, including use of Firefox-only functionality in platform tests, testing a download manager that is not (yet) built for SeaMonkey, and some errors in SeaMonkey sanitizer tests. I could create a patch for those which is currently under review.

The machines will be switched to report to the main SeaMonkey tinderbox waterfall page and be required not to break in future development, once we have fixed those problems and all tests pass.

We have run the unit tests for Linux in our main tinderbox for a while now, but without detailed reporting of numbers, and without coverage of reftests, crashtests, mochitests, chrome or browser tests or other platforms. We now have all those test suites running on two platforms, hopefully adding Mac "soon".

Even a number of about 51500 passing tests doesn't mean that a lot of SeaMonkey-specific code is tested, though. What we're currently are running, are mainly platform tests we share with Firefox - we only have a total of 1 suite and 14 mailnews unit tests being run, and the sanitizer tests account for 31 passed browser-chrome tests (with my patch applies so they actually pass). All other SeaMonkey application code is still not covered by such tests.

What this means is that even though we have machines running the tests, we badly need tests for SeaMonkey code (including mailnews [hint for Thunderbird people reading this]) to be checked in so they get run. Still, we know that the platform still works fine in the configuration used by SeaMonkey, which is good to know and already a major improvement to earlier times.

I'm convinced that amount of automated testing will even improve the jewel of product that SeaMonkey 2 will be. That said, automated testing can only augment manual testing by our community, and never replace it, so please keep on testing our nightly builds - those tests will help to make us not have bad breakage in those nightlies though (at least not for long - it's always a good idea to check for green tinderboxes before downloading and using a nightly, as I hope you know).