The roads I take...

The OpenStreetMap project has launched an all-in-JavaScript map editor called "iD" this week:



While we at Mozilla know you can do a lot of good things in JS these days - after all, we're even launching our own phone OS building fully on HTML+JS, and we have been using more and more JS code to power key functionality in our browsers and other products over the years - it's great to see that complex things like editing maps can be done fully in JS and available for all platforms now, while previously it took proprietary and availability-limited technologies like Flash or Java to do the same thing.

Great work, OpenStreetMap guys!


(And yes, as a contributor to OpenStreetMap and even OSMF member, I am biased, but free and open map data on the web fits Mozilla philosophy pretty well anyhow...)

In the last days, I did some JavaScript performance tests on different SeaMonkey versions on my machine, just to find out where we are heading for SeaMonkey 2.1 in comparison to what we shipped in the past:



I ran 3 tests there: my personal Mandelbrot demo 1.0 with default settings (blue), the pretty common SunSpider benchmark (red), and the new Kraken benchmark (yellow). Note that the scale of Kraken results is different to the scale of the other two.

(Update/Note: I was told that my Mandelbrot demo uses JavaScript 1.7 and HTML5 features that Mozilla 1.8 supported but other browsers still don't, so there's a more cross-browser Mandelbrot demo 2.0 now - that one performs even a tad better on Mozilla trunk than the results here.)

The builds I tested were current pre-2.1 trunk builds, which already have the first stage of the new JaegerMonkey JavaScript engine, 2.0 branch builds without and with the venkman JavaScript Debugger add-on enabled (without JSD means that TraceMonkey can fully work, as activating venkman makes JSD put JavaScript into a non-tracing debug mode) - and finally, the 1.1.19 release build.

Unfortunately, I can't get data for Kraken on 1.1.19, as after a very very long time of running (given the "slow JavaScript" warning is deactivated), it errors out with not finding any JSON support. Comparing to the other tests, it suffices to say that SeaMonkey 1.1.x, coming from the Mozilla 1.8 tree, is very slow - but then, that's the baseline we started from.

SeaMonkey 2.0.x - even with venkman on and JSD knocking tracing off - is 90% or more faster than 1.1.x in those tests we can run in the older version. Suffice to say that Mozilla's JS team has made tremendous improvements between the 1.8 and 1.9.1 branches of the platform. One might think it's hard to go even further after such a jump, and indeed it has become much harder - but things could still get better. First, turning the debugger off and tracing on wins 19% in SunSpider, 13% for Mandelbrot and 2% for Kraken (the latter either doesn't have a lot of tight loops, which is what tracing help most with, or we are bad at tracing them in this version). At this level, SeaMonkey 2.0 looks already nice - but that's still 1-2 years old code, and development continued.

The current pre-2.1 builds of SeaMonkey bring another 61% win on Mandelbrot, 66% on SunSpider and 72% on Kraken - over 2.0 with full tracing! While the basic Mandelbrot set image took over 25 seconds to calculate and display on 1.1 and significantly over a second on 2.0, we're under half a second now for pre-2.1 code. And, as I already mentioned, JaegerMonkey work isn't even finished yet, there's a few more things to come there.

Of course, the question is if what those tests run is what Web sites and applications really need - we know a few things in SunSpider are non-realistic workloads. Kraken tries to look into CPU-intense workloads that could be useful for actual web applications, esp. if they want to to things like image and audio processing. My Mandelbrot set demo may look far-fetched, but then, I can surely see websites or even more web games dynamically generating fractal-based shades and textures based on variable parameters, and there you might have very similar code.

For those people who want the hard facts, here's the numbers for my graph with all details:

• Mozilla/5.0 (X11; Linux i686; rv:2.0b7pre) Gecko/20100917 Firefox/4.0b7pre SeaMonkey/2.1b1pre
  • Mandelbrot: 483.0ms +/- 1.4% (0.481, 0.492, 0.474, 0.487, 0.481)
  • SunSpider: 477.4ms +/- 1.1%
  • Kraken: 11151.6ms +/- 0.5%
• Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.13pre) Gecko/20100913 Lightning/1.0b2pre SeaMonkey/2.0.9pre
  • Mandelbrot: 1421.8ms +/- 5.0% (1.336, 1.401, 1.497, 1.383, 1.492)
  • SunSpider: 1738.0ms +/- 2.6%
  • Kraken: 40428.1ms +/- 1.6%
• Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.13pre) Gecko/20100913 Lightning/1.0b2pre SeaMonkey/2.0.9pre (venkman disabled)
  • Mandelbrot: 1237.6ms +/- 3.3% (1.220, 1.216, 1.301, 1.197, 1.254)
  • SunSpider: 1402.2ms +/- 3.6%
  • Kraken: 39558.9ms +/- 1.5%
• Mozilla/5.0 (X11; U; Linux i686; de-AT; rv:1.8.1.24) Gecko/20100301 SeaMonkey/1.1.19
  • Mandelbrot: 25214.2ms +/- 1.0% (25.176, 24.907, 25.180, 25.216, 25.592)
  • SunSpider: 17501.2ms +/- 1.2%
  • Kraken: Runs slowly for a while, but does break for not finding JSON handling.

All in all, the Mozilla JavaScript team is really enabling a lot of new possibilities here, as now we can do things client-side we needed to do server-side up to now, and that makes completely new dynamics possible. And SeaMonkey fully features those improvements made by the Mozilla community.

Die Frage nach Geschwindigkeitsverbesserungen bei SeaMonkey kam kürzlich auf, und nachdem ein großer Teil der JavaScript-Verbesserungen für die nächste Version dieses Wochenende in Entwicklerversionen eingespielt wurden, hab ich heute mal die populäre SunSpider-Benchmark mit verschiedenen SeaMonkey-Versionen laufen lassen.

Der letzte aus der 1.x-Serie, SeaMonkey 1.1.19, der vor einem Monaten die Wartung dieser Versionen beendete, brauchte auf meiner Maschine 17501ms für diese Tests - das also war unsere Startlinie in Erneuerungen.

Eine aktuelle SeaMonkey 2.0.x-Version kommt auf der gleichen Maschine auf 1738ms mit aktiviertem JavaScript-Debugger (Standardeinstellung) bzw. 1402ms, wenn dieser deaktiviert wird. Dass das Deaktivieren des Debuggers unter 2.0 einen Vorteil bringt, liegt daran, dass dort bei seiner Aktivierung die TraceMonkey-Beschleunigung ausgeschaltet wird - trotzdem ist auch ohne dieser eine gewaltige Verbesserung gegenüber 1.x feststellbar, braucht der Test doch nur mehr 1/10 der vorherigen Zeit.

Aber in der heutigen Welt kann das nicht genug sein, und für weitere Verbesserungen hat SeaMonkey 2.0 bloß eine neue Startlinie gesetzt - die Mozilla-Gemeinde ruht nicht und arbeitet intensiv weiter.

Die heutige SeaMonkey-2.1-Vorversion integriert die neuen Arbeiten an "JaegerMonkey" für noch schnellere JavaScript-Ausführung und landet nun bei 477ms für diesen Test auf meiner Maschine, nochmal auf 1/3 gegenüber 2.0 reduziert!

Dabei muss man bemerken, dass die Arbeiten an JaegerMonkey noch lange nicht fertig sind, einige Dinge können hier noch signifikant verbessert werden, und Mozilla's JavaScript-Engine-Entwickler arbeiten bereits kräftig daran. Ich denke, SeaMonkey bewegt sich (wie Firefox) in die richtige Richtung.

Prompted by Wladmimir's post on the perf cost of venkman, I filed a bug on getting rid of that cost in SeaMonkey, either by fixing the actual problem or not enabling whatever causes this slowdown - even if (what I don't hope) it would mean not shipping venkman in the worst case.

With JS perf tests being the number one rocks/sucks meter of browsers right now, esp. with advanced user and web developers, who are a very major part of our target audience, we cannot afford to look worse than Firefox in SunSpider and similar benchmarks.

But here's the real surprise moment I had in all that:

To get some useful data of what the penalty is and if there a significant difference to the old SeaMonkey 1.1.x series in that, I did run SunSpider in a current comm-central/mozilla-1.9.1-based SeaMonkey 2.0b1pre build and a mozilla-1.8.1-based SeaMonkey 1.1.16pre build:

SeaMonkey 1.1.16pre: 15719.8ms +/- 1.5%

SeaMonkey 2.0b1pre (venkman disabled, JIT.content disabled!): 2751.6ms +/- 2.1%

Now that's impressive. This is interpreted code, without just-in-time compilation, and it's still very significantly faster than the old 1.8.1 JS engine. Good work, SpiderMonkey developers!
And yes, this are results on the same machine, under the same testing conditions, with mostly empty testing profiles.

With JIT, we're even faster, of course:

SeaMonkey 2.0b1pre (venkman disabled): 1954.8ms +/- 9.4%

Unfortunately, the penalty of just having venkman enabled is real though, resulting in a ~17% slowdown in my tests:

SeaMonkey 2.0b1pre (default config): 2289.8ms +/- 10.7%

We really need to find a solution for that, ideally so that we can give our users a JS debugger available in the suite by default but still give everyone at any time where he doesn't want to actually debug JS code the best possible performance.

I continue to be impressed with the work the SpiderMonkey/TraceMonkey team did, though, the current 1.9.1 JS engine is already so much better than what we had in 1.8.1, and it's not even finished up yet.

Everybody is writing about TraceMonkey today, the just-landed-on-trunk version of the SpiderMonkey JavaScript engine with "tracing" JIT compilation for speeding up code that is run repeatedly.

What I wondered is how fast it is in code that is no benchmark but stuff a real application might use. And as I have my "Mandelbrot" fun project as a XULRunner app (see the last paragraph of recent status update post), which naturally does a lot of looping over the same equation, so I thought it might be a good candidate for optimization.

So I fired up the XULRunner app, doing four testruns of "time mandelbrot" (launching a script that runs my XULRunner build from today's trunk code under primitive unix time measuring) and looked at the amount of user and sys time used. I didn't want professional benchmarks, I wanted just a rough estimate of real world numbers, so it was OK that I needed to manually click the "Draw Image" button and manually click "File > Quit" once the CPU usage came down from fully using one core. I only wanted rough numbers to see if there's a visible difference, so I could live with those deficiencies and rounded the average of two runs roughly.

For the unoptimized, heavily JS-object-based approach I already had last week, I got a number that was about the same as I estimated previously - about 96 +/- 2 seconds. When I turned on the javascript.options.jit.chrome pref to run it with TraceMonkey, I saw no difference though, still the same, quite slow run time.
But even though Z = Z.square().add(aC); looks good for the Z = Z² + C equation, it's not that efficient to do two JS object creations for every iteration, so earlier in the week, I replaced that nice code with pure math and simple variables only. That helped performance a lot and saved about 90% of the time, cutting it down to about 9.4 +/- a half seconds.
Having that code activated, I turned on TraceMonkey once again. And now, it really helped: 7 +/- 0.2 seconds! That's about 30% faster!

For already optimized, pure JS math code (which is probably more real-world than nice-looking object stacking), squeezing even 30% more performance out of this calculation is surely nice!

Still, running my old Visual Basic 5 application under Wine has, apart from more features, two advantages to this XULRunner app: For one, it updates the image frame in the UI during calculation and not only at the end of execution, which is better felt performance as you see what it's doing. For the other, it's still 20% faster, using about 5.6 +/- 0.2 seconds with the same instrumentation - and that though people say VB is so slow. Bummer.

So, good work done, TraceMonkey friends. But there's still more potential to use the CPU more efficiently. Keep on the good work!