The roads I take...

Here's a short summary of Mozilla-related work I've done in week 21/2012 (May 21 - 27, 2012):

• CSI:Mozilla / CrashKill:
  Tested the new URL list tab on crash lists and found a bug it caused, which was subsequently fixed on Friday.
  Tried to test db data for explosiveness but found a permissions error that will get fixed soon.
  Figured out a list of top Flash issues to possibly talk about during the stability work week.
  Kept track of the notification for ol Flash versions going live.
  Continued the discussion of crash reporting for the desktop web app runtime with the developers.
  Made explosiveness reports for FennecAndroid use ADUs data, which is available now that we're using data from the DB.
  Helped triaging topcrashes for the new mobile Firefox beta.
  I also worked a bit on some data I wanted to have around for the stability work week.
  Just like every week, watched new/rising crashes, caring that bugs are filed where needed.
• SeaMonkey:
  Reviewed a Data Manager patch that unfortunately will need some more work.
• Various Discussions/Topics:
  Android XUL Fennec display problems, stability work week planning, SeaMonkey infrastructure, switch to MPL2 headers, community event calendar, Gaia UI units, privacy laws and cookies, LGPL and Mozilla code, etc.

It looks a lot like crash data on the new Firefox Beta for Android is progressing nicely, every beta we ship performs better there, reporting in with lower crash rates. Please continue testing them and send crash reports to us when you encounter such problems!

Also, on Friday I'll be flying off to California to spend two weeks in the Mountain View office (the second of those being the stability work week) and then somewhat short of two weeks of vacation time driving through Northern California. If you're in the Bay Area and want to meet, let me know!

Here's a short summary of Mozilla-related work I've done in week 20/2012 (May 14 - 20, 2012):

• CSI:Mozilla / CrashKill:
  Lots of discussions around blocklisting old Java versions, we'll go for an extra soft approach as a first step due to the high usage of Flash.
  Following an error of Adobe that we found out about, there was an interesting couple of Flash reprocessing events that we needed and I pushed through.
  As we shipped our first beta of the new Firefox for Android, I helped a lot in monitoring and looking into the crashes we are seeing there.
  On the weekend, pinged people on a heavily explosive Nightly crasher introduced by JS code landed on Friday. People were responsive even on a Sunday to look into the issue, review and land a fix.
  Stayed involved in the discussion for Java signature improvements.
  Reworked my Flash and explosiveness reports to fetch their data from the Socorro database instead of the CSV files.
  Just like every week, watched new/rising crashes, caring that bugs are filed where needed.
• German L10n:
  I synched up a few strings in core and DOMi L10n to trunk again.
• Various Discussions/Topics:
  SeaMonkey build system work and website infrastructure problems, Android XUL Fennec display problems, stability work week planning, plans for moving merge/uplift day, HTML forms and mobile, web apps on Linux, HTML5 Mario prototype, etc.

The new Firefox for Android is in Beta now, and for phones it's a really huge step, it launches faster than most other apps, and still has all the Firefox power on websites - even including Flash. Yes, we needed to sacrifice our "webby" technologies for UI design to get there, but we'll go even farther with those on our future phone OS.

If you have an Android phone, help testing this beta (that is, if it has an ARMv7 processor, we're still working on a version for ARMv6). We know about a couple of crashes, but we need more reported, including steps to reproduce if possible, to get this one to be as stable as one would expect from a browser baring the "Firefox" brand. Everything we catch now in beta and we can fix here, will not hit the masses when we go to release.

That said, I hear it's pretty good already (I don't have an Android phone by choice, wanted something even more open), so you can feel good about trusting it with your everyday browsing tasks on your Android phone. And at the same time, you'll be doing good by testing it and - in case you crash it - by sending a report we can then look at and fix!

Here's a short summary of Mozilla-related work I've done in week 19/2012 (May 7 - 13, 2012):

• CSI:Mozilla / CrashKill:
  More discussions and forming of a concrete plan for Rapid Betas on Socorro.
  Found an interesting signature escaping problem in Socorro.
  Did some triage of mobile crashes with Noaki and Sheila, with an eye on the upcoming beta.
  Discussed and made a proposal for Java signature improvements.
  Followed up on WebRT and B2G crash reporting process, will probably need some more poking until things actually happen there.
  I tried to move things forward on potentially blocking old Flash versions with actively exploited security vulnerabilities, and involved the security-group in the discussions. I hope we'll come to a conclusion on this now.
  Just like every week, watched new/rising crashes, caring that bugs are filed where needed.
• SeaMonkey:
  When people saw a strange website reversal to old days, I poked IT people to get this resolved and they found a temporary solution. There's a larger problem behind this which doesn't only affect our site and which they are working on.
• German L10n:
  I synched up core and suite L10n to trunk once again.
• Themes:
  While the 2.9 versions of EarlyBlue and LCARStrek got their AMO reviews, I started working on 2.10 versions fitting with current beta builds.
• Various Discussions/Topics:
  More datacenter move fallout, Android XUL Fennec display problems, stability work week planning, etc.

As we did get the OK for our stability "work week" (actually more of a meeting week) finally, I have now booked my flight for that as well as one more week leading up to that where I can meet with Mozilla folks in Mountain View and possibly San Francisco - and slightly more than a week of vacation following the event. I'm traveling on June 1, if you are in the Bay Area and want to meet between then and June 11, let me know!

Here's a short summary of Mozilla-related work I've done in week 18/2012 (April 30 - May 6, 2012):

• CSI:Mozilla / CrashKill:
  Had tons of discussions, wrote an initial spec for Rapid Betas on Socorro and handled some questions that came up on it.
  This also brought up availability of old graph data, for which I filed a bug.
  Some more poking about the top Firefox 14+ crasher, which ended up in a backout of the cause in Aurora, so we only need to get an actual fix in Nightly 15 now.
  Blogged about The Life Cycle of a Crash.
  Just like every week, watched new/rising crashes, caring that bugs are filed where needed.
• Various Discussions/Topics:
  Ongoing datacenter moves, HTML L10n, getting de L10n ready for Fennec Beta, padlock and favicon, WebAPI security discussions, Persona marketing, stability work week planning etc.

I spent a lot of time this last week at a local FLOSS event called "Linuxwochen", which was coupled with Linux Graphics Meeting (LGM) and BSDDay this time. There, I talked to a lot of people, representing Mozilla (after all, I'm the local Mozilla Rep, and I guess I was the only core Mozillian there - was there any other?) and trying to tell people about all the cool stuff we are doing while at the same time helping them with questions and trying to remotely diagnose why some could have memory and responsiveness problems with Firefox (which is still the largest complaint and a persistent image problem).

It would have been really helpful to have a B2G demo device to show them how capable such a system is, as the reaction was quite split when I mentioned that system - one group cheered, another frowned. A lot of reasons for the frowns are those you expect from that crowd we had there - finding that JS is a bad language (if one at all), that JS should be disabled as much as possible and apps would undermine that by not running without JS, or even the mindset that web technologies should stay restricted to a low-privilege canvas withing a heavy browser UI. I tried to, but it's hard to convince anyone with one of those opinions that we'd be on the right track by pushing web apps. The cheers were mostly for the the very open stack or simply noting that we're absolutely spot-on that this is where things are moving anyhow and Mozilla as a player with the mission we have is important there.

I also could convince at least one LGM attendee of switching to Fennec (Beta) on his Android tablet, saying it's by far the best browser he's seen on a tablet so far (yay for Gecko being fast on complex Slashdot pages he used as test cases) and make him cheer even more for me telling that the native builds coming up will improve speed even more, mostly in terms of UI. I'm convinced a couple of people I talked to will try native Fennec once it's in the store, I got at least one to also take a look at its Nightly version.

All in all, a lot of interesting conversation, I learned a lot of what people are thinking about us, communicated a lot of what's happening in Mozilla, and had a lot of fun with developers from Austria and around the world. After those days, I got back to the routine of work somewhat tired, but satisfied.

Someone from outside Mozilla who wants to deploy our crash reporting and statistics system asked me if I can "explain the life cycle of a crash". I found that this might be interesting for more people in- and outside of Mozilla, so here it is.



From the user point of view as well as the existence of a crash report, everything starts with the crash reporter dialog. Of course, a number of things have happened before, but as it's (supposed to be) a "life" cycle, so this point is as good as any other to start with.

Now a "crash" is nothing else other than an unsuspected exit of a running process - usually because execution ran into a state that was not intended and the code does not "know" how to continue executing. When this happens to a Firefox process, the routines for that unplanned exit call up the Breakpad code that fetches data like the "stack" of running function calls, the loaded libraries/modules, activated add-ons and some metadata from the crashed Firefox process, building up a crash report. In addition to that, the dialog prompts the user if the reports should be sent to Mozilla, and if so, for a comment, and email, and inclusion of the address for the currently active browser tab. On clicking any of the "Restart" or "Quit" buttons, the report is being sent (unless the checkbox for sending is deactivated).
In case the crashing process is a plugin container (the separate process Firefox runs for executing plugins), what happens is almost the same, just that the dialog window is replaced by a prompt for sending the report, which is being displayed instead of the plugin right on the web page. This prompt has no possibilities for specifying comments, email, or website addresses, so plugin reports miss those. Also, in case such a plugin container does not react to messages from the host Firefox process for a certain time (currently 45 seconds), that host process kills the plugin process and reports are being sent from the state of both the plugin and browser process, marked as "hangs" with special IDs to match them together on the server side.

Inside a running Firefox browser, the about:crashes page can be used to look up reports that have been generated by this installation. This page contains the report ID and a submission or creation date. If the ID ends up with six digits representing the submission date (in YYMMDD format), then it has been received by the servers correctly and the ID is linked to the server-side page for the report. If the ID ends in a random hex code, it was not received correctly (e.g. because of connection problems) and clicking the link triggers re-submitting it.

The server side of this system we're using at Mozilla is called Socorro. When a crash report is being submitted, it's first being received by a collector, which stores a raw dump of it and signals back to the client - this is where the date at the end of report IDs comes from (replacing the random hex sequence created by the client). Then, reports are being processed - though Socorro "throttles" this processing in certain cases to save disk space and analyzing CPU power. For release versions of Firefox desktop, usually only 10% of the incoming reports are processed, but for development/beta versions or other products, all reports are beings processed, as well as any reports including comments. The processor makes the data of the report more accessible to analysis, its main job is to make the crash stack human-readable and derive a "signature" for the crash. The original stacks in the raw reports are only combinations of names of libraries and addresses inside those, the processor connects that information via symbol files to function names as well as source file names and line numbers. Based on information of what function names are in the exit path after the crash has already been caused, the stack can be cleaned up from those unneeded frames, and based on knowledge of some less interesting function names, the top frame/function and/or some of its callers form the signature. After this treatment, the report is ready for being analyzed.

Every morning, Socorro runs jobs to create reports from the processed crash data of the last (UTC) day. For example, it counts how often a signature has been seen on that day for a certain Firefox version and saves that data to be displayed in the "topcrash" reports (internally called TCBS for "top crashes by signature", probably our most important report format).
To know what versions to produce those aggregations for, Socorro looks at Mozilla's FTP server for newly appearing builds a few times a day and automatically adds them to its list of known versions.
Those daily jobs also incorporate ADI data from the metrics team ("active daily installations", often also called ADU for "active daily users" although it's actually the number of requests to the addon blocklist that every profile should make once a day when being used), so that we can calculate "crash rates", which are displayed even at the crash-stats.mozilla.com front page in a graph to identify how well different versions are doing overall.
In addition, a number of important fields of all the processed reports of that last day for mobile and desktop Firefox are put into machine-readable files (*-pub-crashdata.csv.gz in the date-based sudirectories of the crash_analysis machine) so that some people can run their own reports on them, either for their own analysis or for prototyping functionality that later should make it into Socorro itself. One outcome of that data is also the "Are We Stable Yet?" dashboard I created for a quick Firefox stability overview.



Now that our crash report has made it far enough that it's available in reports, the CrashKill team (which I'm working in) and volunteers helping that effort come into play. We're taking a look at those reports (in Socorro as well as chofmann's and my own places for prototyping new ones) every day to make sure we catch all series of crashes that need someone to look into. As part of our mission is to ensure and improve Firefox stability, we make sure there are bug reports filed in Bugzilla for all important crash issues, work with the QA team to try to get the crashes reproduced where needed to get more data for developers to look into, try to get developers assigned to work on the problems, work with release management to get proper tracking and prioritization of highly-visible crashes as well as fixes landed in stabilization phases/branches, and coordinate with others as needed, e.g. for relations to external partners. On the latter, a number of the crashes we catch are 3rd-party issues that can't be fixed in our code, in those cases we try to work with the respective code vendor (mostly add-on creators, plugin and security suite vendors) to come to a solution, and many of them are quite responsive and fix issues on their side. In extreme cases, where Firefox becomes (mostly) unusable in combination with the 3rd-party software, we can make use of add-on or DLL blocklists that prohibit loading of that software. In cases of crashes with 3D or hardware acceleration features of graphics card drivers, we can block usage of those features with those drivers and fall back to non-accelerated and/or 2D display (except for WebGL, which needs to get deactivated in those cases). Of course, we try other avenues first before deploying one of those blocklists, and are way happier, just like our users, when we can actually find a fix in our code.

Developers sometimes take a look at Socorro themselves to find items to work on, and often start looking into a crash when a bug is being filed for which they might get bugmail - otherwise, CrashKill tries to get the right people involved and looking into the bugs. They look at the crash stack as listed in our crash data (see an example, those are the pages as linked in about:crashes as mentioned before) as well as other data submitted with the crash, and try to figure out how to avoid the problem in our code. When they find a fix, it goes through the usual review process and gets and "checked in" to the code, just like other code changes. At that point, it might make sense to apply the same fix to the Aurora and Beta channels, which are the Firefox versions that are in a testing and stabilization phase - CrashKill and release management will track those and get the needed approvals in place where warranted, so that the fix can go out to more of our users faster than most other code developments and releases become more stable.



From this code, binary builds are generated regularly - every day for the on-the-edge-of-development Nightly channel as well as Aurora, roughly every week for Beta, and every six weeks for releases (unless grave security or stability issues arise, where we'll do an updated release in between with just the fixes for those specific issues). In the building process that creates those binaries, symbol files are created next to the actual builds that are to be delivered to users and testers. Those symbol files are sent to our symbol server, where developers have access to them for debugging - and from where the Socorro processors fetch them to translate addresses in binary files into function names and source lines, as mentioned above. The builds themselves are what we deliver to the user, who now hopefully won't see this particular crash happening again, so its "life" is concluded and everybody is hopefully happier.

Though, in some unfortunate cases, a user might hit a different unexpected exit of the Firefox process, Breakpad triggers, showing the Crash Reporter dialog - and the life cycle of another crash begins...