The roads I take...

Crash signatures derived from the function on the stack are how we group ("bucket") crashes as belonging to a certain issue or bug. They should be precise enough to identify this "bucket" but also short enough so we can handle it as a denominator in lists and when talking about those issues. For some time, we have seen that our signatures are very long-winded and contain parts that make it sometimes even harder to bucket correctly. To fix that, we set out to shorten our crash signatures.

We already completed a first step of this effort in June: After we found that templates in signatures were often fluctuating wildly in crashes that belonged to the same bug, all <sometemplate> parts of crash signatures were replaced by just <T>.

That made a signature like this (from bug 1045509, the [@ …] are our customary delimiters for signatures, not really part of the signature itself though):
[@ nsTArray_base<nsTArrayFallibleAllocator, nsTArray_CopyWithMemutils>::UsesAutoArrayBuffer() | nsTArray_Impl<unsigned char, nsTArrayFallibleAllocator>::SizeOfExcludingThis(unsigned int (*)(void const*)) ]

be shortended to:
[@ nsTArray_base<T>::UsesAutoArrayBuffer() | nsTArray_Impl<T>::SizeOfExcludingThis(unsigned int (*)(void const*)) ]

Which is definitely somewhat better to read and put in tables like topcrash reports, etc. - and we found it did not munge bugs together into the same signature more than previously, at least to our knowledge.


But we found out we can go even further: Different argument lists of functions (mostly due to overloading) did as far as I remember not help us distinguish any bugs in the >4 years I have been working with crashes - but patches changing types of arguments or adding one to a function often made us lose the connection between a bug and the signature. Therefore, we are removing argument lists from the signatures.

The signature listed above will turn out as:
[@ nsTArray_base<T>::UsesAutoArrayBuffer | nsTArray_Impl<T>::SizeOfExcludingThis ]


Today, we have run a script on Bugzilla (see bug 1178094) to update all affected bugs to add the new shortened signature to the Crash Signatures field without sending a ton of bugmail.

We have tested in the last weeks that Socorro crash-stats can create the new shortened signatures fine on their staging setup and that generation of the special "shutdownhang | …" signatures for browser processes that did take more than 60s to shut down and "OOM | …" for out-of-memory crashes do still work in all cases where they worked before.


As all preparation has been done, we will flip the switch on production Socorro crash-stats in the next days, and then those shortened signatures will be created everywhere.


Note that this will impede some stats that are comparing signatures across days, even though we will see to reprocess some crashes to make the watershed be at a UTC day delimiter so that as few stats as possible are disturbed by the change.


Please let me know of any issues with those changes (as well as any other questions about or issues with crash analysis), and thanks to Lars, Byron (glob) and others who helped with those changes!

Originally coming as a request from the Socorro team, we have been pushing for some time to get a better solution for connecting bugs to crash reports/stats than the bug summary - in the form of a separate "Crash Signature" Field in bug reports.

It wasn't just messy up to now for Socorro (crash-stats) to parse the signatures out of headers, we also had a number of bugs where the constraints on the summary length caused problems with adding multiple signatures or a very long signature - not to speak of making summaries hard to read for humans. All that should get solved with the new special-purpose field that also allows greater length of its content.

You might have noticed that this "Crash Signature" field has turned up in many products of Mozilla's Bugzilla in the last days, below the tracking/status/blocking flags on the right above the comments.

The syntax for entering signatures is similar to what we did put into summaries so far: a signature is listed as [@ crash::signature(params) ] - for multiple signatures, we prefer to use a new line for every signature to make the field more readable, though.

We're migrating over the contents from summaries to the new field, so you don't need to copy those manually - some of that happened already, but the script didn't catch all necessary bugs the first time it ran. Also, the work on Socorro to use the new field has not landed on production yet. Both should be done soon (within days), and then we can fully use the new field. Until then, it's best to leave the field alone and keep using the summary (though for those cases where the summary is too small, populating the new field right away is safe).

In the future, i.e. once Socorro uses the new field, we'll also be able to make summaries more human-readable, e.g. "[meta] EnterMethodJIT crashes" or something similar will be far easier on the eyes for bug 595351.

Thanks to everyone who worked and is still working on implementing that!

I have blogged about the plans to move old untouched bugs to UNCONFIRMED some time ago, the SeaMonkey team has agreed to move forward with this in the meantime, and today I went for actually doing the change.

So, that makes it bugmail day. So far, I have handled about 900 of the 2750 bugs, and I've run into a problem with Bugzilla that made me re-post the status change and comment, sometimes multiple times, until I realized that it was actually processing the bugs but only cutting the connection for sending me the feedback. Because of that, some bugs with IDs under 110000 ended up with multiple comments from this. I'm very sorry about that.
I'll do the rest of the bugs when Bugzilla has calmed down a bit and will try to avoid the problem causing those multiple comments.

You can filter the bugmail from this change looking for the string "mass-UNCONFIRM-20090614" and this string can also be queried as part of the comment when looking for those bugs later.

I posted an idea in a triage-related newsgroup post a month ago and lacking any reaction, I brought it up in the recent SeaMonkey Status Meeting, where it was decided to get feedback on it before deciding on it, so please tell me what you think about this:

SeaMonkey inherited a large legacy of of bugs from the Mozilla suite, most of which have an unconfirmed value nowadays, four years after founding the new project. The Bugzilla database reflects those as current bugs though, often bringing them up in queries as if they were current, even though we can't confirm nowadays that they still apply to our current development code or even releases.

We have almost 2800 bugs that haven't seen a comment since the project was started but are still in the NEW state, see the bug query if you don't believe it.

Ideally, we'd go and actively triage every single one of those bugs manually to try to find out if they are still valuable. If that takes only 5 minutes in average per bug, that's 29 8-hour workdays or almost 6 full work weeks for one person, and even if we would have someone with a capacity of so much time to work on SeaMonkey alone, I think we'd have higher priority stuff to spend the time on than finding out if bugs that weren't touch for more than 4 years still have any value.

When I thought about this and about how just mass-resolving bugs without warning is quite rude (we already had such incidents), I realized that we already have an identifier in Bugzilla that tells that "this bug is not confirmed to be valid" that also doesn't mean the bug is resolved for good in any way: UNCONFIRMED.
(That even includes enhancement requests for which we can't tell if they still apply to current code or project plans, by the way.)

So, what I'm proposing is that we mass-move those bugs in the SeaMonkey component that had no comment since 2005-03-10 and are still NEW to the UNCONFIRMED state.

This leaves them open but shows everyone that we are unsure that they are still valid within the SeaMonkey project nowadays without just kicking them off into nowhere land. Everyone who get bugmail about this and considers his pet bug in there to be still valid, can comment on that and/or move the bug back to NEW, which is easy - but instead of making one person go though a pile of rotten stuff to find the good stuff, we push the work to more people and to those who actually care about the specific bugs.

In a few months, when that has likely been sorted out and SeaMonkey 2.0 is out the door as a stable release, we can think about what to do with those bugs that still remain UNCONFIRMED - we might leave them as such or ultimately mass-resolve parts or all of them, but we have months before even thinking about this. For now, the proposal is just to make the state reflect reality and make them UNCONFIRMED.

What do you think about this idea to clean up our bug queries to find relevant stuff more easily?

As reported here before, I asked for help help with triaging SeaMonkey bugs. Back then we had over a thousand unconfirmed non-enhancement bugs in the "Mozilla Application Suite" product that had no activity for more than 6 months. When I looked at the query today, I was quite surprised when it only showed 15 bugs!

Looking at the open bug graph for our Bugzilla product I could confirm what the query had suggested:
The people helping us here have killed about a thousand bugs in 8 weeks!
This is absolutely awesome, thanks and congratulations to everyone helping with this effort!

It would be really cool if we can prolong this Bugzilla cleanup effort and try to reduce unconfirmed and old bugs even more, so the real bugs are easier to find and deal with. In my newsgroup posting from today (web version), I have a few suggestions on what to attack next - I'd welcome your input!

A long time ago, tried to get the "Mozilla Application Suite" product renamed and reorganized for SeaMonkey on Mozilla's Bugzilla system. After some poking in different directions, this triggered discussions about a bigger reorg of bugzilla components and even products, ending in a nice proposal, but this got stalled due to a missing script for making this as smooth as possible (i.e. not disturbing saved queries too much and moving/merging components across products) - and due to fear or broken FF3 planning queries.

Because of this, Bugzilla still largely sucks for SeaMonkey due to wrong components and even a wrong product name being used.

A small step towards correcting this has been done now though (thanks to justdave!): The "Enter Bug" product selection describes "Mozilla Application Suite" as SeaMonkey and uses the SeaMonkey logo for it.

I hope this will enable newbies to Bugzilla to at least find the right product for filing their SeaMonkey bugs under.

And maybe, some time in the future, there is still hope for getting a real bmo reorg done so that the product/components structure fits reality better. I won't stop to hope for better times...

Again and again I stumble over people who seemingly don't understand how a bug reporting system is supposed to be used - usually that's Mozilla's Bugzilla in my case.

Those systems can work very well for reporting and working on bugs, and even though it could be improved (yes, all improvements I'd like are filed as bugs themselves), Bugzilla has grown into a really good such system over the years. To be honest, I've never professionally studied such technologies, and Bugzilla is the only one I know really well, but when you're working with it long enough and think about it, you begin to understand what this is all about.

One common mistake people are making is to go into a bug report that already has 50 or more comments and is pretty lengthy, and then they request some detail change in some part of the work that has already been done, reviewed and landed there. A recent comment in the "new SeaMonkey theme" bug doing exactly that is what leads me to write this blog entry. The correct solution to deal with such issues caused by a different bug (esp. if that one is a lengthy report already) is to file a new "followup" bug that covers exactly that one detailed issue you're seeing and mark this new bug dependent on the original one. Leaving a comment in the original bug can also help to get attention to the new issue, but that comment shouldn't be more than "I found a problem with X caused by this checkin, see bug XXXXXX" (Bugzilla is intelligent enough to link your mentioning of "bug XXXXXX" to the actual bug report).

Another common mistake is to add comments to the bug that don't add anything that wasn't already said there, esp. comments that just tell "I'm seeing this too" or "I want this fixed as well" or even "Adding myself to the CC list". "Me too" is only useful if you tell that you're seeing this on a configuration where it wasn't clear before that it is affected by the bug. If developers already know the cause for the problem or are convinced that this should be fixed, a "me too" comment just wastes their time - they need to read it and get no new information from that read, while they could use the same time to already actually work on a fix. You're probably distracting other people from work with such comments, keep that in mind.

I even believe that discussion of how some implementation should work in concept is usually wrong for a bug report. Bug reports should focus on explaining what exactly is the issue, finding out its root cause, and the actual fix (along with reviews of that fix). That's even true for feature requests. If the design of such a feature is unclear, this should be discussed in some place that allows to do this more in a real discussion style, allows easy editing of drafts, etc. Use a newsgroup, a wiki page, or both, or whatever fits this issue, link that from the bug, and come back to the bug when the discussion has settled on a specific approach.

So, actually, effective use of such a system comes down to:

• Care that there's one bug report per specific issue, avoid the need for multiple patches/fixes in a single bug report.
• If you want to track multiple or general issues, use a tracking bug for that (such a bug has multiple specific reports marked as dependencies and no specific work/fix/patch by itself)
• Only add comments that add further information that helps to resolve the bug
• Care that new people who look at the bug report can easily get informed of what the specific issue is and how far a fix has been progressed
• File followup bugs for any derived specific issues, such as regressions, improvements to already passed (reviewed/checked in) fixes (unless you're very sure of a logical one-line-fix or that a backout is the only chance to fix the derived issue)
• When filing new bugs, make sure that all information someone else (a developer) needs to fix this bug is there. If you're not sure what such a developer needs, describe the details of the problem you found (when/where/how it happened to you, what version of what software you were using for that, what you were doing in detail)
• If a developer tells you to file a followup or take the discussion somewhere else, just do it - even if you don't understand why this may be a good idea. If you request a fix done by someone else, it's usually a good idea to also follow his requests that ease him working/helping on a fix. It's usually a good idea to add a comment on the bug that says "requested followup is bug XXXX" or "I opened a thread for this discussion at http://foo/bar" or "The requested draft is on the wiki at http://foo/wiki/bar". Helping developers by following their requests can magically speed up fixing

I'm sure this set of common mistakes and guidelines is incomplete, but I think if you follow this advice, working with Bugzilla and other bug reporting systems will not only be more fun for everyone, but also will make fixes appear faster as developers can be more foused to specific working tasks.