19 December 2010 - 11:33Tracking Down Flash Crashes

Over the last many months, flash has been crashing for me on any page with flash video when I browse away. This means flash crashes when clicking a link or typing in the location bar. Or probably worst when a site that has a flash intro it crashes when loading the main content of the site.

This seems to only affect the Flash Player 10 plugin on OS X, but it was still a bit tricky finding out what crash reports are related to this particular issue.

– There are many versions of the flash plugin

If a bug is reported with a crash signature like [@ FlashPlayer-10.6@0x4b4b39 ], watching the number of crashes for that signature go down may initially seem like something was fixed, but actually users are upgrading their version of flash and crashing at a different address. Additionally, it’s harder to see the actual number of crashes for this issue at a glance as the reports are spread across many entries.

– FlashPlayer-10.6 plugin version is missing

Adding to the issue of there being many versions, for some reason the version information for FlashPlayer-10.6 shows up in reports as 0.0.0.0. Fortunately there’s an additional debug identifier that is consistently reported for the same version of the plugin.

Below is a listing of some recent versions and debug ids:



10.2.151.49nov 303CDC6F1339C791580EC31A451724F0110


10.1.102.64nov 47F846A865E18B6E54DA2E5B46E7CF5B20


10.1.85.3sep 20392E898E2D7BCE8B02FFCE2287F7DF570


10.1.82.76aug 105295F1D9A734E9FDD5350646BF2615A40


10.1.53.64jun 1028AEE5D0CE3E55DB5AE5787143EC0F960

I gathered these by looking through Adobe’s security bulletins to find the date and version of the new release. With the release date, I could then search for crash reports from before the release and see which reports disappeared. Then from the reports that did disappear, I can record the debug identifier to be for the new version.

– Crash reports not being submitted

Another issue that has recently been fixed was that most people running into these plugin crashes had no option to send a crash report. This was because Firefox by default runs in 64-bit mode and was unable to create a report for 32-bit plugin crashes. This means the frequency of the crashes were massively under-reported and the sparse crash data made it harder to find relevant reports.

– Improving crash-stats triaging

For this particular case, I can think of a couple features that could have made tracking down flash crashes more efficient.

Because the crash signature kept changing from version to version, I had to inspect many crashes to see if the whole stack is similar to other stacks. For one set of crashes, I was looking for AnswerNPP_Destroy in frame #8 to identify the family of crashes. Being able to search in part of the stack could also make it easier for various component teams find relevant reports.

For another set of crashes, the crash reason was always EXC_BAD_ACCESS at address 0x44c. So in addition to being able to filter by crash signature, being able to search by reason and address could have avoided needing to click through many crash signature reports.

There’s a number of bugs filed for these flash crashes like bug 590955, bug 572134, and bug 617469. I’m still not sure if it’s an issue with OOPP or a bug in the plugin, so it’s hard to say if flash will stop crashing by a new Firefox beta release or a plugin update. But just as a heads up, getting the latest Flash Player 10.2(.151.49) beta or Firefox 4 beta8 won’t fix these crashes.

12 Comments | Tags: Development, Mozilla

Comments:

  1. Mark says: 19 Dec 2010 - 12:10

    Thank you, thank you, THANK YOU.

    These flash crashes have been horrible for me and have me questioning why I’m bothering running betas/nightlies.

    Now about the browser itself constantly crashing…

  2. Manoj Mehta says: 19 Dec 2010 - 13:22

    Hey Edward,

    I work on the AWS Console for S3 and we have noticed that loading the Console freezes Firefox 4 betas and nightlies. This isn’t the case with Firefox 3.6 or for that matter, any other browser (Chrome, IE). I have been able to reproduce the freeze/hang on both my Mac and my Windows desktop with yesterday’s nightly.

    Given that about 40% of the visitors to are Firefox users, I’d like to help to ensure that Firefox 4.0 users get the best out of their use of the Console. How can we get started with this process?

    Thanks,
    Manoj

  3. Ed says: 19 Dec 2010 - 14:16

    @Manoj: The process starts with filing bugs, and it seems like you’ve already done so with bug 617430. It seems like it’s a javascript compilation issue as there’s thousands of functions that need to be compiled for the AWS Console.

    I’ve updated the bug as you’ve probably noticed from bugmail. It’s a regression as you pointed out, so we’ll see if it’ll get picked up for Firefox 4 before it’s released.

  4. Ian Thomas (thelem) says: 19 Dec 2010 - 14:35

    It’s not just the nightlies, I’ve seen regular flash video crashes in 3.6.13, normally when starting a video. They seem to have reduced since I disabled hardware accelleration and installed flashblock, but I can’t say if either of those are related.

  5. Ed says: 19 Dec 2010 - 16:18

    @thelem: Curious, what platform are you on? You mentioned hardware acceleration, so Windows 7?

  6. Ian Thomas (thelem) says: 19 Dec 2010 - 17:06

    Snow Leopard, although looking through about:crashes I’ve definitely had crashes since making both those changes (hardware accellaration was because it was causing some BBC streams to play frames in the wrong order, flashblock because I got fed up of overly intrusive and memory hogging adverts).

    I’ve sent 7 crash reports this month, but not been able to find anything in bugzilla or really know what to look for or file. Four crashes were FlashPlayer-10.6@0x2b0ee0, three were FlashPlayer-10.6@0x2ace83

    I’ll email you the report IDs in case that helps.

  7. Ed says: 19 Dec 2010 - 18:18

    @thelem: I’ve filed bug 620307 for the crash that you’re running into. Turns out it’s the most reported crash on OS X for Firefox 3.6.13.

  8. Ian Thomas (thelem) says: 20 Dec 2010 - 1:09

    Thanks, good to know I’m not alone at least.

    I notice you’ve added 3 signatures, but not @ FlashPlayer-10.6@0x2b0ee0. Is that a different bug? Sorry, but I don’t really understand whats important and what isn’t on these crash reports.

  9. John Dowdell says: 20 Dec 2010 - 13:54

    Not sure if you’re mainly trying to get your computer to act like other computers, or if you’re mainly trying to change internal processes.

    If the former, and if you accept that other people do not normally crash when leaving a page, then try the isolation/identification steps here:
    http://blogs.adobe.com/jd/2010/02/troubleshooting_player_stabili.html

    jd/adobe

  10. Ed says: 20 Dec 2010 - 14:12

    @jd: Thanks for the pointer. I was able to crash using the Player Test Page:
    http://ed.agadak.net/playerTest.png

    Just loading the page doesn’t cause the crash, but if I click a link from the test page, and go back and forwards a few times, flash player crashes resulting in the screenshot above.

    Flash version: You have version 10,2,151,49 installed
    Build identifier: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0b9pre) Gecko/20101220 Firefox/4.0b9pre

  11. Ted Mielczarek says: 21 Dec 2010 - 10:03

    For the Windows Flash Player plugin, Adobe provides us with obfuscated PDB files that we can use for crash-stats as well as for local debugging. You might see them in crash reports with crazy symbol names like F.

    For Mac and Linux they haven’t quite worked out a similar solution for us yet, although I’ve heard they’re interested. It would certainly be nice to be able to at least correlate crashes across versions. Socorro is not very good at dealing with crashes in modules for which we don’t have symbols, currently.

  12. Smokey Ardisson says: 23 Dec 2010 - 13:45

    FYI, I currently maintain a mapping of Mac Flash versions and debug/module IDs at https://wiki.mozilla.org/CrashKill/Mac_Flash_Identifiers (this effort was begun by Sam Sidler, and the data is also part of David Baron’s crash data scripts: http://hg.mozilla.org/users/dbaron_mozilla.com/crash-data-tools/) It doesn’t include the release date, but that could be added if it’s useful.

    I suppose I should blog about that, since many people may not know it exists…

    (I also filed a Breakpad bug a few years ago about teaching Breakpad to read the Info.plist for the version number in the absence of a library-style version number, but the bug is rather unloved: http://code.google.com/p/google-breakpad/issues/detail?id=321)


Subscribe without commenting

Add a Comment