Version comparison of 64 bit chrome browser crashes on the latest stable(51.0.2704.63) from the last stable(50.0.2661.102)

https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20cpu.Architecture%3D%27amd64%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27browser%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D&compProp=product.Version&v1=51.0.2704.63&v2=50.0.2661.102

The crashes '[Assert] extensions::ExtensionPrefs::ScopedUpdate<base::DictionaryValue,6>::Create' and content::BrowserMainLoop::PreCreateThreads are the top 1,2 browser crashes on latest stable 51.0.2704.63 

Assigning for further investigation to Václav, who is sheriffing today.

I am not sheriffing today.

Assigning the bug to today's stability sheriff (this time correctly).

Only 34 users were impacted. Most of the crashes (if not all) were in 3rd party dlls. Lot of the crashes had INFECTED.

IMO, this is not a stop ship bug.

The following 2 bugs are being investigated and they have increased in 51.0.2704.63. There is overall increase in all top 20 crashes except WSPSendTo crash.

https://bugs.chromium.org/p/chromium/issues/detail?id=616149 and 
https://bugs.chromium.org/p/chromium/issues/detail?id=614753 

3. StartupBrowserCreator::ProcessCmdLineImpl  from 2.51% to 8.19%	
4. 4	[Assert] extensions::ExtensionPrefs::ScopedUpdate<base::DictionaryValue,6>::Create from 1.07% to 3.50%	

Poring over the stats (focussing on the change from 50.0.2661.102 to 51.0.2704.63 since .79 doesn't have a lot of statistics in Stability.Counts UMA yet):
1. The Windows dashboard [1] shows an increase of +43% in unclean shutdowns per mpl (Windows Stability/Stable/64 bit/51).
2. Drilling down deeper in UMA Stability.Counts and comparing 51.0.2704.63 [2] to 50.0.2661.102 [3] for [win,stable,64bit,not infected] gives a 45% increase (Saturday compared to Saturday).
3. The same comparison just for Win7 [win7,stable,64bit,not infected] shows a 62% Sat-to-Sat increase of 51.0.2704.63 [4] over 50.0.2661.102 [5].
4. But also the inverse selection [!win7,stable,64bit,not infected] still has 39% Sat-to-Sat increase of 51.0.2704.63 [6] over 50.0.2661.102 [7].
5. 50.0.2661.102 is stable over time [5] thus we cannot attribute the increase to changes that affect the underlying population.  Thus there seems a causal connection between rise in unclean shutdowns / mpl and update from 50 to 51.

6. However this seems in conflict with the crash signature comparison for [win7,64bit,stable,not infected] displayed in [8] which gives the fraction of top30 crashes over all crashes as:
50.0.2661.102: 50%
51.0.2704.63: 48%
7. If the 60% increase from 50 to 51 was attributable to a few distinct crash signatures, the top30 crash percentile should increase from 50% to ~70% because of the prominence of the new crash signatures.  No such increase is observed, however.
8. This implies that the opposite hypothesis is true: The overall increase in unclean shutdowns must be attributed to an approximately even increase across all crash signatures.
9. It's hard to think of an intrinsic cause that would increase crashiness across the board.
10. The apparent Win64 unclean shutdown increase does not seem reflected in reported crashes.

My conclusion: Going from 50 to 51, either there is a new source of crashes that escapes crash upload and analysis or there has been a change in unclean shutdown normalization, either affecting how unclean shutdowns or how pageloads are counted.

[1] https://chromedash.googleplex.com/dashboard?dashboard=stability-windows
[2] https://uma.googleplex.com/timeline_v2?sid=5c1616979c9c4760caed32459a7ff858
[3] https://uma.googleplex.com/timeline_v2?sid=4dafe3ee50a96db32999fce8905034a7
[4] https://uma.googleplex.com/timeline_v2?sid=9fadeb4c0e0bbcfcad3420d447b9afcd
[5] https://uma.googleplex.com/timeline_v2?sid=91532acf619a5d645c1a5ccae0e1aad7
[6] https://uma.googleplex.com/timeline_v2?sid=26ff9c73a9f8245d07ffc60fd35b51f6
[7] https://uma.googleplex.com/timeline_v2?sid=a36ac5e988e32b6a25bc5b7d3babfa39
[8] https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20cpu.Architecture%3D%27amd64%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27browser%27%20AND%20custom_data.ChromeCrashProto.channel%3D%27m%27%20AND%20custom_data.ChromeCrashProto.malware_verdict%3Dfalse%20AND%20custom_data.ChromeCrashProto.os_family%3D%27Windows%207%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D&compProp=product.Version&v1=51.0.2704.63&v2=50.0.2661.102

In line with https://crbug.com/615759#c8, I'm tentatively dropping RBS and reducing priority.  The fact that a similar increase is observed both for browser and for renderer crashes seems another argument in favour of the normalization theory.  Removing myself from owner since my sheriff shift ends now.  (Still it would be nice to have some follow-up and gain a better understanding of what has happened.)

Updates:

Bug 614753 is going to take several patches to address fully and is unlikely to ship on M51.

Bug 616149 has had a fix landed and merged into M51, that will reduce crashiness, at the expense of potential other badness (proceeding without successfully reading extension prefs).

I'm removing this from the sheriff queue. If there are additional specific issues that need attention, then feel free to bump it back.

Any update on this please.

Archiving issues older than 2 years with no owner or component.