Users experienced this crash on the following builds:

Win Canary 56.0.2889.0 -  0.67 CPM, 8 reports, 8 clients (signature blink::TraceTrait<blink::HeapVector<blink::Member<blink::MessagePort>,0> >::trace)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Crashes are seen on the latest canary of Windows(56.0.2889.0 - 8 crashes from 8 clients till now)

This is regressed in M56, 
https://chromium.googlesource.com/chromium/src/+log/56.0.2888.0..56.0.2889.0?pretty=fuller&n=10000

suspecting https://chromium.googlesource.com/chromium/src/+/fb7d72c8ec38f54f84c226c07c4c980283687d17

keishi@, could you please check the issue.

Note:
1. Not able to reproduce this crash locally, observed in crash server.
2. Instances are low, hence not adding blocker label.
3. Not observing this crash in other OS, 
https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.name%3D%27blink%3A%3ATraceTrait%3Cblink%3A%3AHeapVector%3Cblink%3A%3AMember%3Cblink%3A%3AMessagePort%3E%2C0%3E%20%3E%3A%3Atrace%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=#samplereports:5,+productversion

This crash still observed on latest canary 56.0.2900.0.so far seeing 11 instances from 11 different clients on Windows OS.

Link to the list of builds 
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27blink%3A%3ATraceTrait%3Cblink%3A%3AHeapVector%3Cblink%3A%3AMember%3Cblink%3A%3AMessagePort%3E%2C0%3E%20%3E%3A%3Atrace%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D

56.0.2900.0	20.75%	11	
56.0.2898.0	30.19%	16	
56.0.2896.3	3.77%	2	
56.0.2894.0	3.77%	2	
56.0.2889.0	41.51%	22

keishi@ Could you please look into this issue.

Thnaks,	

Users experienced this crash on the following builds:

Win Canary 57.0.2925.0 -  0.48 CPM, 3 reports, 3 clients (signature blink::TraceTrait<blink::HeapVector<blink::Member<blink::MessagePort>,0> >::trace)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

This crash is seen in win latest dev 56.0.2924.3 (released 8 hours ago) with 3 instances from 3 different client Ids

57.0.2925.0	2.87%	17	
56.0.2924.3	0.51%	3	
56.0.2918.0	4.39%	26	
56.0.2916.0	4.05%	24	
56.0.2915.0	3.21%	19	
56.0.2914.3	1.35%	8	
56.0.2910.0	3.21%	19	
56.0.2909.0	5.07%	30	
56.0.2908.0	4.22%	25	
56.0.2907.0	0.17%	1	
56.0.2906.0	1.69%	10	
56.0.2902.0	52.70%	312	
56.0.2901.4	1.01%	6	
56.0.2901.0	3.04%	18	
56.0.2900.0	4.05%	24	
56.0.2898.0	3.38%	20	
56.0.2896.3	0.68%	4	
56.0.2894.0	0.34%	2	
56.0.2889.0	4.05%	24	

Link to the builds:
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27blink%3A%3ATraceTrait%3Cblink%3A%3AHeapVector%3Cblink%3A%3AMember%3Cblink%3A%3AMessagePort%3E%2C0%3E%20%3E%3A%3Atrace%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion:1000

keishi@, Could you please take a look

Latest crash rates are as below

56.0.2924.3	31.09%	268	latest dev

Link to the list of builds
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27blink%3A%3ATraceTrait%3Cblink%3A%3AHeapVector%3Cblink%3A%3AMember%3Cblink%3A%3AMessagePort%3E%2C0%3E%20%3E%3A%3Atrace%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion:1000

keishi@ Please look into this issue.

Thanks,

M56 beta launch is next week.Your bug is labelled as Release Block beta, please make sure to land the fix by first week of December.

Is it possible for a GC to strike before the initialization at https://cs.chromium.org/chromium/src/third_party/WebKit/Source/modules/serviceworkers/ExtendableMessageEvent.cpp?l=121 is done?

Nevermind, that's not problematic even if it could.

With ServiceWorkerThread now having its own heap, any cross-heap MessagePortArray references not being handled as-wanted for ExtendableMessageEvent, would be one thing to suspect.

Just to update:

This crash is not seen so far in latest dev version 56.0.2924.10 (released 18 hours ago)

56.0.2924.3	38.52%	374	prev dev

Thanks for the investigation.

M56 Beta promotion is scheduled on Dec 06 , please make sure to get this crash resolved ASAP.

Can we have the latest update on this issue?

I don't see this crash after 56.0.2924.3, currently we have 56.0.2924.18 in production.

Leaving the bug open for couple more releases.

Issue 673061 has been merged into this issue.

Copying the Blocker label from the duped Issue 673061 in C#15 for tracking purpose. Duped Issue 673061 has reported 3 digits of crash instances on every canary build of Windows since #57.0.2946.0.   

This crash has high impact on Chrome's stability.
Signature: blink::TraceTrait<blink::FetchEvent>::trace.
Channel: canary. Platform: win.
Labeling  issue 655926  with ReleaseBlock-Dev.


If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Based on the available crash data, Duped Issue 673061 is #1 renderer crash on the latest Dev(57.0.2950.4 - 159 crashes from 134 clients so far) of Windows.

Link to the crashes:
====================
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27blink%3A%3ATraceTrait%3Cblink%3A%3AFetchEvent%3E%3A%3Atrace%27

Can we get an update on this as this is marked as Dev blocker due to duped Issue 673061 and being #1 renderer crash on the latest Dev(57.0.2950.4). Would be good to have this fixed before next Dev release. 

Friendly ping to get an update on this. Consistently #1 renderer crash on Windows canary/Dev.

 Issue 673871  has been merged into this issue.

This is currently the top renderer crash in the latest canary. keishi@, haraken@, please disable this feature while you investigate the crash. 

Despite the fact that this issue was reported in M-56, it doesn't appear in top-10 crashes in latest beta. How is the rollout for this feature being controlled. Finch? Please update this bug with your roll-out plans so we can track them.

keishi@ is ooo until EOY. haraken@ is ooo today. haraken@ will take a look tomorrow.

Given that the crash is happening when we trace MessagePort, I guess this crash is related to the per-thread heap.

Hmm. There was a big spike at 57.0.2952.0 but no crash at 57.0.2955.0. There is no roll out around the versions.

The interesting fact is that all the crashes have ServiceWorkerGlobalScopeProxy::dispatchFetchEvent on the stack.

I see crashes in 57.0.2955.0 and up to current canary:

57.0.2957.0	0.24%	22	
57.0.2956.0	4.26%	398	
57.0.2955.0	4.74%	443	
57.0.2954.0	3.19%	298	
57.0.2953.0	5.31%	497	
57.0.2952.0	3.74%	350	
57.0.2951.0	3.77%	353	
57.0.2950.4	51.32%	4799	
57.0.2950.0	6.38%	597	
57.0.2949.0	4.22%	395	
57.0.2948.0	4.37%	409	
57.0.2947.0	3.20%	299	
57.0.2946.0	4.47%	418	
55.0.2883.87	0.06%	6
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27blink%3A%3ATraceTrait%3Cblink%3A%3AFetchEvent%3E%3A%3Atrace%27#samplereports:5,productversion:20

Looks like a regression between 57.0.2945.0 and 57.0.2946.0.
https://chromium.googlesource.com/chromium/src/+log/57.0.2945.0..57.0.2946.0?pretty=fuller&n=1000

Could it be https://codereview.chromium.org/2516353002 which touched FetchEvent?

Ohh... comment #28 was a different magic signature. But could be related.

This bug originally tracked blink::TraceTrait<blink::HeapVector<blink::Member<blink::MessagePort>,0> >::trace but now also tracks blink::TraceTrait<blink::FetchEvent>::trace since Issue 673061 was merged here. The FetchEvent one has the high crash rate in recent canaries.

If there is some overlap here, how are per-thread issues diagnosed&debugged in release builds?

> how are per-thread issues diagnosed&debugged in release builds?

Yeah, this is my concern as well.

We have a bunch of DCHECKs to detect cross-thread Members (we need to use CrossThreadPersistents for all cross-thread pointers) but the dchecks work only on debug builds...

ok, i've been repeatedly going over this code to spot potential trouble, without locating any.

How about providing a same-thread-checked Member<> subclass and selectively apply it to diagnose? It seems to be specific to a few Member<MessagePort>s (or Member<MessagePortArray>s) here.

Sounds like a good idea!

https://codereview.chromium.org/2592063002/ does that, with https://codereview.chromium.org/2589333005/ following up with ExtendableMessageEvent checking.

If no one else has determined the root problem here, we can land the latter to get some sanity checking/diagnosis going at least.

If the same-thread condition doesn't hold, then the problem enters if ExtendableMessageEvents are created from scripts, passing along MessagePorts from some other heap. i.e., the usage in the ServiceWorkerGlobalScopeProxy::dispatchExtendableMessageEvent() methods, is non-problematic.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/137ed1ac9cbb2dc5f84f3af169ae0b03b1a2361b

commit 137ed1ac9cbb2dc5f84f3af169ae0b03b1a2361b
Author: sigbjornf <sigbjornf@opera.com>
Date: Thu Dec 22 12:16:15 2016

Verify that ExtendableMessageEvent's message ports reside on the same heap

To potentially help diagnose a GC crash (see associated bug), verify
that ExtendableMessageEvent's MessagePortArray and MessagePorts belong
to the same thread heap as the event object.

R=haraken
BUG= 655926 

Review-Url: https://codereview.chromium.org/2589333005
Cr-Commit-Position: refs/heads/master@{#440390}

[modify] https://crrev.com/137ed1ac9cbb2dc5f84f3af169ae0b03b1a2361b/third_party/WebKit/Source/modules/serviceworkers/ExtendableMessageEvent.cpp
[modify] https://crrev.com/137ed1ac9cbb2dc5f84f3af169ae0b03b1a2361b/third_party/WebKit/Source/modules/serviceworkers/ExtendableMessageEvent.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3d5b1a43aeb21176af2d264d8ec399916e29c4cd

commit 3d5b1a43aeb21176af2d264d8ec399916e29c4cd
Author: sigbjornf <sigbjornf@opera.com>
Date: Thu Dec 22 19:45:20 2016

Verify that FetchEvent::m_request holds same-thread reference.

To diagnose instability seen during GCs of FetchEvents, instrument
the m_request member, checking that we only create event object containing
references to Requests that reside in the current thread's heap.

R=
BUG= 655926 

Review-Url: https://codereview.chromium.org/2594423002
Cr-Commit-Position: refs/heads/master@{#440475}

[modify] https://crrev.com/3d5b1a43aeb21176af2d264d8ec399916e29c4cd/third_party/WebKit/Source/modules/serviceworkers/FetchEvent.h

If either #c36 or #c37 gets at the problem(s) here, expect to see CHECK() crashes against the ExtendableMessageEvent and FetchEvent constructors, respectively, instead.

sigbjornf@: Any update on this post CL from C#37. Would be good to have this fixed before the next scheduled Dev release in 1st week of Jan. 

Just to update there have been no crashes seen post chrome version 57.0.2963.0 on Windows for the magic signature 'blink::TraceTrait<blink::FetchEvent>::trace' duped Issue 673061. 

Removing the Dev blocker therefore.

for reference, changes #c36 and #c37 first appeared in 57.0.2961.0

Re: #c41: if failures in SameThreadCheckedMember<Request>::checkPointer() can be seen in >= 57.0.2963.0 instead, then #c37 had an impact.

Interestingly, I cannot find any crash report that contains 'SameThreadCheckedMember'.

Hmm.

Thanks, it's worth keeping #c37 around for the next Dev release, just in case. But an indication that the cause hasn't been located.

I haven't made any real progress determining cause here, so if anyone wants to give it a go - please do. If it still reproduces.

This crash is not happening after 57.0.2952.0.

https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.magic_signature_1.name%3D%27blink%3A%3ATraceTrait%3Cblink%3A%3AHeapVector%3Cblink%3A%3AMember%3Cblink%3A%3AMessagePort%3E%2C0%3E%20%3E%3A%3Atrace%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=#samplereports:5,productversion

Very encouraging, thanks - I'll go over the changes around that time.

Just to clarify: 57.0.2952.0 is the last revision where the crash was observed.

57.0.2952.0	4.47%	50	
56.0.2924.3	39.68%	444	
56.0.2918.0	2.50%	28	
56.0.2916.0	2.23%	25	
56.0.2909.0	2.68%	30	
56.0.2908.0	2.32%	26	
56.0.2902.0	29.40%	329	
56.0.2900.0	2.14%	24	
56.0.2898.0	1.79%	20	
56.0.2889.0	2.14%	24	

fwiw, I don't see any >= M56 stacks that match on the Opera crashserver.

That M56 has stopped reproducing, is esp. interesting. But after having gone through the Source/ changes after 56.0.2924.3, nothing particular stands out as a targetted fix in this area, or for heap mutation in general.

Won't investigate further, but will back out the diagnosing changes made.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5e3d4684cd45c0aa08d541646e035d4e03bec8af

commit 5e3d4684cd45c0aa08d541646e035d4e03bec8af
Author: sigbjornf <sigbjornf@opera.com>
Date: Thu Jan 12 16:27:40 2017

Revert "Verify that FetchEvent::m_request holds same-thread reference."

Undo r440475's temporary use of SameThreadCheckedMember<>.

R=
BUG= 655926 

Review-Url: https://codereview.chromium.org/2629613002
Cr-Commit-Position: refs/heads/master@{#443257}

[modify] https://crrev.com/5e3d4684cd45c0aa08d541646e035d4e03bec8af/third_party/WebKit/Source/modules/serviceworkers/FetchEvent.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8f00e720aadeb5fdd7b14d86b1420124fbf92f00

commit 8f00e720aadeb5fdd7b14d86b1420124fbf92f00
Author: sigbjornf <sigbjornf@opera.com>
Date: Thu Jan 12 18:37:51 2017

Revert ExtendableMessageEvent same heap verification.

Revert r440390's debug instrumentation of ExtendableMessageEvent.

R=
BUG= 655926 

Review-Url: https://codereview.chromium.org/2623273004
Cr-Commit-Position: refs/heads/master@{#443299}

[modify] https://crrev.com/8f00e720aadeb5fdd7b14d86b1420124fbf92f00/third_party/WebKit/Source/modules/serviceworkers/ExtendableMessageEvent.cpp
[modify] https://crrev.com/8f00e720aadeb5fdd7b14d86b1420124fbf92f00/third_party/WebKit/Source/modules/serviceworkers/ExtendableMessageEvent.h