+V8 folks for the first crash

Looks like potential memory corruption.

We should pause the V8 autoroller per https://chromium.googlesource.com/chromium/src/+/ddb812975b721aee41c39a12ac739eab0e8b053f and see whether rolling it back to the previous hash resolves these issues. Frank, can you do that? I can review your change to src/DEPS.

Also, machenbach@, should we put a Win x64 GPU bot on the client.v8.fyi waterfall? Should we be running these browser tests there? If it turns out this breakage was in fact due to the V8 roll then I'd say so.

- Prime suspect with my limited knowledge maybe:
https://chromium.googlesource.com/v8/v8/+/a2b3a2fbc562584ec298dfe674c97662a125a59e

Is there a chromium trybot for this exact setting?

Is there any sort of stress mode, the test can run in? E.g. a filter that can be passed to the trybot to execute only the test in question, but a few 1000 times in parallel?

Without something like that, I have no clue how to approach this kind of failure. My only option would be to rather skip the test for now. To unblock the rolls.

Ah, the test is not flaky, but fails quite reliable. Then ignore what I said about stress mode, regarding this issue. But a trybot would be necessary to do any kind of bisection.

Revert in progress: https://codereview.chromium.org/2841993002/

Hmm, note also that the first mac and the first win failures above have different V8 roll in their blame list.

Did anybody check what's the intersection of the two?

If I'm not mistaken, the intersection is:
https://chromium.googlesource.com/chromium/src/+log/7961ed28c4513ca0415a28720ff43a5109794f23..b7360f79e32c16690c062f0b44fecd7d5d452c1e

There's no V8 roll in this list. So, if the test failures are related, V8 might be innocent here. But maybe they are distinct...

I don't think the Mac crash is related to the Windows crash. The former does not seem to be related to V8. The latter is due to a corrupted snapshot file. I'm not convinced https://codereview.chromium.org/2841993002/ is the culprit, but let's see whether that solves the issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/6d9ca97cd23dd9bf4c0a29350c2756829f7680c2

commit 6d9ca97cd23dd9bf4c0a29350c2756829f7680c2
Author: hablich <hablich@chromium.org>
Date: Tue Apr 25 19:54:00 2017

Revert of [snapshot] full setup delegate should also be able to deserialize. (patchset #2 id:20001 of https://codereview.chromium.org/2840493002/ )

Reason for revert:
prime suspect for https://bugs.chromium.org/p/chromium/issues/detail?id=714976

Original issue's description:
> [snapshot] full setup delegate should also be able to deserialize.
>
> Also move the responsibility of marking builtins as initialized
> to the deserializer.
>
> R=jkummerow@chromium.org
>
> Review-Url: https://codereview.chromium.org/2840493002
> Cr-Commit-Position: refs/heads/master@{#44802}
> Committed: https://chromium.googlesource.com/v8/v8/+/a2b3a2fbc562584ec298dfe674c97662a125a59e

TBR=jkummerow@chromium.org,yangguo@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.

BUG= chromium:714976 

Review-Url: https://codereview.chromium.org/2841993002
Cr-Commit-Position: refs/heads/master@{#44870}

[modify] https://crrev.com/6d9ca97cd23dd9bf4c0a29350c2756829f7680c2/src/builtins/builtins.h
[modify] https://crrev.com/6d9ca97cd23dd9bf4c0a29350c2756829f7680c2/src/setup-isolate-deserialize.cc
[modify] https://crrev.com/6d9ca97cd23dd9bf4c0a29350c2756829f7680c2/src/setup-isolate-full.cc
[modify] https://crrev.com/6d9ca97cd23dd9bf4c0a29350c2756829f7680c2/src/setup-isolate.h
[modify] https://crrev.com/6d9ca97cd23dd9bf4c0a29350c2756829f7680c2/src/snapshot/deserializer.cc
[modify] https://crrev.com/6d9ca97cd23dd9bf4c0a29350c2756829f7680c2/src/snapshot/snapshot.h

FYI: Roll revert in progress: https://codereview.chromium.org/2844513002/

+adamk, kozy, maybe you can have an eye on this today while MUC is in bed?

The two reverts are in conflict with the latest roll 6.0.83. Well, let's hope for the best and fast-forward ToT than ...

Roll CL: https://codereview.chromium.org/2842863002/

Ok I cancelled the roll back https://codereview.chromium.org/2844513002/.
We'll see if the roll forward helps.

The roll-forward just landed in https://codereview.chromium.org/2842863002/ .

I hope that this fixes the test failures that have been seen on the main chromium.gpu waterfall:

https://build.chromium.org/p/chromium.gpu/builders/Win7%20Release%20%28NVIDIA%29?numbuilds=200

https://build.chromium.org/p/chromium.gpu/builders/Mac%20Release%20%28Intel%29?numbuilds=200

https://build.chromium.org/p/chromium.gpu/builders/Linux%20Release%20%28NVIDIA%29?numbuilds=200

Failures on this waterfall are very serious because they can mean that the tryservers win_chromium_rel_ng, linux_chromium_rel_ng, and mac_chromium_rel_ng are broken. However, for some reason, they don't seem to be.

Looks like the roll forward didn't help. That means Yang's commit is not the culprit. In the mean time also https://codereview.chromium.org/2842903003/ was reverted and rebased for  issue 715391 .

 Issue 715391  has been merged into this issue.

Con: This seems to repro on pure-release only. E.g. not on trybots with dcheck_always_on, and not on pure debug bots either.

Pro: This seems to repro on linux. Maybe we're lucky locally.

Now I'm confused. It looks like the GPU linux release builder sets dcheck_always_on. This means we never test V8 in release as this sets DEBUG in V8:
https://build.chromium.org/p/chromium.gpu/builders/GPU%20Linux%20Builder/builds/86200

This means trybots and release bots are close to being equal, but this doesn't fail on trybots, e.g. https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/430266 is a run with the failing V8 version, the tests run, they don't fail.

This is an incremental build problem. The trybots run chromium and gpu tests together and almost all the time build a large number of different compile targets.

The GPU CI bots only build a small number of targets. A script seems to decide dynamically which targets are needed. This leads to browser_tests sometimes being built and sometimes not, e.g. compare:
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.gpu%2FGPU_Linux_Builder%2F86035%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.gpu%2FGPU_Linux_Builder%2F86050%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout

See first failing build:
https://build.chromium.org/p/chromium.gpu/builders/Linux%20Release%20%28NVIDIA%29/builds/101404

See the builder's build that compiled it:
https://build.chromium.org/p/chromium.gpu/builders/GPU%20Linux%20Builder/builds/86069

In the output you can see that mksnapshot ran:
ACTION //v8:run_mksnapshot(//build/toolchain/linux:clang_x64)

But browser_tests weren't built. The non-sane shapshot error makes sense as the snapshot is not fitting to the browser_tests.

Ergo: The script that decides the build targets has a bug.

Yet another symptom: I pressed clobber on the linux builder (it's a thing one should always be able to do cleanly):
https://build.chromium.org/p/chromium.gpu/builders/GPU%20Linux%20Builder/builds/86212

It fails, and now stays red with:
Traceback (most recent call last):
  File "/b/c/b/linux/src/tools/mb/mb.py", line 61, in Main
    ret = self.args.func()
  File "/b/c/b/linux/src/tools/mb/mb.py", line 299, in CmdGen
    return self.RunGNGen(vals)
  File "/b/c/b/linux/src/tools/mb/mb.py", line 885, in RunGNGen
    ', '.join(runtime_deps_targets))
MBErr: did not generate any of browser_tests.runtime_deps
step returned non-zero exit code: 1

Work-around to build brower_tests regardless of script output:
https://codereview.chromium.org/2841833003/

I think this is caused by a "fix" applied in  http://crbug.com/714336 

Currently browser_tests are commented out as dependency of tab_capture_end2end_tests

The root cause would be:
https://chromium.googlesource.com/chromium/src.git/+/2b3b32f690472ea7bfb3c676adce4f3874bab95c

Win7 Release (NVIDIA) builder is green after sashab's revert (https://codereview.chromium.org/2842903003):
https://build.chromium.org/p/chromium.gpu/builders/Win7%20Release%20%28NVIDIA%29/builds/67249

https://build.chromium.org/p/chromium.gpu/builders/Win7%20Release%20%28NVIDIA%29?numbuilds=200

After reverting, the snapshot fits again to the old browser_tests target.

This all makes sense. https://chromium.googlesource.com/chromium/src.git/+/2b3b32f690472ea7bfb3c676adce4f3874bab95c%5E%21/#F0 removes the browser_tests dependency. The script https://build.chromium.org/p/chromium.gpu/builders/GPU%20Linux%20Builder/builds/86069/steps/get%20compile%20targets%20for%20scripts doesn't output browser_tests anymore (at least in most of the cases). Hence, they are not build and never fit to the snapshot (which is built as other targets need it).

I think my work-around from comment 22 isn't really nice, but also not harmful either. Should just be reverted when the underlying problem is resolved.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1d0ceecfb8153cce088d62bf28d9c94c6601de91

commit 1d0ceecfb8153cce088d62bf28d9c94c6601de91
Author: machenbach <machenbach@chromium.org>
Date: Wed Apr 26 10:47:52 2017

Build browser tests on all gpu release bots

Work-around as browser_tests are erroneously not rebuilt properly on
gpu bots.

BUG= chromium:714976 
TBR=kbr@chromium.org,fjhenigman@chromium.org
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
NOTRY=true

Review-Url: https://codereview.chromium.org/2841833003
Cr-Commit-Position: refs/heads/master@{#467292}

[modify] https://crrev.com/1d0ceecfb8153cce088d62bf28d9c94c6601de91/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/1d0ceecfb8153cce088d62bf28d9c94c6601de91/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/1d0ceecfb8153cce088d62bf28d9c94c6601de91/testing/buildbot/chromium.gpu.json

The work-around above fixes the remaining problems on mac, e.g.:
https://build.chromium.org/p/chromium.gpu/builders/Mac%20Release%20%28Intel%29/builds/63073

The linux gn step is still sour since the clobber however:
https://build.chromium.org/p/chromium.gpu/builders/GPU%20Linux%20Builder

I don't know how to continue with the broken linux gpu builder. I can neither run the recipe locally, nor MB (due to not knowing the inputs). I filed blocking bugs for both.

I was able to contrive a similar error as on the bots locally with
tools/mb/mb.py gen -m chromium.gpu -b 'GPU Linux Builder' --config-file tools/mb/mb_config.pyl --goma-dir ~/goma  --swarming-targets-file=stuff //out/testo

And stuff containing:
content_shell_crash_test
browser_tests
content_browsertests
chrome

Then I get the local error:
Traceback (most recent call last):
  File "tools/mb/mb.py", line 61, in Main
    ret = self.args.func()
  File "tools/mb/mb.py", line 299, in CmdGen
    return self.RunGNGen(vals)
  File "tools/mb/mb.py", line 885, in RunGNGen
    ', '.join(runtime_deps_targets))
MBErr: did not generate any of chrome.runtime_deps

I assume the error on the bots is caused in a similar way, but the content of the swarming targets file is different. Maybe this is also indirectly caused by changing the label in:
https://chromium.googlesource.com/chromium/src.git/+/2b3b32f690472ea7bfb3c676adce4f3874bab95c%5E%21/#F1

Giving up here :(

I lied, I didn't give up :)

I just put into file stuff:
tab_capture_end2end_tests

Running the above mb command worked. Until I removed out/testo. I.e. simulating what the bot did on clobber. Afterwards it also fails locally with:
MBErr: did not generate any of browser_tests.runtime_deps

Changing the label of tab_capture_end2end_tests from //chrome/test:tab_capture_end2end_tests back to //chrome/test:browser_tests fixes the issue locally. But I'm a bit reluctant to do this change as it's a partial revert of https://codereview.chromium.org/2828143007 which attempted to fix something else...

Discussed with troopers that we should attempt to fix the builder first, even if this flushes out the other bug  http://crbug.com/714336  again:
https://codereview.chromium.org/2842963003/

 Issue 715478  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b8d34ab3bfbbc9cd34d48a72bad3fdb0ea5507b4

commit b8d34ab3bfbbc9cd34d48a72bad3fdb0ea5507b4
Author: machenbach <machenbach@chromium.org>
Date: Wed Apr 26 14:10:50 2017

Partial revert of "Attempt to disable tab_capture_end2end_tests on the trybots"

This partially reverts:
https://codereview.chromium.org/2828143007
to fix broken GPU Linux builder.

BUG= 714976 , 714336 
TBR=dpranke@chromium.org
NOTRY=true

Review-Url: https://codereview.chromium.org/2842963003
Cr-Commit-Position: refs/heads/master@{#467318}

[modify] https://crrev.com/b8d34ab3bfbbc9cd34d48a72bad3fdb0ea5507b4/testing/buildbot/gn_isolate_map.pyl

MB step fixed again by above CL:
https://build.chromium.org/p/chromium.gpu/builders/GPU%20Linux%20Builder/builds/86245

Relanding the reverted roll and starting the V8 autoroller afterwards.

Unfortunately, I suspect the partial revert in #c37 is just causing other failures. Really all of the interactions here are buggy one way or another. 

https://codereview.chromium.org/2840953002/ will hopefully be the right fix.

machenbach@, thanks for your efforts to try to fix this last night.

dpranke@, thanks for pushing forward https://codereview.chromium.org/2840953002/ . I hope that this will fix these issues permanently.

This should be fixed now, but I'm going to wait for some builds to cycle through before closing.

Looks good enough so far. Please reopen or file a new bug if you see more issues.

IIUC the builders will now build the browser_tests target explicitly? Please revert the work-around https://codereview.chromium.org/2841833003 if this is the case.

Ack.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a5ab58152fbb8f93dc9653f4ecce0d467fe6bf78

commit a5ab58152fbb8f93dc9653f4ecce0d467fe6bf78
Author: dpranke <dpranke@chromium.org>
Date: Fri Apr 28 07:14:22 2017

Revert of Build browser tests on all gpu release bots (patchset #1 id:1 of https://codereview.chromium.org/2841833003/ )

Reason for revert:
I don't think this CL is still needed after my fix for  crbug.com/714976 .

Original issue's description:
> Build browser tests on all gpu release bots
>
> Work-around as browser_tests are erroneously not rebuilt properly on
> gpu bots.
>
> BUG= chromium:714976 
> TBR=kbr@chromium.org,fjhenigman@chromium.org
> CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
> NOTRY=true
>
> Review-Url: https://codereview.chromium.org/2841833003
> Cr-Commit-Position: refs/heads/master@{#467292}
> Committed: https://chromium.googlesource.com/chromium/src/+/1d0ceecfb8153cce088d62bf28d9c94c6601de91

TBR=fjhenigman@chromium.org,kbr@chromium.org,hablich@chromium.org,phajdan.jr@chromium.org,machenbach@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= chromium:714976 

Review-Url: https://codereview.chromium.org/2846883004
Cr-Commit-Position: refs/heads/master@{#467921}

[modify] https://crrev.com/a5ab58152fbb8f93dc9653f4ecce0d467fe6bf78/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/a5ab58152fbb8f93dc9653f4ecce0d467fe6bf78/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/a5ab58152fbb8f93dc9653f4ecce0d467fe6bf78/testing/buildbot/chromium.gpu.json

 Issue 714928  has been merged into this issue.

Thank you Dirk for fixing this!