Bisecting. Good revision: 6b23040f39c99fa96de142c1e67e685e93082128

FYI, the steps on this bug fixed it for me: https://crbug.com/600425

specifically, running a 'gn clean'.

Thanks for the pointer ellyjones@, it seems like we're just missing build dependencies somewhere.

The failing CHECK happens during startup when trying to verify the snapshot, which fails because the snapshot has been generated by an older version of V8.

`gn clean` forces a clean rebuild.

machenbach@, any idea? Can you help triage?

I can't repro this anymore. The steps I tried:

$ git checkout 359bafdab6c2
$ ninja -j4096 -l12 -C out/release chrome
$ out/release/chrome  # starts successfully.

$ git checkout 31b1e91ff7546e6ec8d0527bf301f47b112b696e
$ gclient sync
$ ninja -j4096 -l12 -C out/release chrome
$ out/release/chrome  # also starts successfully.

One guess: I had a non-master branch checked out and did a `git pull --rebase`. Maybe this messes up file timestamp and confuses the build system. 

Lowering priority further since there's no repro.

bump, I can repro this 100%

syned again, right now at bfe88542d3e3fd92c64c281207b46f13ab4a2513, still repros. This is impacting productivity as it is hard for debugging.

xidachen@, did you try a clean build with `gn clean <build dir>` as suggested in #3? Does it still fail afterwards?

Repro instructions that go from a working build to this CHECK failure would go a long way towards a fix, if you can provide them.

xidachen@, could you please provide your detailed step-by-step repro? For somebody that wants to repro this from scratch, i.e. assume no checkout yet.

Lowering priority.
After a gn clean, I can no longer repro this.

 Issue 764657  has been merged into this issue.

A bit higher prio, since we saw it also on a bot for the first time. Maybe we can try to retrace the steps of several compiles of that bot to repro?

I also just ran into this. Doing a clean build also solved my problem.

Whoever in the future runs into this: Could you try to backup your state before doing the gn clean? Or even better: Retrace the revisions you previously built (and the build targets) so that we can get a repro of this...

 Issue 764137  has been merged into this issue.

As mentioned in https://bugs.chromium.org/p/chromium/issues/detail?id=764137#c2 - one does not have to run "gn clean" and rebuild everything. Just rm the *.bin files.

machenbach: Try syncing to r500514, build, and then sync to r501501 and build again.

Doesn't repro for me. r500514 also doesn't compile. I used now it's parent. My steps were:

build/install-build-deps.sh
git co 4d894378a2b24e^
gn args out/repro
# -> use the two args from this issue's description.
gclient sync
ninja -j2000 -C out/repro chrome
out/repro/chrome
-> No crash

git co 261f33bbf6cedb11d
gclient sync
ninja -j2000 -C out/repro chrome
out/repro/chrome
-> No crash

Any other hints about which compile targets you use for the ninja step? Always the same, or varying?

I can't reproduce it anymore either. :(

Looking at my "git reflog", I was actually at r500521, which revert r500514 for not compiling.

issue 762440 seems to have crash reports in the wild for this

Had a look at  issue 764657 , which is the only known bot error of that kind. Interestingly, the compile step ran:
[1688/7678] ACTION //v8:run_mksnapshot(//build/toolchain/linux:clang_x64)

And in the isolated archive of the failing swarming shard is a snapshot_blob.bin:
https://isolateserver.appspot.com/browse?namespace=default-gzip&hash=706c54ebb6d0ed6c55669a93b5ccda92c4d41984

Is it possible that V8 corrupted snapshot_blob.bin itself due to flaky output of mksnapshot? Are we sure that this comes from snapshot_blob.bin or can it have something to do with v8_context_snapshot.bin or natives_blob.bin?

I just had this issue and archived my build folder. Do you still need some of the files?

I'll discuss with V8 devs first what we could do to get any information from the bins. We also have the isolated archive from the bot from comment 22.

@Yang, Jakob: Can we (somewhat easily) retrieve from the snapshot_blob.bin if it is outdated or if it is corrupted in some other way?

@dullweber: is the last modification time of snapshot_blob.bin and the chrome executable around the same time? Or is one much older than the other?

Yes, snapshot_blob.bin was modified at 10:22, chrome 10:26. 
It was the first compilation after rebasing from some revision from yesterday. 

I tried locally with the build from  issue 764657 . It repros only well through the layout test driver and now when opening just one content_shell.

But the MSAN complaint is interesting:
==10082==WARNING: MemorySanitizer: use-of-uninitialized-value

  Uninitialized value was created by an allocation of 'minor' in the stack frame of function '_ZN2gl12GLSurfaceGLX16InitializeOneOffEv'

'minor' is probably the minor V8 version we use to calculate the version hash. If there are paths on which the field is uninitialized, it might potentially contain garbage, which leads to the check failure.

s/now/not in first line of last comment

@dullweber, is #23 a debug build by any chance?

Its was a release build but I just tried to compile a debug build and it also has the snapshot issue. I'm creating an archive of the directory.

Retracing my recent builds (in order of my building them, "bad" meaning it crashes with this IsSane error):

5e519b23 (r500839) - good
rde1fbac (r502527) - bad
49a21bf7 (r502471) - bad
932daf23 (r502470) - bad
9c2e2f0e (r502373) - bad
7c0b28b3 (r501735) - bad
5e519b23 (r500839) - good

Note: I haven't run gn clean at all yet. Dumping this info as requested in #14. I was going to do a git bisect between r500839 and r501735 (running now).

Bisect results are in. The failure was introduced in 79801d40 (r500854):

    Update V8 to version 6.3.92.
    
    Summary of changes available at:
    https://chromium.googlesource.com/v8/v8/+log/2431ff32..e8a6c622

Which only has a single commit inside of it.

And confirmed that syncing to r502527 and running "gn clean" before a full build resolves the issue.

The commit in comment 31 looks like a red herring.

Maybe this is caused by the static initialization order, which might change during incremental linking? And at the commit above it re-linked a bit for whatever reason, but not everything... leading to an order where the version hash is uninitialized and returns garbage at the time of the check...

I'm also bumping into this in a V8 checkout. I had previously compiled to out/Release and out/Debug, then done git pull, normal work in a branch, all the time compiling only out/Debug. Now when I try to use out/Release/d8, I get this error:

marja@marja:~/code/v8$ out/Release/d8 ../foo306.js 


#
# Fatal error in ../../src/snapshot/snapshot.h, line 31
# Check failed: IsSane().
#

==== C stack trace ===============================

    out/Release/d8(+0x8e1193) [0x55b2c20ae193]
    out/Release/d8(+0x7cea6b) [0x55b2c1f9ba6b]
    out/Release/d8(+0x7cb6ac) [0x55b2c1f986ac]
    out/Release/d8(+0x652c77) [0x55b2c1e1fc77]
    out/Release/d8(+0x65286e) [0x55b2c1e1f86e]
    out/Release/d8(+0xec108) [0x55b2c18b9108]
    out/Release/d8(+0xc7727) [0x55b2c1894727]
    /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f328ab33f45]
    out/Release/d8(+0xbb826) [0x55b2c1888826]
Received signal 4 ILL_ILLOPN 55b2c1f99312
Illegal instruction (core dumped)

Food for thought: ahaas@ had a local build that repro'ed it. What we did:
- build chrome
- confirm that snapshot_blob.bin and chrome have reasonably close time stamps (i.e. seem to have both been built)
- run chrome
-> crashes with check failure above
- build content_shell
- confirm that snapshot_blob.bin and chrome still have the same time stamps from before building content_shell
- run chrome
-> doesn't crash anymore o_O

Any progress here? I've been hitting this fairly frequently over the last week or so, and it's quite annoying. Let me know if there's something I can do to help the next time I get this.

I hit this today as well, I have a build directory that reliably repros, even after a resync and rebuild (didn't gn clean though) that I'm preserving in case we need to look through a repro.

No bandwidth for working on this over the next few days. Maybe a bit time again end of week. Yang, do we have a better owner for now?

I'd attempt to make a blind fix, removing the version initialization. Though that theory doesn't really fit to what we observed in comment 35. It's hard to know if we fixed the problem without a real repro (= step by step instructions in a clean checkout with an empty build dir).

I just ran into this. My .bin file dates:
-rw-r--r-- 1 mattm eng  213670 Sep 25 15:55 out/default/natives_blob.bin
-rw-r--r-- 1 mattm eng 2872760 Sep 25 15:58 out/default/snapshot_blob.bin
-rw-r--r-- 1 mattm eng 3329004 Sep 11 14:49 out/default/v8_context_snapshot.bin

I deleted out/default/v8_context_snapshot.bin and rebuilt, but the file was NOT re-generated:
mattm-linux 16:12:12 /y/src/chrome/src$ rm out/default/v8_context_snapshot.bin
mattm-linux 16:12:19 /y/src/chrome/src$ ninja -j1024 -l42 -C out/default chrome && ./out/default/chrome --log-level=0
ninja: Entering directory `out/default'
ninja: no work to do.



It did stop the CHECK failures, though I now get these warnings instead:

[34256:34303:0925/161238.186437:WARNING:child_process_launcher_helper_posix.cc(118)] Ignoring invalid file v8_context_snapshot.bin

I got basically the same results as mattm. Looking at references to v8_context_snapshot.bin, I see r496290. So I'm guessing understanding what dependencies is missing there might lead to fixing this bug.

If I explicitly run: ninja -C out/buildir v8_context_snapshot.bin, then v8_context_snapshot.bin gets regenerated and now my build is happy. If I remove the file, and build the chrome target, it doesn't get regenerated. Assigning to peria@ who wrote most of tools/v8_context_snapshot/BUILD.gn.

My guess is group("chrome") in chrome/BUILD.gn does not need //tools/v8_context_snapshot in public_deps, since chrome is the top level target that people are building with Ninja. Instead, chrome/child/BUILD.gn and chrome/renderer/BUILD.gn needs deps += //tools/v8_context_snapshot.

BTW, I believe //path/to/target implies //path/to/target:target, so no need for //tools/v8_context_snapshot:v8_context_snapshot.

Hopefully this will fix the bug: https://chromium-review.googlesource.com/#/c/chromium/src/+/683469 - I tossed it at the trybots and I'm going to sleep. -_-

Thanks thestig for the analysis! I have a reliable repro now, which confirms this analysis.

Check out df113380b620a5d in a clean build
gclient sync
gn args out/Release/
-> normal release build
ninja -C out/Release -j1000 chrome
-> v8_context_snapshot.bin missing, chrome works
ninja -C out/Release -j1000 content_shell
-> v8_context_snapshot.bin got built, chrome works

Check out cd0de01f947
gclient sync
ninja -C out/Release -j1000 chrome
-> v8_context_snapshot.bin is still old, chrome crashes!
ninja -C out/Release -j1000 v8_context_snapshot.bin
-> v8_context_snapshot.bin got built, chrome works

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/629406d1e940b14210cfb10dfc040c5145be9c29

commit 629406d1e940b14210cfb10dfc040c5145be9c29
Author: Yang Guo <yangguo@chromium.org>
Date: Tue Sep 26 09:55:53 2017

[snapshot] include version string in the startup snapshot.

This is to easier diagnose build issues involving the snapshot.
Sample error message for mismatching snapshot:

#
# Fatal error in ../../src/snapshot/snapshot-common.cc, line 286
# Version mismatch between V8 binary and snapshot.
#   V8 binary version: 6.3.1 (candidate)
#    Snapshot version: 6.3.0 (candidate)
# The snapshot consists of 2820444 bytes and contains 1 contexts.
#


R=machenbach@chromium.org

Bug:  chromium:764327 
Change-Id: Icdc7aeac77819b113985b424feda814a072d5406
Reviewed-on: https://chromium-review.googlesource.com/684295
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Yang Guo <yangguo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#48161}
[modify] https://crrev.com/629406d1e940b14210cfb10dfc040c5145be9c29/src/snapshot/code-serializer.h
[modify] https://crrev.com/629406d1e940b14210cfb10dfc040c5145be9c29/src/snapshot/serializer-common.h
[modify] https://crrev.com/629406d1e940b14210cfb10dfc040c5145be9c29/src/snapshot/snapshot-common.cc
[modify] https://crrev.com/629406d1e940b14210cfb10dfc040c5145be9c29/src/snapshot/snapshot.h

With the change above we should be able to tell which snapshot is failing. If the snapshot contains more than one context, it's probably the v8_context_snapshot.bin.

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/dc64a7311d9cd92c2480563439d686489c368daf

commit dc64a7311d9cd92c2480563439d686489c368daf
Author: Yang Guo <yangguo@chromium.org>
Date: Tue Sep 26 10:32:41 2017

Revert "[snapshot] include version string in the startup snapshot."

This reverts commit 629406d1e940b14210cfb10dfc040c5145be9c29.

Reason for revert: cross platform builds break.

Original change's description:
> [snapshot] include version string in the startup snapshot.
> 
> This is to easier diagnose build issues involving the snapshot.
> Sample error message for mismatching snapshot:
> 
> #
> # Fatal error in ../../src/snapshot/snapshot-common.cc, line 286
> # Version mismatch between V8 binary and snapshot.
> #   V8 binary version: 6.3.1 (candidate)
> #    Snapshot version: 6.3.0 (candidate)
> # The snapshot consists of 2820444 bytes and contains 1 contexts.
> #
> 
> 
> R=​machenbach@chromium.org
> 
> Bug:  chromium:764327 
> Change-Id: Icdc7aeac77819b113985b424feda814a072d5406
> Reviewed-on: https://chromium-review.googlesource.com/684295
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Yang Guo <yangguo@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#48161}

TBR=machenbach@chromium.org,yangguo@chromium.org

Change-Id: I35a9b575e4f7fe5c45c9dc6f9e774c3e6d30049c
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  chromium:764327 
Reviewed-on: https://chromium-review.googlesource.com/684315
Reviewed-by: Yang Guo <yangguo@chromium.org>
Commit-Queue: Yang Guo <yangguo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#48162}
[modify] https://crrev.com/dc64a7311d9cd92c2480563439d686489c368daf/src/snapshot/code-serializer.h
[modify] https://crrev.com/dc64a7311d9cd92c2480563439d686489c368daf/src/snapshot/serializer-common.h
[modify] https://crrev.com/dc64a7311d9cd92c2480563439d686489c368daf/src/snapshot/snapshot-common.cc
[modify] https://crrev.com/dc64a7311d9cd92c2480563439d686489c368daf/src/snapshot/snapshot.h

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/dc7b2b2ba7b725632154a6a7b938a6e4acdee69d

commit dc7b2b2ba7b725632154a6a7b938a6e4acdee69d
Author: Yang Guo <yangguo@chromium.org>
Date: Tue Sep 26 11:29:44 2017

Reland "[snapshot] include version string in the startup snapshot."

This is a reland of 629406d1e940b14210cfb10dfc040c5145be9c29
Original change's description:
> [snapshot] include version string in the startup snapshot.
> 
> This is to easier diagnose build issues involving the snapshot.
> Sample error message for mismatching snapshot:
> 
> #
> # Fatal error in ../../src/snapshot/snapshot-common.cc, line 286
> # Version mismatch between V8 binary and snapshot.
> #   V8 binary version: 6.3.1 (candidate)
> #    Snapshot version: 6.3.0 (candidate)
> # The snapshot consists of 2820444 bytes and contains 1 contexts.
> #
> 
> 
> R=machenbach@chromium.org
> 
> Bug:  chromium:764327 
> Change-Id: Icdc7aeac77819b113985b424feda814a072d5406
> Reviewed-on: https://chromium-review.googlesource.com/684295
> Reviewed-by: Michael Achenbach <machenbach@chromium.org>
> Commit-Queue: Yang Guo <yangguo@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#48161}

Bug:  chromium:764327 
Change-Id: I3721689824e0a6909eede86d0829dc258ae40c4d
Reviewed-on: https://chromium-review.googlesource.com/684494
Reviewed-by: Michael Achenbach <machenbach@chromium.org>
Commit-Queue: Yang Guo <yangguo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#48164}
[modify] https://crrev.com/dc7b2b2ba7b725632154a6a7b938a6e4acdee69d/src/snapshot/code-serializer.h
[modify] https://crrev.com/dc7b2b2ba7b725632154a6a7b938a6e4acdee69d/src/snapshot/serializer-common.h
[modify] https://crrev.com/dc7b2b2ba7b725632154a6a7b938a6e4acdee69d/src/snapshot/snapshot-common.cc
[modify] https://crrev.com/dc7b2b2ba7b725632154a6a7b938a6e4acdee69d/src/snapshot/snapshot.h
[modify] https://crrev.com/dc7b2b2ba7b725632154a6a7b938a6e4acdee69d/src/version.cc
[modify] https://crrev.com/dc7b2b2ba7b725632154a6a7b938a6e4acdee69d/test/cctest/test-version.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bf05f2bdfbe8645b6c20af7e013a7555b9fbd06e

commit bf05f2bdfbe8645b6c20af7e013a7555b9fbd06e
Author: Lei Zhang <thestig@chromium.org>
Date: Tue Sep 26 22:00:40 2017

Fix chrome/ dependencies for //tools/v8_context_snapshot.

Exposing //tools/v8_context_snapshot as a public dependency for the
"chrome" target does not do anything useful when one is building the
"chrome" target.

BUG= 764327 

Change-Id: Ia1b6bc6cf643a4981085cf58c72be7c1f3dd44b4
Reviewed-on: https://chromium-review.googlesource.com/683469
Reviewed-by: Hitoshi Yoshida <peria@chromium.org>
Commit-Queue: Lei Zhang <thestig@chromium.org>
Cr-Commit-Position: refs/heads/master@{#504493}
[modify] https://crrev.com/bf05f2bdfbe8645b6c20af7e013a7555b9fbd06e/chrome/BUILD.gn
[modify] https://crrev.com/bf05f2bdfbe8645b6c20af7e013a7555b9fbd06e/chrome/child/BUILD.gn
[modify] https://crrev.com/bf05f2bdfbe8645b6c20af7e013a7555b9fbd06e/chrome/installer/mini_installer/BUILD.gn
[modify] https://crrev.com/bf05f2bdfbe8645b6c20af7e013a7555b9fbd06e/chrome/renderer/BUILD.gn

Hopefully that's it.

thestig@, thank you for handling this.