Issue 870584 link

Starred by 2 users

Issue metadata

Status:	Fixed
Owner:	yangguo@chromium.org
Closed:	Nov 9
Cc:	yukishiino@chromium.org mlippautz@chromium.org tikuta@chromium.org yangguo@chromium.org jgruber@chromium.org peria@chromium.org haraken@chromium.org verwa...@chromium.org r...@chromium.org
Components:	Blink>JavaScript>Runtime Build
EstimatedDays:	----
NextAction:	----
OS:	----
Pri:	3
Type:	Bug

Blocking:
issue 330260
issue 869348

v8_context_snapshot.bin is non-deterministic

Project Member Reported by tikuta@chromium.org, Aug 3

Issue description

Everytime I run v8_context_snapshot_generator, it generates different v8_context_snapshot.bin binaries.

This can be blocker of test exectuion caching on swarming.

With discussion to peria@-san, we noticed that there are some uninitialized byte suffix in version string 
https://cs.chromium.org/chromium/src/v8/src/snapshot/snapshot.h?l=221&rcl=f357c021ff4c925ea4c00ae9620d8d66b134b788

But seems more non-deterministic things in v8_context_snapshot.bin.

I confirmed that v8's snapshot_blob.bin did not become different files with 2 times clean build comparison.

Showing comments 3 - 102 of 102 Older ›

Comment 3 by jgruber@chromium.org, Aug 3

Cc: -yangguo@chromium.org
Owner: jgruber@chromium.org
Status: Assigned (was: Available)

Taking this for now so it doesn't get lost.

Comment 4 by yangguo@chromium.org, Aug 3

Any sort of non-determinism during context initialization could cause the snapshot to become non-deterministic as well.

One way I have been able to debug this kind of issue is to compare two snapshots, and figure out the first diverging byte at offset N. Then set a conditional break point to v8::internal::SnapshotByteSink::Put to break when we write the N-th byte. The stack trace at the break should be able to give some clues.

Comment 5 by peria@chromium.org, Aug 3

#2; The comment in the header file is wrong. It says using 128 bytes for version string, but actually 64 bytes are used (kVersionStringLength = 64), and it is well zero-initialized.

Latter parts look different in every build.

Comment 6 by tikuta@chromium.org, Aug 3

> Otherwise, I don't have any immediate suspicions. Is the repro just trigger a standard chrome build process on ToT? Do you have links to test failures?

There is no builder checking the determinisity of v8_context_snapshot.bin

My goal here is v8_context_snapshot_generator generates the same binary when we run its multiple times.


> I recently fixed a potential nondeterminism related to off-heap trampolines [1]. Maybe that is related? You could check by setting v8_enable_embedded_builtins=false in BUILD.gn.

Specifying v8_enable_embedded_builtins=false does not fix that.

I used following args.gn

```
dcheck_always_on = true
ffmpeg_branding = "Chrome"
# goma_dir = "/b/c/goma_client"
is_component_build = false
is_debug = false
proprietary_codecs = true
strip_absolute_paths_from_debug_symbols = true
symbol_level = 1
use_goma = true

use_dummy_lastchange=true

# For https://bugs.chromium.org/p/chromium/issues/detail?id=870584
v8_enable_embedded_builtins=false

```

and got following results.

tikuta@tikuta:~/chromium/src/out/linux_chromium_rel_ng$ python ../../build/gn_run_binary.py ./v8_context_snapshot_generator --output_file=v8_context_snapshot.bin
tikuta@tikuta:~/chromium/src/out/linux_chromium_rel_ng$ sha256sum v8_context_snapshot.bin
180b679348c4c3e64c0823fa29a41ed8a58670e286d4c56401d8502e0eb581f9  v8_context_snapshot.bin
tikuta@tikuta:~/chromium/src/out/linux_chromium_rel_ng$ python ../../build/gn_run_binary.py ./v8_context_snapshot_generator --output_file=v8_context_snapshot.bin
tikuta@tikuta:~/chromium/src/out/linux_chromium_rel_ng$ sha256sum v8_context_snapshot.bin 
0017f15785c0ee726085833e337f4211ae1cfba8965756c806979a637b344923  v8_context_snapshot.bin

Comment 7 by peria@chromium.org, Aug 3

Let me follow-up my comment.

> Latter parts look different in every build.
"Latter parts" includes offsets to built-in and contexts. It means some serialized data can be different.

"every build" means every clean build.  Running same v8_context_snapshot_generator seems to generate same binary file.

Comment 8 by tikuta@chromium.org, Aug 3

> "every build" means every clean build.  Running same v8_context_snapshot_generator seems to generate same binary file.

Actually running same v8_context_snapshot_generator seems to generate *different* binary file?

Comment 9 by jgruber@chromium.org, Aug 3

I verified locally that the V8 snapshot is identical across clean builds:

$ autoninja -C out/release d8
$ sha256sum out/release/*_blob.bin
fc39a1d453593f86c9acb986e961d40e7d62c1a486c68beff9ac9c18d71e8db2  out/release/natives_blob.bin
23adaf024ab0d9cd1f3407c4fd4acbab5aa98a17d08991c7efa64388b116e1fe  out/release/snapshot_blob.bin
$ autoninja -C out/release -t clean
$ autoninja -C out/release d8
$ sha256sum out/release/*_blob.bin
fc39a1d453593f86c9acb986e961d40e7d62c1a486c68beff9ac9c18d71e8db2  out/release/natives_blob.bin
23adaf024ab0d9cd1f3407c4fd4acbab5aa98a17d08991c7efa64388b116e1fe  out/release/snapshot_blob.bin

Comment 10 by jgruber@chromium.org, Aug 3

Each v8_context_snapshot_generator run generates a different .bin on my machine. The diff, from what I've seen, starts at offset 0x48 or 0x54:

$ out/release/v8_context_snapshot_generator --output_file=out/release/v8_context_snapshot_this.bin
$ out/release/v8_context_snapshot_generator --output_file=out/release/v8_context_snapshot_that.bin
$ cmp -l out/release/v8_context_snapshot_{this,that}.bin | gawk '{printf "%08X %02X %02X\n", $1-1, strtonum(0$2), strtonum(0$3)}' | head
00000048 90 80
0000004C CC BC
00000050 60 70
00000054 D4 FC
00000060 10 00
000002BC 9E 96
000002BD 4A FC
000002BE 7B 80
000002BF 7A AF
000002EC 9E B6

Comment 11 by jgruber@chromium.org, Aug 3

Owner: tikuta@chromium.org

If I set --predictable in V8, the generated blobs are again identical:

$ out/release/v8_context_snapshot_generator --output_file=out/release/v8_context_snapshot_this.bin 
$ out/release/v8_context_snapshot_generator --output_file=out/release/v8_context_snapshot_that.bin 
$ sha256sum out/release/v8_context_snapshot*.bin 
f72a0f4cdfbca49d059c86d25240ae218db7deff6b0795d89bc69cf1445c51ad  out/release/v8_context_snapshot_that.bin
f72a0f4cdfbca49d059c86d25240ae218db7deff6b0795d89bc69cf1445c51ad  out/release/v8_context_snapshot_this.bin

Here's what I changed:

diff --git a/src/flag-definitions.h b/src/flag-definitions.h
index 41961102ed..ad76efb85c 100644
--- a/src/flag-definitions.h
+++ b/src/flag-definitions.h
@@ -1367,7 +1367,7 @@ DEFINE_IMPLICATION(print_all_code, code_comments)
 // Predictable mode related flags.
 //
 
-DEFINE_BOOL(predictable, false, "enable predictable mode")
+DEFINE_BOOL(predictable, true, "enable predictable mode")
 DEFINE_IMPLICATION(predictable, single_threaded)
 DEFINE_NEG_IMPLICATION(predictable, memory_reducer)
 DEFINE_VALUE_IMPLICATION(single_threaded, wasm_num_compilation_tasks, 0)


The flag ensures that V8's heap layout is identical across runs. Could you set v8::internal::FLAG_predictable = true in v8_context_snapshot_generator.cc? Assigning back for next steps.

Comment 12 by tikuta@chromium.org, Aug 3

Status: Started (was: Assigned)

#11, Thank you for investigation!
I'll set the flag.

Comment 13 by tikuta@chromium.org, Aug 3

Owner: peria@chromium.org

I somehow failed to do this.

Project Member

Comment 14 by bugdroid1@chromium.org, Aug 3

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/51c23e5a3a821d63278b751489bec0efc5afbffc

commit 51c23e5a3a821d63278b751489bec0efc5afbffc
Author: Hitoshi Yoshida <peria@chromium.org>
Date: Fri Aug 03 11:20:12 2018

V8ContextSnapshot: Set predictable flag in v8_context_snapshot_generator

In general, V8 uses some random numbers not to make their environment
non-deterministic. However, it prevents us from caching
v8_context_snapshot.bin in test runs.

This CL sets "--predictable" flag in the snapshot generator to fix
the random seed and generate identical snapshot files in every run.


Bug:  870584 
Change-Id: I90e60d9636cbd31263658a6fa19c942d89442568
Reviewed-on: https://chromium-review.googlesource.com/1161853
Commit-Queue: Hitoshi Yoshida <peria@chromium.org>
Commit-Queue: Yuki Shiino <yukishiino@chromium.org>
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Yuki Shiino <yukishiino@chromium.org>
Cr-Commit-Position: refs/heads/master@{#580504}
[modify] https://crrev.com/51c23e5a3a821d63278b751489bec0efc5afbffc/tools/v8_context_snapshot/v8_context_snapshot_generator.cc

Comment 15 by tikuta@chromium.org, Aug 3

Status: Verified (was: Started)

Thanks, I confirmed that v8_context_snapshot.bin generated deterministically in non-component build.

Although it is not yet deterministic in component build, that is not problem this time.

Comment 16 by thakis@chromium.org, Aug 6

This is still failing on the Windows Deterministic bot:

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Windows%20deterministic/9666

v8_context_snapshot.bin                                 : DIFFERENT (unexpected): 4 out of 1527736 bytes are different (0.00%)
  0x151e00  : 06ad1706b117c0824300009ec07ca9c602f3c0000000009f9e1675020a00b111 '........C....|............u.....'
              06ad1706b117c0824300009ec07ca9ca02f3c0000000009f9e1675020a00b111 '........C....|............u.....'
                                             ^
  0x152e00  : 06211f020cf606251f06291fc0826d0000841d58e09ec0b095f502f0c0000000 '.!.....%..)...m....X............'
              06211f020cf606251f06291fc0826d0000841d58e09ec0b095f902f0c0000000 '.!.....%..)...m....X............'
                                                                 ^
  0x15cb00  : 5ac6020c2503c000000000021c16d0c10a0000000200000016e901063d22c0ce 'Z...%.......................="..'
              5aca020c2503c000000000021c16d0c10a0000000200000016e901063d22c0ce 'Z...%.......................="..'
                 ^
  0x15cb20  : 040000020c16950706fd330601349e1675029ee0c1f45ac602000000009e8402 '..........3..4..u.....Z.........'
              040000020c16950706fd330601349e1675029ee0c1f45aca02000000009e8402 '..........3..4..u.....Z.........'

Comment 17 by thakis@chromium.org, Aug 6

Status: Started (was: Verified)

Comment 18 by thakis@chromium.org, Aug 6

Blocking: 330260

Comment 19 by tikuta@chromium.org, Aug 6

Owner: tikuta@chromium.org

I will investigate this.

Comment 20 by tikuta@chromium.org, Aug 7

I confirmed that v8_context_snapshot.bin become non deterministic with deterministic builder's args.gn

```
is_component_build = false
is_debug = false
strip_absolute_paths_from_debug_symbols = true
symbol_level = 1
target_cpu = "x86"
use_goma = true
```

But ifI changed symbol_level to 1, the same v8_context_snapshot_generator seems to genearates the same binary.


PS C:\src\chromium\src> Get-FileHash .\out\deterministic\v8_context_snapshot.bin

Algorithm       Hash                                                                   Path
---------       ----                                                                   ----
SHA256          A22760758AB3284BC3F274779963A2EE4FA9D670002A6BD2899040CCF57D08E1       C:\src\chromium\src\out\deterministic\v8_context_snapshot.bin


PS C:\src\chromium\src> del .\out\deterministic\v8_context_snapshot.bin
PS C:\src\chromium\src> ninja -C .\out\deterministic\ -j1000 v8_context_snapshot.bin
ninja: Entering directory `.\out\deterministic\'
[1/1] ACTION //tools/v8_context_snapshot:generate_v8_context_snapshot(//build/toolchain/win:win_clang_x86)
PS C:\src\chromium\src> Get-FileHash .\out\deterministic\v8_context_snapshot.bin

Algorithm       Hash                                                                   Path
---------       ----                                                                   ----
SHA256          A22760758AB3284BC3F274779963A2EE4FA9D670002A6BD2899040CCF57D08E1       C:\src\chromium\src\out\deterministic\v8_context_snapshot.bin


PS C:\src\chromium\src> del .\out\deterministic\v8_context_snapshot_generator.exe
PS C:\src\chromium\src> ninja -C .\out\deterministic\ -j1000 v8_context_snapshot.bin
ninja: Entering directory `.\out\deterministic\'
[2/2] ACTION //tools/v8_context_snapshot:generate_v8_context_snapshot(//build/toolchain/win:win_clang_x86)
PS C:\src\chromium\src> Get-FileHash .\out\deterministic\v8_context_snapshot.bin

Algorithm       Hash                                                                   Path
---------       ----                                                                   ----
SHA256          6643272EF16A160740DD5A24570A98BE81760CF05E664332C921D388C1576D96       C:\src\chromium\src\out\deterministic\v8_context_snapshot.bin


PS C:\src\chromium\src>


It is something miraculous for me.

Comment 21 by tikuta@chromium.org, Aug 7

Owner: ----
Status: Available (was: Started)

#20

I meant to paste this for args.gn.

```
is_component_build = false
is_debug = false
strip_absolute_paths_from_debug_symbols = true
symbol_level = 0 # not 1
target_cpu = "x86"
use_goma = true
```

I don't come up the way for further investigation.
Maybe higher symbol_level for snapshot generator eventually fix this.

Comment 22 by jgruber@chromium.org, Aug 7

#21, where does indeterminism come in? Is it already in the V8 snapshot with symbol_level = 0? Does it start at a particular byte position (see the command in #10)? Yang also suggested a way of debugging in #4.

Comment 23 by tikuta@chromium.org, Aug 7

Owner: tikuta@chromium.org
Status: Assigned (was: Available)

#22, windows deterministic builder does not show the difference of natives_blob.bin  and snapshot_blob.bin.

And v8_context_snapshot.bin difference seems to be around here.

v8_context_snapshot.bin                                 : DIFFERENT (unexpected): 8 out of 1528124 bytes are different (0.00%)
  0x151ee0  : 0000d019000006a917020cf106ad1706b117c0824300009ec0bcb84a02f3c000 '....................C......J....'
              0000d019000006a917020cf106ad1706b117c0824300009ec0bcb84b03f3c000 '....................C......K....'
                                                                     ^ ^                                   ^
  0x152f00  : e09ec0b0a67902f0c0000000009f9e1675020a007116c000000000024c16d0c1 '.....y..........u...q.......L...'
              e09ec0b0a67a03f0c0000000009f9e1675020a007116c000000000024c16d0c1 '.....z..........u...q.......L...'
                         ^ ^                                                         ^
  0x15cc00  : 9f9ef7c0a00500009ec0346a4a020c2503c000000000021c16d0c10a00000002 '..........4jJ..%................'
              9f9ef7c0a00500009ec0346a4b030c2503c000000000021c16d0c10a00000002 '..........4jK..%................'
                                       ^ ^                                                  ^
  0x15cc40  : 346a4a02000000009e84021498c0a29601000615340619349e84e4021498c0a2 '4jJ.................4..4........'
              346a4b03000000009e84021498c0a29601000615340619349e84e4021498c0a2 '4jK.................4..4........'
                   ^ ^                                                            ^

I will investigate this as suggested in #4. Thanks

Project Member

Comment 24 by bugdroid1@chromium.org, Aug 7

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ead8646549210f386a00ae7ced0517d8defcfa7b

commit ead8646549210f386a00ae7ced0517d8defcfa7b
Author: Takuto Ikuta <tikuta@chromium.org>
Date: Tue Aug 07 14:42:53 2018

Add v8_context_snapshot.bin to deterministic whitelist

This is to fix Deterministic Linux (dbg) builder.

Removed mini_installer.exe and previous_version_mini_installer.exe too.

Bug:  871760 ,  870584 
Change-Id: Ieb85e844ceb9d5430f3870b26f98ffbe095ad539
Reviewed-on: https://chromium-review.googlesource.com/1165269
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Avi Drissman <avi@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#581221}
[modify] https://crrev.com/ead8646549210f386a00ae7ced0517d8defcfa7b/tools/determinism/deterministic_build_whitelist.pyl

Comment 25 by tikuta@chromium.org, Aug 13

I yet seeing non-deterministic v8_context_snapshot.bin seems come from v8's snapshot_blob.bin

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/162678
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/162679

Above builds produced different snapshot_blob.bin's
https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=168205e27e89c6827fd9b7ec2ed2bcb4fe54fbf0&as=snapshot_blob.bin
https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=d064fe2cf9607b03e6d343aeb8ee0216556a237d&as=snapshot_blob.bin

But without using something like text viewer, it is hard to find the place of non-deterministic place.
v8 folks, do you have something like that so that we don't need to do hard debug?

Comment 26 by jgruber@chromium.org, Aug 13

Hm I don't know what's happening there. I just tried again locally, using the same build flags as the bot in #25, and the snapshot blob is deterministic:

$ autoninja -C out/crbug-870584/ d8
$ sha256sum out/crbug-870584/snapshot_blob.bin
f374519b9fe0faada905537e4941af1fdcf98db74a46cef6fa8f2756148ed16f  out/crbug-870584/snapshot_blob.bin
$ autoninja -C out/crbug-870584/ d8 -t clean
$ autoninja -C out/crbug-870584/ d8 
$ sha256sum out/crbug-870584/snapshot_blob.bin
f374519b9fe0faada905537e4941af1fdcf98db74a46cef6fa8f2756148ed16f  out/crbug-870584/snapshot_blob.bin

Build args:

$ cat out/crbug-870584/args.gn 
# Build arguments go here.
# See "gn args <out_dir> --list" for available build arguments.
dcheck_always_on = true
is_component_build = false
is_debug = false
strip_absolute_paths_from_debug_symbols = true
symbol_level = 0
use_goma = true

Revision:

538bd6c7090fb4ad3c73675c5db28e17b4c3dd33 (plus some unrelated local changes of mine on top)

I also had a quick look at the snapshots from #25, the diff looks interesting:

$ cmp -l ~/Downloads/snapshot_blob.bin ~/Downloads/snapshot_blob\ \(1\).bin | gawk '{printf "%08X %02X %02X\n", $1-1, strtonum(0$2), strtonum(0$3)}' | head
00159C75 F0 E8
00159CA1 E8 F0
00159CEC E8 F0
00159E8C F0 E8
00159EAA E8 F0
00159EEF E8 F0
00159F0D E8 F0
00159F52 E8 F0
00159F69 E8 F0
00159FE9 E8 F0

Comment 27 by tikuta@chromium.org, Aug 13

I tried to compare mksnapshot binaries from 2 different builds by logging in to the VM's.

https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/162805
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/162807

And I found that mksnapshot binary in https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/162805 does not have symbol FLAG_enable_one_shot_optimization, I logged in to the machine and check what's the v8's HEAD revision in the machine.
That pointed out https://chromium.googlesource.com/v8/v8/+log/c1226cea1ec11d5b766226c912c475647a731274

So this non-determinism is now looks infra issue, rather than build itself.
This may related to that code search for v8 is not latest (Currently codesearch indexed https://chromium.googlesource.com/v8/v8.git/+/4fe3de13fc5c2592559bb5e49e1a81a6d09a2efd)

Comment 28 by tikuta@chromium.org, Aug 13

Filed codesearch bug https://bugs.chromium.org/p/chromium/issues/detail?id=873638

Comment 29 by jgruber@chromium.org, Aug 13

#27: Not sure I understand. Did you find out why the two bot builds from #25 produced different blobs?

Comment 30 by tikuta@chromium.org, Aug 13

Ah, sorry I overlooked that linux_chromium_rel_ng builder tried patch from v8 projects in
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/162807
Please ignore #27.

Comment 31 by tikuta@chromium.org, Aug 14

#26,

Culplit of non-determinism is CreateRegExpLiteral generated around
https://cs.chromium.org/chromium/src/v8/src/snapshot/builtin-serializer.cc?l=44&rcl=4fe3de13fc5c2592559bb5e49e1a81a6d09a2efd

I got different snapshot_blob.bin from below tries.
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/163646
https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/163647

820b27700f8a4ab633ae0e22319f05a9d75f68c5 (different from my local) vs 22621613deae5c5bb91a6f6eae9fbed02096ea78 (same with my local)

I think CreateRegExpLiteral generates below sequence in my local.
00159dd0  00 cc 4c 89 4d d0 4c 89  75 e0 48 89 5d f0

But different snapshot_blob.bin has following sequence.
00159dd0  00 cc 4c 89 4d d0 4c 89  75 e0 48 89 5d e8 

See attached screenshot of vimdiff applied to `hexdump -C` output.

I use following code to find culplit place.
https://chromium-review.googlesource.com/c/v8/v8/+/1174292


v8 folks, do you know how CreateRegExpLiteral is serialized to snapshot_blob.bin?

	diff.png 244 KB View Download

Comment 32 by jgruber@chromium.org, Aug 14

There shouldn't be anything special about the CreateRegExpLiteral bytecode handler. See [0] and [1] for definition and body.

[0] https://cs.chromium.org/chromium/src/v8/src/interpreter/interpreter-generator.cc?l=2318&rcl=4fe3de13fc5c2592559bb5e49e1a81a6d09a2efd
[1] https://cs.chromium.org/chromium/src/v8/src/builtins/builtins-constructor-gen.cc?l=296&rcl=4fe3de13fc5c2592559bb5e49e1a81a6d09a2efd

I still cannot repro locally. #31, were you able to produce different snapshots from the same revision? Do you have repro instructions? 

Or is it only triggered when building on different bots ("swarm1069-c4" vs. "swarm1017-c4" in #31)?

Comment 33 by tikuta@chromium.org, Aug 14

#32, 

This is not reproduced locally too.
Only happened on buildbot and I'm trying to find the place generating different binary.

Comment 34 by petermarshall@chromium.org, Aug 16

Cc: -petermarshall@chromium.org

Comment 35 by thakis@chromium.org, Sep 7

Any news? I think we're waiting for jgruber to reply?

Comment 36 by jgruber@chromium.org, Sep 10

Nothing has changed on my end, the V8 snapshot is deterministic on my machine. The latest WIP was mentioned in #33 by tikuta.

Comment 37 by thakis@google.com, Oct 12

I just tried this myself and had no issues reproducing locally. (jgruber, you're trying this on Windows, yes? On Linux static library builds, things are fine.)

C:\src\chrome\src>type out\gnrel\args.gn
symbol_level = 1
use_goma = true
is_debug = false

C:\src\chrome\src>type out\gnrel2\args.gn
symbol_level = 1
use_goma = true
is_debug = false


Then

C:\src\chrome\src>..\..\tim\tim ninja -C out\gnrel v8_context_snapshot.bin -j200
ninja: Entering directory `out\gnrel'
[17660/17660] ACTION //tools/v8_context_snapshot:generate_v8_context_snapshot(//build/toolchain/win:win_clang_x64)

peak memory: 602.38MB
real: 40m10.844s
qpc: 2410845048us

C:\src\chrome\src>..\..\tim\tim ninja -C out\gnrel2 v8_context_snapshot.bin -j200
ninja: Entering directory `out\gnrel2'
[17660/17660] ACTION //tools/v8_context_snapshot:generate_v8_context_snapshot(//build/toolchain/win:win_clang_x64)

peak memory: 602.45MB
real: 17m44.110s
qpc: 1064089405us


And:

thakis@thakis22-w MINGW64 /c/src/chrome/src (master)
$ cmp out/gnrel/v8_context_snapshot.bin out/gnrel2/v8_context_snapshot.bin
out/gnrel/v8_context_snapshot.bin out/gnrel2/v8_context_snapshot.bin differ: char 9, line 1



You're not seeing this locally?

Comment 38 by thakis@chromium.org, Oct 12

Locally I'm seeing diffs in bytes 8-16, then some at 0x88bcf, 0x89f96, 0x96fc3, 0x97014 (just a few bytes each). The file starts like so:

00000000: 0300 0000 0100 0000 81fd 4f95 0721 5f28  ..........O..!_(
00000010: 372e 312e 3236 3100 0000 0000 0000 0000  7.1.261.........
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000050: 70c9 0500 4890 0700 106f 0800 8004 0a00  p...H....o......
00000060: 7603 dec0 0700 0000 e8c8 0500 9876 0080  v............v..


Looks like this describes the container:

https://cs.chromium.org/chromium/src/v8/src/snapshot/snapshot.h?g=0&l=220

  // Snapshot blob layout:
  // [0] number of contexts N
  // [1] rehashability
  // [2] checksum part A
  // [3] checksum part B
  // [4] (128 bytes) version string
  // [5] offset to builtins
  // [6] offset to context 0
  // [7] offset to context 1
  // ...
  // ... offset to context N - 1
  // ... startup snapshot data
  // ... builtin snapshot data
  // ... context 0 snapshot data
  // ... context 1 snapshot data
(the version string is only 64 byte, the comment is wrong)


That header says we have 3 contexts, rehashable is set to 1, then the 8 different bytes are two checksums (so those should go away automatically once the other differences go away), 64 bytes version string, then builtins at 0x5c970, context data at 0x79048, 0x86f70 and 0xa0480.

So the different areas are all in context snapshot 1 (starting counting at 0 -- i.e. the middle of the 3).

For the approach in comment 4, one has to look at data_.size() relative to context 1's start -- i.e. look at data that's written not at 0x88bcf but at 0x88bcf - 0x86f70 if I understand things right (---wait, I think each snapshot has a small header too, at least the 0xC0DE0000 ^ external_refs magic number? If so, that would need to be adjusted too.)

Comment 39 by thakis@chromium.org, Oct 12

This is probably the header for one snapshot data

  // The data header consists of uint32_t-sized entries:
  // [0] magic number and (internal) external reference count
  // [1] number of reservation size entries
  // [2] payload length
  // ... reservations
  // ... serialized payload

For

00000060: 7603 dec0 0700 0000 e8c8 0500 9876 0080  v............v..


above, this means 0x376 external references (looks like this is always the number for all snapshot datas), 7 reservation size entries, payload length of 0x5c8e8. Looks like each reservation is a 32-bit bitfield (31 bit size, 1 bit "is last?").

The header of context 1 starts:

00086f10: 7603 dec0 3a00 0000 7894 0100 0000 0080  v...:...x.......

So 0x3a reservation sizes, and a data size of 0x19478. So this has a header size of 12 bytes + 0x3a*4 bytes = 244 bytes == 0xf4 bytes. So that needs to be subtacted form the offsets above too. So I think the first difference is at 0x88bcf - 0x86f70 - 0xf4 (?)

Comment 40 by thakis@chromium.org, Oct 12

Sorry, I typod the offset of context 1. It's at 0x88bcf - 0x86f10 - 0xf4

Comment 41 by thakis@chromium.org, Oct 12

Oh, looks like snapshot data has padding after the reservations, see POINTER_SIZE_ALIGN in https://cs.chromium.org/chromium/src/v8/src/snapshot/snapshot-common.cc?dr=CSs&g=0&l=684

Vector<const byte> SnapshotData::Payload() const {
  uint32_t reservations_size =
      GetHeaderValue(kNumReservationsOffset) * kUInt32Size;
  uint32_t padded_payload_offset =
      POINTER_SIZE_ALIGN(kHeaderSize + reservations_size);
  const byte* payload = data_ + padded_payload_offset;
  uint32_t length = GetHeaderValue(kPayloadLengthOffset);
  DCHECK_EQ(data_ + size_, payload + length);
  return Vector<const byte>(payload, length);
}

Don't believe comments, I suppose...

Comment 42 by thakis@chromium.org, Oct 12

Here's the lhs and rhs of the first diff:

00088bc0: 4d17 c000 0000 00c1 2100 009d c0f0 1dc5  M.......!.......
00088bd0: 42f6 7f00 00f6 c000 0000 0000 0000 009e  B...............


00088bc0: 4d17 c000 0000 00c1 2100 009d c0f0 1dc6  M.......!.......
00088bd0: 13f7 7f00 00f6 c000 0000 0000 0000 009e  ................


The 0x7ff712c6 / 0x7ff642c5 looks a bit like a NaN...

Comment 43 by jgruber@chromium.org, Oct 15

Wow thanks thakis. Could you perhaps set a conditional breakpoint (see #4) on the first diff and post the stack trace? That might show what we are currently serializing.

Comment 44 by thakis@chromium.org, Oct 16

I think this is the stack:
0:000> k
 # Child-SP          RetAddr           Call Site
00 000000e7`0112b620 00007ff7`6abdfa1d v8_context_snapshot_generator!v8::internal::SnapshotByteSink::Put+0x1f [C:\src\chrome\src\v8\src\snapshot\snapshot-source-sink.cc @ 20] 
01 000000e7`0112b680 00007ff7`6abdb983 v8_context_snapshot_generator!v8::internal::SnapshotByteSink::PutRaw+0x2d [C:\src\chrome\src\v8\src\snapshot\snapshot-source-sink.cc @ 64] 
02 000000e7`0112b6d0 00007ff7`6abdb5c8 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeContent+0xf3 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 715] 
03 000000e7`0112b740 00007ff7`6abd7614 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeObject+0xd8 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 669] 
04 000000e7`0112b7a0 00007ff7`6abd8b4b v8_context_snapshot_generator!v8::internal::PartialSerializer::SerializeObject+0x394 [C:\src\chrome\src\v8\src\snapshot\partial-serializer.cc @ 117] 
05 000000e7`0112b860 00007ff7`6aa2a0f5 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::VisitPointers+0x8b [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 769] 
06 000000e7`0112b8e0 00007ff7`6a9f11fb v8_context_snapshot_generator!v8::internal::BodyDescriptorApply<v8::internal::CallIterateBody,void,v8::internal::Map *,v8::internal::HeapObject *,int,v8::internal::ObjectVisitor *>+0x305 [C:\src\chrome\src\v8\src\objects-body-descriptors-inl.h @ 843] 
07 000000e7`0112b990 00007ff7`6abdb96c v8_context_snapshot_generator!v8::internal::HeapObject::IterateBody+0x1b [C:\src\chrome\src\v8\src\objects.cc @ 3719] 
08 000000e7`0112b9c0 00007ff7`6abdb5c8 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeContent+0xdc [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 713] 
09 000000e7`0112ba30 00007ff7`6abd7614 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeObject+0xd8 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 669] 
0a 000000e7`0112ba90 00007ff7`6abd8b4b v8_context_snapshot_generator!v8::internal::PartialSerializer::SerializeObject+0x394 [C:\src\chrome\src\v8\src\snapshot\partial-serializer.cc @ 117] 
0b 000000e7`0112bb50 00007ff7`6aa2a21d v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::VisitPointers+0x8b [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 769] 
0c 000000e7`0112bbd0 00007ff7`6a9f11fb v8_context_snapshot_generator!v8::internal::BodyDescriptorApply<v8::internal::CallIterateBody,void,v8::internal::Map *,v8::internal::HeapObject *,int,v8::internal::ObjectVisitor *>+0x42d [C:\src\chrome\src\v8\src\objects-body-descriptors-inl.h @ 835] 
0d 000000e7`0112bc80 00007ff7`6abdb96c v8_context_snapshot_generator!v8::internal::HeapObject::IterateBody+0x1b [C:\src\chrome\src\v8\src\objects.cc @ 3719] 
0e 000000e7`0112bcb0 00007ff7`6abdb5c8 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeContent+0xdc [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 713] 
0f 000000e7`0112bd20 00007ff7`6abd7614 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeObject+0xd8 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 669] 
10 000000e7`0112bd80 00007ff7`6abd8b4b v8_context_snapshot_generator!v8::internal::PartialSerializer::SerializeObject+0x394 [C:\src\chrome\src\v8\src\snapshot\partial-serializer.cc @ 117] 
11 000000e7`0112be40 00007ff7`6aa2a0f5 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::VisitPointers+0x8b [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 769] 
12 000000e7`0112bec0 00007ff7`6a9f11fb v8_context_snapshot_generator!v8::internal::BodyDescriptorApply<v8::internal::CallIterateBody,void,v8::internal::Map *,v8::internal::HeapObject *,int,v8::internal::ObjectVisitor *>+0x305 [C:\src\chrome\src\v8\src\objects-body-descriptors-inl.h @ 843] 
13 000000e7`0112bf70 00007ff7`6abdb96c v8_context_snapshot_generator!v8::internal::HeapObject::IterateBody+0x1b [C:\src\chrome\src\v8\src\objects.cc @ 3719] 
14 000000e7`0112bfa0 00007ff7`6abdb5c8 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeContent+0xdc [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 713] 
15 000000e7`0112c010 00007ff7`6abd7614 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeObject+0xd8 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 669] 
16 000000e7`0112c070 00007ff7`6abdb873 v8_context_snapshot_generator!v8::internal::PartialSerializer::SerializeObject+0x394 [C:\src\chrome\src\v8\src\snapshot\partial-serializer.cc @ 117] 
17 000000e7`0112c130 00007ff7`6abdb544 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializePrologue+0x243 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 404] 
18 000000e7`0112c1b0 00007ff7`6abd7614 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeObject+0x54 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 656] 
19 000000e7`0112c210 00007ff7`6abd8b4b v8_context_snapshot_generator!v8::internal::PartialSerializer::SerializeObject+0x394 [C:\src\chrome\src\v8\src\snapshot\partial-serializer.cc @ 117] 
1a 000000e7`0112c2d0 00007ff7`6aa2a0f5 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::VisitPointers+0x8b [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 769] 
1b 000000e7`0112c350 00007ff7`6a9f11fb v8_context_snapshot_generator!v8::internal::BodyDescriptorApply<v8::internal::CallIterateBody,void,v8::internal::Map *,v8::internal::HeapObject *,int,v8::internal::ObjectVisitor *>+0x305 [C:\src\chrome\src\v8\src\objects-body-descriptors-inl.h @ 843] 
1c 000000e7`0112c400 00007ff7`6abdb96c v8_context_snapshot_generator!v8::internal::HeapObject::IterateBody+0x1b [C:\src\chrome\src\v8\src\objects.cc @ 3719] 
1d 000000e7`0112c430 00007ff7`6abdb5c8 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeContent+0xdc [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 713] 
1e 000000e7`0112c4a0 00007ff7`6abd7614 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeObject+0xd8 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 669] 
1f 000000e7`0112c500 00007ff7`6abd8b4b v8_context_snapshot_generator!v8::internal::PartialSerializer::SerializeObject+0x394 [C:\src\chrome\src\v8\src\snapshot\partial-serializer.cc @ 117] 
20 000000e7`0112c5c0 00007ff7`6a9f11fb v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::VisitPointers+0x8b [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 769] 
21 000000e7`0112c640 00007ff7`6abdb96c v8_context_snapshot_generator!v8::internal::HeapObject::IterateBody+0x1b [C:\src\chrome\src\v8\src\objects.cc @ 3719] 
22 000000e7`0112c670 00007ff7`6abdb5c8 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeContent+0xdc [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 713] 
23 000000e7`0112c6e0 00007ff7`6abd7614 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeObject+0xd8 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 669] 
24 000000e7`0112c740 00007ff7`6abd8b4b v8_context_snapshot_generator!v8::internal::PartialSerializer::SerializeObject+0x394 [C:\src\chrome\src\v8\src\snapshot\partial-serializer.cc @ 117] 
25 000000e7`0112c800 00007ff7`6a9f11fb v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::VisitPointers+0x8b [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 769] 
26 000000e7`0112c880 00007ff7`6abdb96c v8_context_snapshot_generator!v8::internal::HeapObject::IterateBody+0x1b [C:\src\chrome\src\v8\src\objects.cc @ 3719] 
27 000000e7`0112c8b0 00007ff7`6abdb5c8 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeContent+0xdc [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 713] 
28 000000e7`0112c920 00007ff7`6abd7614 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeObject+0xd8 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 669] 
29 000000e7`0112c980 00007ff7`6abd8b4b v8_context_snapshot_generator!v8::internal::PartialSerializer::SerializeObject+0x394 [C:\src\chrome\src\v8\src\snapshot\partial-serializer.cc @ 117] 
2a 000000e7`0112ca40 00007ff7`6aa2a104 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::VisitPointers+0x8b [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 769] 
2b 000000e7`0112cac0 00007ff7`6a9f11fb v8_context_snapshot_generator!v8::internal::BodyDescriptorApply<v8::internal::CallIterateBody,void,v8::internal::Map *,v8::internal::HeapObject *,int,v8::internal::ObjectVisitor *>+0x314 [C:\src\chrome\src\v8\src\objects-body-descriptors-inl.h @ 843] 
2c 000000e7`0112cb70 00007ff7`6abdb96c v8_context_snapshot_generator!v8::internal::HeapObject::IterateBody+0x1b [C:\src\chrome\src\v8\src\objects.cc @ 3719] 
2d 000000e7`0112cba0 00007ff7`6abdb5c8 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeContent+0xdc [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 713] 
2e 000000e7`0112cc10 00007ff7`6abd7614 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeObject+0xd8 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 669] 
2f 000000e7`0112cc70 00007ff7`6abdb873 v8_context_snapshot_generator!v8::internal::PartialSerializer::SerializeObject+0x394 [C:\src\chrome\src\v8\src\snapshot\partial-serializer.cc @ 117] 
30 000000e7`0112cd30 00007ff7`6abdb544 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializePrologue+0x243 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 404] 
31 000000e7`0112cdb0 00007ff7`6abd7614 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeObject+0x54 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 656] 
32 000000e7`0112ce10 00007ff7`6abd8b4b v8_context_snapshot_generator!v8::internal::PartialSerializer::SerializeObject+0x394 [C:\src\chrome\src\v8\src\snapshot\partial-serializer.cc @ 117] 
33 000000e7`0112ced0 00007ff7`6aa2a0f5 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::VisitPointers+0x8b [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 769] 
34 000000e7`0112cf50 00007ff7`6a9f11fb v8_context_snapshot_generator!v8::internal::BodyDescriptorApply<v8::internal::CallIterateBody,void,v8::internal::Map *,v8::internal::HeapObject *,int,v8::internal::ObjectVisitor *>+0x305 [C:\src\chrome\src\v8\src\objects-body-descriptors-inl.h @ 843] 
35 000000e7`0112d000 00007ff7`6abdb96c v8_context_snapshot_generator!v8::internal::HeapObject::IterateBody+0x1b [C:\src\chrome\src\v8\src\objects.cc @ 3719] 
36 000000e7`0112d030 00007ff7`6abdb5c8 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeContent+0xdc [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 713] 
37 000000e7`0112d0a0 00007ff7`6abd7614 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeObject+0xd8 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 669] 
38 000000e7`0112d100 00007ff7`6abdb873 v8_context_snapshot_generator!v8::internal::PartialSerializer::SerializeObject+0x394 [C:\src\chrome\src\v8\src\snapshot\partial-serializer.cc @ 117] 
39 000000e7`0112d1c0 00007ff7`6abdb544 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializePrologue+0x243 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 404] 
3a 000000e7`0112d240 00007ff7`6abd7614 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeObject+0x54 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 656] 
3b 000000e7`0112d2a0 00007ff7`6abd8b4b v8_context_snapshot_generator!v8::internal::PartialSerializer::SerializeObject+0x394 [C:\src\chrome\src\v8\src\snapshot\partial-serializer.cc @ 117] 
3c 000000e7`0112d360 00007ff7`6a9f11fb v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::VisitPointers+0x8b [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 769] 
3d 000000e7`0112d3e0 00007ff7`6abdb96c v8_context_snapshot_generator!v8::internal::HeapObject::IterateBody+0x1b [C:\src\chrome\src\v8\src\objects.cc @ 3719] 
3e 000000e7`0112d410 00007ff7`6abdb5c8 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeContent+0xdc [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 713] 
3f 000000e7`0112d480 00007ff7`6abd7614 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::ObjectSerializer::SerializeObject+0xd8 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 669] 
40 000000e7`0112d4e0 00007ff7`6abd8a97 v8_context_snapshot_generator!v8::internal::PartialSerializer::SerializeObject+0x394 [C:\src\chrome\src\v8\src\snapshot\partial-serializer.cc @ 117] 
41 000000e7`0112d5a0 00007ff7`6a521f44 v8_context_snapshot_generator!v8::internal::Serializer<v8::internal::DefaultSerializerAllocator>::VisitRootPointers+0x57 [C:\src\chrome\src\v8\src\snapshot\serializer.cc @ 117] 
42 000000e7`0112d5f0 00007ff7`6abd6ee3 v8_context_snapshot_generator!v8::internal::RootVisitor::VisitRootPointer+0x14 [C:\src\chrome\src\v8\src\visitors.h @ 70] 
43 000000e7`0112d620 00007ff7`6a4fa89b v8_context_snapshot_generator!v8::internal::PartialSerializer::Serialize+0xf3 [C:\src\chrome\src\v8\src\snapshot\partial-serializer.cc @ 48] 
44 000000e7`0112d6a0 00007ff7`6b65897f v8_context_snapshot_generator!v8::SnapshotCreator::CreateBlob+0xb6b [C:\src\chrome\src\v8\src\api.cc @ 815] 
45 000000e7`0112f1c0 00007ff7`6b4a1abd v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x15f [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 316] 
46 000000e7`0112f380 00007ff7`6a4f10b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
47 000000e7`0112f3b0 00007ff7`6cc2ee88 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
48 000000e7`0112f740 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
49 000000e7`0112f780 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
4a 000000e7`0112f7b0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Comment 45 by thakis@chromium.org, Oct 16

(For the line numbers: I'm at v8 25a2b998e2c5b85a7456ec08)

Comment 46 by thakis@chromium.org, Oct 16

My uninformed hunch is that something in https://chromium.googlesource.com/chromium/src/+blame/master/third_party/blink/renderer/bindings/core/v8/v8_context_snapshot.cc creates a blink js object that contains a nan which happens to be created nondeterministically, and then that gets serialized out.

If that's true, the next step is to figure out which object the nan belongs to. Any good suggestions for things to try?

Comment 47 by jgruber@chromium.org, Oct 17

What you should be able to do is print the object causing the snapshot difference while debugging. For gdb we have a bunch of helper macros, see 

https://cs.chromium.org/chromium/src/v8/tools/gdbinit?l=7&rcl=deda0b4d243f3af7523c96eb8bfcff28a285be18

job <object passed to SerializeObject> would be your best bet here.

And once you've found the problematic object, you could use watchpoints on its memory location (ideally addresses will be deterministic with --predictable) to find out where it is created / altered.

Thanks for debugging, it's really a big help.

Comment 48 by thakis@chromium.org, Oct 17

Notes to self:

- had to make v8_internal_Print_Object __declspec(dllexport) so that the linker doesn't strip it

- `x v8_context_snapshot_generator!*internal_Print*` verifies the symbol is there

- i clicked stack frame 3 (which showed me that this->object_ is at 0x000006c2`aa1ed461 and ran `.call v8_context_snapshot_generator!_v8_internal_Print_Object(0x000006c2`aa1ed461)` and then stepped into that. It printed: "0x06c2aa1ed461 <JSObject>". Looks like this just prints the address, which I already knew. Trying to set a watchpoint at that address now.

Comment 49 by thakis@chromium.org, Oct 17

I set an access breakpoint, here's where the write is:

0:000> ba w 1 0x000006c2`aa1ed461
0:000> g
ModLoad: 00007ffa`0a500000 00007ffa`0a52d000   C:\Windows\System32\IMM32.DLL
*** WARNING: Unable to verify checksum for v8_context_snapshot_generator.exe
Breakpoint 0 hit
v8_context_snapshot_generator!v8::internal::EvacuateVisitorBase::RawMigrateObject+0x34f:
00007ff7`7bdfa02f 410f110c06      movups  xmmword ptr [r14+rax],xmm1 ds:000006c2`aa1ed470=00000000000000000000000000000000
0:000> k
 # Child-SP          RetAddr           Call Site
00 0000005b`3c53cc10 00007ff7`7bdf8d82 v8_context_snapshot_generator!v8::internal::EvacuateVisitorBase::RawMigrateObject<v8::internal::EvacuateVisitorBase::kFast>+0x34f [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 1174] 
01 0000005b`3c53cc80 00007ff7`7bde5058 v8_context_snapshot_generator!v8::internal::EvacuateNewSpaceVisitor::Visit+0x132 [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 1257] 
02 0000005b`3c53cd10 00007ff7`7bde4a00 v8_context_snapshot_generator!v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor,v8::internal::MajorNonAtomicMarkingState>+0x5c8 [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 2674] 
03 0000005b`3c53cea0 00007ff7`7bde45f7 v8_context_snapshot_generator!v8::internal::FullEvacuator::RawEvacuatePage+0x2b0 [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 2493] 
04 0000005b`3c53cfb0 00007ff7`7bdfaa2e v8_context_snapshot_generator!v8::internal::Evacuator::EvacuatePage+0x177 [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 2398] 
05 0000005b`3c53d0c0 00007ff7`7bdd886a v8_context_snapshot_generator!v8::internal::PageEvacuationTask::RunInParallel+0x1be [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 2517] 
06 0000005b`3c53d200 00007ff7`7bdd8e9c v8_context_snapshot_generator!v8::internal::ItemParallelJob::Task::RunInternal+0x5a [C:\src\chrome\src\v8\src\heap\item-parallel-job.cc @ 45] 
07 0000005b`3c53d240 00007ff7`7bde72dc v8_context_snapshot_generator!v8::internal::ItemParallelJob::Run+0x4ac [C:\src\chrome\src\v8\src\heap\item-parallel-job.cc @ 94] 
08 0000005b`3c53d3a0 00007ff7`7bde6ee6 v8_context_snapshot_generator!v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator,v8::internal::MarkCompactCollector>+0x3bc [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 2550] 
09 0000005b`3c53d4a0 00007ff7`7bddd2c7 v8_context_snapshot_generator!v8::internal::MarkCompactCollector::EvacuatePagesInParallel+0x2b6 [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 2611] 
0a 0000005b`3c53d570 00007ff7`7bddadd0 v8_context_snapshot_generator!v8::internal::MarkCompactCollector::Evacuate+0x3f7 [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 2731] 
0b 0000005b`3c53d700 00007ff7`7bdb4af1 v8_context_snapshot_generator!v8::internal::MarkCompactCollector::CollectGarbage+0x110 [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 471] 
0c 0000005b`3c53d750 00007ff7`7bdb28d1 v8_context_snapshot_generator!v8::internal::Heap::MarkCompact+0x101 [C:\src\chrome\src\v8\src\heap\heap.cc @ 1778] 
0d 0000005b`3c53d7d0 00007ff7`7bdb1055 v8_context_snapshot_generator!v8::internal::Heap::PerformGarbageCollection+0x611 [C:\src\chrome\src\v8\src\heap\heap.cc @ 1625] 
0e 0000005b`3c53d990 00007ff7`7bdb1556 v8_context_snapshot_generator!v8::internal::Heap::CollectGarbage+0x555 [C:\src\chrome\src\v8\src\heap\heap.cc @ 1288] 
0f 0000005b`3c53dac0 00007ff7`7b9da041 v8_context_snapshot_generator!v8::internal::Heap::CollectAllAvailableGarbage+0xc6 [C:\src\chrome\src\v8\src\heap\heap.cc @ 1128] 
10 0000005b`3c53dbe0 00007ff7`7cb389bf v8_context_snapshot_generator!v8::SnapshotCreator::CreateBlob+0x311 [C:\src\chrome\src\v8\src\api.cc @ 720] 
11 0000005b`3c53f700 00007ff7`7c981afd v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x15f [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 316] 
12 0000005b`3c53f8c0 00007ff7`7b9d10b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
13 0000005b`3c53f8f0 00007ff7`7e10eec8 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
14 0000005b`3c53fc80 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
15 0000005b`3c53fcc0 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
16 0000005b`3c53fcf0 00000000`00000000 ntdll!RtlUserThreadStart+0x21


(I also confirmed that this->object_ was indeed at that address later.)

Comment 50 by thakis@chromium.org, Oct 17

Does that stack mean that the object is moving out of the nursery into the more permanent space? If so, how do I figure out what created the object in the nursery?

Comment 51 by thakis@chromium.org, Oct 17

Cc: r...@chromium.org

src_addr in RawMigrateObject is 0x0000071f`42893030, so let's set an access breakpoint for that too:



0:000> ba w 1 0x0000071f`42893030
0:000> g
ModLoad: 00007ffa`0a500000 00007ffa`0a52d000   C:\Windows\System32\IMM32.DLL
*** WARNING: Unable to verify checksum for v8_context_snapshot_generator.exe
Breakpoint 0 hit
v8_context_snapshot_generator!v8::internal::EvacuateVisitorBase::RawMigrateObject+0x40f:
00007ff7`7bdfa0ef 410f110c06      movups  xmmword ptr [r14+rax],xmm1 ds:0000071f`42893040=00000000000000000000000000000000
0:000> k
 # Child-SP          RetAddr           Call Site
00 000000b6`fd4fce90 00007ff7`7bdf8ef3 v8_context_snapshot_generator!v8::internal::EvacuateVisitorBase::RawMigrateObject<v8::internal::EvacuateVisitorBase::kFast>+0x40f [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 1174] 
01 000000b6`fd4fcf00 00007ff7`7bde5058 v8_context_snapshot_generator!v8::internal::EvacuateNewSpaceVisitor::Visit+0x2a3 [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 1265] 
02 000000b6`fd4fcf90 00007ff7`7bde4a00 v8_context_snapshot_generator!v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor,v8::internal::MajorNonAtomicMarkingState>+0x5c8 [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 2674] 
03 000000b6`fd4fd120 00007ff7`7bde45f7 v8_context_snapshot_generator!v8::internal::FullEvacuator::RawEvacuatePage+0x2b0 [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 2493] 
04 000000b6`fd4fd230 00007ff7`7bdfaa2e v8_context_snapshot_generator!v8::internal::Evacuator::EvacuatePage+0x177 [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 2398] 
05 000000b6`fd4fd340 00007ff7`7bdd886a v8_context_snapshot_generator!v8::internal::PageEvacuationTask::RunInParallel+0x1be [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 2517] 
06 000000b6`fd4fd480 00007ff7`7bdd8e9c v8_context_snapshot_generator!v8::internal::ItemParallelJob::Task::RunInternal+0x5a [C:\src\chrome\src\v8\src\heap\item-parallel-job.cc @ 45] 
07 000000b6`fd4fd4c0 00007ff7`7bde72dc v8_context_snapshot_generator!v8::internal::ItemParallelJob::Run+0x4ac [C:\src\chrome\src\v8\src\heap\item-parallel-job.cc @ 94] 
08 000000b6`fd4fd620 00007ff7`7bde6ee6 v8_context_snapshot_generator!v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator,v8::internal::MarkCompactCollector>+0x3bc [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 2550] 
09 000000b6`fd4fd720 00007ff7`7bddd2c7 v8_context_snapshot_generator!v8::internal::MarkCompactCollector::EvacuatePagesInParallel+0x2b6 [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 2611] 
0a 000000b6`fd4fd7f0 00007ff7`7bddadd0 v8_context_snapshot_generator!v8::internal::MarkCompactCollector::Evacuate+0x3f7 [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 2731] 
0b 000000b6`fd4fd980 00007ff7`7bdb4af1 v8_context_snapshot_generator!v8::internal::MarkCompactCollector::CollectGarbage+0x110 [C:\src\chrome\src\v8\src\heap\mark-compact.cc @ 471] 
0c 000000b6`fd4fd9d0 00007ff7`7bdb28d1 v8_context_snapshot_generator!v8::internal::Heap::MarkCompact+0x101 [C:\src\chrome\src\v8\src\heap\heap.cc @ 1778] 
0d 000000b6`fd4fda50 00007ff7`7bdb1055 v8_context_snapshot_generator!v8::internal::Heap::PerformGarbageCollection+0x611 [C:\src\chrome\src\v8\src\heap\heap.cc @ 1625] 
0e 000000b6`fd4fdc10 00007ff7`7bdb1556 v8_context_snapshot_generator!v8::internal::Heap::CollectGarbage+0x555 [C:\src\chrome\src\v8\src\heap\heap.cc @ 1288] 
0f 000000b6`fd4fdd40 00007ff7`7b9da041 v8_context_snapshot_generator!v8::internal::Heap::CollectAllAvailableGarbage+0xc6 [C:\src\chrome\src\v8\src\heap\heap.cc @ 1128] 
10 000000b6`fd4fde60 00007ff7`7cb389bf v8_context_snapshot_generator!v8::SnapshotCreator::CreateBlob+0x311 [C:\src\chrome\src\v8\src\api.cc @ 720] 
11 000000b6`fd4ff980 00007ff7`7c981afd v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x15f [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 316] 
12 000000b6`fd4ffb40 00007ff7`7b9d10b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
13 000000b6`fd4ffb70 00007ff7`7e10eec8 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
14 000000b6`fd4fff00 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
15 000000b6`fd4fff40 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
16 000000b6`fd4fff70 00000000`00000000 ntdll!RtlUserThreadStart+0x21


Huh, that's the same stack :-/ So I guess it isn't nursery promotion. Anyhow, here src_address is 0x0000022e`2563f450 so let's try again there:

0:000> ba w 1 0x0000022e`2563f450
0:000> g
ModLoad: 00007ffa`0a500000 00007ffa`0a52d000   C:\Windows\System32\IMM32.DLL
*** WARNING: Unable to verify checksum for v8_context_snapshot_generator.exe
Breakpoint 0 hit
v8_context_snapshot_generator!v8::internal::Factory::AllocateRawWithAllocationSite+0x59:
00007ff7`7bd8f589 85ed            test    ebp,ebp
0:000> k
 # Child-SP          RetAddr           Call Site
00 00000052`2e32e7c0 00007ff7`7bd94ec5 v8_context_snapshot_generator!v8::internal::Factory::AllocateRawWithAllocationSite+0x59 [C:\src\chrome\src\v8\src\heap\factory.cc @ 135] 
01 00000052`2e32e810 00007ff7`7bec574c v8_context_snapshot_generator!v8::internal::Factory::NewJSObjectFromMap+0x25 [C:\src\chrome\src\v8\src\heap\factory.cc @ 2901] 
02 00000052`2e32e860 00007ff7`7b9d718e v8_context_snapshot_generator!v8::internal::JSObject::New+0x6c [C:\src\chrome\src\v8\src\objects.cc @ 1352] 
03 00000052`2e32e8d0 00007ff7`7b9d6339 v8_context_snapshot_generator!v8::internal::`anonymous namespace'::InstantiateObject+0x2be [C:\src\chrome\src\v8\src\api-natives.cc @ 405] 
04 00000052`2e32ea20 00007ff7`7b9d61ce v8_context_snapshot_generator!v8::internal::`anonymous namespace'::InstantiateFunction+0x119 [C:\src\chrome\src\v8\src\api-natives.cc @ 0] 
05 00000052`2e32eb80 00007ff7`7b9ed9cc v8_context_snapshot_generator!v8::internal::ApiNatives::InstantiateFunction+0x5e [C:\src\chrome\src\v8\src\api-natives.cc @ 554] 
06 00000052`2e32ec00 00007ff7`7cdeb4dd v8_context_snapshot_generator!v8::FunctionTemplate::GetFunction+0x11c [C:\src\chrome\src\v8\src\api.cc @ 6358] 
07 00000052`2e32ed00 00007ff7`7cb3921b v8_context_snapshot_generator!blink::V8ObjectConstructor::CreateInterfaceObject+0x5d [C:\src\chrome\src\third_party\blink\renderer\platform\bindings\v8_object_constructor.cc @ 83] 
08 00000052`2e32eea0 00007ff7`7cb391f1 v8_context_snapshot_generator!blink::`anonymous namespace'::CreateInterfaceObject+0x7b [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 39] 
09 00000052`2e32ef20 00007ff7`7cb391f1 v8_context_snapshot_generator!blink::`anonymous namespace'::CreateInterfaceObject+0x51 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 36] 
0a 00000052`2e32efa0 00007ff7`7cb38c8f v8_context_snapshot_generator!blink::`anonymous namespace'::CreateInterfaceObject+0x51 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 36] 
0b 00000052`2e32f020 00007ff7`7cb3896d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshotForWorld+0x29f [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 494] 
0c 00000052`2e32f270 00007ff7`7c981afd v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x10d [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 308] 
0d 00000052`2e32f430 00007ff7`7b9d10b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
0e 00000052`2e32f460 00007ff7`7e10eec8 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
0f 00000052`2e32f7f0 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
10 00000052`2e32f830 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
11 00000052`2e32f860 00000000`00000000 ntdll!RtlUserThreadStart+0x21


Looks like we got somewhere! I don't know v8 well enough to know where exactly though. (rnk: what's with the "api-natives.cc @ 0" at stack 4?) /me goes read api-natives.cc...

Comment 52 by r...@chromium.org, Oct 18

Line zero sounds like the new source location the optimized debug info people are using to try to smooth out line stepping. It might be applied if a call got tail merged by simplifycfg. It could also be bad inlined call frame info.

Comment 53 by jgruber@chromium.org, Oct 18

Cc: yangguo@chromium.org

The last stack trace from #51 indeed looks interesting. Seems like the nondeterministic object is created in InstantiateFunction (https://cs.chromium.org/chromium/src/v8/src/api-natives.cc?l=483&rcl=bad9daf231f81a73da1b7b5a2a481892d92cc05c). Yang, any ideas?

#48: I forgot to mention that _v8_internal_Print_Object will not be very useful in a release build without enabling

v8_enable_object_print = true

I usually add 

v8_enable_disassembler = true

for printing code objects as well.

Comment 54 by yangguo@chromium.org, Oct 18

Objects created from templates may have embedder fields. My suspicion is that we don't initialize these embedder fields. Could you test whether this patch fixes this issue?

diff --git a/src/api-natives.cc b/src/api-natives.cc
index 96abd127c0..7aeaf4b85e 100644
--- a/src/api-natives.cc
+++ b/src/api-natives.cc
@@ -407,6 +407,10 @@ MaybeHandle<JSObject> InstantiateObject(Isolate* isolate,
       JSObject::New(constructor, new_target, Handle<AllocationSite>::null()),
       JSObject);
 
+  for (int i = 0; i < info->embedder_field_count(); i++) {
+    object->SetEmbedderField(i, Smi::kZero);
+  }
+
   if (is_prototype) JSObject::OptimizeAsPrototype(object);
 
   ASSIGN_RETURN_ON_EXCEPTION(


It might generally be great if we had a good way to test determinism :/

Comment 55 by mlippautz@chromium.org, Oct 18

Cc: mlippautz@chromium.org

I was just checking embedder field initialization

NewJSObjectFromMap
-> InitializeJSObjectFromMap
-> InitializeJSObjectBody
-> JSObject::InitializeBody [1]

Here we do loop over internal aka embedder fields and we set their value to undefined, so I am unsure whether it's really embedder fields.

[1] https://cs.chromium.org/chromium/src/v8/src/objects/js-objects-inl.h?sq=package:chromium&g=0&l=405

Comment 56 by thakis@chromium.org, Oct 18

Thanks for all the ideas, I'll try them today :-)

> It might generally be great if we had a good way to test determinism :/

There's a "linux deterministic" bot on the main waterfall, so that ensures we're deterministic on Windows. I'm currently trying to get the windows build to be 100% deterministic. Once I've figured this bug here out I'll probably create a "win deterministic" main waterfall too. So there's some checking already and there will be more soon.

Comment 57 by thakis@chromium.org, Oct 19

With the patch in comment 54:

[1018/200224.369:FATAL:v8_context_snapshot.cc(344)] Check failed: field_type != InternalFieldType::kNone (  vs.  )
Backtrace:
        base::debug::StackTrace::StackTrace [0x00007FF73039B874+36]
        logging::LogMessage::~LogMessage [0x00007FF72FFF6B02+98]
        blink::V8ContextSnapshot::SerializeInternalField [0x00007FF730359C65+373]
        v8::internal::PartialSerializer::SerializeEmbedderFields [0x00007FF72F8D717D+589]
        v8::internal::PartialSerializer::Serialize [0x00007FF72F8D6F03+259]
        v8::SnapshotCreator::CreateBlob [0x00007FF72F1FA83B+2923]
        blink::V8ContextSnapshot::TakeSnapshot [0x00007FF73035965F+351]
        blink::WebV8ContextSnapshot::TakeSnapshot [0x00007FF7301A1E5D+13]
        main [0x00007FF72F1F10B1+177]
        __scrt_common_main_seh [0x00007FF73192FB08+268] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283)
        BaseThreadInitThunk [0x00007FFA0B121FE4+20]
        RtlUserThreadStart [0x00007FFA0C86CB31+33]

./v8_context_snapshot_generator failed with exit code 0x80000003

Comment 58 by thakis@chromium.org, Oct 19

Here's the object print with those gn settings:

000006C2AA1ED461: [JS_API_OBJECT_TYPE] in OldSpace
 - map: 0x06435770e051 <Map(HOLEY_ELEMENTS)> [DictionaryProperties]
 - prototype: 0x06c2aa1ed121 <Object map = 000006435770E0A1>
 - elements: 0x06e16c082cf1 <FixedArray[0]> [HOLEY_ELEMENTS]
 - embedder fields: 1
 - properties: 0x06c2aa1ed481 <NameDictionary[389]> {
   #lookupPrefix: 0x06c2aa185011 <JSFunction lookupPrefix (sfi = 000006C2AA184FD9)> (data, dict_index: 42, attrs: [WEC])
   #compareDocumentPosition: 0x06c2aa184ec1 <JSFunction compareDocumentPosition (sfi = 000006C2AA184E89)> (data, dict_index: 40, attrs: [WEC])
   #insertBefore: 0x06c2aa185209 <JSFunction insertBefore (sfi = 000006C2AA1851D1)> (data, dict_index: 45, attrs: [WEC])
   #CDATA_SECTION_NODE: 4 (data, dict_index: 5, attrs: [_E_])
   #ENTITY_REFERENCE_NODE: 5 (data, dict_index: 6, attrs: [_E_])
   #normalize: 0x06c2aa184c21 <JSFunction normalize (sfi = 000006C2AA184BE9)> (data, dict_index: 36, attrs: [WEC])
   #isConnected: 0x06c2aa184991 <AccessorPair> (accessor, dict_index: 23, attrs: [_EC])
   #DOCUMENT_TYPE_NODE: 10 (data, dict_index: 11, attrs: [_E_])
   #childNodes: 0x06c2aa1849f1 <AccessorPair> (accessor, dict_index: 27, attrs: [_EC])
   #firstChild: 0x06c2aa184a09 <AccessorPair> (accessor, dict_index: 28, attrs: [_EC])
   #ATTRIBUTE_NODE: 2 (data, dict_index: 3, attrs: [_E_])
   #isDefaultNamespace: 0x06c2aa185161 <JSFunction isDefaultNamespace (sfi = 000006C2AA185129)> (data, dict_index: 44, attrs: [WEC])
   #ELEMENT_NODE: 1 (data, dict_index: 2, attrs: [_E_])
   #lookupNamespaceURI: 0x06c2aa1850b9 <JSFunction lookupNamespaceURI (sfi = 000006C2AA185081)> (data, dict_index: 43, attrs: [WEC])
   #nodeValue: 0x06c2aa184a69 <AccessorPair> (accessor, dict_index: 32, attrs: [WEC])
   #nodeName: 0x06c2aa184961 <AccessorPair> (accessor, dict_index: 21, attrs: [_EC])
   #parentNode: 0x06c2aa1849c1 <AccessorPair> (accessor, dict_index: 25, attrs: [_EC])
   #lastChild: 0x06c2aa184a21 <AccessorPair> (accessor, dict_index: 29, attrs: [_EC])
   #DOCUMENT_POSITION_DISCONNECTED: 1 (data, dict_index: 14, attrs: [_E_])
   #ENTITY_NODE: 6 (data, dict_index: 7, attrs: [_E_])
   0x06e16c086dd9 <Symbol: Symbol.toStringTag>: 0x078b8e8a7db9 <String[4]: Node> (data, dict_index: 1, attrs: [__C])
   #nextSibling: 0x06c2aa184a51 <AccessorPair> (accessor, dict_index: 31, attrs: [_EC])
   #appendChild: 0x06c2aa1852b1 <JSFunction appendChild (sfi = 000006C2AA185279)> (data, dict_index: 46, attrs: [WEC])
   #DOCUMENT_POSITION_CONTAINS: 8 (data, dict_index: 17, attrs: [_E_])
   #baseURI: 0x06c2aa184979 <AccessorPair> (accessor, dict_index: 22, attrs: [_EC])
   #isEqualNode: 0x06c2aa184d71 <JSFunction isEqualNode (sfi = 000006C2AA184D39)> (data, dict_index: 38, attrs: [WEC])
   #hasChildNodes: 0x06c2aa184ad1 <JSFunction hasChildNodes (sfi = 000006C2AA184A99)> (data, dict_index: 34, attrs: [WEC])
   #getRootNode: 0x06c2aa184b79 <JSFunction getRootNode (sfi = 000006C2AA184B41)> (data, dict_index: 35, attrs: [WEC])
   #DOCUMENT_POSITION_IMPLEMENTATION_SPECIFIC: 32 (data, dict_index: 19, attrs: [_E_])
   #removeChild: 0x06c2aa185401 <JSFunction removeChild (sfi = 000006C2AA1853C9)> (data, dict_index: 48, attrs: [WEC])
   #DOCUMENT_NODE: 9 (data, dict_index: 10, attrs: [_E_])
   #DOCUMENT_POSITION_CONTAINED_BY: 16 (data, dict_index: 18, attrs: [_E_])
   #PROCESSING_INSTRUCTION_NODE: 7 (data, dict_index: 8, attrs: [_E_])
   #COMMENT_NODE: 8 (data, dict_index: 9, attrs: [_E_])
   #isSameNode: 0x06c2aa184e19 <JSFunction isSameNode (sfi = 000006C2AA184DE1)> (data, dict_index: 39, attrs: [WEC])
   #NOTATION_NODE: 12 (data, dict_index: 13, attrs: [_E_])
   #DOCUMENT_POSITION_PRECEDING: 2 (data, dict_index: 15, attrs: [_E_])
   #parentElement: 0x06c2aa1849d9 <AccessorPair> (accessor, dict_index: 26, attrs: [_EC])
   #contains: 0x06c2aa184f69 <JSFunction contains (sfi = 000006C2AA184F31)> (data, dict_index: 41, attrs: [WEC])
   #replaceChild: 0x06c2aa185359 <JSFunction replaceChild (sfi = 000006C2AA185321)> (data, dict_index: 47, attrs: [WEC])
   #nodeType: 0x06c2aa184949 <AccessorPair> (accessor, dict_index: 20, attrs: [_EC])
   #DOCUMENT_POSITION_FOLLOWING: 4 (data, dict_index: 16, attrs: [_E_])
   #ownerDocument: 0x06c2aa1849a9 <AccessorPair> (accessor, dict_index: 24, attrs: [_EC])
   #TEXT_NODE: 3 (data, dict_index: 4, attrs: [_E_])
   #cloneNode: 0x06c2aa184cc9 <JSFunction cloneNode (sfi = 000006C2AA184C91)> (data, dict_index: 37, attrs: [WEC])
   #constructor: 0x06c2aa1854a9 <JSFunction Node (sfi = 000006C2AA185471)> (data, dict_index: 49, attrs: [W_C])
   #previousSibling: 0x06c2aa184a39 <AccessorPair> (accessor, dict_index: 30, attrs: [_EC])
   #DOCUMENT_FRAGMENT_NODE: 11 (data, dict_index: 12, attrs: [_E_])
   #textContent: 0x06c2aa184a81 <AccessorPair> (accessor, dict_index: 33, attrs: [WEC])
 }
 - embedder fields = {
    00007FF7E630C4F0
 }

Comment 59 by jgruber@chromium.org, Oct 19

#58: So are embedder fields the source of nondeterminism? That would already help narrow things down a lot.

Comment 60 by thakis@chromium.org, Oct 19

How do I check? The suggestion in comment 54 didn't work, see 57.

Comment 61 by jgruber@chromium.org, Oct 19

Printing the object also prints the contents of embedder fields, see the bottom of #58. I was wondering if that matches the relevant byte pattern written into the snapshot.

Comment 62 by thakis@chromium.org, Oct 19

Sorry for the silly question.

Yes, it seems to match! The embedder field is 00007FF7E630C4F0, and 0x88bcc onward in that dump reads c0f0 c430 e6f7 7f00 00f6, which is that number in little endian (and one additional byte on each end).

Comment 63 by jgruber@chromium.org, Oct 19

Right, the last step is finding out where this embedder field is set. Per #55 it looks like V8 initializes it correctly, my theory is that the nondeterministic value comes from blink (not an expert on embedder fields though).

Comment 64 by yangguo@chromium.org, Oct 19

Maybe set a break point to v8::internal::JSObject::SetEmbedderField?

Comment 65 by thakis@chromium.org, Oct 19

blink has fairly few callers to Set.*InternalField: https://cs.chromium.org/search/?q=set%5Ba-zA-Z%5D*internalfield+file:blink+-file:bindings/tests+-file:win-debug+-file:android-debug+-file:chromeos-debug&sq=package:chromium&type=cs

Comment 66 by jgruber@chromium.org, Oct 19

We know the problematic object don't we? Would a watchpoint on the embedder field help?

Comment 67 by thakis@chromium.org, Oct 19

SetEmbedderField gets inlined and I can't break on it. I set a breakpoint on Set*InternalField (`bm v8_context_snapshot_generator!*Set*InternalField*`); here are all calls to it where SetInternalFieldCount() was called with n=1:


0:000> k
 # Child-SP          RetAddr           Call Site
00 0000006b`b28ff208 00007ff7`1fe31f16 v8_context_snapshot_generator!v8::ObjectTemplate::SetInternalFieldCount [C:\src\chrome\src\v8\src\api.cc @ 1968] 
01 0000006b`b28ff210 00007ff7`200a0a11 v8_context_snapshot_generator!blink::V8DOMConfiguration::InitializeDOMInterfaceTemplate+0xe6 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 816] 
02 0000006b`b28ff290 00007ff7`200a7a4a v8_context_snapshot_generator!blink::V8Window::installV8WindowTemplate+0x71 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_window.cc @ 12123] 
03 0000006b`b28ff430 00007ff7`1fe3207c v8_context_snapshot_generator!blink::V8WindowPartial::installV8WindowTemplate+0x2a [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\modules\v8\v8_window_partial.cc @ 3040] 
04 0000006b`b28ff4d0 00007ff7`2008a1b0 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
05 0000006b`b28ff560 00007ff7`1fdfdf25 v8_context_snapshot_generator!blink::V8Window::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_window.cc @ 12887] 
06 0000006b`b28ff5a0 00007ff7`1fdfddad v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshotForWorld+0xf5 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 471] 
07 0000006b`b28ff7f0 00007ff7`1fc46f5d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x10d [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 308] 
08 0000006b`b28ff9b0 00007ff7`1ec510b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
09 0000006b`b28ff9e0 00007ff7`213d4348 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
0a 0000006b`b28ffd70 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
0b 0000006b`b28ffdb0 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
0c 0000006b`b28ffde0 00000000`00000000 ntdll!RtlUserThreadStart+0x21




0:000> k
 # Child-SP          RetAddr           Call Site
00 0000006b`b28ff058 00007ff7`1fe31f16 v8_context_snapshot_generator!v8::ObjectTemplate::SetInternalFieldCount [C:\src\chrome\src\v8\src\api.cc @ 1968] 
01 0000006b`b28ff060 00007ff7`2004e227 v8_context_snapshot_generator!blink::V8DOMConfiguration::InitializeDOMInterfaceTemplate+0xe6 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 816] 
02 0000006b`b28ff0e0 00007ff7`1fe3207c v8_context_snapshot_generator!blink::installV8NodeTemplate+0x57 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_node.cc @ 999] 
03 0000006b`b28ff190 00007ff7`2004a900 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
04 0000006b`b28ff220 00007ff7`1ffbd418 v8_context_snapshot_generator!blink::V8Node::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_node.cc @ 1100] 
05 0000006b`b28ff260 00007ff7`1ffc0d84 v8_context_snapshot_generator!blink::V8Document::installV8DocumentTemplate+0x38 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_document.cc @ 7562] 
06 0000006b`b28ff300 00007ff7`1fe3207c v8_context_snapshot_generator!blink::V8DocumentPartial::installV8DocumentTemplate+0x24 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\modules\v8\v8_document_partial.cc @ 103] 
07 0000006b`b28ff360 00007ff7`1ffa6900 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
08 0000006b`b28ff3f0 00007ff7`1ffe3ca8 v8_context_snapshot_generator!blink::V8Document::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_document.cc @ 7940] 
09 0000006b`b28ff430 00007ff7`1fe3207c v8_context_snapshot_generator!blink::installV8HTMLDocumentTemplate+0x38 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_html_document.cc @ 125] 
0a 0000006b`b28ff4d0 00007ff7`1ffe3920 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
0b 0000006b`b28ff560 00007ff7`1fdfdf25 v8_context_snapshot_generator!blink::V8HTMLDocument::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_html_document.cc @ 182] 
0c 0000006b`b28ff5a0 00007ff7`1fdfddad v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshotForWorld+0xf5 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 471] 
0d 0000006b`b28ff7f0 00007ff7`1fc46f5d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x10d [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 308] 
0e 0000006b`b28ff9b0 00007ff7`1ec510b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
0f 0000006b`b28ff9e0 00007ff7`213d4348 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
10 0000006b`b28ffd70 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
11 0000006b`b28ffdb0 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
12 0000006b`b28ffde0 00000000`00000000 ntdll!RtlUserThreadStart+0x21



0:000> k
 # Child-SP          RetAddr           Call Site
00 0000006b`b28ff1d8 00007ff7`1fe31f16 v8_context_snapshot_generator!v8::ObjectTemplate::SetInternalFieldCount [C:\src\chrome\src\v8\src\api.cc @ 1968] 
01 0000006b`b28ff1e0 00007ff7`1ffbd435 v8_context_snapshot_generator!blink::V8DOMConfiguration::InitializeDOMInterfaceTemplate+0xe6 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 816] 
02 0000006b`b28ff260 00007ff7`1ffc0d84 v8_context_snapshot_generator!blink::V8Document::installV8DocumentTemplate+0x55 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_document.cc @ 7563] 
03 0000006b`b28ff300 00007ff7`1fe3207c v8_context_snapshot_generator!blink::V8DocumentPartial::installV8DocumentTemplate+0x24 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\modules\v8\v8_document_partial.cc @ 103] 
04 0000006b`b28ff360 00007ff7`1ffa6900 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
05 0000006b`b28ff3f0 00007ff7`1ffe3ca8 v8_context_snapshot_generator!blink::V8Document::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_document.cc @ 7940] 
06 0000006b`b28ff430 00007ff7`1fe3207c v8_context_snapshot_generator!blink::installV8HTMLDocumentTemplate+0x38 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_html_document.cc @ 125] 
07 0000006b`b28ff4d0 00007ff7`1ffe3920 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
08 0000006b`b28ff560 00007ff7`1fdfdf25 v8_context_snapshot_generator!blink::V8HTMLDocument::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_html_document.cc @ 182] 
09 0000006b`b28ff5a0 00007ff7`1fdfddad v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshotForWorld+0xf5 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 471] 
0a 0000006b`b28ff7f0 00007ff7`1fc46f5d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x10d [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 308] 
0b 0000006b`b28ff9b0 00007ff7`1ec510b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
0c 0000006b`b28ff9e0 00007ff7`213d4348 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
0d 0000006b`b28ffd70 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
0e 0000006b`b28ffdb0 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
0f 0000006b`b28ffde0 00000000`00000000 ntdll!RtlUserThreadStart+0x21



0:000> k
 # Child-SP          RetAddr           Call Site
00 0000006b`b28ff3a8 00007ff7`1fe31f16 v8_context_snapshot_generator!v8::ObjectTemplate::SetInternalFieldCount [C:\src\chrome\src\v8\src\api.cc @ 1968] 
01 0000006b`b28ff3b0 00007ff7`1ffe3cc5 v8_context_snapshot_generator!blink::V8DOMConfiguration::InitializeDOMInterfaceTemplate+0xe6 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 816] 
02 0000006b`b28ff430 00007ff7`1fe3207c v8_context_snapshot_generator!blink::installV8HTMLDocumentTemplate+0x55 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_html_document.cc @ 125] 
03 0000006b`b28ff4d0 00007ff7`1ffe3920 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
04 0000006b`b28ff560 00007ff7`1fdfdf25 v8_context_snapshot_generator!blink::V8HTMLDocument::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_html_document.cc @ 182] 
05 0000006b`b28ff5a0 00007ff7`1fdfddad v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshotForWorld+0xf5 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 471] 
06 0000006b`b28ff7f0 00007ff7`1fc46f5d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x10d [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 308] 
07 0000006b`b28ff9b0 00007ff7`1ec510b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
08 0000006b`b28ff9e0 00007ff7`213d4348 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
09 0000006b`b28ffd70 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
0a 0000006b`b28ffdb0 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
0b 0000006b`b28ffde0 00000000`00000000 ntdll!RtlUserThreadStart+0x21



0:000> k
 # Child-SP          RetAddr           Call Site
00 0000006b`b28ff208 00007ff7`1fe31f16 v8_context_snapshot_generator!v8::ObjectTemplate::SetInternalFieldCount [C:\src\chrome\src\v8\src\api.cc @ 1968] 
01 0000006b`b28ff210 00007ff7`200a0a11 v8_context_snapshot_generator!blink::V8DOMConfiguration::InitializeDOMInterfaceTemplate+0xe6 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 816] 
02 0000006b`b28ff290 00007ff7`200a7a4a v8_context_snapshot_generator!blink::V8Window::installV8WindowTemplate+0x71 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_window.cc @ 12123] 
03 0000006b`b28ff430 00007ff7`1fe3207c v8_context_snapshot_generator!blink::V8WindowPartial::installV8WindowTemplate+0x2a [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\modules\v8\v8_window_partial.cc @ 3040] 
04 0000006b`b28ff4d0 00007ff7`2008a1b0 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
05 0000006b`b28ff560 00007ff7`1fdfdf25 v8_context_snapshot_generator!blink::V8Window::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_window.cc @ 12887] 
06 0000006b`b28ff5a0 00007ff7`1fdfddd0 v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshotForWorld+0xf5 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 471] 
07 0000006b`b28ff7f0 00007ff7`1fc46f5d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x130 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 308] 
08 0000006b`b28ff9b0 00007ff7`1ec510b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
09 0000006b`b28ff9e0 00007ff7`213d4348 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
0a 0000006b`b28ffd70 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
0b 0000006b`b28ffdb0 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
0c 0000006b`b28ffde0 00000000`00000000 ntdll!RtlUserThreadStart+0x21


0:000> k
 # Child-SP          RetAddr           Call Site
00 0000006b`b28ff058 00007ff7`1fe31f16 v8_context_snapshot_generator!v8::ObjectTemplate::SetInternalFieldCount [C:\src\chrome\src\v8\src\api.cc @ 1968] 
01 0000006b`b28ff060 00007ff7`2004e227 v8_context_snapshot_generator!blink::V8DOMConfiguration::InitializeDOMInterfaceTemplate+0xe6 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 816] 
02 0000006b`b28ff0e0 00007ff7`1fe3207c v8_context_snapshot_generator!blink::installV8NodeTemplate+0x57 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_node.cc @ 999] 
03 0000006b`b28ff190 00007ff7`2004a900 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
04 0000006b`b28ff220 00007ff7`1ffbd418 v8_context_snapshot_generator!blink::V8Node::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_node.cc @ 1100] 
05 0000006b`b28ff260 00007ff7`1ffc0d84 v8_context_snapshot_generator!blink::V8Document::installV8DocumentTemplate+0x38 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_document.cc @ 7562] 
06 0000006b`b28ff300 00007ff7`1fe3207c v8_context_snapshot_generator!blink::V8DocumentPartial::installV8DocumentTemplate+0x24 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\modules\v8\v8_document_partial.cc @ 103] 
07 0000006b`b28ff360 00007ff7`1ffa6900 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
08 0000006b`b28ff3f0 00007ff7`1ffe3ca8 v8_context_snapshot_generator!blink::V8Document::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_document.cc @ 7940] 
09 0000006b`b28ff430 00007ff7`1fe3207c v8_context_snapshot_generator!blink::installV8HTMLDocumentTemplate+0x38 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_html_document.cc @ 125] 
0a 0000006b`b28ff4d0 00007ff7`1ffe3920 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
0b 0000006b`b28ff560 00007ff7`1fdfdf25 v8_context_snapshot_generator!blink::V8HTMLDocument::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_html_document.cc @ 182] 
0c 0000006b`b28ff5a0 00007ff7`1fdfddd0 v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshotForWorld+0xf5 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 471] 
0d 0000006b`b28ff7f0 00007ff7`1fc46f5d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x130 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 308] 
0e 0000006b`b28ff9b0 00007ff7`1ec510b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
0f 0000006b`b28ff9e0 00007ff7`213d4348 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
10 0000006b`b28ffd70 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
11 0000006b`b28ffdb0 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
12 0000006b`b28ffde0 00000000`00000000 ntdll!RtlUserThreadStart+0x21



0:000> k
 # Child-SP          RetAddr           Call Site
00 0000006b`b28ff1d8 00007ff7`1fe31f16 v8_context_snapshot_generator!v8::ObjectTemplate::SetInternalFieldCount [C:\src\chrome\src\v8\src\api.cc @ 1968] 
01 0000006b`b28ff1e0 00007ff7`1ffbd435 v8_context_snapshot_generator!blink::V8DOMConfiguration::InitializeDOMInterfaceTemplate+0xe6 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 816] 
02 0000006b`b28ff260 00007ff7`1ffc0d84 v8_context_snapshot_generator!blink::V8Document::installV8DocumentTemplate+0x55 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_document.cc @ 7563] 
03 0000006b`b28ff300 00007ff7`1fe3207c v8_context_snapshot_generator!blink::V8DocumentPartial::installV8DocumentTemplate+0x24 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\modules\v8\v8_document_partial.cc @ 103] 
04 0000006b`b28ff360 00007ff7`1ffa6900 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
05 0000006b`b28ff3f0 00007ff7`1ffe3ca8 v8_context_snapshot_generator!blink::V8Document::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_document.cc @ 7940] 
06 0000006b`b28ff430 00007ff7`1fe3207c v8_context_snapshot_generator!blink::installV8HTMLDocumentTemplate+0x38 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_html_document.cc @ 125] 
07 0000006b`b28ff4d0 00007ff7`1ffe3920 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
08 0000006b`b28ff560 00007ff7`1fdfdf25 v8_context_snapshot_generator!blink::V8HTMLDocument::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_html_document.cc @ 182] 
09 0000006b`b28ff5a0 00007ff7`1fdfddd0 v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshotForWorld+0xf5 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 471] 
0a 0000006b`b28ff7f0 00007ff7`1fc46f5d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x130 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 308] 
0b 0000006b`b28ff9b0 00007ff7`1ec510b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
0c 0000006b`b28ff9e0 00007ff7`213d4348 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
0d 0000006b`b28ffd70 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
0e 0000006b`b28ffdb0 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
0f 0000006b`b28ffde0 00000000`00000000 ntdll!RtlUserThreadStart+0x21




0:000> k
 # Child-SP          RetAddr           Call Site
00 0000006b`b28ff3a8 00007ff7`1fe31f16 v8_context_snapshot_generator!v8::ObjectTemplate::SetInternalFieldCount [C:\src\chrome\src\v8\src\api.cc @ 1968] 
01 0000006b`b28ff3b0 00007ff7`1ffe3cc5 v8_context_snapshot_generator!blink::V8DOMConfiguration::InitializeDOMInterfaceTemplate+0xe6 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 816] 
02 0000006b`b28ff430 00007ff7`1fe3207c v8_context_snapshot_generator!blink::installV8HTMLDocumentTemplate+0x55 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_html_document.cc @ 125] 
03 0000006b`b28ff4d0 00007ff7`1ffe3920 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
04 0000006b`b28ff560 00007ff7`1fdfdf25 v8_context_snapshot_generator!blink::V8HTMLDocument::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_html_document.cc @ 182] 
05 0000006b`b28ff5a0 00007ff7`1fdfddd0 v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshotForWorld+0xf5 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 471] 
06 0000006b`b28ff7f0 00007ff7`1fc46f5d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x130 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 308] 
07 0000006b`b28ff9b0 00007ff7`1ec510b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
08 0000006b`b28ff9e0 00007ff7`213d4348 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
09 0000006b`b28ffd70 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
0a 0000006b`b28ffdb0 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
0b 0000006b`b28ffde0 00000000`00000000 ntdll!RtlUserThreadStart+0x21



0:000> k
 # Child-SP          RetAddr           Call Site
00 0000006b`b28ff3a8 00007ff7`1fe31f16 v8_context_snapshot_generator!v8::ObjectTemplate::SetInternalFieldCount [C:\src\chrome\src\v8\src\api.cc @ 1968] 
01 0000006b`b28ff3b0 00007ff7`1ffe3cc5 v8_context_snapshot_generator!blink::V8DOMConfiguration::InitializeDOMInterfaceTemplate+0xe6 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 816] 
02 0000006b`b28ff430 00007ff7`1fe3207c v8_context_snapshot_generator!blink::installV8HTMLDocumentTemplate+0x55 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_html_document.cc @ 125] 
03 0000006b`b28ff4d0 00007ff7`1ffe3920 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
04 0000006b`b28ff560 00007ff7`1fdfdf25 v8_context_snapshot_generator!blink::V8HTMLDocument::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_html_document.cc @ 182] 
05 0000006b`b28ff5a0 00007ff7`1fdfddd0 v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshotForWorld+0xf5 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 471] 
06 0000006b`b28ff7f0 00007ff7`1fc46f5d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x130 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 308] 
07 0000006b`b28ff9b0 00007ff7`1ec510b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
08 0000006b`b28ff9e0 00007ff7`213d4348 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
09 0000006b`b28ffd70 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
0a 0000006b`b28ffdb0 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
0b 0000006b`b28ffde0 00000000`00000000 ntdll!RtlUserThreadStart+0x21



That doesn't look very useful. Next, `bm v8_context_snapshot_generator!*NewExternal`. That triggers only twice:

0:000> k
 # Child-SP          RetAddr           Call Site
00 00000034`5d34ee18 00007ff7`1ec72318 v8_context_snapshot_generator!v8::internal::Factory::NewExternal [C:\src\chrome\src\v8\src\heap\factory.cc @ 2553] 
01 00000034`5d34ee20 00007ff7`200a0b16 v8_context_snapshot_generator!v8::External::New+0x78 [C:\src\chrome\src\v8\src\api.cc @ 6423] 
02 00000034`5d34eeb0 00007ff7`200a7a4a v8_context_snapshot_generator!blink::V8Window::installV8WindowTemplate+0x176 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_window.cc @ 12152] 
03 00000034`5d34f050 00007ff7`1fe3207c v8_context_snapshot_generator!blink::V8WindowPartial::installV8WindowTemplate+0x2a [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\modules\v8\v8_window_partial.cc @ 3040] 
04 00000034`5d34f0f0 00007ff7`2008a1b0 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
05 00000034`5d34f180 00007ff7`1fdfdf25 v8_context_snapshot_generator!blink::V8Window::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_window.cc @ 12887] 
06 00000034`5d34f1c0 00007ff7`1fdfddad v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshotForWorld+0xf5 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 471] 
07 00000034`5d34f410 00007ff7`1fc46f5d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x10d [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 308] 
08 00000034`5d34f5d0 00007ff7`1ec510b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
09 00000034`5d34f600 00007ff7`213d4348 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
0a 00000034`5d34f990 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
0b 00000034`5d34f9d0 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
0c 00000034`5d34fa00 00000000`00000000 ntdll!RtlUserThreadStart+0x21


00 00000034`5d34ee18 00007ff7`1ec72318 v8_context_snapshot_generator!v8::internal::Factory::NewExternal [C:\src\chrome\src\v8\src\heap\factory.cc @ 2553] 
01 00000034`5d34ee20 00007ff7`200a0b16 v8_context_snapshot_generator!v8::External::New+0x78 [C:\src\chrome\src\v8\src\api.cc @ 6423] 
02 00000034`5d34eeb0 00007ff7`200a7a4a v8_context_snapshot_generator!blink::V8Window::installV8WindowTemplate+0x176 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_window.cc @ 12152] 
03 00000034`5d34f050 00007ff7`1fe3207c v8_context_snapshot_generator!blink::V8WindowPartial::installV8WindowTemplate+0x2a [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\modules\v8\v8_window_partial.cc @ 3040] 
04 00000034`5d34f0f0 00007ff7`2008a1b0 v8_context_snapshot_generator!blink::V8DOMConfiguration::DomClassTemplate+0x8c [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_dom_configuration.cc @ 833] 
05 00000034`5d34f180 00007ff7`1fdfdf25 v8_context_snapshot_generator!blink::V8Window::domTemplate+0x20 [C:\src\chrome\src\out\gnrelsym\gen\third_party\blink\renderer\bindings\core\v8\v8_window.cc @ 12887] 
06 00000034`5d34f1c0 00007ff7`1fdfddd0 v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshotForWorld+0xf5 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 471] 
07 00000034`5d34f410 00007ff7`1fc46f5d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x130 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 308] 
08 00000034`5d34f5d0 00007ff7`1ec510b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
09 00000034`5d34f600 00007ff7`213d4348 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
0a 00000034`5d34f990 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
0b 00000034`5d34f9d0 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
0c 00000034`5d34fa00 00000000`00000000 ntdll!RtlUserThreadStart+0x21


(Yes, that's the same stack twice. windbg did say 

0:000> g
Breakpoint 1 hit
v8_context_snapshot_generator!v8::internal::Factory::NewExternal:
00007ff7`1f023550 4157            push    r15

twice though.)


This looks maybe more promising.

Comment 68 by thakis@chromium.org, Oct 19

The value passed to NewExternal() is 0x00007ff7`21bac5b0, which is different from the embedder field 00007FF72160C4F0, so this is probably not it.

Comment 69 by thakis@google.com, Oct 19

re 66: good suggestion, let me try this. To find the offset where to set my watchpoint, I added the offset output like

  if (obj->GetEmbedderFieldCount() > 0) {
    os << "\n - embedder fields: " << obj->GetEmbedderFieldCount() << " " << obj->GetEmbedderFieldOffset(0);
  }


to JSObjectPrintHeader. With that, the .call from 48 prints:

 - embedder fields: 1 24

The object moves around quite a bit through gcs (and I don't want to tweak gc so that the snapshot output doesn't change much), so I'll try watchpoints at this offset relative to a bunch of addresses.

Comment 70 by thakis@google.com, Oct 19

(re 69: since GetEmbedderFieldOffset() is inlined everywhere, I couldn't just call it from the debugger)

Comment 71 by thakis@google.com, Oct 19

(re 69: since GetEmbedderFieldOffset() is inlined everywhere, I couldn't just call it from the debugger)

Comment 72 by thakis@google.com, Oct 20

The dance from comment 51 with an offset of 24 / 0x18 leads to:

0:000> ba w 1 0x0000022e`2563f468
0:000> g
ModLoad: 00007ffa`0a500000 00007ffa`0a52d000   C:\Windows\System32\IMM32.DLL
*** WARNING: Unable to verify checksum for v8_context_snapshot_generator.exe
Breakpoint 0 hit
v8_context_snapshot_generator!v8::internal::Factory::InitializeJSObjectBody+0x163:
00007ff6`798a1ed3 4883c6f8        add     rsi,0FFFFFFFFFFFFFFF8h
0:000> k
 # Child-SP          RetAddr           Call Site
00 00000013`a337e940 00007ff6`7989a81f v8_context_snapshot_generator!v8::internal::Factory::InitializeJSObjectBody+0x163 [C:\src\chrome\src\v8\src\heap\factory.cc @ 2885] 
01 00000013`a337e9b0 00007ff6`799d9a7c v8_context_snapshot_generator!v8::internal::Factory::NewJSObjectFromMap+0x7f [C:\src\chrome\src\v8\src\heap\factory.cc @ 2912] 
02 00000013`a337ea00 00007ff6`794d718e v8_context_snapshot_generator!v8::internal::JSObject::New+0x6c [C:\src\chrome\src\v8\src\objects.cc @ 1352] 
03 00000013`a337ea70 00007ff6`794d6339 v8_context_snapshot_generator!v8::internal::`anonymous namespace'::InstantiateObject+0x2be [C:\src\chrome\src\v8\src\api-natives.cc @ 405] 
04 00000013`a337ebc0 00007ff6`794d61ce v8_context_snapshot_generator!v8::internal::`anonymous namespace'::InstantiateFunction+0x119 [C:\src\chrome\src\v8\src\api-natives.cc @ 0] 
05 00000013`a337ed20 00007ff6`794ed9cc v8_context_snapshot_generator!v8::internal::ApiNatives::InstantiateFunction+0x5e [C:\src\chrome\src\v8\src\api-natives.cc @ 556] 
06 00000013`a337eda0 00007ff6`7a9309bd v8_context_snapshot_generator!v8::FunctionTemplate::GetFunction+0x11c [C:\src\chrome\src\v8\src\api.cc @ 6358] 
07 00000013`a337eea0 00007ff6`7a67e6fb v8_context_snapshot_generator!blink::V8ObjectConstructor::CreateInterfaceObject+0x5d [C:\src\chrome\src\third_party\blink\renderer\platform\bindings\v8_object_constructor.cc @ 83] 
08 00000013`a337f040 00007ff6`7a67e6d1 v8_context_snapshot_generator!blink::`anonymous namespace'::CreateInterfaceObject+0x7b [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 39] 
09 00000013`a337f0c0 00007ff6`7a67e6d1 v8_context_snapshot_generator!blink::`anonymous namespace'::CreateInterfaceObject+0x51 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 36] 
0a 00000013`a337f140 00007ff6`7a67e14f v8_context_snapshot_generator!blink::`anonymous namespace'::CreateInterfaceObject+0x51 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 36] 
0b 00000013`a337f1c0 00007ff6`7a67de2d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshotForWorld+0x29f [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 495] 
0c 00000013`a337f410 00007ff6`7a4c6fbd v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x10d [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 308] 
0d 00000013`a337f5d0 00007ff6`794d10b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
0e 00000013`a337f600 00007ff6`7bc543c8 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
0f 00000013`a337f990 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
10 00000013`a337f9d0 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
11 00000013`a337fa00 00000000`00000000 ntdll!RtlUserThreadStart+0x21


Breaks on the last line here:

  if (in_progress) {
    filler = *one_pointer_filler_map();
  } else {
    filler = *undefined_value();
  }
  obj->InitializeBody(*map, start_offset, *undefined_value(), filler);


0:000> g
Breakpoint 0 hit
v8_context_snapshot_generator!v8::Object::SetAlignedPointerInInternalField+0x8a:
00007ff7`5442014a 4883c428        add     rsp,28h
0:000> k
 # Child-SP          RetAddr           Call Site
00 00000002`daeff180 00007ff7`55860b64 v8_context_snapshot_generator!v8::Object::SetAlignedPointerInInternalField+0x8a [C:\src\chrome\src\v8\src\api.cc @ 5854] 
01 00000002`daeff1d0 00007ff7`555ae73b v8_context_snapshot_generator!blink::V8ObjectConstructor::CreateInterfaceObject+0x1c4 [C:\src\chrome\src\third_party\blink\renderer\platform\bindings\v8_object_constructor.cc @ 112] 
02 00000002`daeff370 00007ff7`555ae711 v8_context_snapshot_generator!blink::`anonymous namespace'::CreateInterfaceObject+0x7b [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 39] 
03 00000002`daeff3f0 00007ff7`555ae711 v8_context_snapshot_generator!blink::`anonymous namespace'::CreateInterfaceObject+0x51 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 36] 
04 00000002`daeff470 00007ff7`555ae18f v8_context_snapshot_generator!blink::`anonymous namespace'::CreateInterfaceObject+0x51 [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 36] 
05 00000002`daeff4f0 00007ff7`555ade6d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshotForWorld+0x29f [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 495] 
06 00000002`daeff740 00007ff7`553f701d v8_context_snapshot_generator!blink::V8ContextSnapshot::TakeSnapshot+0x10d [C:\src\chrome\src\third_party\blink\renderer\bindings\core\v8\v8_context_snapshot.cc @ 308] 
07 00000002`daeff900 00007ff7`544010b1 v8_context_snapshot_generator!blink::WebV8ContextSnapshot::TakeSnapshot+0xd [C:\src\chrome\src\third_party\blink\renderer\core\exported\web_v8_context_snapshot.cc @ 13] 
08 00000002`daeff930 00007ff7`56b84408 v8_context_snapshot_generator!main+0xb1 [C:\src\chrome\src\tools\v8_context_snapshot\v8_context_snapshot_generator.cc @ 59] 
09 00000002`daeffcc0 00007ffa`0b121fe4 v8_context_snapshot_generator!__scrt_common_main_seh+0x10c [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 283] 
0a 00000002`daeffd00 00007ffa`0c86cb31 KERNEL32!BaseThreadInitThunk+0x14
0b 00000002`daeffd30 00000000`00000000 ntdll!RtlUserThreadStart+0x21


If I add this printf:


    if (prototype_object->InternalFieldCount() ==
        kV8PrototypeInternalFieldcount) {
fprintf(stderr, "setting value %p\n", type);  // XXX
      prototype_object->SetAlignedPointerInInternalField(
          kV8PrototypeTypeIndex, const_cast<WrapperTypeInfo*>(type));
    }

then that prints 3 of the 4 values which end up being different. So hooray, we
found where most of the differing data comes from. Namely, it's 

&V8HTMLDocument::wrapperTypeInfo->parent_type->parent_type (for the first
of the 3 values; the other two ones are probably the direct parent and
V8HTMLDocument::wrapperTypeInfo itself? ...yes, this printf prints the same
values:

fprintf(stderr, "direct: %p %p %p\n", &V8HTMLDocument::wrapperTypeInfo, &V8Document::wrapperTypeInfo, &V8Node::wrapperTypeInfo);

), which is
V8Node::wrapperTypeInfo. This is a global placed by the linker, so I
suppose I need to investigate why the linker decided to put that at
a different address in different builds.

Hm, the two generated v8_context_snapshot_generator.exes are identical
though (if synced up enough to have my /pdbaltpath: and /pdbsourcepath: changes)? I'll poke some more.

Comment 73 by thakis@google.com, Oct 20

The two binaries are as I said identical (as are snapshot_blob.bin and natives_blob.bin; I copied over an arbitrary one and swapping them out doesn't make a difference). And yet:

C:\src\chrome\src\out>v8_context_snapshot_generator.exe --output_file=v8_context_snapshot.bin
external 0000022E25610D29
direct: 00007FF7A29B9720 00007FF7A2F60128 00007FF7A29C1E00
setting value 00007FF7A29C1E00
setting value 00007FF7A2F60128
setting value 00007FF7A29B9720
external 0000022E25657329

C:\src\chrome\src\out>v8_context_snapshot_generator.exe --output_file=v8_context_snapshot.bin
external 0000022E25610D29
direct: 00007FF7A29B9720 00007FF7A2F60128 00007FF7A29C1E00
setting value 00007FF7A29C1E00
setting value 00007FF7A2F60128
setting value 00007FF7A29B9720
external 0000022E25657329

C:\src\chrome\src\out>copy gnrel2\v8_context_snapshot_generator.exe .
Overwrite .\v8_context_snapshot_generator.exe? (Yes/No/All): yes
        1 file(s) copied.

C:\src\chrome\src\out>v8_context_snapshot_generator.exe --output_file=v8_context_snapshot.bin
external 0000022E25610D29
direct: 00007FF7219A9720 00007FF721F50128 00007FF7219B1E00
setting value 00007FF7219B1E00
setting value 00007FF721F50128
setting value 00007FF7219A9720
external 0000022E25657329


So the same executable, just copied in from another location, produces different output when run (??). Might be related to ASLR somehow, but I'm still a bit confused at the moment.

Comment 74 by thakis@chromium.org, Oct 20

Looks like these 3 addresses (but nothing else really) changes every time I copy the executable over, even if I copy it over from gnrel\ each time:

C:\src\chrome\src\out>copy gnrel\v8_context_snapshot_generator.exe .
Overwrite .\v8_context_snapshot_generator.exe? (Yes/No/All): yes
        1 file(s) copied.

C:\src\chrome\src\out>v8_context_snapshot_generator.exe --output_file=v8_context_snapshot.bin
external 0000022E25610D29
direct: 00007FF70B6D9720 00007FF70BC80128 00007FF70B6E1E00
setting value 00007FF70B6E1E00
setting value 00007FF70BC80128
setting value 00007FF70B6D9720
external 0000022E25657329

C:\src\chrome\src\out>v8_context_snapshot_generator.exe --output_file=v8_context_snapshot.bin
external 0000022E25610D29
direct: 00007FF70B6D9720 00007FF70BC80128 00007FF70B6E1E00
setting value 00007FF70B6E1E00
setting value 00007FF70BC80128
setting value 00007FF70B6D9720
external 0000022E25657329

C:\src\chrome\src\out>copy gnrel2\v8_context_snapshot_generator.exe .
Overwrite .\v8_context_snapshot_generator.exe? (Yes/No/All): yes
        1 file(s) copied.

C:\src\chrome\src\out>v8_context_snapshot_generator.exe --output_file=v8_context_snapshot.bin
external 0000022E25610D29
direct: 00007FF7526B9720 00007FF752C60128 00007FF7526C1E00
setting value 00007FF7526C1E00
setting value 00007FF752C60128
setting value 00007FF7526B9720
external 0000022E25657329

C:\src\chrome\src\out>v8_context_snapshot_generator.exe --output_file=v8_context_snapshot.bin
external 0000022E25610D29
direct: 00007FF7526B9720 00007FF752C60128 00007FF7526C1E00
setting value 00007FF7526C1E00
setting value 00007FF752C60128
setting value 00007FF7526B9720
external 0000022E25657329

C:\src\chrome\src\out>v8_context_snapshot_generator.exe --output_file=v8_context_snapshot.bin
external 0000022E25610D29
direct: 00007FF7526B9720 00007FF752C60128 00007FF7526C1E00
setting value 00007FF7526C1E00
setting value 00007FF752C60128
setting value 00007FF7526B9720
external 0000022E25657329

C:\src\chrome\src\out>copy gnrel\v8_context_snapshot_generator.exe .
Overwrite .\v8_context_snapshot_generator.exe? (Yes/No/All): yes
        1 file(s) copied.

C:\src\chrome\src\out>v8_context_snapshot_generator.exe --output_file=v8_context_snapshot.bin
external 0000022E25610D29
direct: 00007FF73BE09720 00007FF73C3B0128 00007FF73BE11E00
setting value 00007FF73BE11E00
setting value 00007FF73C3B0128
setting value 00007FF73BE09720
external 0000022E25657329

Comment 75 by thakis@chromium.org, Oct 20

Just running `touch v8_context_snapshot_generator.exe` is enough to change these values. I guess the aslr seed depends on file write time or something.

I guess my question now is: Why does this affect only 4 pointers, and not much more?

Comment 76 by thakis@chromium.org, Oct 20

Maybe v8 does something smart internally for serializing pointers and the API SetAlignedPointerInInternalField() is just broken? Looks like there are only very few non-test (non-pdfium) calls to it (i.e. from blink: https://cs.chromium.org/search/?q=SetAlignedPointerInInternalField+-file:test+-file:pdfium+file:blink&sq=package:chromium&type=cs -- and one passes nullptr which is obviously fine), maybe they're all incorrect?

Comment 77 by thakis@chromium.org, Oct 20

Here's a small stand-alone demo showing that what blink is currently doing can't possibly be deterministic due to aslr:

C:\src\chrome\src>type test.cc
#include <stdio.h>
int foo = 0;
int main() {
  fprintf(stderr, "p %p\n", &foo);
}

C:\src\chrome\src>cl /nologo test.cc /link /dynamicbase
test.cc

C:\src\chrome\src>test
p 00007FF61E53F9A0

C:\src\chrome\src>test
p 00007FF61E53F9A0

# This is how you do `touch test.exe` in cmd:
C:\src\chrome\src>type nul >> test.exe

C:\src\chrome\src>test
p 00007FF66F0EF9A0

Note that the output is different after updating the exe's mtime.


v8 folks, how does the rest of the snapshot writing deal with this?

Comment 78 by thakis@chromium.org, Oct 20

(Why isn't this a problem on linux? My guess -- I need to check -- is because we only link chrome and mksnapshot with `-pie` but not v8_context_snapshot_generator, so main exe symbols aren't aslr'd there.)

Comment 79 by thakis@chromium.org, Oct 20

With this patch, this reproduces with me on linux too, with snapshot diffs in exactly the same places:

$ git diff
diff --git a/tools/v8_context_snapshot/BUILD.gn b/tools/v8_context_snapshot/BUILD.gn
index 771fc0df366c..408126a103c5 100644
--- a/tools/v8_context_snapshot/BUILD.gn
+++ b/tools/v8_context_snapshot/BUILD.gn
@@ -98,6 +98,8 @@ if (use_v8_context_snapshot) {
       "v8_context_snapshot_generator.cc",
     ]
 
+    ldflags = [ "-pie" ]
+
     deps = [
       "//gin:gin",
       "//mojo/core/embedder",



V8 folks, is this enough for you to investigate a way forward? (And once this is fixed, we should land that patch on linux, to make linux and win more alike, and to make the context generator more like the chrome binary. I'm a bit surprised the snapshot works at all in chrome given that it writes out absolute addresses.)

Comment 80 by thakis@chromium.org, Oct 20

(In a component build, .sos are always linked with with pic in our build, so this is also why v8_context_snapshot.bin isn't deterministic in component builds on linux -- the diffs are again exactly in the same places. So fixing this aslr issue will make the snapshot deterministic on static release win builds (haven't checked component there), in component builds on linux, and in static builds on linux where the generator is linked with -pie)

Comment 81 by thakis@chromium.org, Oct 21

Related: https://codereview.chromium.org/54823002/

Comment 82 by thakis@chromium.org, Oct 21

Also related: https://codereview.chromium.org/2178093003

Comment 83 by thakis@chromium.org, Oct 21

Blockedon: 881417

I did some more reading, and I think the answer to the question "how could this work?" is that it doesn't. 3 of the 4 written pointers are never read, and https://chromium-review.googlesource.com/c/chromium/src/+/1292385 removes those. And I think the fourth pointer causes an actual crash bug, issue 881417.

Project Member

Comment 84 by bugdroid1@chromium.org, Oct 22

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c9dd6b7d8f9a39ae13d5fa88dd8808ee5e53a876

commit c9dd6b7d8f9a39ae13d5fa88dd8808ee5e53a876
Author: Nico Weber <thakis@chromium.org>
Date: Mon Oct 22 02:07:45 2018

blink: Remove kV8PrototypeTypeIndex.

The value is never read, and storing it makes context snapshot writing
nondeterministic.

A comment claims that the mere presence of the field is needed to identify
prototypes, with a link to https://bugs.webkit.org/show_bug.cgi?id=110436
The code in that patch does indeed use v8PrototypeInternalFieldcount to
identify prototypes (see CustomElementHelpers::findWrapperType in that
patch), but the corresponding code in modern blink is
V0CustomElementConstructorBuilder::HasValidPrototypeChainFor which
instead uses a pointer comparison with
script_state_->PerContextData()->PrototypeForType(type). However,
V0CustomElementConstructorBuilder::PrototypeIsValid() does check
`prototype_->InternalFieldCount()` to check if a prototype is
already registered -- so keep kV8PrototypeInternalFieldcount around
and just keep the uninitialized default value (the code only cares
about the number of fields being > 0, not about the value of the field).

TBR=peria (who lg'd this, but Gerrit lost the LG)

Bug:  870584 
Change-Id: If5474f37499121047c4b0232a85435f139e5cbc0
Reviewed-on: https://chromium-review.googlesource.com/c/1292385
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#601464}
[modify] https://crrev.com/c9dd6b7d8f9a39ae13d5fa88dd8808ee5e53a876/third_party/blink/renderer/bindings/core/v8/v8_dom_configuration.cc
[modify] https://crrev.com/c9dd6b7d8f9a39ae13d5fa88dd8808ee5e53a876/third_party/blink/renderer/platform/bindings/v8_object_constructor.cc
[modify] https://crrev.com/c9dd6b7d8f9a39ae13d5fa88dd8808ee5e53a876/third_party/blink/renderer/platform/bindings/wrapper_type_info.h

Comment 85 by jgruber@chromium.org, Oct 22

#79 yes I can take it from here. Thanks again for investigating! 👍

Comment 86 by jgruber@chromium.org, Oct 22

Cc: tikuta@chromium.org
Owner: jgruber@chromium.org

Comment 87 by thakis@chromium.org, Oct 22

Awesome, thanks!

Just for completeness: The last pointer is written by the SetAlignedPointerInInternalField() call in CreatePlainWrapper(), and then redundantly again by the call to SetAlignedPointerInInternalFields() (note: plural) right after CreatePlainWrapper() is called. (But it sounds the issue is that these pointers are supposed to be converted to enum values at serialization time and then back at deserialization time, so maybe this isn't that interesting.)

Comment 88 by thakis@chromium.org, Oct 22

(For folks following along at home,  issue 617892  describes how blink snapshot pointers in internal fields are supposed to work.)

Comment 89 by yangguo@chromium.org, Oct 22

Owner: yangguo@chromium.org

I think I know what the issue is.

When serializing internal fields, we defer to the embedded via callback. But we nevertheless serialize the content verbatim. During deserializer we write the verbatim content first, and then call the embedders via callback to overwrite. Copying verbatim is the reason we get non determinism. But this does not explain the crash bug 881417. I'll cook up a fix.

Comment 90 by yangguo@chromium.org, Oct 22

So the API actually looks like this:

During serialization, if we encounter an object that has embedder fields, we remember it. Later, we call the embedder's serialization callback for every one of these objects in a batch and get the data back that is serialized by the embedder.

During deserialization, we call the embedder's deserialization callback with the data previously provided by the serialization callback. The embedder can then set the field if they like.

The issue is that the embedder field can either store another V8 heap object, or an aligned pointer. We cannot really tell whether it's an aligned pointer or a Smi though. That' why we serialize the content verbatim anyways, just in case the embedder is storing an integer there and does not do anything special to serialize it.

I could change the API so that the embedder's callback has to signal whether it serialized the embedder field, or needs V8 to deal with it.

Or I could expect the embedder to deal with the Smi case. But that would expose an implementation detail to the embedder.

@jgruber: WDYT?

Comment 91 by yangguo@chromium.org, Oct 22

As discussed offline, we will clear the embedder field if the serializer callback returns data. If it returns empty data, V8 will serialize the field as a normal heap reference or smi.

Comment 92 by jgruber@chromium.org, Oct 22

As discussed offline, clearing the field if the callback returns data sgtm. 

So the expectations for usage of embedder fields are:

- Either they store a reference to a V8 heap object with deterministic value across mksnapshots,
- or to a smi with deterministic value across mksnapshots,
- or to a smi with nondeterministic value that is replaced by the callback (in which case we clear the value).

Is that right?

Comment 93 by yangguo@chromium.org, Oct 22

Yes.

Project Member

Comment 94 by bugdroid1@chromium.org, Oct 22

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1d3d3c5a8e33953a62a0c724a153772702d84501

commit 1d3d3c5a8e33953a62a0c724a153772702d84501
Author: Nico Weber <thakis@chromium.org>
Date: Mon Oct 22 17:22:34 2018

Make v8_context_snapshot.bin writing deterministic.

V8 currently writes the raw values for embedder fields always,
so don't put pointers (which differ even between runs of the same
binary, due to ASLR) in there until that's fixed. Instead, put
a magic integer there and transform that to a pointer at
deserialization time.

(Also remove a redundant call from CreatePlainWrapper() --
the caller of CreatePlainWrapper() already sets the embedder
field, no need to do that twice. This part is behavior-preserving.)

Bug:  870584 
Change-Id: I6b29286ec8c7a68a662d51c36b5642b968f2e343
Reviewed-on: https://chromium-review.googlesource.com/c/1293791
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Kentaro Hara <haraken@chromium.org>
Reviewed-by: Hans Wennborg <hans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#601628}
[modify] https://crrev.com/1d3d3c5a8e33953a62a0c724a153772702d84501/third_party/blink/renderer/bindings/core/v8/v8_context_snapshot.cc

Project Member

Comment 95 by bugdroid1@chromium.org, Oct 22

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ff82c14543a8c8a791fac7b4f07bbf254b0e86ea

commit ff82c14543a8c8a791fac7b4f07bbf254b0e86ea
Author: Nico Weber <thakis@chromium.org>
Date: Mon Oct 22 18:26:23 2018

Build v8_context_snapshot_generator with -pie on linux.

After https://chromium-review.googlesource.com/c/1293791,
Deterministic Linux should stay green with -pie, and it
makes that bot more like its dbg variant (.sos already
use position-independent code and ASLR) and like the
Windows deterministic bot.

Bug:  870584 
Change-Id: I730437f3d1160016d3cdf78921990cf2d4b9a40a
Reviewed-on: https://chromium-review.googlesource.com/c/1294289
Reviewed-by: Hans Wennborg <hans@chromium.org>
Reviewed-by: Kentaro Hara <haraken@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#601660}
[modify] https://crrev.com/ff82c14543a8c8a791fac7b4f07bbf254b0e86ea/tools/v8_context_snapshot/BUILD.gn

Project Member

Comment 96 by bugdroid1@chromium.org, Oct 22

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9e109ccead3f29557fefd845add16bbdec419528

commit 9e109ccead3f29557fefd845add16bbdec419528
Author: Nico Weber <thakis@chromium.org>
Date: Mon Oct 22 20:42:18 2018

Remove v8_context_snapshot.bin (and deps) from deterministic build whilelist file.

Writing v8_context_snapshot.bin is now deterministic.

Bug:  870584 
Change-Id: If70b304db316e4245b093ccdd31fb9aa0dae05ff
Reviewed-on: https://chromium-review.googlesource.com/c/1294304
Reviewed-by: Erik Chen <erikchen@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#601715}
[modify] https://crrev.com/9e109ccead3f29557fefd845add16bbdec419528/tools/determinism/deterministic_build_whitelist.pyl

Comment 97 by thakis@chromium.org, Oct 22

Blockedon: -881417

Comment 98 by thakis@chromium.org, Oct 23

This is now basically fixed. Once yangguo's v8 change is in, we're going to be able to remove a blink-side hack.

Project Member

Comment 99 by bugdroid1@chromium.org, Nov 6

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/4495cba3d689e9ee136850550fd864ddf011b81e

commit 4495cba3d689e9ee136850550fd864ddf011b81e
Author: Yang Guo <yangguo@chromium.org>
Date: Tue Nov 06 11:48:19 2018

[snapshot] clear internal fields that are serialized by callback

This is to ensure the snapshot is deterministic.

Internal fields can contain:
- reference to heap object
- embedder-defined aligned pointer
- a smi

The latter two are not distinguishable by V8, so if the serializer
callback returns non-zero value, we consider it to be an aligned pointer
and clear it to ensure that the snapshot does not contain memory
addresses that may not be deterministic. If the callback returns
{ nullptr, 0 } as result, we consider it to be a smi or some in-place
data that we then serialize verbatim.


R=jgruber@chromium.org

Bug:  chromium:870584 
Change-Id: I3cf9abf135ffd28d8138fa32636b12596b076e13
Reviewed-on: https://chromium-review.googlesource.com/c/1304441
Commit-Queue: Yang Guo <yangguo@chromium.org>
Reviewed-by: Jakob Kummerow <jkummerow@chromium.org>
Reviewed-by: Jakob Gruber <jgruber@chromium.org>
Cr-Commit-Position: refs/heads/master@{#57272}
[modify] https://crrev.com/4495cba3d689e9ee136850550fd864ddf011b81e/include/v8.h
[modify] https://crrev.com/4495cba3d689e9ee136850550fd864ddf011b81e/src/objects/js-objects-inl.h
[modify] https://crrev.com/4495cba3d689e9ee136850550fd864ddf011b81e/src/objects/js-objects.h
[modify] https://crrev.com/4495cba3d689e9ee136850550fd864ddf011b81e/src/snapshot/partial-serializer.cc
[modify] https://crrev.com/4495cba3d689e9ee136850550fd864ddf011b81e/src/snapshot/partial-serializer.h
[modify] https://crrev.com/4495cba3d689e9ee136850550fd864ddf011b81e/src/snapshot/snapshot-source-sink.cc
[modify] https://crrev.com/4495cba3d689e9ee136850550fd864ddf011b81e/src/snapshot/snapshot-source-sink.h
[modify] https://crrev.com/4495cba3d689e9ee136850550fd864ddf011b81e/test/cctest/test-serialize.cc

Comment 100 by yangguo@chromium.org, Nov 6

@thakis, we now clear the embedder field if the serialization callback produced results. So it should be safe to revert your fix once V8 rolls into Chrome the next time.

Project Member

Comment 101 by bugdroid1@chromium.org, Nov 9

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d2acccc10cf8afd5a3709f5fd562d700d6b96979

commit d2acccc10cf8afd5a3709f5fd562d700d6b96979
Author: Nico Weber <thakis@chromium.org>
Date: Fri Nov 09 16:08:31 2018

Remove v8_context_snapshot.bin determinism hack.

Reverts parts of
https://chromium-review.googlesource.com/c/chromium/src/+/1293791 that should
no longer be needed after https://chromium-review.googlesource.com/c/1304441

Bug:  870584 
Change-Id: I76fa59eb5c88c34cd222f2a40c983a4f4e615b27
Reviewed-on: https://chromium-review.googlesource.com/c/1328823
Reviewed-by: Kentaro Hara <haraken@chromium.org>
Reviewed-by: Hitoshi Yoshida <peria@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#606859}
[modify] https://crrev.com/d2acccc10cf8afd5a3709f5fd562d700d6b96979/third_party/blink/renderer/bindings/core/v8/v8_context_snapshot.cc

Comment 102 by thakis@chromium.org, Nov 9

Status: Fixed (was: Assigned)

Showing comments 3 - 102 of 102 Older ›

►

Issue 870584 link

Issue metadata

v8_context_snapshot.bin is non-deterministic

Issue description

Comment 3 by jgruber@chromium.org, Aug 3

Comment 3 Deleted

Comment 4 by yangguo@chromium.org, Aug 3

Comment 4 Deleted

Comment 5 by peria@chromium.org, Aug 3

Comment 5 Deleted

Comment 6 by tikuta@chromium.org, Aug 3

Comment 6 Deleted

Comment 7 by peria@chromium.org, Aug 3

Comment 7 Deleted

Comment 8 by tikuta@chromium.org, Aug 3

Comment 8 Deleted

Comment 9 by jgruber@chromium.org, Aug 3

Comment 9 Deleted

Comment 10 by jgruber@chromium.org, Aug 3

Comment 10 Deleted

Comment 11 by jgruber@chromium.org, Aug 3

Comment 11 Deleted

Comment 12 by tikuta@chromium.org, Aug 3

Comment 12 Deleted

Comment 13 by tikuta@chromium.org, Aug 3

Comment 13 Deleted

Comment 14 by bugdroid1@chromium.org, Aug 3

Comment 14 Deleted

Comment 15 by tikuta@chromium.org, Aug 3

Comment 15 Deleted

Comment 16 by thakis@chromium.org, Aug 6

Comment 16 Deleted

Comment 17 by thakis@chromium.org, Aug 6

Comment 17 Deleted

Comment 18 by thakis@chromium.org, Aug 6

Comment 18 Deleted

Comment 19 by tikuta@chromium.org, Aug 6

Comment 19 Deleted

Comment 20 by tikuta@chromium.org, Aug 7

Comment 20 Deleted

Comment 21 by tikuta@chromium.org, Aug 7

Comment 21 Deleted

Comment 22 by jgruber@chromium.org, Aug 7

Comment 22 Deleted

Comment 23 by tikuta@chromium.org, Aug 7

Comment 23 Deleted

Comment 24 by bugdroid1@chromium.org, Aug 7

Comment 24 Deleted

Comment 25 by tikuta@chromium.org, Aug 13

Comment 25 Deleted

Comment 26 by jgruber@chromium.org, Aug 13

Comment 26 Deleted

Comment 27 by tikuta@chromium.org, Aug 13

Comment 27 Deleted

Comment 28 by tikuta@chromium.org, Aug 13

Comment 28 Deleted

Comment 29 by jgruber@chromium.org, Aug 13

Comment 29 Deleted

Comment 30 by tikuta@chromium.org, Aug 13

Comment 30 Deleted

Comment 31 by tikuta@chromium.org, Aug 14

Comment 31 Deleted

Comment 32 by jgruber@chromium.org, Aug 14

Comment 32 Deleted

Comment 33 by tikuta@chromium.org, Aug 14

Comment 33 Deleted

Comment 34 by petermarshall@chromium.org, Aug 16

Comment 34 Deleted

Comment 35 by thakis@chromium.org, Sep 7

Comment 35 Deleted

Comment 36 by jgruber@chromium.org, Sep 10

Comment 36 Deleted

Comment 37 by thakis@google.com, Oct 12

Comment 37 Deleted

Comment 38 by thakis@chromium.org, Oct 12

Comment 38 Deleted

Comment 39 by thakis@chromium.org, Oct 12

Comment 39 Deleted

Comment 40 by thakis@chromium.org, Oct 12

Comment 40 Deleted