Could this be related to the random crashes of the brotli executable in  Issue 739916 ? Could something have changed in the compiler toolchain to cause (perhaps) uninitialized memory to result in random crashes?

+clang folks; not sure which compiler toolchain is used on this bot

it uses clang, but is there any reason to believe this should be due to a compiler bug rather than due to a v8 bug? most failures are due to code bugs.

The last Clang update was on 21 November, so it seems more likely that it's something in v8 that's changed.

It's possible this is related to  bug 739916  (which is a duplicate of  bug 644525 ) but it's difficult to tell. The error reporting code in mksnapshot doesn't print the error code, and even then we'd really need to see the code bytes to be sure.

The unique signature of  bug 644525  is that we end up with code bytes that contain zero, which then causes a crash. And, relinking with no other changes solves the problem.

Disabling incremental linking for specific binaries may reduce the frequency, but it does not solve it. Linking of mksnapshot.exe is slow enough that we should leave incremental linking enabled.

TL;DR - we need a crash dump along with a copy of the binary that crashed and the PDB.

Seemed to happen again in:
https://ci.chromium.org/buildbot/chromium/Win%20x64/17310
https://ci.chromium.org/buildbot/chromium.chrome/Google%20Chrome%20Win/24904

Output from Win 64:
FAILED: gen/v8/snapshot.cc snapshot_blob.bin 
C:/b/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../v8/tools/run.py ./mksnapshot --turbo_instruction_scheduling --startup_src gen/v8/snapshot.cc --random-seed 314159265 --startup_blob snapshot_blob.bin
==== C stack trace ===============================
	v8::internal::compiler::MachineOperatorGlobalCache::MachineOperatorGlobalCache [0x000000013FE70003+8387]
	v8::base::CallOnceImpl [0x00000001403F6CA3+51]
	v8::internal::compiler::MachineOperatorBuilder::MachineOperatorBuilder [0x000000013FE6B4F0+112]
	v8::internal::compiler::RawMachineAssembler::RawMachineAssembler [0x000000013FE95453+147]
	v8::internal::compiler::CodeAssemblerState::CodeAssemblerState [0x000000013FD9B3BB+219]
	v8::internal::SetupIsolateDelegate::SetupBuiltinsInternal [0x00000001403B973E+42478]
	v8::internal::SetupIsolateDelegate::SetupBuiltinsInternal [0x00000001403AF203+179]
	v8::internal::Isolate::Init [0x000000014006B075+13557]
	v8::SnapshotCreator::SnapshotCreator [0x000000013FC79DF8+200]
	v8::V8::CreateSnapshotDataBlob [0x000000013FC7A960+80]
	main [0x000000013FC7132A+298]
	__scrt_common_main_seh [0x00000001404DD1C9+285] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283)
	BaseThreadInitThunk [0x00000000779D59CD+13]
	RtlUserThreadStart [0x0000000077C0B891+33]

Output from Google Chrome Win:
FAILED: gen/v8/snapshot.cc snapshot_blob.bin 
C:/b/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../v8/tools/run.py ./mksnapshot --turbo_instruction_scheduling --startup_src gen/v8/snapshot.cc --random-seed 314159265 --startup_blob snapshot_blob.bin

Tend to agree with c#3... peria@ is there a reason V8 snapshotting can still cause build failures? I feel like I've gotten these in the past and fixed by running `gn clean`.

oops readd kbr

CC'ing yangguo@ from V8 team; not sure who on V8 knows the snapshot generator the best.

Building the snapshot is the first time the V8 binary gets executed. So if the build is corrupted, this is often the first place that catches it. For some reason we get a crash. It is very deterministic, so I'm very surprised it would fail.

Is there any source of non-determinism on the windows bot that could cause arbitrary binary to crash? Maybe race conditions with writing compiled binaries out to disk and executing it? I'm just guessing here.

Aha. From the brotli.exe  Issue 739916  it looks like this may be the same linker bug as in  Issue 644525 .

What is the frequency of these crashes?

Absent other information I have to assume that this is a duplicate of 644525. I am making progress on that bug although the most likely explanation is an OS bug, which is not exactly encouraging.