FYI: Mine is only for ChromeOS, so if this is for Win, it should be unrelated.

My patch changed the code in a python tool that run tests on Chrome, so it's extremely unlikely to be the culprit.

Latest builds are green again, now suspecting that this was not due to code change, and instead is due to some kind of flakiness.

I think we can see if something like this happens again, and close if not.

Also thanks hidehiko@ and nednguyen@ for taking a look :-)

This was resolved today, although I forgot exactly how

This appears to be happening again; @qyearsley, do you remember anything about how this was fixed before?

https://build.chromium.org/p/chromium/builders/Win/builds/57257

[38872/64403] ACTION //v8:run_mksnapshot(//build/toolchain/win:x86)
FAILED: gen/v8/snapshot.cc snapshot_blob.bin 
C:/b/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../v8/tools/run.py ./mksnapshot --startup_src gen/v8/snapshot.cc --random-seed 314159265 --startup_blob snapshot_blob.bin
==== C stack trace ===============================
	(No symbol) [0x00880000]
	v8::internal::Heap::SetUp [0x00EAF890+1264]
	v8::internal::Isolate::Init [0x00F22374+1044]
	v8::SnapshotCreator::SnapshotCreator [0x00C88A86+150]
	v8::V8::CreateSnapshotDataBlob [0x00C8ABB1+33]
	main [0x00C81690+192]
	__scrt_common_main_seh [0x011F96FB+249] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253)
	BaseThreadInitThunk [0x7531338A+18]
	RtlInitializeExceptionChain [0x771C9A02+99]
	RtlInitializeExceptionChain [0x771C99D5+54]

Unfortunately I don't remember clearly -- I think that at this time (last week) Dirk mentioned that something about v8 snapshot being flaky.

It appears as though earlier, the next build became green again later without any relevant fixes/reverts being committed in chromium/src.

I couldn't find any related bug about the //v8:run_mksnapshot action failing flakily; if it is flaky, then we may want to file a separate bug or make this bug focused on that problem.

yangguo, are you the right person to ping about v8 snapshot flaking?

Yes.

With previous build flakiness we were never able to reproduce locally. No idea what's going on :(

Marking available since I'm not the right person to own this, and it appears there are no immediate next actions here that we know of.

This is still on the sheriff bug list (label Sheriff-Chromium) which might be helpful just to let sheriffs know that this is an issue. Sheriffs: If this bug isn't helpful then consider removing the Sheriff-Chromium label.

This crash happened today in  Google Chrome Win Build bot.
https://build.chromium.org/p/chromium.chrome/builders/Google%20Chrome%20Win/builds/20610

Just now, we got: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium%2FWin%2F59604%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout

[44564/66303] ACTION //v8:run_mksnapshot(//build/toolchain/win:x86) FAILED: gen/v8/snapshot.cc snapshot_blob.bin C:/b/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../v8/tools/run.py ./mksnapshot --startup_src gen/v8/snapshot.cc --random-seed 314159265 --startup_blob snapshot_blob.bin ==== C stack trace =============================== v8::internal::Heap::CreateInitialObjects [0x00743000+3792] v8::internal::SetupIsolateDelegate::SetupHeapInternal [0x007439D1+17] v8::internal::SetupIsolateDelegate::SetupHeap [0x006F1FB1+17] v8::internal::Isolate::Init [0x0053D734+9380] v8::SnapshotCreator::SnapshotCreator [0x002B89F5+245] v8::V8::CreateSnapshotDataBlob [0x002BACD1+33] main [0x002B1590+192] __scrt_common_main_seh [0x0081BD1A+248] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283) BaseThreadInitThunk [0x76E7338A+18] RtlInitializeExceptionChain [0x77429A02+99] RtlInitializeExceptionChain [0x774299D5+54]

Happened again: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium%2FWin%2F59725%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout

[44595/66403] ACTION //v8:run_mksnapshot(//build/toolchain/win:x86)
FAILED: gen/v8/snapshot.cc snapshot_blob.bin 
C:/b/depot_tools/win_tools-2_7_6_bin/python/bin/python.exe ../../v8/tools/run.py ./mksnapshot --startup_src gen/v8/snapshot.cc --random-seed 314159265 --startup_blob snapshot_blob.bin
==== C stack trace ===============================
	v8::internal::Factory::NewRawOneByteString [0x00450232+274]
	v8::internal::Heap::CreateInitialObjects [0x006E56B0+2704]
	v8::internal::SetupIsolateDelegate::SetupHeapInternal [0x006E6616+278]
	v8::internal::SetupIsolateDelegate::SetupHeap [0x00694AE1+17]
	v8::internal::Isolate::Init [0x004DF5A4+9380]
	v8::SnapshotCreator::SnapshotCreator [0x00258865+245]
	v8::V8::CreateSnapshotDataBlob [0x0025ABA1+33]
	main [0x00251590+192]
	__scrt_common_main_seh [0x007BE85A+248] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:283)
	BaseThreadInitThunk [0x76D0338A+18]
	RtlInitializeExceptionChain [0x77659A02+99]
	RtlInitializeExceptionChain [0x776599D5+54]

IMHO, short-term solution to this is to add retries, so that these failures stop causing compile failures for Chromium developers.

Mid-term solution is to add monitoring so that we know when the script fails. We should capture the stack trace, bot name, builder name and any other metadata that would help us identify whether this is a problem with the code or with the machines running it (this script never fails on V8 waterfall).

Long-term solution will depend on the results of the monitoring. It may require fixing bugs in the mksnapsot binary or in the underlying V8 compiler.

WDYT?

Another observation: All reports in this issue and the blocked issue are regarding the Win and Win64 Chromium clobber builders. The issue has not been reported for any other builder yet.

P.S. The current stack trace does not contain the error message and by comparing stacktraces from different builds, it looks like they are trimmed by something, so the error may actually happen deeper in the call hierarchy than it may appear.

P.P.S. After looking at further 2 builds, it seems that stack traces are not the same, which makes it seems that the error is happening regardless of the code that is running.

Also the error that was reported in July is different and probably unrelated with current failures.

Another data point:
https://ci.chromium.org/buildbot/chromium.chrome/Google%20Chrome%20Win/25852

Looks like this still isn't resolved. How is this bug different from  Issue 700525 ?

Lets mark this as dupe. Both bugs have a rich history with some good and some different examples. From an infrastructure p-o-v it's the same thing, just possibly different root causes for the flaky crashes.

I also filed the particular crash now as separate v8 bug:
 https://crbug.com/v8/7301