The current error on the tree related to build 8912 is being tracked by  issue 922025 , FYI.

The public waterfall problem in 922025 found and reverted a culprit:
https://bugs.chromium.org/p/chromium/issues/detail?id=922025#c5

I went through the history of the CQ failures, and the failures do start slightly after the culprit CL landed. This suggest that the failures were likely caused by the culprit CL. That being said...the symptoms on the CQ appear so different than the public waterfall problem that I wonder if we're seeing two different problems [especially since we've seen this particular problem 1 month ago]. 

The GPU CL was reverted in  bug 922025 , but only made the GPU-related ASAN failures go away; just seeing a different crash (now in ~GraphicsContext while painting).

I could not reproduce the problem locally but I got these error messages: https://bugs.chromium.org/p/chromium/issues/detail?id=289453

Can we grab the build dir from the bot?

Most of the build is pretty deterministic by now. The one thing we know isn't is ld64. If we have some ODR violation somewhere and ld64 parallelism makes it nondeterministic which version is pulled in, that might result in what we're seeing.

CCI trooper here - what do you want to pull from which bot? Let me know and I'll see what I can do.

From the chat - I'm shutting down https://chromium-swarm.appspot.com/bot?id=vm73-m9&sort_stats=total%3Adesc which will stop https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.13%20Tests temporarily. The bot will finish https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.13%20Tests/8931 and then terminate. I'll grab the resulting build dir and restart the bot.

The bot finished the build and terminated. I'll be downloading the files shortly.

The build dir is 28GB. That'll take a while...

Created 11GB tarball. Downloading it from Golo is another ~30min... I'll probably then upload it to an internal Google Drive or something.

Tree closures should be P-0, no?

This reminds me of a previous issue (https://bugs.chromium.org/p/chromium/issues/detail?id=900405#c13) that dpranke@, tbansal@, horo@, thakis@, and I previously did a bunch of debugging on. The only conclusion we came to at the time is that there were stale .o files that ninja was failing to rebuild, even though its dependencies changed.

Unfortunately, we never figured out the root cause, and we 'resolved' the issue by clobbering the affected bots.

Clobbering Mac Builder:

./swarming.py trigger -S chromium-swarm.appspot.com -d pool luci.chromium.ci -d id vm74-m9 --named-cache builder_ea778128bf936f6571437fe2d8c833f98214957b2868d01f3dd50f6fcbe3d309_v2 cache/builder --raw-cmd -- /bin/rm -rf cache/builder

Task: https://chromium-swarm.appspot.com/user/task/426c6fb59137ef10

Mac Builder completed its build: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20Builder/94314

Mac testers are starting, e.g.:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac10.13%20Tests/8933

Clobbering "Mac FYI GPU ASAN Release":

./swarming.py trigger -S chromium-swarm.appspot.com -d pool luci.chromium.ci -d id build91-m9 --named-cache builder_cab593cb012d77b5769967b178802a9590f1f6cab32db18f3fd9b34757dcee62_v2 cache/builder --raw-cmd -- /bin/rm -rf cache/builder

https://chromium-swarm.appspot.com/user/task/426ccc1691d75d10

Clobbering "Mac Asan 64 Builder" (2 bots):

./swarming.py trigger -S chromium-swarm.appspot.com -d pool luci.chromium.ci -d id vm259-m9 --tags clobber:mac-asan-64-builder --named-cache builder_f6e6bada9972e10bc0841fcf0461958d34fecfc7e713a2f894c66d00b195e6ab_v2 cache/builder --raw-cmd -- /bin/rm -rf cache/builder

./swarming.py trigger -S chromium-swarm.appspot.com -d pool luci.chromium.ci -d id vm262-m9 --tags clobber:mac-asan-64-builder --named-cache builder_f6e6bada9972e10bc0841fcf0461958d34fecfc7e713a2f894c66d00b195e6ab_v2 cache/builder --raw-cmd -- /bin/rm -rf cache/builder
Triggered task: sergeyberezin@google.com/id=vm262-m9_pool=luci.chromium.ci

Tasks: https://chromium-swarm.appspot.com/tasklist?c=name&c=state&c=created_ts&c=duration&c=pending_time&c=pool&c=bot&et=1547597640000&f=clobber-tag%3Amac-asan-64-builder&l=50&n=true&s=created_ts%3Adesc&st=1547511240000

For posterity, the theory was that infra failures on the Builder bot last night left partial, stale or corrupted .o's, or messed up the ninja dependencies in a way that they didn't get rebuilt. The Tester bots just kept picking up those broken builds and crashing with them.

Elly noted that "yesterday night, two back to back "infra failure"s on this bot, where the build output simply stops partway through".

first complete build after that: 2019-01-14 7:28 PM (EST)
first dead mac10.13 tests run: 2019-01-14 8:10 PM (EST)".

Similarly the "Mac ASan 64 Tests" and "Mac FYI GPU ASAN Release" bots had infra failures yesterday, just prior to going flaky then solid red. So they've been clobbered as well.

I was a sheriff at that moment. Yup, there was a massive LUCI outage right before these failures started. Would be good to write a postmortem.

If we do write a postmortem, I've saved the Hangouts Chat log here:
https://docs.google.com/document/d/1CdJ_8RXaWVn2oKTc0TK1JQzI7p3gatuKd4aMh05eWvc/edit?usp=sharing

(we didn't turn history on, so it would otherwise have been purged after 24 hours)

In the "jumbo builds didn't build correctly"  bug 921967  we found 6 empty dependency slots in ninja's dependency databases on the bot we investigated. The object file didn't even depend on the cc file itself and nothing could make those object files rebuild. It matches the theory you have, but we have no idea how it happened.

FTR, if still needed, a copy of build dir from #c10 is here: https://drive.google.com/file/d/13GUYGUbjSluvNIgNpM5MoUD_n7EP6GVu/view?usp=sharing

(I'll likely delete it from my drive in a few weeks, please make a copy if you need it).