For some reason, it seems like user expectations have been broken on the ref build since about 11/30.

See: ref_build_stoppage.png.

Given that this is the ref build and not ToT, there are a couple of possibilities that I see, each of which I'll address separately:

### The reference build was updated ###

Nope. The last reference build update was on 10/11, long before the stoppage. https://cs.corp.google.com/github/catapult-project/catapult/common/py_utils/py_utils/chrome_binaries.json?q=p:github+f:%5Ecatapult-project/catapult/+chrome_binaries.json&dr

### The stoppage was caused by a Javascript-side Catapult change ###

Nope. The changelog from the first broken build (https://chromium.googlesource.com/chromium/src/+log/679ce83f762e4774b5721704257baa5e71119baa%5E..3b5e1a345a490034e4588ce4e512bfa10de413f3?pretty=fuller) shows only one Catapult roll, containing two CLs: https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/b38578d92404..59c9c1d73830

Neither CL seems remotely related to metrics.

### The stoppage was caused by a benchmark or infrastructure change ###

Possibly: searching for "tools/perf" in the changelog only yields one changelist that *might* be related: https://codereview.chromium.org/2540213002. That changelist enabled swarming on the Mac perf bots.

This data stoppage only occurred on Mac (which that changelist affected), but that doesn't mean as much as it might sound like: for some reason, we've apparently *never* successfully identified video rail stages in Windows ref traces, whereas we have on Mac. Regardless, investigating this CL seems worthwhile, and I'm going to ask eyaich@ about it.
 

Deleted: ref_build_stoppage.png 164 KB

	ref_build_stoppage.png 164 KB View Download