All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=703651

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZykwELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImVDaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9Ub3RhbCB0cmFjZSBzaXplIGluIGJ5dGVzX3N1bQwLEg1TdG9wcGFnZUFsZXJ0GPfOGww,agxzfmNocm9tZXBlcmZykwELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImVDaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9Ub3RhbCB0cmFjZSBzaXplIGluIGJ5dGVzX3N0ZAwLEg1TdG9wcGFnZUFsZXJ0GPfOGww,agxzfmNocm9tZXBlcmZykwELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImVDaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9Ub3RhbCB0cmFjZSBzaXplIGluIGJ5dGVzX21pbgwLEg1TdG9wcGFnZUFsZXJ0GPfOGww,agxzfmNocm9tZXBlcmZykwELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImVDaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9Ub3RhbCB0cmFjZSBzaXplIGluIGJ5dGVzX21heAwLEg1TdG9wcGFnZUFsZXJ0GPfOGww,agxzfmNocm9tZXBlcmZylQELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImdDaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9Ub3RhbCB0cmFjZSBzaXplIGluIGJ5dGVzX2NvdW50DAsSDVN0b3BwYWdlQWxlcnQY984bDA,agxzfmNocm9tZXBlcmZykwELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImVDaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9Ub3RhbCB0cmFjZSBzaXplIGluIGJ5dGVzX2F2ZwwLEg1TdG9wcGFnZUFsZXJ0GPfOGww,agxzfmNocm9tZXBlcmZymQELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImtDaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9NYXggbnVtYmVyIG9mIGV2ZW50cyBwZXIgc2Vjb25kX3N1bQwLEg1TdG9wcGFnZUFsZXJ0GPfOGww,agxzfmNocm9tZXBlcmZymQELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImtDaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9NYXggbnVtYmVyIG9mIGV2ZW50cyBwZXIgc2Vjb25kX3N0ZAwLEg1TdG9wcGFnZUFsZXJ0GPfOGww,agxzfmNocm9tZXBlcmZymQELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImtDaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9NYXggbnVtYmVyIG9mIGV2ZW50cyBwZXIgc2Vjb25kX21pbgwLEg1TdG9wcGFnZUFsZXJ0GPfOGww,agxzfmNocm9tZXBlcmZymQELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImtDaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9NYXggbnVtYmVyIG9mIGV2ZW50cyBwZXIgc2Vjb25kX21heAwLEg1TdG9wcGFnZUFsZXJ0GPfOGww,agxzfmNocm9tZXBlcmZymwELEhNTdG9wcGFnZUFsZXJ0UGFyZW50Im1DaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9NYXggbnVtYmVyIG9mIGV2ZW50cyBwZXIgc2Vjb25kX2NvdW50DAsSDVN0b3BwYWdlQWxlcnQY984bDA,agxzfmNocm9tZXBlcmZymQELEhNTdG9wcGFnZUFsZXJ0UGFyZW50ImtDaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9NYXggbnVtYmVyIG9mIGV2ZW50cyBwZXIgc2Vjb25kX2F2ZwwLEg1TdG9wcGFnZUFsZXJ0GPfOGww,agxzfmNocm9tZXBlcmZynAELEhNTdG9wcGFnZUFsZXJ0UGFyZW50Im5DaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9NYXggZXZlbnQgc2l6ZSBpbiBieXRlcyBwZXIgc2Vjb25kX3N1bQwLEg1TdG9wcGFnZUFsZXJ0GPfOGww,agxzfmNocm9tZXBlcmZynAELEhNTdG9wcGFnZUFsZXJ0UGFyZW50Im5DaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9NYXggZXZlbnQgc2l6ZSBpbiBieXRlcyBwZXIgc2Vjb25kX3N0ZAwLEg1TdG9wcGFnZUFsZXJ0GPfOGww,agxzfmNocm9tZXBlcmZynAELEhNTdG9wcGFnZUFsZXJ0UGFyZW50Im5DaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9NYXggZXZlbnQgc2l6ZSBpbiBieXRlcyBwZXIgc2Vjb25kX21pbgwLEg1TdG9wcGFnZUFsZXJ0GPfOGww,agxzfmNocm9tZXBlcmZynAELEhNTdG9wcGFnZUFsZXJ0UGFyZW50Im5DaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9NYXggZXZlbnQgc2l6ZSBpbiBieXRlcyBwZXIgc2Vjb25kX21heAwLEg1TdG9wcGFnZUFsZXJ0GPfOGww,agxzfmNocm9tZXBlcmZyngELEhNTdG9wcGFnZUFsZXJ0UGFyZW50InBDaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9NYXggZXZlbnQgc2l6ZSBpbiBieXRlcyBwZXIgc2Vjb25kX2NvdW50DAsSDVN0b3BwYWdlQWxlcnQY984bDA,agxzfmNocm9tZXBlcmZynAELEhNTdG9wcGFnZUFsZXJ0UGFyZW50Im5DaHJvbWl1bVBlcmYvY2hyb21pdW0tcmVsLW1hYzExLWFpci90cmFjaW5nLnRyYWNpbmdfd2l0aF9kZWJ1Z19vdmVyaGVhZC9NYXggZXZlbnQgc2l6ZSBpbiBieXRlcyBwZXIgc2Vjb25kX2F2ZwwLEg1TdG9wcGFnZUFsZXJ0GPfOGww


Bot(s) for this bug's original alert(s):

chromium-rel-mac11-air

It seems a bit suspicious that this stopped working when https://codereview.chromium.org/2714533002 was landed.

This is probably related to https://bugs.chromium.org/p/chromium/issues/detail?id=703291&can=1&q=reporter%3Ame%20tracing_with_debug_overhead&colspec=ID%20Pri%20M%20Stars%20ReleaseBlock%20Component%20Status%20Owner%20Summary%20OS%20Modified.

In the latest build (https://build.chromium.org/p/chromium.perf/builders/Mac%20Air%2010.11%20Perf/builds/651), I see this benchmark producing data just fine in https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=463fea6ddae8fefeabbec42ea8eaf1505653e402 

Simon: is this a problem with the dashboard?

Ned: in your link, I actually don't see any values with names like 'Max event size in bytes per second_avg', did the names change?

Ben/Charlie: did any of you refactor the tracing size metric recently?

The names did indeed change: they now use hacker_style instead of "Regular English style" and are named more consistently with the other metrics. This happened in Ben's change here: https://github.com/catapult-project/catapult/commit/4c3dbb5cccad3b0ff1e5ddcb0538556f49def0fb

In https://chromeperf.appspot.com/report?sid=a8cfc68f7c2ac7a9bf44ef139ec1658f5b901bfb5d8c72e1bf680e062747720d, you can see that there's still a trace_size_avg metric that's reporting data just fine, which replaces "Total trace size in bytes_avg", which is very funkily named.

In the future, is there some group that we should message when we expect metrics to stop reporting data? For example, https://codereview.chromium.org/2755863003/ disables a whole bunch of aggregate metrics on the tracing metric that aren't useful (trace_size_stddev), so we should expect a bunch of data stoppage alerts on those soon.

I think perf-sheriffs@chromium.org is the best group to alert, and putting a clear note on the CL description is super helpful.

But the weird thing here is that the dashboard thinks the stoppage happens in this range:
https://chromium.googlesource.com/chromium/src/+log/0a6d72780a0317bf6d1bfcabc023123e5cdfb454%5E..62514bb6d6cfe433a87cdd1faed4fbcdfc789859?pretty=fuller

Which doesn't include that CL. I checked and the problem is that that build was interrupted, and that CL landed in the next build. So normally the sheriff would have seen this in the diff from the debug button, but this is kind of an edge case.

Good to know! I'll definitely keep that in mind in the future and when reviewing CLs.