+simonhatch to check my analysis:

Here are the buildbot status pages for the failed bisects:
https://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/2218
https://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/2225
https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/clankium-low-end-phone-perf-bisect/builds/306
https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/clankium-low-end-phone-perf-bisect/builds/309
https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/clankium-low-end-phone-perf-bisect/builds/323

These are all timing out after 24 hours, except for one which appears to hit the hour step timeout.

Looks like these are coming from the v8 memory sheriff, which has the following alerting configs:

*/*/*/memory:chrome:renderer_processes:reported_by_chrome:v8:heap:allocated_objects_size_max/*
*/*/*/memory:chrome:renderer_processes:reported_by_chrome:v8:heap:effective_size_avg/*
*/*/*/memory:chrome:renderer_processes:reported_by_chrome:v8:heap:effective_size_max/*

The idea is to alert on any test that measures v8 heap. But there are two different kinds of tests:

Most telemetry tests have a format like:
Master/Bot/Benchmark/Metric/Page

But the system health ones have a format like:
Master/Bot/Benchmark/Metric/CATEGORY/Page

The system_health ones appear to be timing out, because multiple pages are being run by the bisect, since you're monitoring the entire category. The other monitoring for system_health benchmarks monitors the page.

I think the right way to do this is to specify the monitoring per-benchmark. Now that we have made some headway on go/why-merge-benchmarks, do we know all the names of the benchmarks we're trying to monitor for memory? As benchmark harnesses like media add memory metrics, these are added to the v8 memory sheriffing rotation. Is that intended? If not, I recommend specifing exact benchmarks to monitor, with configs that match the page.

In the short term, I will re-kick the bisects on those bugs with a specific page filter.

cc Ulan for feedback on monitoring config

Re: #c1

Yeah that sounds right, most of those are hitting the global 24 hr limit and we don't have a good way of surfacing that back to bisect yet, crbug.com/674945

https://uberchromegw.corp.google.com/i/internal.tryserver.clankium/builders/clankium-low-end-phone-perf-bisect/builds/309 is actually failing with an exception somewhere in buildbot code, we've seen that failure before, reported it already in crbug.com/728293

> As benchmark harnesses like media add memory metrics, these are added to the v8 memory sheriffing rotation. Is that intended?
Yes, the idea was to send v8 memory alerts for all benchmarks that use the "memoryMetric" to v8 memory sheriff rotation to reduce load on the general performance sheriff rotation.

If we specify the exact list of benchmarks to monitor, then we would have to keep that list in sync with benchmarks that use the "memoryMetric" manually?

Oh, I see!

In the short term, we do have to hard-code the benchmark names, but we're adding very few new benchmarks so it shouldn't be too much of an issue. The benchmarks currently producing data are:

media.android.tough_video_cases_tbmv2
media.tough_video_cases_tbmv2
memory.long_running_idle_gmail_background_tbmv2
memory.long_running_idle_gmail_tbmv2
oortonline_tbmv2
v8.browsing_desktop
v8.browsing_desktop_classic
v8.browsing_mobile
v8.browsing_mobile_classic
v8.infinite_scroll-classic_tbmv2
v8.infinite_scroll_tbmv2
v8.mobile_infinite_scroll-classic_tbmv2
v8.mobile_infinite_scroll_tbmv2

Do you want the metrics monitored for all of them?

In the long term, we are working towards coming up with a better way of configuring monitoring in Q3/Q4. It should be easy to see and change configurations, with code review, but there are a lot of edge cases to work through. We'll definitely remember this one and come back to you when we have a proposed design!

> Do you want the metrics monitored for all of them?
Yep.

> In the long term, we are working towards coming up with a better way of configuring monitoring in Q3/Q4. It should be easy to see and change configurations, with code review, but there are a lot of edge cases to work through.
Awesome, looking forward to it :)

Updated all the monitoring in the v8 memory sheriff rotation to list by-benchmark for now, so that we always alert on individual stories. This should make bisect faster and also catch problems that "average out" on multiple pages.