📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/1406f7dfc40000

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/14e8d3c0240000

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/158f4800240000

😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/1406f7dfc40000

All of the attempts failed. See the individual attempts for details on each error.

😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/14e8d3c0240000

All of the attempts failed. See the individual attempts for details on each error.

😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/158f4800240000

All of the attempts failed. See the individual attempts for details on each error.

Seems renaming scheduler/base tracing category from renderer.scheduler to sequence_manager has caused that difference.
https://cs.chromium.org/chromium/src/tools/perf/benchmarks/oortonline.py?type=cs&q=renderer%5C.scheduler+file:perf&sq=package:chromium&l=38

Do we need TaskQueueManagerImpl and folks' traces for that benchmark?

📍 Pinpoint job started.
https://pinpoint-dot-chromeperf.appspot.com/job/14f48930240000

My recommendations here  is to reproduce the change locally, adjust the categories in the benchmarks until you get to a 100% match in your local results2.html, and then push those changes to the corresponding benchmarks.

For more details about this bug, here is the suspected graph changes:
https://chromeperf.appspot.com/report?sid=3bcb3d712a287b364424d4966990a7f89c2951fe97e2d8e36b547a4de3e58c7d&start_rev=554669&end_rev=557812

The suspected CL causing those changes:
https://chromium-review.googlesource.com/c/chromium/src/+/1052130

The suspected reason for the changed values is that the original category was used to calculate a set of TBMv2 metrics.
The metrics can be traced from the following search:
https://cs.chromium.org/search/?q=renderer.scheduler+f:tools/perf&type=cs

Also CCing dtu for the Pinpoint failures.

😿 Pinpoint job stopped with an error.
https://pinpoint-dot-chromeperf.appspot.com/job/14f48930240000

All of the attempts failed. See the individual attempts for details on each error.

Looking at Pinpoint failures:

Mac: I see "Could not find values matching: histogram: benchmark_duration"
The results histograms contain "benchmark_total_duration", not "benchmark_duration"
+eakuefner for advice

Linux: I see an error while running the test.
Emily fixed this on Monday. (https://chromium-review.googlesource.com/1057350)
But the fix was in Chromium, so if you bisect on old Chromium, you won't have the fix.
I have a workaround for this, but we would run into the "benchmark_duration" issue.

Back in office. Regarding the point about benchmark_duration and benchmark_total_duration, I think the best thing to do is probably to look for benchmark_total_duration if benchmark_duration is specified as the metric on which to bisect and we're reading histogram results, and vice-versa.

Yes, I can confirm that https://chromium-review.googlesource.com/1052130 causes the difference. So I don't see a problem then :)

Deleted: results_after.html 1.2 MB

results_after.html 1.2 MB View Download

Deleted: results_before.html 1.2 MB

results_before.html 1.2 MB View Download

From what I understand,
The benchmarks in question are currently reporting better values because that are inspecting less events (since they are checking the renderer.scheduler category that is shrinking), which makes the benchmark less accurate, and maybe measuring the wrong thing?
I think that might be OK, as long as you add the new category to those benchmarks.
I don't think that has happened yet since:
https://cs.chromium.org/chromium/src/tools/perf/measurements/task_execution_time.py?q=renderer.scheduler+f:tools/perf&dr=C&l=32 still has many places in the code pointing to the old category.

It might not be straight forward since the new category is more generic.

I think we should be able to add the new category to the list here:

https://cs.chromium.org/chromium/src/tools/perf/measurements/task_execution_time.py?rcl=0d3c6ea5cb8cb0afb1794b46dfeea8e8955a5cd9&l=25

I *think* the metrics should already be filtering the matching events by the thread they're interested in, so the fact that the category is more generic doesn't hurt.

Greg, could you put together a patch for this please?

What benchmark is using TaskExecutionTime? I thought we delete that