All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=776491

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=44cd37549faebc71f1fd08c33900361f3df42f61af4b288e7cef3493af5896b1


Bot(s) for this bug's original alert(s):

chromium-rel-mac11

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8965279721295020496

=== BISECT JOB RESULTS ===
NO Perf regression found

Bisect Details
  Configuration: mac_10_11_perf_bisect
  Benchmark    : system_health.common_desktop
  Metric       : total:500ms_window:renderer_eqt_max/load_chrome/load_chrome_blank

Revision             Result                  N
chromium@508541      8.89957 +- 61.5152      21      good
chromium@508633      8.35211 +- 56.9483      21      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.chrome.blank system_health.common_desktop

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8965279721295020496


For feedback, file a bug with component Speed>Bisection

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8965202172181642192

Retrying for a different alert in the group.

=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : Greg Kerr
  Commit : fcd7b94faa40dfc3e4e4be26f4e854dd14ef650c
  Date   : Thu Oct 12 22:48:50 2017
  Subject: Enable Finch Trial Test for MacV2Sandbox.

Bisect Details
  Configuration: mac_retina_perf_bisect
  Benchmark    : system_health.common_desktop
  Metric       : total:500ms_window:renderer_eqt_max/load_chrome/load_chrome_blank
  Change       : 682.89% | 1.1456514795 -> 8.96920668833

Revision             Result                   N
chromium@508507      1.14565 +- 0.740648      6      good
chromium@508511      1.05693 +- 1.26757       6      good
chromium@508512      1.0478 +- 1.10523        6      good
chromium@508513      9.1479 +- 30.7267        6      bad       <--
chromium@508514      8.47558 +- 28.9986       6      bad
chromium@508520      8.7049 +- 28.9147        6      bad
chromium@508533      8.71413 +- 28.3933       6      bad
chromium@508559      9.29809 +- 28.7734       6      bad
chromium@508611      8.96921 +- 26.415        6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.chrome.blank system_health.common_desktop

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8965202172181642192


For feedback, file a bug with component Speed>Bisection

I want to track this separately.

A couple observations:

1) Why is the original bad bisect time faster than the good time?
chromium@508541      8.89957 +- 61.5152      21      good
chromium@508633      8.35211 +- 56.9483      21      bad

2) It got 682% slower in the next bisect? Is that a glitch?

3) At any rate, I reproduced this locally with my performance improvements patch, and the local results are identical. Hopefully once I land that patch all these benchmarks will return to baseline.

Re #7:

1) The bisects ran on different bots. mac_10_11_perf_bisect is a desktop machine running 10.11. mac_retina_perf_bisect is a retina laptop also running 10.11. Maybe there is something different about the laptop config?

2) +tdresser for eqt metric, I know this was just turned on, any ideas why we have such a large % regression?

Re #2), this isn't that crazy. There was an improvement of about this magnitude back in July.

My guess is that this is a legitimate regression.

Thanks for the info. Any idea why the dashboards page is empty: https://chromeperf.appspot.com/group_report?bug_id=776491 ?

I'll want to monitor then when the improvement CL lands.

It's probably because all the alerts assigned to this bug were re-assigned to  crbug.com/775378  by the bisect job in #6 (the link you sent is searching alerts by bug_id).

simonhatch@ or sullivan@: is there a way of undoing #6? This has happened more than once that two different issues are marked as dups by bisect jobs and we wanted to undo it.

You can re-triage the alerts to the bug you want them on.

Ok, thanks for the information, as long as I can find the alerts somewhere this is fine.

I've had good luck with the other regression bugs, but I think I need some help understanding this one. What exactly does this measure ("EQT")?

I couldn't find any measurable difference here, with my CL or without, so I tried on the bots and saw the same result.

When run with a pageset-repeat of 5, it shows that reverting my CL actually makes this benchmark slightly slower than TOT, which doesn't add up. Given that this is a 700% regression, it shouldn't be hard to reproduce.

The runs are here:
https://build.chromium.org/p/tryserver.chromium.perf/builders/mac_10_11_perf_bisect/builds/1874
https://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/2663

Is it possible something else entirely caused this slowdown in the metrics?

This metric measures main thread responsiveness, as described here:
https://docs.google.com/document/d/1Vgu7-R84Ym3lbfTRi98vpdspRr1UwORB4UV-p9K1FF0/edit#heading=h.qtmvlls54hz

We only recently added it to the dashboard, so it's possible there are still some issues to work out.

The next step here is to get a trace, but I'm having a hard time getting a trace off the dashboard.

If there are no other regressions, I'm fine with this not blocking your release.
If that seems reasonable to you, feel free to reassign this bug to me (to dig into the metric quality concerns).

Thanks. I'm still working through some other issues with my changes so I'll leave this bug assigned to me for now and we'll re-assess this at the end.

To get traces off the dashboard:
1) Go to the debug link, since it looks like the alerts have been re-triaged:
https://chromeperf.appspot.com/group_report?sid=44cd37549faebc71f1fd08c33900361f3df42f61af4b288e7cef3493af5896b1
2) Click on a point before the regression, and click "View trace from this run ": https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/load_chrome_blank_2017-10-12_21-58-45_65855.html
3) Click on a point after the regression, and click "View trace from this run ":
https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/load_chrome_blank_2017-10-13_03-39-21_70951.html 

I need additional categories to make sense of the trace.

There is a large task present in the "after" trace that isn't present in the "before" trace, but I can't identify what it is.

I assume the "Debug Trace" button is supposed to handle the use case of getting traces with additional categories?

Yep! Simon is working on fixing it for pinpoint.

Is this still an issue?

Friendly ping

Thanks for the ping. I've got a CL up for this, and in the meantime I've deployed a version of Pinpoint with that Cl that surfaces the traces.

I think the debug trace button is fully working now? So we can get a trace? Tim, is that still on your plate?

Kicked off traces:
Before: https://pinpoint-dot-chromeperf.appspot.com/job/129d0da8840000
After:  https://pinpoint-dot-chromeperf.appspot.com/job/1483b698840000

I don't see anything in the pinpoint UI showing the revision it's running on though.

I sat there refreshing a few times until I realized the chart was far off to the right.

You should be able to click a point on the chart and see the revision it's on.

Filed:

https://github.com/catapult-project/catapult/issues/4180 - For the chart off to the right
https://github.com/catapult-project/catapult/issues/4181 - For doing a quick check on the revisions

Marking WontFix since the Finch data suggests the mild performance regressions is warranted for the security improvements.