Issue 778807  has been merged into this issue.

Note that a bunch of benchmarks' runtime duration regressed heavily as well: https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgx5ilswoM

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8964636886560192128

 issue 778135  tracks general perf impact of the switch.

last time we switched (in m62), we didn't see this. Did change anything?

Can anyone profile telemetry_unittests with both compilers and check where the difference is?

s/Did change anything/Did anything change wrt how telemetry_unittests are built/

While waiting for the bisect, here are the before & after traces:

Before: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/browse_media_youtube_2017-10-24_06-57-54_56880.html

After: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/browse_media_youtube_2017-10-24_13-03-24_7400.html


I notice that in before trace, DesktopBrowserBackend.GetSystemInfo was taking 4ms, while in the after trace, DesktopBrowserBackend.GetSystemInfo was taking 5000ms (1000X regression)

+kbr@ in case you notice any slow down in GPU tests

We have 118 hits for 'GetSystemInfo', pointing to several different symbols. Which one are you referring to? (Can't see your traces, off corp atm)

GetSystemInfo relates to https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/browser/browser.py?rcl=5da4837140ae16586b2b9097efbb08a7f4726a05&l=339

It essentially makes a devtool connection & fetch 'SystemInfo.getInfo' https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/system_info_backend.py?rcl=5da4837140ae16586b2b9097efbb08a7f4726a05&l=14

Do you know which c++ code this ends up calling?

I believe that ends up calling https://cs.chromium.org/chromium/src/content/browser/devtools/protocol/system_info_handler.cc

jbudorick, can you link to a build that shows the slow telemetry_unittests behaviour, and ideally which test it is that slowed down?


> I notice that in before trace, DesktopBrowserBackend.GetSystemInfo was taking 4ms, while in the after trace, DesktopBrowserBackend.GetSystemInfo was taking 5000ms (1000X regression)

Thanks! That suggests it's hitting the kGPUInfoWatchdogTimeoutMs timeout: https://cs.chromium.org/chromium/src/content/browser/devtools/protocol/system_info_handler.cc?rcl=f0dfc858de0d129a4e8a013ae9f870b710fc2d80&l=31

=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : Zhenyao Mo
  Commit : 0728a85617eae06098373349af80c28ce26b60fd
  Date   : Tue Oct 24 11:16:56 2017
  Subject: One step further into moving GPU feature decisions to GPU process

Bisect Details
  Configuration: win_8_perf_bisect
  Benchmark    : smoothness.tough_scrolling_cases
  Metric       : benchmark_duration/benchmark_duration
  Change       : 45.13% | 11.9583611098 -> 17.3554694447

Revision             Result                   N
chromium@511078      11.9584 +- 2.22448       6      good
chromium@511087      11.9043 +- 1.801         6      good
chromium@511088      11.8134 +- 1.04186       6      good
chromium@511089      17.1544 +- 1.11707       6      bad       <--
chromium@511090      17.2969 +- 0.750019      6      bad
chromium@511092      17.3596 +- 1.09234       6      bad
chromium@511096      17.1842 +- 0.8886        6      bad
chromium@511114      17.4095 +- 1.22446       6      bad
chromium@511150      17.3555 +- 1.37406       6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests smoothness.tough_scrolling_cases

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8964636886560192128


For feedback, file a bug with component Speed>Bisection

#13: a passing run from Monday, pre-regression, where both swarming shards complete in <6 minutes: https://build.chromium.org/p/chromium.win/builders/Win7%20Tests%20%281%29/builds/72715

a passing run from Tuesday, post-regression, where both swarming shards complete in >20 minutes: https://build.chromium.org/p/chromium.win/builders/Win7%20Tests%20%281%29/builds/72749

Haven't looked into the particulars of what slowed down.

Issue 777565 has been merged into this issue.
Issue 777569 has been merged into this issue.
Issue 777963 has been merged into this issue.
Issue 777964 has been merged into this issue.
Issue 777970 has been merged into this issue.