All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=706873

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgnOPPrwsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgnMra7wkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7MO-vQsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7MXu7wgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7K2esgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgrIGLqgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7KK3twkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgrI3AmAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgrNnejggM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7NTlrQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7Ias5AgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7JSpvwkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7IuLvAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgrPLgiQgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7KaMsAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgnMDcvAsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgrK6Y3wkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgrPfEnggM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7OSh4ggM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgrK-l9QoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7JmEugoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgrL7h-goM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgrN6A8AoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7Ij1sQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7KD47QkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7Nz28wgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7KbqqQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgrPnuqgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgrPf96QoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgrMbm9QkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgrK6YnwkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgrMmL9QsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7M_ngwkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7L-PogoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7I6Z-ggM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg7NL6tgoM


Bot(s) for this bug's original alert(s):

chromium-rel-win10
chromium-rel-win7-dual
chromium-rel-win7-x64-dual
chromium-rel-win8-dual

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8983677396771830560

=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : sugoi
  Commit : 7ae2ba1892a8b4dee8d7c231ac0e0fbe66588912
  Date   : Sat Mar 25 13:59:47 2017
  Subject: Solving some telemetry tests' slowness

Bisect Details
  Configuration: win_x64_perf_bisect
  Benchmark    : page_cycler_v2.intl_ar_fa_he
  Metric       : timeToFirstContentfulPaint_avg/pcv1-cold/http___www.islamweb.net_
  Change       : 123.10% | 605.059166667 -> 1349.90883333

Revision             Result                  N
chromium@459639      605.059 +- 218.968      6      good
chromium@459646      570.542 +- 65.4935      6      good
chromium@459650      532.223 +- 98.2581      6      good
chromium@459651      1712.26 +- 1149.12      6      bad       <--
chromium@459652      1347.82 +- 1007.27      6      bad
chromium@459653      1349.91 +- 1860.72      6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http...www.islamweb.net. page_cycler_v2.intl_ar_fa_he

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8983677396771830560

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6421788185067520


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

sugoi, can you take a look at this too?

So, from what I can see, almost all of these regressions are actually a return to the normal state.

4 of them seem to show an actual regression, all from the same website (farsnews.com). One possibility would be that this page may contain content that requires a GPU to render. If that's the case, then it would simply have not rendered that portion of the page at all before, and it would now render it using SwiftShader, which would explain why there's a difference (I can see the current page contains many animations, but I don't know what the recording contains).

Another possibility could be that some media is using the GPU when it shouldn't be to render itself (SwiftShader is for WebGL only). I'm working on a cl that would ensure that the GpuProcessHost contains the same info as the GpuDataManager here:
https://codereview.chromium.org/2781993002/
I don't think this is the problem we are seeing here, though.

It's likely the problem comes from having a GPU process when there was none before, but at this moment I don't know exactly what the issue is.

Thanks! Adding page_cycler_v2 owners kouhei and ksakamoto: anything more we should investigate here?

It's interesting that all affected tests are "cache cold" tests (pcv1-cold).

I'd love to see traces, but it looks like traces for only pcv1-warm runs are uploaded to cloud storage?

+Ethan, Ned: Comment 7 seems correct, I looked at the chartjson and it does seem we're only uploading traces for pcv1-warm: https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FWin_10_Perf%2F584%2F%2B%2Frecipes%2Fsteps%2Fpage_cycler_v2.intl_ja_zh_Dashboard_Upload%2F0%2Flogs%2Fjson.output%2F0

Any ideas?

Hmhh, this is because the "trace": {...} are keyed by page names, so there is no wonder  the entries of cold vs warm are merged together.

The trace should still be uploaded in the log. But this lead me tho think we probably should just kill the grouping key idea. It has been causing us too many headache.

I dig the log in https://chromium-swarm.appspot.com/task?id=351f3049f7144410&refresh=10&show_raw=1 to get a cold trace after the regression here: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_0-2017-03-25_15-16-17-47757.html

Thanks Ned, good to know pcv1-warm traces can be found from logs if necessary. :)

The 4 regressions from farsnews.com seem to have been recovered, so I think we can close this bug.

Closing per #11