Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9007157752248707344

We reverted the GN flip and I don't see the numbers going back up, so I doubt that's it.

=== Auto-CCing suspected CL author sebmarchand@chromium.org ===

Hi sebmarchand@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Disable Full-WPO for the x64 official build (non-PGO).
Author  : sebmarchand
Commit description:
  
Disable Full-WPO for the x64 official builds (this doesn't
affect the PGO builds).

This should help me to get an official
"non-PGO & non-WPO" build that I can compare against an official PGO
build.

Also I'm not sure if it's worth keeping WPO turned on for the official
builds at the moment ? We don't ship these builds (we ship the PGO ones)
and so this only slowdown the builders that build an official x64 build
for testing purposes (especially considering that they need to link all the unittests with Full-WPO)

BUG= 490934 , 617982

Review-Url: https://codereview.chromium.org/2142603002
Cr-Commit-Position: refs/heads/master@{#405164}
Commit  : 0d74d7f71c93bfc3b32653de7958ff84d173937d
Date    : Wed Jul 13 15:42:28 2016


===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N  Good?
chromium@405135  455658  590.134  5  good
chromium@405151  455674  775.894  5  good
chromium@405161  452279  7684.99  5  good
chromium@405163  454205  2479.76  5  good
chromium@405164  302921  235.366  5  bad    <--
chromium@405166  295298  4657.42  5  bad
chromium@405171  297828  328.895  5  bad
chromium@405206  307006  854.933  5  bad

Bisect job ran on: winx64nvidia_perf_bisect
Bug ID: 628212

Test Command: .\src\out\Release_x64\angle_perftests.exe --test-launcher-print-test-stdio=always --test-launcher-jobs=1
Test Metric: IndexDataManger_run/score
Relative Change: 32.62%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64nvidia_perf_bisect/builds/1717
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9007157752248707344


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5653981943037952

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Sorry for that, I've disabled Full-WPO for the 64-bit build of Chrome (it has been turned on ~1 month ago) because we don't ship this configuration anymore (we ship PGO instead) and it has a huge impact on the build time on the perf bots, so it's not worth keeping it. 

Also, having the perf bots look at the performance numbers with WPO turned on probably doesn't make sense, the impact of the WPO optimizations can't really be affected by a single CL and so it just change our baseline.

Seb, are you saying you get a worse idea of "does this CL regress performance" with WPO turned on?

Yeah, for 2 reasons:
- The build time is significantly longer, so it'll take more time to bisect and to find the culprit.
- The baseline is a smaller score, so a CL that has a X % perf impact will result in a smaller absolute variation.

Hm, those reasons aren't that compelling to me. Will WPO on or off affect noise in the test results?

Well, WPO will just make the score look better but that's all. As we don't ship this configuration it'll just give us a better score on the bot but it won't really measure what we're shipping and it won't help finding a regression.