All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=772732

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=ad8238a2722b92e81f1b18cc5d66eebf9abf6562d45a70bd61bd0da3c6c5b3c2


Bot(s) for this bug's original alert(s):

chromium-rel-win7-gpu-nvidia

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8966288161280778992

=== Auto-CCing suspected CL author jdoerrie@chromium.org ===

Hi jdoerrie@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : jdoerrie
  Commit : 76cee9c97af4ab083737857679afe1053065149c
  Date   : Fri Oct 06 22:42:42 2017
  Subject: Rewrite base::Value::GetType to base::Value::type on Linux

Bisect Details
  Configuration: winx64nvidia_perf_bisect
  Benchmark    : angle_perftests
  Metric       : Uniforms_gl_null_400_vec4/score
  Change       : 39.06% | 11644.5 -> 7096.5

Revision             Result                  N
chromium@507152      11644.5 +- 493.811      6      good
chromium@507194      12480.7 +- 3540.67      6      good
chromium@507200      11737.0 +- 1621.25      6      good
chromium@507203      11955.8 +- 970.675      6      good
chromium@507204      7168.0 +- 287.44        6      bad       <--
chromium@507205      7066.83 +- 405.368      6      bad
chromium@507215      7148.33 +- 369.496      6      bad
chromium@507235      7096.5 +- 216.494       6      bad

To Run This Test
  .\src\out\Release_x64\angle_perftests.exe --test-launcher-print-test-stdio=always --test-launcher-jobs=1

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8966288161280778992


For feedback, file a bug with component Speed>Bisection

jdoerrie: would your change affect the test harness performance?

I didn't have a detailed look at the results yet, but I think it is highly unlikely my change caused the regression. My change only renamed usages of |base::Value::GetType()| to |base::Value::type()|. Both of these functions are equivalent and have the exact same implementation: https://codesearch.chromium.org/chromium/src/base/values.h?l=148-149&rcl=e5a5438e01450e1aea1df4cb6fc20de938432760

Looking at the graphs linked in the first post it seemed like the performance recovered. Maybe it was just a flake or another culprit was found and reverted?

It's a bit alarming we can have 40% regressions that are flakes.. closing as WontFix and cc'ing Speed team.

Both bisect & dashboard confirm the regression, so this is not a flake. 

I can see there are two actions we should take for this bug:
1) The benchmark's metric is correct, and this is a subtle performance bug.
2) The benchmark's metric is unreliable, and we should remove it.

jmadill@, since you are the owner of the benchmark, can you take lead in investigating whether this is (1) or (2)?

The things that could affect the benchmark's results are a) a change in the test harness which affects the results, b) huge variation between runs because of unrelated code changes affecting the binary or test runs somehow, or c) some kind of temporary change in the test machine that causes variation.

The benchmark is very reliable when I run it locally - but I use a script to increase iteration counts, make sure my system's environment is the same, and I don't use the test harness in Chrome (just ordinary GoogleTest).

I thought it might also be a change that could be in a prior or following chromium revision (that would need a nudge) but I didn't see any ANGLE rolls there, and the regression was strong enough it didn't see to be likely.

I mainly work in ANGLE, only dabbling in infra. Please feel free to follow up, so I'll leave it as available.  Or feel free to close it out. If it was a flake somehow due to the change in #3 it seems to have been resolved.

To clarify, the benchmark was run on a continuous build, and it shows a dip in the chart here:
https://chromeperf.appspot.com/group_report?bug_id=772732

Then that exact dip was reproduced when running the benchmark on a completely different machine with the same config in the bisect in #3. The bisect shows that the dip started with the culprit CL.

No changes to the framework were seen in the range.

To me, this points to a problem with the measurement in the test itself, not the framework.

OK, could I ask for help trying to reproduce the regression locally? There are no ANGLE changes in the regression range, and I'm at home today (actually it's a holiday in Canada) on a slow computer so building Chrome is a large endeavour. The exact test I'd do is try to confirm if the CL in comment #3 (76cee9c97af4ab083737857679afe1053065149c) also causes a regression locally. I agree though with jdoerrie that it would be very bizzarre.

Unless there's some other reason that could cause the measurement to be incorrect.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8966191328206456816

jmadill@, to reproduce the results locally, you can fetch the isolate to run the test:

Before regression:
https://chromium-swarm.appspot.com/user/task/390b5c6545021f10

After regression:
https://chromium-swarm.appspot.com/task?id=390ec49361504110


If you open these two links, you can see instruction on the page that looks like:

"Run this task locally:
python swarming.py reproduce -S chromium-swarm.appspot.com 390ec49361504110"

swarming.py script is in https://cs.chromium.org/chromium/src/tools/swarming_client/swarming.py


To tinker with 76cee9c97af4ab083737857679afe1053065149c & see how that affect the perf data locally, you will need to build locally. 

Kicked off bisect for the perf improvement, to gain more information.

If it does turn out to be pure variation in test results, it's a little discouraging that variation can be almost 100%. This particular machine does use a pretty old GPU, with old drivers, that we're in the process of upgrading. So that could be one improvement.

Another improvement we could consider is if we can devote more test time to increasing the iteration count - either just doubling it manually right now (or something similar) or by devoting a few bots specifically to ANGLE. It's not something I was going to bring up for a while but it could be a longer-term fix to high variance.

Looks like I can reliably reproduce the performance regression with the isolates. I'll try reproducing it locally but could use some help with that given my slow computer. 

Ned, were those isolates from the narrow bisect or the continuous builder?

I grab those isolates from the continuous builder.

Assign to  jmadill@ given the regression is reproducible.

I was not able to reproduce the regression in the bisected CL - I saw substantial variation at times but ultimately nothing consistent. Seeing as the CL only barely changes one or two relevant files to the test harness (https://chromium-review.googlesource.com/c/chromium/src/+/702456/2/base/json/json_writer.cc) and there's no ANGLE CLs in the regression range, I think the problem is the compiler is optimizing the binary slightly differently and producing better or worse results.

There's really no resolution here that I can do. I'm sure the perf team is very familiar with how difficult these issues can be.

=== Auto-CCing suspected CL author tzik@chromium.org ===

Hi tzik@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : tzik
  Commit : 7aacb29fbd32ce5b447b4c50b20ef7508caffbfb
  Date   : Sun Oct 08 23:26:14 2017
  Subject: Use DCHECK instead of assert on scoped_refptr assertions

Bisect Details
  Configuration: winx64nvidia_perf_bisect
  Benchmark    : angle_perftests
  Metric       : Uniforms_gl_null_400_vec4/score
  Change       : 61.83% | 7123.16666667 -> 11527.3333333

Revision             Result                  N
chromium@507308      7123.17 +- 330.295      6      good
chromium@507309      7144.33 +- 304.827      6      good
chromium@507310      12091.2 +- 3007.48      6      bad       <--
chromium@507311      11527.3 +- 840.256      6      bad

To Run This Test
  .\src\out\Release_x64\angle_perftests.exe --test-launcher-print-test-stdio=always --test-launcher-jobs=1

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8966191328206456816


For feedback, file a bug with component Speed>Bisection

Sorry tzik, that was actually a perf improvement not a regression.