All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=649994

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICghc6OrwkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICghZqFqgoM


Bot(s) for this bug's original alert(s):

android-galaxy-s5
android-nexus6

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9000579440972945632

=== Auto-CCing suspected CL author mvstanton@chromium.org ===

Hi mvstanton@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : [TypeFeedbackVector] special ic slots for interpreter compare/binary ops.
Author  : mvstanton
Commit description:
  
Full code uses patching ICs for this feedback, and the interpreter uses
the type feedback vector. It's a good idea to code the vector slots
appropriately as ICs so that the runtime profiler can better gauge if
the function is ready for tiering up from Ignition to TurboFan.

As is, the feedback is stored in "general" slots which can't be
characterized by the runtime profiler into feedback states.

This CL addresses that problem. Note that it's also important to
carefully exclude these slots from the profiler's consideration when
determining if you want to optimize from Full code.

BUG=

Review-Url: https://codereview.chromium.org/2342853002
Cr-Commit-Position: refs/heads/master@{#39555}
Commit  : b88d132f4cbc4a7f4de106542ae5895079049070
Date    : Tue Sep 20 13:54:51 2016


===== TESTED REVISIONS =====
Revision                       Mean    Std Dev  N  Good?
chromium@420006                686.8   2.77489  5  good
chromium@420026                679.4   7.09225  5  good
chromium@420027                686.8   5.63028  5  good
chromium@420027,v8@b88df1c9e4  676.0   8.33667  5  good
chromium@420027,v8@05a00a93b1  678.2   3.03315  5  good
chromium@420027,v8@b88d132f4c  897.6   1.94936  5  bad    <--
chromium@420027,v8@e05e735a8a  994.6   2.70185  5  bad
chromium@420028                1000.4  3.36155  5  bad
chromium@420029                993.8   4.54973  5  bad
chromium@420031                996.6   4.27785  5  bad
chromium@420036                997.4   4.5607   5  bad
chromium@420046                767.8   3.34664  5  bad
chromium@420086                768.4   5.17687  5  bad

Bisect job ran on: android_s5_perf_bisect
Bug ID: 649994

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests sunspider
Test Metric: Total/Total
Relative Change: 11.88%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1031
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9000579440972945632


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6183108078993408

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Perf sheriff ping

mvstanton@, do you think that your CL above could have affected the results of the sunspider benchmark on android?

Friendly perf-sheriff ping, any update?

Hmm, it does look like it from the graphs. But these new IC slots are only used in the interpreter which shouldn't be running here.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8997662786016904992

triggered some non-archive bisect builds. If they show the same result we can rule out device flakyness.

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1125
Failure reason: the build has failed due to infrastructure failure.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8997660729168060928

=== Auto-CCing suspected CL author mvstanton@chromium.org ===

Hi mvstanton@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : [TypeFeedbackVector] special ic slots for interpreter compare/binary ops.
Author  : mvstanton
Commit description:
  
Full code uses patching ICs for this feedback, and the interpreter uses
the type feedback vector. It's a good idea to code the vector slots
appropriately as ICs so that the runtime profiler can better gauge if
the function is ready for tiering up from Ignition to TurboFan.

As is, the feedback is stored in "general" slots which can't be
characterized by the runtime profiler into feedback states.

This CL addresses that problem. Note that it's also important to
carefully exclude these slots from the profiler's consideration when
determining if you want to optimize from Full code.

BUG=

Review-Url: https://codereview.chromium.org/2342853002
Cr-Commit-Position: refs/heads/master@{#39555}
Commit  : b88d132f4cbc4a7f4de106542ae5895079049070
Date    : Tue Sep 20 13:54:51 2016


===== TESTED REVISIONS =====
Revision                       Mean    Std Dev  N  Good?
chromium@420006                854.4   7.76531  5  good
chromium@420026                848.8   5.93296  5  good
chromium@420027                846.4   8.32466  5  good
chromium@420027,v8@b88df1c9e4  850.8   5.63028  5  good
chromium@420027,v8@05a00a93b1  850.2   4.32435  5  good
chromium@420027,v8@b88d132f4c  1163.4  6.91375  5  bad    <--
chromium@420027,v8@e05e735a8a  1215.2  6.01664  5  bad
chromium@420028                1220.8  5.26308  5  bad
chromium@420029                1222.0  10.2225  5  bad
chromium@420031                1226.4  4.15933  5  bad
chromium@420036                1226.2  5.06952  5  bad
chromium@420046                899.4   3.84708  5  bad
chromium@420086                899.8   3.96232  5  bad

Bisect job ran on: android_nexus6_perf_bisect
Bug ID: 649994

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests sunspider
Test Metric: Total/Total
Relative Change: 5.31%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus6_perf_bisect/builds/2696
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8997660729168060928


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5779266514452480

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

This CL introduced a special kind of IC for interpreted code and we aren't even using that. However the CL had a bug, fixed a few hours later.

Funny, one of the two graphs recovered completely:

                                    before during after
Android-nexus6/sunspider/Total    - 850    900    863
Android-galaxy-s5/sunspider/Total - 689    771    806

So android-galaxy-s5 we have some problem still (which got even worse)...

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8997652353360623168

===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@420006  812.611  128.324  18  good
chromium@420119  899.056  140.35   18  bad

Bisect job ran on: android_s5_perf_bisect
Bug ID: 649994

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests sunspider
Test Metric: Total/Total
Relative Change: 11.21%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1126
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8997652353360623168


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5061382326714368

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

We've decommissioned the s5 bot. Not sure if we can make progress here?

Probably not.

These will all be invalid as they are specific to Samsung S5s, which we do not trust to give us a signal and have subsequently removed from the waterfalls. If you think this regression is real, please re-open.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982580376663200816