This is the failure image for http://apeha.ru: https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Galaxy%20S5%20Perf%20(2)/builds/2168/steps/page_cycler.intl_hi_ru/logs/stdio. It looks shockingly like the "Welcome to your new Samsung Galaxy S5" screen in Russian.

... aaaaand this is the failure screen on Nexus 5X: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/profiler-file-id_1-2016-04-11_04-04-1012655.png. It looks like Chrome is crashing. Let's see what the return code bisect turns up.

Bisect started here: https://chromeperf.appspot.com/buildbucket_job_status/9015655227001637840

Ethan suggested lowering the repeat count to 5, which I did here: https://chromeperf.appspot.com/buildbucket_job_status/9015653032818127456

....buuut, then I accidentally made the timeout wrong. Fixed that and limited this to techcrunch.com (which was repeatedly failing) here: https://chromeperf.appspot.com/buildbucket_job_status/9015652330974178832

Yes, this is a comedy of errors on my part. Sigh.

=== Auto-CCing suspected CL author skyostil@chromium.org ===

Hi skyostil@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : variations: Also enable the RenderingPipelineThrottling feature
Author  : skyostil
Commit description:
  
In addition to enabling the RenderingPipelineThrottling field trial,
also enable the corresponding feature to actually test the feature on
the waterfall.

BUG= 487937 

Review URL: https://codereview.chromium.org/1845223002

Cr-Commit-Position: refs/heads/master@{#384293}
Commit  : 904b8f7f89b62c48bdaf78553fc3b0830f3ac16c
Date    : Thu Mar 31 16:04:23 2016


===== TESTED REVISIONS =====
Revision                Exit Code   Std. Dev.   Num Values  Good?
chromium@384274         0           N/A         5           good
chromium@384285         0           N/A         5           good
chromium@384290         0           N/A         5           good
chromium@384292         0           N/A         5           good
chromium@384293         1           N/A         5           bad         <-
chromium@384295         1           N/A         5           bad
chromium@384316         1           N/A         5           bad
chromium@384358         1           N/A         5           bad
chromium@384442         1           N/A         5           bad
chromium@384610         1           N/A         5           bad

Bisect job ran on: android_nexus5X_perf_bisect
Bug ID: 602340

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --also-run-disabled-tests rasterize_and_record_micro.top_25_smooth
Test Metric: pixels_rasterized/pixels_rasterized
Relative Change: Zero to non-zero
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5X_perf_bisect/builds/74
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9015653032818127456


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=602340

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : variations: Also enable the RenderingPipelineThrottling feature
Author  : skyostil
Commit description:
  
In addition to enabling the RenderingPipelineThrottling field trial,
also enable the corresponding feature to actually test the feature on
the waterfall.

BUG= 487937 

Review URL: https://codereview.chromium.org/1845223002

Cr-Commit-Position: refs/heads/master@{#384293}
Commit  : 904b8f7f89b62c48bdaf78553fc3b0830f3ac16c
Date    : Thu Mar 31 16:04:23 2016


===== TESTED REVISIONS =====
Revision                Exit Code   Std. Dev.   Num Values  Good?
chromium@384274         0           N/A         5           good
chromium@384285         0           N/A         5           good
chromium@384290         0           N/A         5           good
chromium@384292         0           N/A         5           good
chromium@384293         1           N/A         5           bad         <-
chromium@384295         1           N/A         5           bad
chromium@384316         1           N/A         5           bad
chromium@384358         1           N/A         5           bad
chromium@384442         1           N/A         5           bad
chromium@384610         1           N/A         5           bad

Bisect job ran on: android_nexus5X_perf_bisect
Bug ID: 602340

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --also-run-disabled-tests rasterize_and_record_micro.top_25_smooth
Test Metric: pixels_rasterized/http___techcrunch.com
Relative Change: Zero to non-zero
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5X_perf_bisect/builds/76
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9015652330974178832


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=602340

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Sami, are you looking at this failure? Also seems to be the cause for  bug 601765 .

So rasterize_and_record_micro was definitely broken by my patch (https://codereview.chromium.org/1845223002), but I later fixed it with https://codereview.chromium.org/1898813002. The remaining failure seems to about gmail failing to load, which seems unrelated.

Of course the test is passing perfectly fine locally, so I'll kick off another return code bisect with the later range.

Kicked off the bisect. (The dashboard didn't seem to give me a link to it :P)

Friendly sheriff ping,
Did the last bisect ran successfully?

(According to the Perf Bot Sheriffing guide, Pri-1 build bot bugs should be pinged daily. Please close the issue if it's already been fixed or lower its priority.)

If it ran successfully, it should have updated the bug. Should probably kick off another.

Hmm, so I kicked off a return code bisect here between 384295 and 392898 (ToT):

https://chromeperf.appspot.com/report?sid=97dec3b17fdf38d7b60144c5778967352ad5325ec013c32bf6e3e183cb383a33

The "VIEW JOB" link popped up, but clicking on it just takes me back to that same page.

The bot is now green. The failed benchmarks are either running ok now or tracked by other bugs. Closing this bug.