Also cc-ing Dave, Roberto, and Charlie who all looked into this on their shifts.

The range is actually much wider: r384275-r384610

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/502afb658d9b946b123ced6d0c91c444adb23f19

commit 502afb658d9b946b123ced6d0c91c444adb23f19
Author: petrcermak <petrcermak@chromium.org>
Date: Wed Apr 20 17:43:37 2016

Disable rasterize_and_record_micro.top_25_smooth on Linux and Android

Rationale: The benchmark is causing persistent redness on one Linux and
several Android perf bots:

https://build.chromium.org/p/chromium.perf/builders/Linux%20Perf%20(5)
https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf%20(2)
https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20(2)
https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus7v2%20Perf%20(2)
https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus6%20Perf%20(2)
https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus9%20Perf%20(2)
https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus9%20Perf%20(2)

TBR=skyostil
BUG= 600377 , 605120 
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq

Review URL: https://codereview.chromium.org/1907463002

Cr-Commit-Position: refs/heads/master@{#388528}

[modify] https://crrev.com/502afb658d9b946b123ced6d0c91c444adb23f19/tools/perf/benchmarks/rasterize_and_record_micro.py

Manually bisecting the issue is much slower than I expected. Anyway, it seems to me that I managed to narrow the range down to r384568-r384610. If anyone wants to continue before tomorrow, here are the commands that I used:

gclient sync -n -r src@HASH_TO_TEST
git cl patch 1851843002 && gclient runhooks && cr install chrome_public
tools/perf/run_benchmark -v --browser=android-chromium smoothness.image_decoding_case

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/523707adcb03a0fa6123831eaa8cad0048f911a4

commit 523707adcb03a0fa6123831eaa8cad0048f911a4
Author: skyostil <skyostil@chromium.org>
Date: Thu Apr 21 10:10:53 2016

Revert of Disable rasterize_and_record_micro.top_25_smooth on Linux and Android (patchset #2 id:20001 of https://codereview.chromium.org/1907463002/ )

Reason for revert:
Benchmark fixed with https://codereview.chromium.org/1898813002.

Original issue's description:
> Disable rasterize_and_record_micro.top_25_smooth on Linux and Android
>
> Rationale: The benchmark is causing persistent redness on one Linux and
> several Android perf bots:
>
> https://build.chromium.org/p/chromium.perf/builders/Linux%20Perf%20(5)
> https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf%20(2)
> https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20(2)
> https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus7v2%20Perf%20(2)
> https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus6%20Perf%20(2)
> https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus9%20Perf%20(2)
> https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus9%20Perf%20(2)
>
> TBR=skyostil
> BUG= 600377 , 605120 
> CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq

TBR=skyostil@google.com,petrcermak@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 600377 , 605120 

Review URL: https://codereview.chromium.org/1903903003

Cr-Commit-Position: refs/heads/master@{#388734}

[modify] https://crrev.com/523707adcb03a0fa6123831eaa8cad0048f911a4/tools/perf/benchmarks/rasterize_and_record_micro.py

Still bisecting this...

I found that all the key_mobile_sites[_smooth] test failures were due to the following v8 patch (+cc slan):

https://chromium.googlesource.com/v8/v8/+/86357d5235ceba61c151f0b6e509bcb365860454

In the meantime, these test failures were fixed in the 388802:388814 range (https://chromium.googlesource.com/chromium/src/+log/76d38030ee8557b813673881a7a6804fbb70b14a..c740e34d19834b978e7f6cfc91ec1d07b5e64b03, I suspect it was a V8 roll https://chromium.googlesource.com/chromium/src/+/673432ba1135aad6e63b682ea2b0112750257b80).

This leaves the following tests crashing:

https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf%20(1)
  sunspider

https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf%20(2):
  smoothness.image_decoding_cases
  rasterize_and_record_micro.top_25_smooth
  webrtc.webrtc_smoothness

https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf%20(3)
  v8.google

+cc slan

Unfortunately, I don't have the bandwidth to look into the remaining issues this week. To reproduce the key_mobile_sites[_smooth] issues, I used the following gn config:

ffmpeg_branding = "Chrome"
is_chrome_branded = true
is_debug = false
is_official_build = true
proprietary_codecs = true
symbol_level = 1
target_os = "android"
use_goma = true

And used the attached script to test individual Chrome revisions (note that I use cr with the out folder set to out_android32/Release):

./try_revision.sh 388804

Deleted: try_revision.sh 1016 bytes

try_revision.sh 1016 bytes View Download

Thanks, Petr! Assigning to Fadi (current sheriff)

Note: The change in #8 has been reverted to fix  crbug.com/604422 

This was picked up in today's v8 roll: https://chromium.googlesource.com/chromium/src/+/48d42d5a041a1257c3a98f898f385379b7ce74f5

Fadi, are you still looking at this? I'm still seeing a lot of those failures.

Actually I got busy, but I am back on it Today. I will update the status on it at end of day.

Thanks Fadi :-)

I'm not quite sure what to do here since there are different steps that are red on different builders; some builders have just a few steps red (e.g. https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus6%20Perf%20%281%29) and some have nothing failing currently (e.g. https://build.chromium.org/p/chromium.perf/builders/Android%20One%20Perf%20%283%29)

And the set of tests that are failing have changed since April.

Is the solution just to go through all of the benchmarks that appear to be failing in the past 5-10 runs and disable them?

First, I am filing bugs for all offline devices like  http://crbug.com/609497   http://crbug.com/609499  and  http://crbug.com/609507 

Next, I plan to disable the ones that are benchmark failing and rename them to Fix and re-enable.

Given that many of the benchmarks started failing simultaneously, I don't think that this is a benchmark issue. Instead, I think that we should find the patch that caused all this. I know that it's hard, but by disabling the benchmarks, we ignore something that might be a genuine Chrome bug (rather than a benchmark bug).

Thanks for the feedback, I will create a new list of benchmarks, I think this time it is much smaller, and then decide, I like your approach better.

I have checked all the consitent failures, and there is 4 tests falling across devices:
smoothness.image_decoding_cases
rasterize_and_record_micro.top_25_smooth
webrtc.webrtc_smoothness

And more recently:
smoothness.gpu_rasterization.tough_pinch_zoom_cases

I have created this spreadsheet to track Android failures, if you are perf sheriff and not sure whether you should file a bug for an Android failure, please consult this sheet:
https://docs.google.com/spreadsheets/d/1rP8vL9U1iOMZXDsoTYkrQNv7b3iw0Cw8d4ummYM3klM/edit#gid=0

I will disable them despite them being possible chrome bugs as it makes the job of chromium sheriff very hard.
I will create a separate fix and re-enable bug for each to track individually.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fd1d68375707e9dbfd7a5804b879828778e9276f

commit fd1d68375707e9dbfd7a5804b879828778e9276f
Author: fmeawad <fmeawad@chromium.org>
Date: Sat May 07 12:44:14 2016

[ChromiumPerf] Disable failing android benchmarks on ChromiumPerf

BUG= 605120 , 610015 , 610018 , 610019 , 610021 
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq

Review-Url: https://codereview.chromium.org/1960863002
Cr-Commit-Position: refs/heads/master@{#392255}

[modify] https://crrev.com/fd1d68375707e9dbfd7a5804b879828778e9276f/tools/perf/benchmarks/rasterize_and_record_micro.py
[modify] https://crrev.com/fd1d68375707e9dbfd7a5804b879828778e9276f/tools/perf/benchmarks/smoothness.py
[modify] https://crrev.com/fd1d68375707e9dbfd7a5804b879828778e9276f/tools/perf/benchmarks/webrtc.py

Pri-0 bugs are critical regressions or serious emergencies, and this bug has not been updated in three days. Could you please provide an update, or adjust the priority to a more appropriate level if applicable?

If a fix is in active development, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Perf Bot sheriff ping: Looks like after disabling failing tests, android red sea is turning green, although the root cause of these failures still need to be fixed.
Fadi@ Is it ok to downgrade this priority to P1?  

Degrading to Pri-2 as the tests are already disabled.

Future sheriffs can use the spreadsheet created by Fadi in comment 20 to track the status.

The original regressions tracked in this issue are now fixed.