Escalating priority due to the nastiness on the optional android trybots and GPU FYI bots. CC'ing two people on the latest Catapult roll.

Recent rolls:

https://codereview.chromium.org/2562673002
https://codereview.chromium.org/2558203002
https://codereview.chromium.org/2560633003

this is likely related to  issue 672382 

Yeah, probably a duplicate. Merging.

Note: the fact that android_optional_gpu_tests_rel is so flaky now probably means that we should temporarily remove it from the CQ_INCLUDE_TRYBOTS lines from various presubmit and roll scripts. Stephen, could you do a code search for that bot and remove it, filing a new bug about it and pointing to it?

One of the places is in Skia infra.
https://skia-review.googlesource.com/c/2420/

Maybe it's easier to remove all the tests from the bot?

It would be useful in the future to have the ability to make *_optional_gpu_tests_rel experimental, like https://codereview.chromium.org/2559313002 does for android_n5x_swarming_rel.

Yes, that would be useful -- unfortunately it's not easy. Only if a bot is on a given CQ can it be marked experimental. These are being added via CQ_INCLUDE_TRYBOTS.

I agree, the easiest thing might be to just remove all tests from "Android Release (Nexus 5X)" for the moment. That will have the effect of removing them from android_optional_gpu_tests_rel, which mirrors that waterfall bot per tools/build/scripts/slave/recipe_modules/chromium_tests/trybots.py . There's no way to exclude them from android_optional_gpu_tests_rel and not the waterfall bot. If you'd like to put together a CL which does that I'll gladly review it.

Jamie and I took a shot at this, but realized we don't really have the knowledge here.

ynovikov@, is this something you could take on (disabling all the tests on N5X)?

Actually, Jamie is going to give it a shot after all.

In review: https://codereview.chromium.org/2562133002/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9b5a07979aa2dcfd2698646be9b3c8a6ed6768dd

commit 9b5a07979aa2dcfd2698646be9b3c8a6ed6768dd
Author: jmadill <jmadill@chromium.org>
Date: Fri Dec 09 22:57:22 2016

Disable all GPU tests on N5X bot temporarily.

This bot config is having stability problems - disable all tests until
the issue is resolved.

BUG= 672502 
NOTRY=true
NOPRESUBMIT=true
R=kbr@chromium.org
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2562133002
Cr-Commit-Position: refs/heads/master@{#437676}

[modify] https://crrev.com/9b5a07979aa2dcfd2698646be9b3c8a6ed6768dd/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/9b5a07979aa2dcfd2698646be9b3c8a6ed6768dd/testing/buildbot/chromium.gpu.fyi.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/da8379afe5d31e6f166e53865fc1f69fac2f3bda

commit da8379afe5d31e6f166e53865fc1f69fac2f3bda
Author: ynovikov <ynovikov@chromium.org>
Date: Wed Dec 14 21:43:29 2016

Revert of Disable all GPU tests on N5X bot temporarily. (patchset #1 id:1 of https://codereview.chromium.org/2562133002/ )

Reason for revert:
N5X problems might be fixed. Reverting to test that assumption.

Original issue's description:
> Disable all GPU tests on N5X bot temporarily.
>
> This bot config is having stability problems - disable all tests until
> the issue is resolved.
>
> BUG= 672502 
> NOTRY=true
> NOPRESUBMIT=true
> R=kbr@chromium.org
> CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
>
> Committed: https://crrev.com/9b5a07979aa2dcfd2698646be9b3c8a6ed6768dd
> Cr-Commit-Position: refs/heads/master@{#437676}

TBR=kbr@chromium.org,jmadill@chromium.org
NOTRY=true
NOPRESUBMIT=true
BUG= 672502 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2577793002
Cr-Commit-Position: refs/heads/master@{#438636}

[modify] https://crrev.com/da8379afe5d31e6f166e53865fc1f69fac2f3bda/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/da8379afe5d31e6f166e53865fc1f69fac2f3bda/testing/buildbot/chromium.gpu.fyi.json

AFAICT this still happens:
https://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28Nexus%205X%29/builds/5124
https://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28Nexus%205X%29/builds/5114
https://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28Nexus%205X%29/builds/5113

No, this bug had a very different failure pattern.
As to the failures in #16, I think WebglConformance_conformance_textures_image_bitmap_* should be marked as flaky on Nexus 5X.

P1 bug ping, what's the remaining work here?

Since  issue 672382  is fixed, then this is too.