Can anyone tell me how to update the expectations for these?

This failure is distressing. It looks like both of the NVIDIA bots on the chromium.gpu.fyi waterfall started failing:
https://luci-milo.appspot.com/buildbot/chromium.gpu.fyi/Android%20Release%20%28NVIDIA%20Shield%20TV%29/?limit=200
https://luci-milo.appspot.com/buildbot/chromium.gpu.fyi/Android%20Release%20%28Nexus%209%29/?limit=200

Here are the first failing builds on each:
https://luci-milo.appspot.com/buildbot/chromium.gpu.fyi/Android%20Release%20%28Nexus%209%29/7437
https://luci-milo.appspot.com/buildbot/chromium.gpu.fyi/Android%20Release%20%28NVIDIA%20Shield%20TV%29/2165

This looks like a bug in the JavaScript engine where either the compiler or garbage collector are doing something wrong (corrupting an argument to the WebGLRenderingContext.attachShader method) and the bindings are managing to catch it rather than crash.

This might be attributable to this V8 roll:
https://chromium.googlesource.com/v8/v8/+log/99a6c585..74cef601

but that only shows up in the blamelist for the Nexus 9 bot, and not the Shield TV bot.

Might be attributable to this ANGLE roll:
https://chromium.googlesource.com/angle/angle.git/+log/27a6063..70c95fa

but that only shows up in the blamelist for the Shield TV bot and not the Nexus 9 bot.

Couldn't reproduce on my personal Shield tablet.

A colleague loaned me a Nexus 9 which I flashed to NMF26F (NYC MR1, release-keys) and ran Chromium at r485695; wasn't able to reproduce the failures. Still trying.

The N9 on the waterfall is still running Android M (I think because of stability issues with N when it was last tested) but the Shield TV is running N.

These failures are concerning because they've reliably regressed and there's no good explanation for them.

V8 team: do you have a Nexus 9 on which you might try reproducing these failures?

Suppressing the failures in https://chromium-review.googlesource.com/568909/ .

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1fedb7496bda617c92432b17c6c5f89417d3d380

commit 1fedb7496bda617c92432b17c6c5f89417d3d380
Author: Kenneth Russell <kbr@chromium.org>
Date: Thu Jul 13 00:11:48 2017

Suppress attachShader API failures on Android/NVIDIA devices.

BUG= 740769 
TBR=zmo@chromium.org

Change-Id: I73c1a95ad06c6f5e0eecd32c6be691840f26afce
Reviewed-on: https://chromium-review.googlesource.com/568909
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#486171}
[modify] https://crrev.com/1fedb7496bda617c92432b17c6c5f89417d3d380/content/test/gpu/gpu_tests/webgl_conformance_expectations.py

I flashed a Nexus 9 to a userdebug build of N2G47S and ran webgl_conformance_tests via Telemetry against a Chromium build at r485695 with basically the same command line arguments as on the bots, but with --test-filter=conformance_ogles_GL_functions . The failures seen on these bots didn't reproduce.

I'm still quite concerned about this regression as it seems that it could be a bug in either the register allocator or garbage collector, but I'm unable to reproduce it locally and think that others won't be able to, either. The suppressions above have taken effect on the N9 bot and will on the Shield shortly, so I'm closing this as WontFix.

I have Nexus9 and will try to reproduce after flashing it. Did you use the following command line?
./content/test/gpu/run_gpu_integration_test.py webgl_conformance --show-stdout --browser=android-chromium --passthrough -v --extra-browser-args="--enable-logging=stderr --js-flags=--expose-gc" /tmp/tmpVsOFhw.json

> but with --test-filter=conformance_ogles_GL_functions . 
Crashes I investigated in the past never reproduced with --test-filter. They required running the whole test suite.

Telemetry supports auto bisects based on the return code. Is it possible to bisect on the bot here?

Ulan, thanks for trying to repro this. Given how reliably reproducible it is on the bots, I would have thought the individual tests would fail, since the browser restarts after each failure.

I've never tried using the bisect bots with our Telemetry based suite. However, unfortunately, these devices are one-offs on the chromium.gpu.fyi waterfall, and I don't think we have bisect bots in that configuration.

One more wrinkle. The Nexus 9 on the waterfall is running Android M -- I think there were stability problems with N. I only tried Android N on the Nexus 9. However, the Shield TV is running a developer version of Android N and it reproduced the problem in exactly the same way.

There's a chance this may be a problem specific to NVIDIA's OpenGL ES driver where for some reason one of our shaders now fails to compile. It's possible I didn't test with the right driver version to provoke the bug.

Thanks again for taking a look. Let me assign this to you and downgrade it. You may need to remove the expectations I added in 1fedb7496bda617c92432b17c6c5f89417d3d380 in order to easily see the failures. Please close as WontFix again if you can't reproduce.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/88ab20412467348e8b4387cca3ee48c5ea302379

commit 88ab20412467348e8b4387cca3ee48c5ea302379
Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org>
Date: Fri Jul 14 01:23:09 2017

Roll src/third_party/catapult/ 6daeed6d7..2569648cd (14 commits)

https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/6daeed6d7311..2569648cdd4f

$ git log 6daeed6d7..2569648cd --date=short --no-merges --format='%ad %ae %s'
2017-07-13 simonhatch Pinpoint - First pass new job dialog
2017-07-13 benjhayden Fix CSVBuilder.
2017-07-13 benjhayden Reserve names for split diagnostics.
2017-07-13 dtu [pinpoint] Correct device affinity when multiple executions are started together.
2017-07-13 eakuefner [Catapult] Fix bin_checks PRESUBMIT not allowing deletions in bin/
2017-07-13 nednguyen Revert of Add logging to debug battor not running (patchset #1 id:1 of https://codereview.chromium.org/2869913002/ )
2017-07-13 sullivan Add support for additional flag-specified tracing categories.
2017-07-13 eakuefner [Dashboard] Make add_histograms an ApiRequestHandler
2017-07-13 etienneb Validate code-identifiers from traces/modules when available
2017-07-13 eakuefner [Tracing] Fix bin/run_tests
2017-07-13 kbr Fix run_browser_tests.py on Android.
2017-07-13 sullivan Add APIs to list the timeseries for a benchmark, and download them.
2017-07-13 sullivan Add alert timestamps, bisect start, and convert bug timestamps to ISO format.
2017-07-13 nednguyen Add smoke test for webpagereplay_go_server

Created with:
  roll-dep src/third_party/catapult
BUG= 719447 , 739783 , 740769 


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls


CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=sullivan@chromium.org

Change-Id: I44b1b5541e2c258f061e3138318a8c1af9cdcbc1
Reviewed-on: https://chromium-review.googlesource.com/570836
Reviewed-by: <catapult-deps-roller@chromium.org>
Commit-Queue: <catapult-deps-roller@chromium.org>
Cr-Commit-Position: refs/heads/master@{#486599}
[modify] https://crrev.com/88ab20412467348e8b4387cca3ee48c5ea302379/DEPS

I run the whole test suite on Nexus9 flashed with N2G47S at TOT (850575d) with revert of 1fedb74.

WebglConformance_conformance_ogles_GL_functions_functions failures did not reproduce. I am going to close this bug.

FWIW, I did reproduce an instance of  crbug.com/725559  on several other tests (log attached):
	07-13 13:51:00.610  8055  8123 E v8      : 
	07-13 13:51:00.610  8055  8123 E v8      : 
	07-13 13:51:00.610  8055  8123 E v8      : #
	07-13 13:51:00.610  8055  8123 E v8      : # Fatal error in ../../v8/src/compiler/register-allocator-verifier.cc, line 251
	07-13 13:51:00.610  8055  8123 E v8      : # 
	07-13 13:51:00.610  8055  8123 E v8      : Check failed: Immediately after CommitAssignmentPhase..
	07-13 13:51:00.610  8055  8123 E v8      : 
	07-13 13:51:00.610  8055  8123 E v8      : #


Command line:
./content/test/gpu/run_gpu_integration_test.py webgl_conformance --show-stdout --browser=android-chromium --passthrough -v --extra-browser-args="--enable-logging=stderr --js-flags=--expose-gc" /tmp/tmpVsOFhw.json

GN flags:
target_os="android"
use_goma = true
is_debug = false
dcheck_always_on = true

Deleted: webgl.txt 2.1 MB

webgl.txt 2.1 MB View Download

Thanks for trying Ulan.

Happily, these tests seem to be passing on the bots now.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a5f88b7e35a567c40e25e21b285ec7e8c687e6d6

commit a5f88b7e35a567c40e25e21b285ec7e8c687e6d6
Author: James Darpinian <jdarpinian@chromium.org>
Date: Mon Nov 12 23:09:19 2018

Cleaning up WebGL test failure expectations.

Many of these are leftover failure expectations from fixed bugs.
Some were fixed on some platforms but not others. There are some
new failures that were hidden by previous broad suppressions. Also
some tests are only failing because we haven't updated the drivers
on the bots (http://crbug.com/887241).

Bug:  757097 ,  838133 ,  672380 , 625738,  angleproject:2325 
Bug:  534697 , 849572, 625738, 703779,  angleproject:2142 
Bug: 757098, 644740, 705865, 602688,  angleproject:2103 
Bug: 903903,  740769 , 662644,  680720 ,  angleproject:1932 
Bug:  709874 , 887241,  625365 , angleproject:2952
Change-Id: I4e39e5a09d31d788d8a8615ad06d1ec88f064f00
Reviewed-on: https://chromium-review.googlesource.com/c/1325197
Commit-Queue: James Darpinian <jdarpinian@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#607367}
[modify] https://crrev.com/a5f88b7e35a567c40e25e21b285ec7e8c687e6d6/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py
[modify] https://crrev.com/a5f88b7e35a567c40e25e21b285ec7e8c687e6d6/content/test/gpu/gpu_tests/webgl_conformance_expectations.py
[modify] https://crrev.com/a5f88b7e35a567c40e25e21b285ec7e8c687e6d6/content/test/gpu/gpu_tests/webgl_conformance_expectations_unittest.py