Failure on Nexus 9:

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20FYI%20Release%20%28Nexus%209%29/2223

Will apply suppression to all Android.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e57301731de28f9571f4c777b888d2bc7cafbc35

commit e57301731de28f9571f4c777b888d2bc7cafbc35
Author: Jamie Madill <jmadill@chromium.org>
Date: Thu Apr 19 19:52:09 2018

Suppress flaky WebGL video tests on Android.

These tests appear to be flaky accross multiple configs. See the
crbug for more information.

Also merge these expectations with a Qualcomm expectation that can
be more general.

No-Try: True
Tbr: kbr@chromium.org
Bug: 834933, 752291
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: If54639a060eec63efe99247b9fac5ee1a94eff65
Reviewed-on: https://chromium-review.googlesource.com/1019874
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Cr-Commit-Position: refs/heads/master@{#552129}
[modify] https://crrev.com/e57301731de28f9571f4c777b888d2bc7cafbc35/content/test/gpu/gpu_tests/webgl_conformance_expectations.py

The flaky failure of WebglConformance_conformance_textures_video_tex_2d_rgba_rgba_unsigned_byte comes immediately after an expected failure of WebglConformance_conformance_textures_video_tex_2d_alpha_alpha_unsigned_byte where the GPU process crashes:

	04-18 20:15:31.178  9252  9268 F chromium: [FATAL:gles2_cmd_copy_texture_chromium.cc(287)] Check failed: false. 
	04-18 20:15:31.178  9252  9268 F chromium: #00 0x0000007f5db498ab /data/app/org.chromium.chrome-1/base.apk+0x00000000029768ab
	04-18 20:15:31.178  9252  9268 F chromium: #01 0x0000007f606e0daf /data/app/org.chromium.chrome-1/base.apk+0x000000000550ddaf
...

The browser is then restarted and this is the first test run after the restart.

Here is the log excerpt from https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Android%20FYI%20Release%20%28Nexus%205X%29/2383 
 where WebglConformance_conformance_textures_video_tex_2d_rgba_rgba_unsigned_byte failed:

AssertionError: shouldBe 0,255,0
at (4, 4) expected: 0,255,0 was 0,0,0

testing: video/mp4; codecs="avc1.42E01E, mp4a.40.2"
Testing texImage2D with flipY=true bindingTarget=TEXTURE_2D
Checking lower left corner
FAIL shouldBe 0,255,0
at (4, 4) expected: 0,255,0 was 0,0,0
Checking upper left corner
shouldBe 255,0,0
at (4, 24) expected: 255,0,0 was 0,0,0
FAIL shouldBe 255,0,0
at (4, 24) expected: 255,0,0 was 0,0,0
Testing texImage2D with flipY=false bindingTarget=TEXTURE_2D
Checking lower left corner
Checking upper left corner
Testing texSubImage2D with flipY=true bindingTarget=TEXTURE_2D
Checking lower left corner
Checking upper left corner
Testing texSubImage2D with flipY=false bindingTarget=TEXTURE_2D
Checking lower left corner
Checking upper left corner

Not sure whether something might have changed in the Android media stack recently which could have caused failures like this – i.e., if the events related to videos are delivered in a slightly different order than before?

+frank,cj for surfaces work; though I'm not sure it's enabled on Android. Frank did turn on MediaCodecVideoDecoder by default recently too though.

The NOTREACHED() being hit is concerning, it could end up as a security issue depending on where the invalid enum comes from

i don't believe that the surfaces work is turned on for android.

the flakiness dashboard doesn't have any data earlier than the 18th. 
 unfortunately, that's also when i turned on mcvd by default.

so, don't know which came first.

locally, with MCVD enabled, i've gotten a repro 1/8 attempts.

i'm retrying with MCVD off.

still fails with MCVD off, so i think that it's unrelated.

liberato@: could you tell me how to reproduce this flakiness? This is what I did:

- Ran with Canary 68.0.3400.0 on Android
- Nexus 5X, OPM1.171019.011 (maybe I should have flashed it to an earlier version?)
- Set the autoplay policy to "No user gesture is required" in about:flags
- Served up a local copy of https://github.com/KhronosGroup/WebGL on port 8000
- adb reverse tcp:8000 tcp:8000
- Repeatedly reloaded the test

No failures.

I guess I should have flashed to a userdebug build so I can run it via Telemetry.

But can you tell me if there are obvious bugs in https://github.com/KhronosGroup/WebGL/blob/master/sdk/tests/js/webgl-test-utils.js#L2896 where our heuristics are wrong for telling whether or not the video is playing and has a non-blank frame?

Please assign back to me with your answers. Thanks.

 Issue 834924  has been merged into this issue.

 Issue 834892  has been merged into this issue.

i used a n5x with 8.0.0 userdebug on it.  i built ChromePublic.apk normally (release, dchecks not enabled), installed it, and ran via telemetry with "android-chromium" as the browser selection.  i used a test filter to get 8 tests only.  i don't remember the exact filter string, and bash history is stuck in my local terminal so i can't get it out easily via ssh.

i'll update with the telemetry command line when i'm back in the office monday.

one thing i realized is that i was using adb_chrome_public_command_line to turn on / off MCVD for my testing.  does telemetry override that somehow?  if so, then maybe i didn't test anything. 

kbr@: the command was:

content/test/gpu/run_gpu_integration_test.py webgl_conformance --show-stdout --browser=android-chromium --test-filter=textures_video_tex_2d

i ran it 8-10 times to see a failure.

also, for webgl-test-utils.js, i don't know if timeupdated guarantees that a video frame is ready.  we'll fire the first one (i think) when our buffering state changes, but that doesn't seem like it makes any guarantees that a frame has made it out of the decoder.

it's subtle.  i'm often wrong about these things.

=> dalecurtis for the correct answer (please see c#9 "but can you tell me")

No bugs, but a more efficient implementation is something like this:

https://cs.chromium.org/chromium/src/third_party/WebKit/LayoutTests/media/suspend-util.js?l=17

I.e. elide waiting for playing and timeupdate and instead just efficiently poll currentTime > 0. timeupdate events are supposed to fire at most every ~250ms per spec IIRC, so waiting for it may result in waiting a bit more time and if you're just going to wait for timeupdate after the playing event, no reason to do both.

liberato@: I flashed a N5X to MMB29Q userdebug (same as the bots) and built ChromePublic.apk at 48376006e317f7a3f4b0f6daf8bde9cbae19671c . Then applied the attached patch to turn off the flaky expectations. (A few of the tex-2d tests are known to fail on Android right now.) It took ~15 tries but I finally got a failure:

[4/8] gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest.WebglConformance_conformance_textures_video_tex_2d_rgb_rgb_unsigned_byte failed unexpectedly 11.0642s:
  
  Traceback (most recent call last):
    _RunGpuTest at content/test/gpu/gpu_tests/gpu_integration_test.py:132
      self.RunActualGpuTest(url, *args)
    RunActualGpuTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:188
      getattr(self, test_name)(test_path, *args[1:])
    _RunConformanceTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:202
      self._CheckTestCompletion()
    _CheckTestCompletion at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:198
      self.fail(self._WebGLTestMessages(self.tab))
    fail at /usr/lib/python2.7/unittest/case.py:410
      raise self.failureException(msg)
  AssertionError: shouldBe 0,255,0
  at (4, 4) expected: 0,255,0 was 0,0,0
  
  testing: video/mp4; codecs="avc1.42E01E, mp4a.40.2"
  Testing texImage2D with flipY=true bindingTarget=TEXTURE_2D
  Checking lower left corner
  FAIL shouldBe 0,255,0
  at (4, 4) expected: 0,255,0 was 0,0,0
  Checking upper left corner
  shouldBe 255,0,0
  at (4, 24) expected: 255,0,0 was 0,0,0
  FAIL shouldBe 255,0,0
  at (4, 24) expected: 255,0,0 was 0,0,0
....


If you can more reliably reproduce this then can you try modifying webgl-test-utils.js's startPlayingAndWaitForVideo locally to use the technique from suspend-util.js instead and see if it's more reliable? If so we'll change the WebGL conformance suite.

Thanks for your help.

Deleted: expectations.patch 1.9 KB

expectations.patch 1.9 KB Download

 Issue 907184  has been merged into this issue.