Also on FYI waterfall:
https://ci.chromium.org/buildbot/chromium.gpu.fyi/Linux%20FYI%20Debug%20%28NVIDIA%29/396

ccameron@, any chance that this is related to https://chromium-review.googlesource.com/949980? That's the only thing that seems to be relevant in recent commits.

From these four runs:

https://chromium-swarm.appspot.com/task?id=3c1a85ae7efc3810&refresh=10&show_raw=1
https://chromium-swarm.appspot.com/task?id=3c19eefe3310de10&refresh=10&show_raw=1
https://chromium-swarm.appspot.com/task?id=3c19dea4600a9f10&refresh=10&show_raw=1
https://chromium-swarm.appspot.com/task?id=3c19ce2b77b82d10&refresh=10&show_raw=1

here are the pixel diffs:

http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?42e60f7292e75a5ab0b4bb0ed7b4795fa2cdec0c_Linux_Debug_NVIDIA__telemetry
http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?42864292ecfb1a33c65aeafc9a9271140cf85638_Linux_Debug_NVIDIA__telemetry
http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?d159d6562c4c8fc9cbd135a3a8544ead29b03e27_Linux_Debug_NVIDIA__telemetry
http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?b7948feeb1b5902802218c5dfc51a6af47075fd5_Linux_Debug_NVIDIA__telemetry

The failure mode looks like either a slightly incorrect video frame was grabbed, or the frame was offset by a pixel or two.

Why would this be flaky only on the Debug bot? Could there be some sort of race condition in layout or painting? Adding more components.

A pretty conservative regression range would indicate that whatever change broke this test landed between the following two builds:

https://ci.chromium.org/buildbot/chromium.gpu/Linux%20Debug%20%28NVIDIA%29/91480

and

https://ci.chromium.org/buildbot/chromium.gpu/Linux%20Debug%20%28NVIDIA%29/91468

kylechar@: could you dig into the regression range between those two builds and see if there's anything suspicious? The web tools are terrible at plugging in Chromium commit positions and getting blamelists.

CL Range:
https://chromium.googlesource.com/chromium/src/+log/1cffa8f71564086c45cae117d9321ec52458c6be%5E..caeb7aedfb5d7fa8bfca515bd80cb0f833909d70?pretty=fuller&n=

I don't see anything obviously wrong in the blamelist so I'm trying to reproduce. The test doesn't pass with my normal GN args on a local build so I'm rebuilding now.

I built with the args from "GPU Linux Builder (dbg)" and I'm running it locally in a loop. The test is consistently passing which points to something race-y. I'll have to get access to the bot and running there to reproduce, or otherwise just speculatively revert things.

ccameron: https://crrev.com/c/949980 seems like the mostly likely candidate for change in position. Is there anything race-y in that CL?

No luck reproducing locally as the test ran overnight without failure, so that's not going to work.

http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?42e60f7292e75a5ab0b4bb0ed7b4795fa2cdec0c_Linux_Debug_NVIDIA__telemetry

I took a closer look at the pixel output and the source video. The result isn't a pixel shift, it's the second frame in the video, 

The video doesn't autoplay for me locally. Is it somehow autoplaying on the bot or is it starting at the second frame somehow?

Same failure on Linux Release (NVIDIA) now too.

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Release%20(NVIDIA)/123102

+kbr I'm not sure who is the right person for video related bugs?

Dale: was there a recent change in the behavior of video elements – perhaps just GPU-accelerated ones – that would cause them to autoplay, or display their second rather than first frame, when they should be displaying their first frame?

Going to mark the test Flaky for now.

No, nothing should have changed here recently. There have been a few bugs cropping up lately about out of order frames being grabbed. I'm not sure what's going on yet.

+tguilbert who was looking into one of these issues.

This should be software decoded video and I recently changed where frame copying is done, so this could be an issue with GpuMemoryBuffer copies.

+dcastagna, is it possible for copies to complete out of order from the raster pool?

Hmmmm. In that case I wonder if it could be the same root cause as  Issue 819914 . There's a reliable repro for that and it occurs only with 2D canvas.

That said the presence of issue 820240 which does not use GpuMemoryBuffers implies that this is a different issue...

There's also  issue 814120  then.

Ah, I think this is definitely out of order copies from the GMB frame pool:

I constructed a small test to check for out of order frames coming back from the frame pool and it failed, so will take a look.

Probably  issue 819914  is the same.

I think they might be out of order, there is currently nothing that guarantees they are in order.
IIRC the assumption we made with the pool was that we'd have only one videoframe copy in flight, but now that we moved it, that assumption is not valid anymore. :/

dcastagna@ said he'd take on making the GMB pool sequential since we use it in quite a few places that may suffer from similar hidden issues currently.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/621948db9c10e76f6b3032d938242598a48a7269

commit 621948db9c10e76f6b3032d938242598a48a7269
Author: Yuly Novikov <ynovikov@chromium.org>
Date: Fri Mar 09 04:14:33 2018

Update Pixel_Video_MP4 expectations

Fails on Nexus 5X, Flaky on Linux NVIDIA

BUG=819635,820240
TBR=kbr@chromium.org

Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: Ie1daa75a8144d7bbcd314cba228fc528e6a4956a
Reviewed-on: https://chromium-review.googlesource.com/955959
Commit-Queue: Yuly Novikov <ynovikov@chromium.org>
Reviewed-by: Yuly Novikov <ynovikov@chromium.org>
Cr-Commit-Position: refs/heads/master@{#542019}
[modify] https://crrev.com/621948db9c10e76f6b3032d938242598a48a7269/content/test/gpu/gpu_tests/pixel_expectations.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4484282d93e0da93befa7e04ee29343634ca5b6f

commit 4484282d93e0da93befa7e04ee29343634ca5b6f
Author: Daniele Castagna <dcastagna@chromium.org>
Date: Fri Mar 09 22:35:09 2018

media: Make GMBVP frame delivery in order

GpuMemoryBufferVideoFramePool was initially used by VideoRendererImpl,
and it assumed at most one copy at a time would be requested.
If a new copy was requested before the previous one finished, there
was no guarantee that the frame ready callback would be called in the
same order as the copy request.

crrev.com/947383 moved the copy to GMB, and the pool, in an earlier
stage of video playback. Now more than on copy could be requested,
and the pool might deliver videoframes out of order.

This patch adds the guarantee that GMBVP will deliver frames in the
same orders the copy were requested.

Bug: 819635
Change-Id: Ic8fd5e95bcda59c836e6be65eaa3891ace0b1aa7
Reviewed-on: https://chromium-review.googlesource.com/955912
Commit-Queue: Daniele Castagna <dcastagna@chromium.org>
Reviewed-by: Dale Curtis <dalecurtis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#542254}
[modify] https://crrev.com/4484282d93e0da93befa7e04ee29343634ca5b6f/media/video/gpu_memory_buffer_video_frame_pool.cc
[modify] https://crrev.com/4484282d93e0da93befa7e04ee29343634ca5b6f/media/video/gpu_memory_buffer_video_frame_pool_unittest.cc

The VP9 test is flaky too: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_rel_ng/247886

That's for a CL which reenables the tests after incrementing version numbers so new reference images should be generated: https://chromium-review.googlesource.com/c/chromium/src/+/1214364

The diff in bot results is all black: http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?399751fcb4af665abcebd83a37a0cae007e16956_linux_chromium_rel_ng_telemetry

But generating a diff via imagemagick locally shows subtle differences all over the image.

I'll mark the VP9 test as flaky too.

Deleted: Pixel_Video_VP9_gen.png 127 KB

	Pixel_Video_VP9_gen.png 127 KB View Download

Deleted: Pixel_Video_VP9_ref.png 127 KB

	Pixel_Video_VP9_ref.png 127 KB View Download

Deleted: diff.png 6.9 KB

	diff.png 6.9 KB View Download

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1a9264ddffb97e645ae936840846a741dfefee24

commit 1a9264ddffb97e645ae936840846a741dfefee24
Author: Sunny Sachanandani <sunnyps@chromium.org>
Date: Tue Dec 11 00:52:48 2018

Re-enable video pixel tests after rebaselining

The tests were marked as failing and version numbers were incremented in
https://crrev.com/c/1182222, but I forgot to re-enable the tests.

Bug: 869677, 913138, 858826, 819635, 774809
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: I7fdb28c3dbaa7aad4137b16d8f9bf8fe15fe9a08
Reviewed-on: https://chromium-review.googlesource.com/c/1214364
Commit-Queue: Sunny Sachanandani <sunnyps@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#615353}
[modify] https://crrev.com/1a9264ddffb97e645ae936840846a741dfefee24/content/test/gpu/gpu_tests/pixel_expectations.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b188cd82b0f877078b2a7f87169a4a16a1f9507d

commit b188cd82b0f877078b2a7f87169a4a16a1f9507d
Author: Jamie Madill <jmadill@chromium.org>
Date: Tue Dec 11 16:38:53 2018

Revert "Re-enable video pixel tests after rebaselining"

This reverts commit 1a9264ddffb97e645ae936840846a741dfefee24.

Reason for revert: rebaseline didn't seem to work.

Failing builds:

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win7%20FYI%20Release%20%28AMD%29/3655
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win7%20FYI%20Release%20%28NVIDIA%29/3470
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20FYI%20Release%20%28Intel%20HD%20630%29/2954

Failing tests:

gpu_tests.pixel_integration_test.PixelIntegrationTest.Pixel_DirectComposition_Video_VP9
gpu_tests.pixel_integration_test.PixelIntegrationTest.Pixel_Video_VP9
gpu_tests.pixel_integration_test.PixelIntegrationTest.Pixel_DirectComposition_Video_MP4
gpu_tests.pixel_integration_test.PixelIntegrationTest.Pixel_Video_MP4

Original change's description:
> Re-enable video pixel tests after rebaselining
> 
> The tests were marked as failing and version numbers were incremented in
> https://crrev.com/c/1182222, but I forgot to re-enable the tests.
> 
> Bug: 869677, 913138, 858826, 819635, 774809
> Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
> Change-Id: I7fdb28c3dbaa7aad4137b16d8f9bf8fe15fe9a08
> Reviewed-on: https://chromium-review.googlesource.com/c/1214364
> Commit-Queue: Sunny Sachanandani <sunnyps@chromium.org>
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#615353}

TBR=kbr@chromium.org,sunnyps@chromium.org

Change-Id: I553c10ca8022a90a0afa73ffafa44aa5bbf4e2d1
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 869677, 913138, 858826, 819635, 774809
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Reviewed-on: https://chromium-review.googlesource.com/c/1372234
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Cr-Commit-Position: refs/heads/master@{#615553}
[modify] https://crrev.com/b188cd82b0f877078b2a7f87169a4a16a1f9507d/content/test/gpu/gpu_tests/pixel_expectations.py