Frank and I have been thinking about this from the context of Android. This would definitely be a good thing to have for as many platforms as possible. 

We do have AV Analysis tests which check the actual pixels https://av-analysis.corp.google.com/#/get-testpasslist/2/Video%20Stack , but they do not run on the waterfall, so results come back much later and are not integrated with Chromium infrastructure.

Let us know if you have any specific plans or ideas or would like to discuss.

i'm going to be starting on something to allow pixel tests on android, but it will involve screen capture since that's the only way that we can get the contents of overlays.

are you doing something like that, or is this read-back from the compositor?

if the former, then we might want to figure out if there's common stuff.

I'm hoping to add some pixel tests to  content/test/gpu/gpu_tests/pixel_test_pages.py. Currently we grab the result screenshot using a readback from the compositor, but I'm working on making it do a real screen capture instead ( bug 702023 ).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/434832f73131bcf29f81a268b28211e64329d61d

commit 434832f73131bcf29f81a268b28211e64329d61d
Author: jbauman <jbauman@chromium.org>
Date: Thu Mar 30 00:35:48 2017

Add GPU integration pixel tests for VP9 and MP4 video.

Ensure that the compositor draws these correctly.

BUG= 704389 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2769063006
Cr-Commit-Position: refs/heads/master@{#460598}

[modify] https://crrev.com/434832f73131bcf29f81a268b28211e64329d61d/chrome/test/BUILD.gn
[add] https://crrev.com/434832f73131bcf29f81a268b28211e64329d61d/content/test/data/gpu/pixel_video_mp4.html
[add] https://crrev.com/434832f73131bcf29f81a268b28211e64329d61d/content/test/data/gpu/pixel_video_vp9.html
[modify] https://crrev.com/434832f73131bcf29f81a268b28211e64329d61d/content/test/gpu/gpu_tests/pixel_expectations.py
[modify] https://crrev.com/434832f73131bcf29f81a268b28211e64329d61d/content/test/gpu/gpu_tests/pixel_integration_test.py
[modify] https://crrev.com/434832f73131bcf29f81a268b28211e64329d61d/content/test/gpu/gpu_tests/pixel_test_pages.py
[modify] https://crrev.com/434832f73131bcf29f81a268b28211e64329d61d/content/test/gpu/gpu_tests/trace_integration_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c6c435ba5ba3620440a88283237370b8c8d19781

commit c6c435ba5ba3620440a88283237370b8c8d19781
Author: jbauman <jbauman@chromium.org>
Date: Thu Mar 30 20:58:01 2017

Enable pixel video tests

All the reference images should exist now.

BUG= 704389 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2786063004
Cr-Commit-Position: refs/heads/master@{#460871}

[modify] https://crrev.com/c6c435ba5ba3620440a88283237370b8c8d19781/content/test/gpu/gpu_tests/pixel_expectations.py

Thanks for adding these John!

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1c0d3dd9ac676ade69a0418a089b79f1fa50205c

commit 1c0d3dd9ac676ade69a0418a089b79f1fa50205c
Author: sugoi <sugoi@chromium.org>
Date: Fri Mar 31 13:31:23 2017

Revert of Enable pixel video tests (patchset #1 id:1 of https://codereview.chromium.org/2786063004/ )

Reason for revert:
Broke "Win10 Release (NVIDIA)" and "Win10 Debug (NVIDIA)" bots. See:
https://build.chromium.org/p/chromium.gpu.fyi/builders/Win10%20Debug%20(NVIDIA)
https://build.chromium.org/p/chromium.gpu.fyi/builders/Win10%20Release%20(NVIDIA)

Original issue's description:
> Enable pixel video tests
>
> All the reference images should exist now.
>
> BUG= 704389 
> CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
>
> Review-Url: https://codereview.chromium.org/2786063004
> Cr-Commit-Position: refs/heads/master@{#460871}
> Committed: https://chromium.googlesource.com/chromium/src/+/c6c435ba5ba3620440a88283237370b8c8d19781

TBR=kbr@chromium.org,jbauman@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 704389 

Review-Url: https://codereview.chromium.org/2784353004
Cr-Commit-Position: refs/heads/master@{#461104}

[modify] https://crrev.com/1c0d3dd9ac676ade69a0418a089b79f1fa50205c/content/test/gpu/gpu_tests/pixel_expectations.py

I think the reason it's failing on Win10 is that we don't have separate baselines for windows 10 and windows 7, and we use different video processors (DXVA vs. cc's shaders) so the results are slightly different. It may be ok to just increase the tolerance there.

Also broken were https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Debug%20%28NVIDIA%29/builds/36781
https://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28Nexus%205%29/builds/4851 and
https://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28Pixel%20C%29/builds/5266

The linux debug NVIDIA and the Pixel C failed to complete the VP9 test, which is pretty weird. We have ref pictures for those platforms. MP4 worked there.

Nexus 5 failed when trying to crop the image from 1080 pixels wide to 1200. Maybe 400 pixels wide is too wide for the screen, accounting for its device scale factor. Maybe going down to 300 pixels wide would be good.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/062ee74fcfe10a4141eedfdaf37cbf7e9fe1b21f

commit 062ee74fcfe10a4141eedfdaf37cbf7e9fe1b21f
Author: jbauman <jbauman@chromium.org>
Date: Mon Apr 03 20:43:02 2017

Use separate pixel reference images for windows 10 and 7

We use different video processors on these platforms, so the results
are allowed to be slightly different.

BUG= 704389 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2794713002
Cr-Commit-Position: refs/heads/master@{#461524}

[modify] https://crrev.com/062ee74fcfe10a4141eedfdaf37cbf7e9fe1b21f/content/test/gpu/gpu_tests/cloud_storage_integration_test_base.py
[modify] https://crrev.com/062ee74fcfe10a4141eedfdaf37cbf7e9fe1b21f/content/test/gpu/gpu_tests/pixel_expectations.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/70b38bac56e4c561d4e8c6197fc5f222eb59055a

commit 70b38bac56e4c561d4e8c6197fc5f222eb59055a
Author: jbauman <jbauman@chromium.org>
Date: Tue Apr 04 01:45:56 2017

Capture only a 300x300 area with video pixel tests.

The Nexus 5's screen isn't big enough for a 400 pixel wide capture, when
accounting for the device scale factor. The tests are already marked as
failing and should be re-enabled once baselines are generated.

BUG= 704389 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2795053003
Cr-Commit-Position: refs/heads/master@{#461608}

[modify] https://crrev.com/70b38bac56e4c561d4e8c6197fc5f222eb59055a/content/test/gpu/gpu_tests/pixel_test_pages.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d9ff98eb4126be34772cb51a6431ee645162e970

commit d9ff98eb4126be34772cb51a6431ee645162e970
Author: jbauman <jbauman@chromium.org>
Date: Tue Apr 04 18:50:22 2017

Re-enable pixel tests on windows 10.

New baselines should have landed now.

BUG= 704389 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2792243002
Cr-Commit-Position: refs/heads/master@{#461789}

[modify] https://crrev.com/d9ff98eb4126be34772cb51a6431ee645162e970/content/test/gpu/gpu_tests/pixel_expectations.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/45c4581da9985f74c34aef5456786d5cc1773790

commit 45c4581da9985f74c34aef5456786d5cc1773790
Author: jbauman <jbauman@chromium.org>
Date: Tue Apr 04 23:30:52 2017

Re-enable pixel video tests.

The tests have been resized to work on the Nexus 5, and new windows 10
baselines were generated to handle its different video processing
system. VP9 tests are still disabled on linux and android NVIDIA cards
because we were seeing hangs there.

BUG= 704389 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2797843002
Cr-Commit-Position: refs/heads/master@{#461895}

[modify] https://crrev.com/45c4581da9985f74c34aef5456786d5cc1773790/content/test/gpu/gpu_tests/pixel_expectations.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2b277506fe914467d4bf3f0e3b1d807586d93408

commit 2b277506fe914467d4bf3f0e3b1d807586d93408
Author: jbauman <jbauman@chromium.org>
Date: Thu Apr 06 23:02:09 2017

Check for readyState in video pixel tests

Otherwise it's possible the video could be loaded before this code runs,
in which case the canplaythrough event won't happen and the test will hang.

TBR=kbr@chromium.org
BUG= 704389 

Review-Url: https://codereview.chromium.org/2803823005
Cr-Commit-Position: refs/heads/master@{#462667}

[modify] https://crrev.com/2b277506fe914467d4bf3f0e3b1d807586d93408/content/test/data/gpu/pixel_video_mp4.html
[modify] https://crrev.com/2b277506fe914467d4bf3f0e3b1d807586d93408/content/test/data/gpu/pixel_video_vp9.html

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/67e8bdf6ec24a9c345a6978a65338c7497cb4f59

commit 67e8bdf6ec24a9c345a6978a65338c7497cb4f59
Author: jbauman <jbauman@chromium.org>
Date: Fri Apr 07 21:34:58 2017

Add video pixel tests for DirectComposition layers.

The --enable-direct-composition-layers flag will force chrome to use
DirectComposition layers for hardware-decoded video (assuming
DirectComposition is actually supported on the system).

BUG= 704389 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2805173002
Cr-Commit-Position: refs/heads/master@{#463008}

[modify] https://crrev.com/67e8bdf6ec24a9c345a6978a65338c7497cb4f59/content/test/gpu/gpu_tests/pixel_expectations.py
[modify] https://crrev.com/67e8bdf6ec24a9c345a6978a65338c7497cb4f59/content/test/gpu/gpu_tests/pixel_integration_test.py
[modify] https://crrev.com/67e8bdf6ec24a9c345a6978a65338c7497cb4f59/content/test/gpu/gpu_tests/pixel_test_pages.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cbe9e8bb2512a3195b85efb50d3d80d9cf6dfa03

commit cbe9e8bb2512a3195b85efb50d3d80d9cf6dfa03
Author: jbauman <jbauman@chromium.org>
Date: Sat Apr 08 00:16:08 2017

Reenable Pixel_Video_VP9 on NVIDIA linux and android.

The change to check readyState in the test seems to have fixed the
flakiness.

TBR=kbr@chromium.org
BUG= 704389 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2800053005
Cr-Commit-Position: refs/heads/master@{#463066}

[modify] https://crrev.com/cbe9e8bb2512a3195b85efb50d3d80d9cf6dfa03/content/test/gpu/gpu_tests/pixel_expectations.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/875fdf2dbc234ca80ded492af8671f3194f8239b

commit 875fdf2dbc234ca80ded492af8671f3194f8239b
Author: jbauman <jbauman@chromium.org>
Date: Fri Apr 14 23:53:41 2017

Bump revision number for Pixel_DirectComposition_* tests.

Due to a bug (fixed in r464601) the reference image for Intel was
incorrect. Bump the revision to regenerate it.

TBR=kbr@chromium.org
BUG= 704389 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2818233002
Cr-Commit-Position: refs/heads/master@{#464831}

[modify] https://crrev.com/875fdf2dbc234ca80ded492af8671f3194f8239b/content/test/gpu/gpu_tests/pixel_test_pages.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ee8be9881c62d56bdee4fd0dd282dd18d9f4cf37

commit ee8be9881c62d56bdee4fd0dd282dd18d9f4cf37
Author: jbauman <jbauman@chromium.org>
Date: Wed Apr 26 05:53:05 2017

Enable Pixel_DirectComposition_Video_* tests.

Keep Pixel_DirectComposition_Video_MP4 disabled on Intel, because it
seems to have some sort of bug.

BUG= 704389 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2827833002
Cr-Commit-Position: refs/heads/master@{#467249}

[modify] https://crrev.com/ee8be9881c62d56bdee4fd0dd282dd18d9f4cf37/content/test/gpu/gpu_tests/pixel_expectations.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/eba7a8783c19c40fbb8306c15a486114eef69637

commit eba7a8783c19c40fbb8306c15a486114eef69637
Author: jbauman <jbauman@chromium.org>
Date: Sat May 27 08:36:43 2017

Bump version number of Pixel_DirectComposition_Video_MP4

Pixel_DirectComposition_Video_MP4_v2_win10_8086_1912_msaa.png is bad
but the bug causing that issue is probably fixed (r472749), so bump the
version to regenerate it.

BUG= 704389 
TBR=kbr@chromium.org
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2913463002
Cr-Commit-Position: refs/heads/master@{#475244}

[modify] https://crrev.com/eba7a8783c19c40fbb8306c15a486114eef69637/content/test/gpu/gpu_tests/pixel_expectations.py
[modify] https://crrev.com/eba7a8783c19c40fbb8306c15a486114eef69637/content/test/gpu/gpu_tests/pixel_test_pages.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9f39f94b9552de0584d65d78cd633fc470ee6a8b

commit 9f39f94b9552de0584d65d78cd633fc470ee6a8b
Author: jbauman <jbauman@chromium.org>
Date: Sat May 27 22:24:00 2017

Re-enable Pixel_DirectComposition_Video_MP4

The baseline for the new version should be generated now.

BUG= 704389 
TBR=kbr@chromium.org
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2905413003
Cr-Commit-Position: refs/heads/master@{#475253}

[modify] https://crrev.com/9f39f94b9552de0584d65d78cd633fc470ee6a8b/content/test/gpu/gpu_tests/pixel_expectations.py