https://wmatrix.googleplex.com/unfiltered?hide_missing=True&releases=tot&tests=video_GlitchDetection

Vin is going to work on this test next quarter.

Would it be better to use the term disable instead of remove? :-) A failing test can be fixed.

The thing is video_GlitchDetection requires two things -- golden images and Chameleon boards. It makes the test not easy to scale and maintain. We can see the test only run on five devices. If a test does not catch any regression for a year, usually it's a sign the test is not that useful. Tom mentioned a way not to use golden images. I forgot what it was. Tom. What was your idea of not using golden image for the test?

I prefer removing tests because some partners do not know and still try to run the tests.

One could manually create special videos with large 16x16 binary timestamps encoded in them (say top row). That way one can keep capturing images, convert the large timestamp pixels back into numbers and make sure these are not grossly disordered.
1) No need for goldens.
2) May or may not be a need for Chameleon boards. I guess as alternatives our autotest screenshots are too slow and not useful for this. Maybe one could point a webcam at the screen but that requires some more serious (en-?)coding.

The test is an end-to-end test. It is different with other unit tests/integration tests, which are short and easy to run but limited to only testing the video codec. You may modify the suite attribute to split it from the unit/integration tests as they have different properties and test purposes.

Having golden video is not an issue. I agree we have to scale it more other boards. Mussa and I discussed some ideas how to scale:
 * Use ffmepg to analyse the video file during every test, such that don't need to pre-compute the golden checksum for one specific video;
 * Set Chameleon (external monitor) to the same screen resolution as the video resolution to reduce other noises;
 * Do color calibration during every test to scale to different hardware.

Vin may work on the above works to scale the test to all boards.

What I mentioned to Wucheng a way without using golden images, is for measuring video quality. The implementation is to detect some video codec compression artifact like ringing, posterization, block boundary, etc. It is targeted to test two-end video communication (like Hangout) under different conditions (like workload/network bandwidth). However, our team has no resource to do it this quarter. Will revisit it once we have more resource.

By the way, the video_GlitchDetection test detected some regressions, like:
  https://bugs.chromium.org/p/chromium/issues/detail?id=580736

If the test can't be scaled for partners, I agree with Tom  that that can be fixed by separating this test from other tests. Please see my email about the bvt_perbuild suite to the group.

>> If a test does not catch any regression for a year, usually it's a sign the test is not that useful. 

I don't agree on this :-)

This is a very difficult to implement right test. But it also is a very high value test if it works reliably. How about we create a doc and discuss what can be done to improve it?

I drafted a document to collect ideas:
  https://docs.google.com/a/google.com/document/d/1DZPX6YFm-doNPjMiu8kehwuWKKllbKnX_dAoqusZ9LY/edit?usp=sharing

Feel free to edit/give comments.

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

We are going to move to AV-Analysis-Service as our solution to video testing. The scores assigned by the cloud service should help find glitches during playback. Marking as won't fix; will reopen if needed.