Happened on veyron_minnie on 10814.0.0 too:
http://ubercautotest.corp.google.com/tko/retrieve_logs.cgi?job=/results/211513735-chromeos-test/chromeos4-row9-rack10-host19

Seems to have passed after a retry?

We shall have a finer grain error message to distinguish the three condition apart.

Saw this failure last night in HWTest [bvt-cq] on peach_pit-, peppy-, and tricky-chrome-pfq.  Here's the failed master pfq run with links to the individual failed builders:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8939700450600602400

Same error message:
video_ChromeHWDecodeUsed.vp8          [ FAILED ]
video_ChromeHWDecodeUsed.vp8            ERROR: Media.GpuVideoDecoderInitializeStatus not loaded or histogram bucket not found or histogram bucket found at < 100%
video_ChromeHWDecodeUsed.vp8            retry_count: 1

+This week's gardeners, to keep an eye on it

Upping to P1 since this has happened several times now. 

deanliao: could you ptal or help us find a new owner?

deanliao: this actually looks like it's happening consistently now on the builders

Looking into it.

For peach_pit, it failed because the histogram is not expected:
07/29 20:43:58.081 DEBUG|histogram_verifier:0028| chrome://histograms/Media.GpuVideoDecoderInitializeStatus:
Histogram: Media.GpuVideoDecoderInitializeStatus recorded 5 samples, mean = 12.0 (flags = 0x41)
0   ------------------O                                                       (1 = 20.0%)
1   ... 
15  ------------------------------------------------------------------------O (4 = 80.0%) {20.0%}

It expected all GpuVideoDecoderInitializeStatus are zero. But 4 out of 5 status are 15 instead of 0.

Status 15: DECODER_ERROR_NOT_SUPPORTED

All three boards' failure reasons are the same. 4 unexpected DECODER_ERROR_NOT_SUPPORTED in chrome://histograms/Media.GpuVideoDecoderInitializeStatus

Last known good image: (R70-10919.0.0-rc2, 70.0.3505.0)
First bad image version: (R70-10920.0.0-rc1, 70.0.3506.0)

Found a peach_pit DUT, it is locally reproducible. 

Bisecting https://chromium.googlesource.com/chromium/src/+log/70.0.3505.0..70.0.3506.0?n=10000

In DUT, after running autotest from host, you may run it locally:
/usr/local/autotest/bin/autotest_client /usr/local/autotest/tests/video_ChromeHWDecodeUsed/control.vp8

The culprit commit: https://chromium-review.googlesource.com/c/chromium/src/+/1153832
ChromeOS OOBE: Add Welcome screen animations.

It could be because something wrong playing OOBE video so that histograms/Media.GpuVideoDecoderInitializeStatus has DECODER_ERROR_NOT_SUPPORTED state counted.

I didn't find a way to reset histogram before test starts. I propose we can add a method to obtain and verify diff of histogram before and after the test.

Of course, why the OOBE animations cause DECODER_ERROR_NOT_SUPPORTED needs investigation.

alemate: apparently the culprit is a commit you landed recently? (see #13)

Should we revert it to make the tree green?

Indeed reverting the culprit commit should make the tree green again. On video team side, two things to follow:
1) figure out why the OOBE animation introduces DECODER_ERROR_NOT_SUPPORTED
2) make test more robust by checking diff of histogram before and after test item.

Histogram is now a different issue 869189.

I filed issue 869243 to "investigate why the OOBE animation introduces DECODER_ERROR_NOT_SUPPORTED".

Assigning to video test owner to investigate option 2: "make test more robust by checking diff of histogram before and after test item."

Why should we revert animations if it's bug in test + video decoding? Animations are playing OK.

The root cause of the failure is that video_ChromeHWDecodeUsed test checks histograms only after test. So if pre-test histograms has counter unexpected to the test, it will fail no matter what the test itself is ok or not.

We should expect histogram diff instead. Working on it.

It is part of  https://crbug.com/844755  effort. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/3e14531037de06976b4fc50fe33b48d7fa899a58

commit 3e14531037de06976b4fc50fe33b48d7fa899a58
Author: Dean Liao <deanliao@chromium.org>
Date: Wed Aug 01 21:52:30 2018

cros/video: add HistogramDiffer

In the past, some video tests, e.g. video_ChromeHWDecodeUsed,
only expect post-test histogram. It can get false negative result
if histogram before test has something unexpected, e.g. error
status code. To make the test pre-test histogram invariant,
HistogramDiffer take two histogram snapshots before and after test,
then calculates their difference.

BUG= chromium:845372 
TEST=./histogram_verifier_unittest.py
Pass video_ChromeHWDecodeUsed on peach_pit R70-10920.0.0-rc1.

Change-Id: I86faec85dd47908d988433e46ba9c30dffc06b6e
Reviewed-on: https://chromium-review.googlesource.com/1156707
Commit-Ready: Shuo-Peng Liao <deanliao@google.com>
Tested-by: Shuo-Peng Liao <deanliao@google.com>
Reviewed-by: Hirokazu Honda <hiroh@chromium.org>

[modify] https://crrev.com/3e14531037de06976b4fc50fe33b48d7fa899a58/client/site_tests/video_ChromeHWDecodeUsed/video_ChromeHWDecodeUsed.py
[add] https://crrev.com/3e14531037de06976b4fc50fe33b48d7fa899a58/client/cros/video/histogram_verifier_unittest.py
[add] https://crrev.com/3e14531037de06976b4fc50fe33b48d7fa899a58/client/cros/video/common.py
[modify] https://crrev.com/3e14531037de06976b4fc50fe33b48d7fa899a58/client/cros/video/histogram_verifier.py

We actually need this in M69.

I mean we need #20 in M69.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/3a26463be800cba6d09559c31c6f90a90796f707

commit 3a26463be800cba6d09559c31c6f90a90796f707
Author: Dean Liao <deanliao@chromium.org>
Date: Fri Aug 17 18:17:58 2018

cros/video: use  HistogramDiffer to expect histogram change.

It is a follow-up commit of https://crrev.com/c/1156707.

BUG= chromium:845372 
TEST=./histogram_verifier_unittest.py
Run autotest video_ChromeHWDecodeUsed, video_ChromeRTCHWDecodeUsed,
video_ChromeVidResChangeHWDecode, video_MediaRecorderHWEncodeUsed,
and video_MediaRecorderPerf.

Change-Id: I743f00d76a37dc67889bbdb505854767013fe892
Reviewed-on: https://chromium-review.googlesource.com/1177402
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Shuo-Peng Liao <deanliao@google.com>
Reviewed-by: Hirokazu Honda <hiroh@chromium.org>

[modify] https://crrev.com/3a26463be800cba6d09559c31c6f90a90796f707/client/site_tests/video_ChromeVidResChangeHWDecode/video_ChromeVidResChangeHWDecode.py
[modify] https://crrev.com/3a26463be800cba6d09559c31c6f90a90796f707/client/site_tests/video_ChromeHWDecodeUsed/video_ChromeHWDecodeUsed.py
[modify] https://crrev.com/3a26463be800cba6d09559c31c6f90a90796f707/client/site_tests/video_MediaRecorderPerf/video_MediaRecorderPerf.py
[modify] https://crrev.com/3a26463be800cba6d09559c31c6f90a90796f707/client/cros/video/histogram_verifier_unittest.py
[modify] https://crrev.com/3a26463be800cba6d09559c31c6f90a90796f707/client/site_tests/video_MediaRecorderHWEncodeUsed/video_MediaRecorderHWEncodeUsed.py
[modify] https://crrev.com/3a26463be800cba6d09559c31c6f90a90796f707/client/cros/video/histogram_verifier.py
[modify] https://crrev.com/3a26463be800cba6d09559c31c6f90a90796f707/client/site_tests/video_ChromeRTCHWDecodeUsed/video_ChromeRTCHWDecodeUsed.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/a36b878559862fe31c20af485c8c8883a7bb5113

commit a36b878559862fe31c20af485c8c8883a7bb5113
Author: Dean Liao <deanliao@chromium.org>
Date: Tue Aug 21 17:33:59 2018

Reland "cros/video: use HistogramDiffer to expect histogram change."

This reverts commit 3cdc7663539d8c7f6cf8ada119a012c7e8dfef9d,
which is a reland of 3a26463be800cba6d09559c31c6f90a90796f707.

It adds retry mechanism for histogram_verifier.expect_sole_bucket() as
its replacement target, histogram_verifier.verify(), has retry
mechanism.

Also, adds expect_histogram_grow() with observation peirod.

BUG= chromium:845372 , chromium:875699 
TEST=./histogram_verifier_unittest.py
Run autotest video_ChromeHWDecodeUsed, video_ChromeRTCHWDecodeUsed,
video_ChromeVidResChangeHWDecode, video_MediaRecorderHWEncodeUsed,
and video_MediaRecorderPerf.

Change-Id: Ic15b14234a86fb222a3947c75fb572fdba3c4083
Reviewed-on: https://chromium-review.googlesource.com/1180792
Commit-Ready: Shuo-Peng Liao <deanliao@google.com>
Tested-by: Shuo-Peng Liao <deanliao@google.com>
Reviewed-by: Hirokazu Honda <hiroh@chromium.org>

[modify] https://crrev.com/a36b878559862fe31c20af485c8c8883a7bb5113/client/site_tests/video_ChromeVidResChangeHWDecode/video_ChromeVidResChangeHWDecode.py
[modify] https://crrev.com/a36b878559862fe31c20af485c8c8883a7bb5113/client/site_tests/video_ChromeHWDecodeUsed/video_ChromeHWDecodeUsed.py
[modify] https://crrev.com/a36b878559862fe31c20af485c8c8883a7bb5113/client/site_tests/video_MediaRecorderPerf/video_MediaRecorderPerf.py
[modify] https://crrev.com/a36b878559862fe31c20af485c8c8883a7bb5113/client/cros/video/histogram_verifier_unittest.py
[modify] https://crrev.com/a36b878559862fe31c20af485c8c8883a7bb5113/client/site_tests/video_MediaRecorderHWEncodeUsed/video_MediaRecorderHWEncodeUsed.py
[modify] https://crrev.com/a36b878559862fe31c20af485c8c8883a7bb5113/client/cros/video/histogram_verifier.py
[modify] https://crrev.com/a36b878559862fe31c20af485c8c8883a7bb5113/client/site_tests/video_ChromeRTCHWDecodeUsed/video_ChromeRTCHWDecodeUsed.py

To fix flaky test because of wrongly expecting histogram snapshot after test instead of examining histogram difference before and after the test, we need to merge both https://crrev.com/c/1156707 and https://crrev.com/c/1180792

This bug requires manual review: We are only 12 days from stable.
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), kariahda@(iOS), cindyb@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Issue 869189 has been merged into this issue.

 Issue 845055  has been merged into this issue.

Merge approved, M69.

cros tryjob -g 1186012 -g 1186013 --branch release-R69-10895.B daisy-paladin-tryjob peppy-paladin-tryjob
failed at build_package

Asking 
https://groups.google.com/a/google.com/forum/#!topic/chromeos-infra-discuss/Y6ep-UDba_Q

Also need to merge this https://crrev.com/c/1186923

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/079f964d5569375e75adf18e375381a0fdbc7c5c

commit 079f964d5569375e75adf18e375381a0fdbc7c5c
Author: Dean Liao <deanliao@chromium.org>
Date: Thu Aug 23 17:58:37 2018

cros/video: add HistogramDiffer

In the past, some video tests, e.g. video_ChromeHWDecodeUsed,
only expect post-test histogram. It can get false negative result
if histogram before test has something unexpected, e.g. error
status code. To make the test pre-test histogram invariant,
HistogramDiffer take two histogram snapshots before and after test,
then calculates their difference.

BUG= chromium:845372 
TEST=./histogram_verifier_unittest.py
Pass video_ChromeHWDecodeUsed on peach_pit R70-10920.0.0-rc1.

Change-Id: I86faec85dd47908d988433e46ba9c30dffc06b6e
Previous-Reviewed-on: https://chromium-review.googlesource.com/1156707
(cherry picked from commit 544d6d0fa47c8e6f97ce2dbe2d1bfcf4d53df203)
Reviewed-on: https://chromium-review.googlesource.com/1186012
Commit-Queue: Shuo-Peng Liao <deanliao@google.com>
Tested-by: Shuo-Peng Liao <deanliao@google.com>
Reviewed-by: Hirokazu Honda <hiroh@chromium.org>

[modify] https://crrev.com/079f964d5569375e75adf18e375381a0fdbc7c5c/client/site_tests/video_ChromeHWDecodeUsed/video_ChromeHWDecodeUsed.py
[add] https://crrev.com/079f964d5569375e75adf18e375381a0fdbc7c5c/client/cros/video/histogram_verifier_unittest.py
[add] https://crrev.com/079f964d5569375e75adf18e375381a0fdbc7c5c/client/cros/video/common.py
[modify] https://crrev.com/079f964d5569375e75adf18e375381a0fdbc7c5c/client/cros/video/histogram_verifier.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/cc94f101deecbbd4325c287fe6592e76ae1fe6ee

commit cc94f101deecbbd4325c287fe6592e76ae1fe6ee
Author: Dean Liao <deanliao@chromium.org>
Date: Thu Aug 23 17:58:38 2018

Reland "cros/video: use HistogramDiffer to expect histogram change."

This reverts commit 3cdc7663539d8c7f6cf8ada119a012c7e8dfef9d,
which is a reland of 3a26463be800cba6d09559c31c6f90a90796f707.

It adds retry mechanism for histogram_verifier.expect_sole_bucket() as
its replacement target, histogram_verifier.verify(), has retry
mechanism.

Also, adds expect_histogram_grow() with observation peirod.

BUG= chromium:845372 , chromium:875699 
TEST=./histogram_verifier_unittest.py
Run autotest video_ChromeHWDecodeUsed, video_ChromeRTCHWDecodeUsed,
video_ChromeVidResChangeHWDecode, video_MediaRecorderHWEncodeUsed,
and video_MediaRecorderPerf.

Change-Id: Ic15b14234a86fb222a3947c75fb572fdba3c4083
Previous-Reviewed-on: https://chromium-review.googlesource.com/1180792
(cherry picked from commit 4f4af6bd9e3f13196a9c76bb5215efad4ecf6103)
Reviewed-on: https://chromium-review.googlesource.com/1186013
Commit-Queue: Shuo-Peng Liao <deanliao@google.com>
Tested-by: Shuo-Peng Liao <deanliao@google.com>
Reviewed-by: Hirokazu Honda <hiroh@chromium.org>

[modify] https://crrev.com/cc94f101deecbbd4325c287fe6592e76ae1fe6ee/client/site_tests/video_ChromeVidResChangeHWDecode/video_ChromeVidResChangeHWDecode.py
[modify] https://crrev.com/cc94f101deecbbd4325c287fe6592e76ae1fe6ee/client/site_tests/video_ChromeHWDecodeUsed/video_ChromeHWDecodeUsed.py
[modify] https://crrev.com/cc94f101deecbbd4325c287fe6592e76ae1fe6ee/client/site_tests/video_MediaRecorderPerf/video_MediaRecorderPerf.py
[modify] https://crrev.com/cc94f101deecbbd4325c287fe6592e76ae1fe6ee/client/cros/video/histogram_verifier_unittest.py
[modify] https://crrev.com/cc94f101deecbbd4325c287fe6592e76ae1fe6ee/client/site_tests/video_MediaRecorderHWEncodeUsed/video_MediaRecorderHWEncodeUsed.py
[modify] https://crrev.com/cc94f101deecbbd4325c287fe6592e76ae1fe6ee/client/cros/video/histogram_verifier.py
[modify] https://crrev.com/cc94f101deecbbd4325c287fe6592e76ae1fe6ee/client/site_tests/video_ChromeRTCHWDecodeUsed/video_ChromeRTCHWDecodeUsed.py

deanliao@ pinged for the additional CL that needs to be included. Approved for M69.

Verify procedure:

1. cros flash R69-10895.33.0 image to a peach_pit DUT from cautotest lab.
2. Run autotest video_ChromeHWDecodeUsed.vp8 and saw test failed.
3. Apply the three commits: https://crrev.com/c/1186923 https://crrev.com/c/1186012 https://crrev.com/c/1186013 to DUT
4. Run autotest video_ChromeHWDecodeUsed.vp8 again to see test pass.

The reason I need https://crrev.com/c/1186923 is because it has a Timer class which is used by https://crrev.com/c/1186013

https://luci-scheduler.appspot.com/jobs/chromeos/release-R69-10895.B-master-release#

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Verified using https://stainless.corp.google.com/search?view=matrix&row=model&col=test&first_date=2018-08-28&last_date=2018-08-30&test=video_ChromeHWDecodeUsed&build=R69-10895.40.0&exclude_cts=false&exclude_not_run=false&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false