Unable to build the Chrome from our end, hence adding TE-NeedsTriageHelp label for further triage of the issue

I've been looking at those failures recently.  It's not always the same set of failing tests, but in the particular cases I've looked at the problem as I see it is the following:

- DecoderSelector tries to initialize DecryptingVideoDecoder and it fails: StubCdm::InitializeVideoDecoder() always fails
- DecoderSelector then successfully initializes DecryptingDemuxerStream and a non-decrypting VideoDecoder
- Playback starts, StubCdm::Decrypt() fails (as always), putting the pipeline in the error state
- StubCdm::UpdateSession() also fails (as always), generating the EME_UPDATE_FAILED title that is actually expected by the test
- Because of the way Utils.setResultInTitle() in //media/test/data/eme_player_js/utils.js works, only the first of the errors matters
- There is a race between those two errors. In my case, the demuxer error "wins" most of the time

@xhwang, do you run these tests with the stub CDM or the full Widevine CDM? I guess the full CDM eliminates the race because the decryption actually works. OTOH the license server is only available on Linux, and also only for Chrome, because the server is sadly not present in the public repo. And if the license cannot be updated the full CDM will also fail, right?

I'm not sure what the best course of action here is. With the stub CDM and no license server there's quite little that can really be tested anyway. Maybe we could change the stub to pretend the decoder initialization is successful and to generate some dummy (e.g., black/silent) frames in DecryptAndDecodeFrame()/DecryptAndDecodeSamples()?

Note that downstream projects cannot really test it with the full CDM and the license server.

We run it with the real Widevine CDM and I never expect stub CDM to pass these tests.

In other words, these test only works in the case where the CDM is bundled (e.g. in Google Chrome), not component updated.

The easiest solution would be to restrict these tests to only run when the CDM is bundled. WDYT?

There's no point in keeping tests that are expected to fail, so if we don't find a better way we'll indeed have to disable them.

Quick question before I explore other options though: Does the test suffer from the race problem when testing with the real CDM but without the license server? Or is the decoding somehow suspended until the session update either succeeds or fails?

In most cases we don't have the license server and we just stop the test after session update etc. In some cases we have a simplified license server to test playback though.

https://cs.chromium.org/chromium/src/chrome/browser/media/encrypted_media_browsertest.cc?rcl=c4dd10518c291c20a9881e6720d8b9a9505c2010&l=159

I may have found a simple tweak to the stub code that stabilizes the test, see https://chromium-review.googlesource.com/c/chromium/src/+/831026.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c42add43e4cc9bc096f4e4a1eadc065b27f0b8ca

commit c42add43e4cc9bc096f4e4a1eadc065b27f0b8ca
Author: Wojciech Dzierżanowski <wdzierzanowski@opera.com>
Date: Mon Dec 18 07:56:08 2017

Avoid race in MSE_Widevine tests

When run with the stub Widevine CDM, these tests suffer from a race
between the result of a failed StubCdm::Decrypt() and a failed
StubCdm::UpdateSession().  The tests expect the latter, but the former
often arrives first.

By allowing the stub to InitializeVideoDecoder() successfully and
returning cdm::kNoKey upon DecryptAndDecodeFrame() attempt we avoid the
DecryptingDemuxerStream errors altogether, so there is no race anymore.

Bug:  768745 
Test: 'browser_tests --gtest_filter=*MSE_Widevine*' pass when run with the stub Widevine CDM

Change-Id: I35a35d006d118f8f7f9a8a64cf9334017644f4ea
Reviewed-on: https://chromium-review.googlesource.com/831026
Reviewed-by: Xiaohan Wang <xhwang@chromium.org>
Commit-Queue: Wojciech Dzierżanowski <wdzierzanowski@opera.com>
Cr-Commit-Position: refs/heads/master@{#524663}
[modify] https://crrev.com/c42add43e4cc9bc096f4e4a1eadc065b27f0b8ca/media/cdm/stub/stub_cdm.cc