A good number of these look like media/ EME tests, should probably have someone from that team take a look.

+jrummell (for EME) & wolenetz (for MSE). 

John, Matt, can you guys take a look at the above tests?

The ExternalClearKey tests (probably all the EncryptedMediaTests listed) register a pepper plugin by appending something like "--register-pepper-plugins="libclearkeycdmadapter.so#ClearKey CDM#ClearKey CDM 0.1.0.0#0.1.0.0;application/x-ppapi-clearkey-cdm"" to the command line, and then use it. Looking at the logs for issue 627959 I see "FAIL: NotSupportedError Unsupported keySystem", which indicates that the plugin wasn't registered (or failed to load).

I'm not sure what the change for Flash is, but it appears that the registration of the other pepper plugins shouldn't be delayed at startup.

Just a comment about the tests not running as part of CQ. I just checked a job that recently ran (https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/271453) and in the stdio log for "browser_tests (with patch) on Ubuntu-12.04" I see:
[904/969] MSE_ExternalClearKey/EncryptedMediaTest.Playback_VideoAudio_WebM_Opus/0 (3667 ms)
[905/969] SRC_ExternalClearKey/EncryptedMediaTest.Playback_Multiple_VideoAudio_WebM/0 (6814 ms)
[906/969] MSE_ExternalClearKey/EncryptedMediaTest.Playback_VideoClearAudio_WebM_Opus/0 (3667 ms)
(and lots more). So the tests do run in CQ.

Sorry, I misspoke. I meant that the /bots/ on which the failures occurred do not run as part of CQ (i.e. official.desktop.continuous: win_trunk, mac_trunk).

(Indeed, the change that broke the tests did pass CQ including linux_chromium_rel_ng, and AFAIK that bot remained happy.)

John and I chatted about this in person, and decided it doesn't make too much sense and it would be good to verify there is an issue here before proceeding.

So, I've created https://codereview.chromium.org/2195103002/ to demonstrate this. My expectations are:

(1) It passes CQ dry run.
(2) But when you build and run browser_tests locally with an officially branded build (and an internal checkout), the tests fail.

If that's not the case, then maybe I misunderstood the problem here and we need to think about exactly what is going wrong. I'm trying this out now, will post results Monday (my Mac build takes way too long...).

My expectations in #5 seem to be fulfilled: The CL passes CQ dry run, but running browser_tests locally (with a Chrome-branded internal build) gives me:

57 tests failed:
    ECKEncryptedMediaTest.CDMCrashDuringDecode (../../chrome/browser/media/encrypted_media_browsertest.cc:614)
    ECKEncryptedMediaTest.CDMExpectedCrash (../../chrome/browser/media/encrypted_media_browsertest.cc:620)
    ECKEncryptedMediaTest.FileIOTest (../../chrome/browser/media/encrypted_media_browsertest.cc:627)
    ECKEncryptedMediaTest.LoadLoadableSession (../../chrome/browser/media/encrypted_media_browsertest.cc:632)
    ECKEncryptedMediaTest.LoadUnknownSession (../../chrome/browser/media/encrypted_media_browsertest.cc:636)
    EncryptedMediaSupportedTypesExternalClearKeyTest.Audio_MP4 (../../chrome/browser/media/encrypted_media_supported_types_browsertest.cc:559)
    EncryptedMediaSupportedTypesExternalClearKeyTest.Audio_WebM (../../chrome/browser/media/encrypted_media_supported_types_browsertest.cc:517)
    EncryptedMediaSupportedTypesExternalClearKeyTest.Basic (../../chrome/browser/media/encrypted_media_supported_types_browsertest.cc:457)
    EncryptedMediaSupportedTypesExternalClearKeyTest.Video_MP4 (../../chrome/browser/media/encrypted_media_supported_types_browsertest.cc:536)
    EncryptedMediaSupportedTypesExternalClearKeyTest.Video_WebM (../../chrome/browser/media/encrypted_media_supported_types_browsertest.cc:498)
    MSE_ExternalClearKey/EncryptedMediaTest.ConfigChangeVideo/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.FrameSizeChangeVideo/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.InvalidResponseKeyError/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.Playback_AudioClearVideo_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.Playback_AudioOnly_MP4/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.Playback_AudioOnly_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.Playback_AudioOnly_WebM_Opus/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.Playback_ClearVideo_WEBM_EncryptedAudio_MP4/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.Playback_EncryptedVideo_MP4_ClearAudio_WEBM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.Playback_EncryptedVideo_WEBM_EncryptedAudio_MP4/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.Playback_Multiple_VideoAudio_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.Playback_VP9Video_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.Playback_VideoAudio_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.Playback_VideoAudio_WebM_Opus/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.Playback_VideoClearAudio_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.Playback_VideoClearAudio_WebM_Opus/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKey/EncryptedMediaTest.Playback_VideoOnly_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.ConfigChangeVideo/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.FrameSizeChangeVideo/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.InvalidResponseKeyError/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.Playback_AudioClearVideo_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.Playback_AudioOnly_MP4/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.Playback_AudioOnly_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.Playback_AudioOnly_WebM_Opus/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.Playback_ClearVideo_WEBM_EncryptedAudio_MP4/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.Playback_EncryptedVideo_MP4_ClearAudio_WEBM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.Playback_EncryptedVideo_WEBM_EncryptedAudio_MP4/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.Playback_Multiple_VideoAudio_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.Playback_VP9Video_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.Playback_VideoAudio_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.Playback_VideoAudio_WebM_Opus/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.Playback_VideoClearAudio_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.Playback_VideoClearAudio_WebM_Opus/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    MSE_ExternalClearKeyDecryptOnly/EncryptedMediaTest.Playback_VideoOnly_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    PepperContentSettingsSpecialCasesPluginsBlockedTest.Normal (../../chrome/browser/content_settings/content_settings_browsertest.cc:472)
    SRC_ExternalClearKey/EncryptedMediaTest.FrameSizeChangeVideo/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    SRC_ExternalClearKey/EncryptedMediaTest.InvalidResponseKeyError/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    SRC_ExternalClearKey/EncryptedMediaTest.Playback_AudioClearVideo_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    SRC_ExternalClearKey/EncryptedMediaTest.Playback_AudioOnly_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    SRC_ExternalClearKey/EncryptedMediaTest.Playback_AudioOnly_WebM_Opus/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    SRC_ExternalClearKey/EncryptedMediaTest.Playback_Multiple_VideoAudio_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    SRC_ExternalClearKey/EncryptedMediaTest.Playback_VP9Video_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    SRC_ExternalClearKey/EncryptedMediaTest.Playback_VideoAudio_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    SRC_ExternalClearKey/EncryptedMediaTest.Playback_VideoAudio_WebM_Opus/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    SRC_ExternalClearKey/EncryptedMediaTest.Playback_VideoClearAudio_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    SRC_ExternalClearKey/EncryptedMediaTest.Playback_VideoClearAudio_WebM_Opus/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
    SRC_ExternalClearKey/EncryptedMediaTest.Playback_VideoOnly_WebM/0 (../../chrome/browser/media/encrypted_media_browsertest.cc:415)
1 test timed out:
    PepperContentSettingsSpecialCasesTest.Baseline (../../chrome/browser/content_settings/content_settings_browsertest.cc:457)

Hey folks,

Any updates?  We're quickly approaching the branch point for M54 (Aug 25th - ~ 2 weeks), so we are getting a bit down to the wire.  

Who would be the right owner for this (John R., since the failures seem to be in EME tests)?

Thanks in advance for helping to bring this into a close, getting Flash deployed exclusively over component update is critical to our ability to keep the plugin and our users secure and up to date.

Hey Justin, do you have any insights on this code?

Sorry, nope.

Adding Brett and Yuzhu, who may have some distant memories... of the land of Pepper Flash.

I'm investigating why ExternalClearKey doesn't get loaded on official builds (which I can repro on my Windows box). However, if others have insight into how the other tests work (and why they fail with the change), it might make isolating the issue easier.

ChromePluginTest.InstalledPlugins
PolicyTest.DisabledPluginsExceptions
PolicyTest.EnabledPlugins
PepperContentSettingsSpecialCasesPluginsBlockedTest.Normal
PolicyTest.DisabledPlugins

Sorry, I should have updated it, I fixed those tests as part of fixing other pieces of code, only the media tests listed in comment #6 are still failing for me. (And the PepperContentSettingsSpecialCasesTest just seems flaky.)

I found the problem. RegisterPepperFlashWithChrome() goes through and removes older registrations of Flash. The check to see if the plugin is Flash is [1]:
  return plugin.is_pepper_plugin() &&
         (plugin.pepper_permissions & ppapi::PERMISSION_FLASH);

However, pepper plugins registered by the command line get all permissions [2]:
    // Command-line plugins get full permissions.
    plugin.permissions = ppapi::PERMISSION_ALL_BITS;

So this is finding the registration for ExternalClearKey and removing it, resulting in the test being unable to load it.

Will investigate how the test for Flash can be improved (only the particular permission, use the name, or something else).

[1] https://cs.chromium.org/chromium/src/chrome/browser/component_updater/pepper_flash_component_installer.cc?l=105
[2] https://cs.chromium.org/chromium/src/content/common/pepper_plugin_list.cc?l=128

Wow! Great find!!

I agree that we should just be able to use the plugin name; it looks like all sites that register the plugin use that name (and I can't think of any reason we would have non-Flash plugins registered with that name). I'll update my CL to do this.

Nice job John!  Thank you for digging into that, excellent excellent excellent :).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/21d7beb1e2c02a8103d23126cf22838241fe794d

commit 21d7beb1e2c02a8103d23126cf22838241fe794d
Author: waffles <waffles@chromium.org>
Date: Mon Aug 22 22:50:20 2016

This change takes out the loading of the Flash bundle from chrome_content_client.cc on all platforms except Linux and ChromeOS, in preparation for removing Flash from the installation bundle on Windows and Mac. This should not result in any user-visible behavior changes, as the component updater will load the bundled component on Windows and Mac.

The change to the component installer is necessary to not remove command-line specified pepper plugins - see the bug for details.

BUG= 628320 
TEST=https://docs.google.com/document/d/1iTQiaqjuHsKV4cPqSOet-eJKWb2SsJLp2ieDj_Mul4s/edit?pref=2&pli=1

Review-Url: https://codereview.chromium.org/2195103002
Cr-Commit-Position: refs/heads/master@{#413565}

[modify] https://crrev.com/21d7beb1e2c02a8103d23126cf22838241fe794d/chrome/browser/component_updater/pepper_flash_component_installer.cc
[modify] https://crrev.com/21d7beb1e2c02a8103d23126cf22838241fe794d/chrome/common/chrome_content_client.cc