I don't know of a recent change that would cause this level of failure. 

I suspect this is a product change/breakage. Let me know if I can help.

Matt,
Can you take a look? All most half of MSE and all EME conformance tests failed. This is very unusual.
I have tried beta build on the same device, all tests pass as expected. This is a recent regression.

I don't have O on a device that I can readily try.
I'm also unaware of any major MSE change in that range; perhaps something in AVDA changed (sandersd@ is out).
Dale, any ideas?
Yining, can you try bisecting please to narrow down the regression range? Also, is this regression *only* on Android O?

=>liberato who's making changes in these areas

We've already cut M61, so this is RBB for the 61 beta.

Pls apply appropriate OS label.

I just tried a few repro attempts on non-Android-O (attempting only a subset of the reported failing tests: MSE(Opus): Test: 36~41, and MSE (Media) Test: 100~102:

Linux Chrome: Unable to repro on either 59.0.3071.115 (Official Build) (64-bit) or ToT (M62) Debug.
Android M Nexus 7 (2013): Unable to repro on either 59.0.3071.125 (Official Build) (32-bit) or 61.0.3162.3 (Official Build) dev (32-bit)

I haven't tried to repro on Android O. liberato@, please take a look.
Also, +xhwang@ w.r.t. EME conformance failures. yiningc@, can you please provide a quick example of such an EME conformance failure too?

Did EME tests succeed on previous Chrome versions and start to fail on M61?

I have tried on Android K, M, and O
this issue only repro on Android O. On the other two android OS, both MSE and EME conformance tests pass appropriately on 61.0.3162.3 build. But on Andoid O, there are massive MSE and EME failures the Chrome 61.0.3162.3.
the regression on Android O happened somewhere between 61.0.3136.4 and 61.0.3162.3.

will take a look.

MSE core (21) fails yiningc@'s pixel using chrome dev 3162.3 .
my locally build ToT and also locally built chromium 61.0.3162.0 both pass.

will try building .3 exactly.

Hmm, if this is O specifically it may not be ReleaseBlock-Beta since it's unreleased, but I'm not sure what our release schedule is.  +cmasso who is in charge of Android M61.

61.0.3162.3 (93e40b30f8) built locally runs fine.  so, it's something about the official dev build.

--no-experiments: chrome dev build still fails.
is_official_build: local build still works.

trying build is_chrome_branded.

is_chrome_branded local build works.

i still have a lot of variations in chrome://about in the dev build that aren't in my local build.  i'll try that next.

network folks: in the failed case, i'm seeing this error in the official dev build:

Assert failed: XHR length is (32072) which should be (828645)

any idea what would be different between an official dev build and my locally built (is_chrome_branded, is_official_build)?

variations: setting the channel to either canary or dev didn't cause my local build to fail.

dalecurtis@ suggested unzipping the apks.

min/target sdks are different, 16 / 23 locally and 24 / 26 in the dev build (why?).  i set mine to 24 / 26 locally, but still works fine.

anyway, i downloaded 61.0.3163.13 dev from the play store onto my n5x with OPM1.170726, and it works fine.  i upgraded chrome dev on the pixel to the same version, and it fails.  it has OPR1.170420.

so, i'm going to install that on the n5x and see what happens.

i ended up installing OPM on both devices.  nothing changed.

i tried canary 62.0.3166.3 from the play store, and it works on the pixel.

so, to summarize:

A. 61 dev from play store works on N5x, not on pixel xl.  both have OPM.
B. 61 dev built locally (official, chrome branded) works on both N5x and Pixel XL.
C. 62 canary from play store works on Pixel.

i can't explain all of those at once.  A seems like it's related to the device.  B seems like it's something about how it's built / packaged.  C seems like it's something we fixed in chrome.

hubbe@ suggested data saver, which i enabled manually.  didn't break the working cases on the pixel.

Hmm, will reach out to clank-team@ to see if anyone has any ideas.

i'm really wondering if it's not related to finch in some way, and my tests above were insufficient to rule it out.

as in, looking at A,B,C as ruling things out:

- it can't be the OS, since A
- it can't be a bad CL in chrome, since B
- it can't be the hardware, since B
- it can't be packaging / signing since C

what's left?  it's chrome dev on this particular device.  that sounds like finch to me.

i installed an internal build of chrome on the pixel.  i had to uninstall chrome-dev to do it, and it worked.   i then upgraded in-place to the play store version, and it still works.

luckily, i captured all the variations before and after:

good: AdsDisableActiveView-Experiment                                           
 bad: AdsDisableActiveView-Control                                              
good: AppBannerTriggering-CheckOnPageLoad                                       
 bad: AppBannerTriggering-Control                                               
good: BackgroundTracing-AggressiveLowMemorySignalStats                          
 bad: BackgroundTracing-default                                                 
good: ChromeHome-Enabled_Button                                                 
 bad: ChromeHome-Default                                                        
good: DisablePreferCompositingToLCDTextOnLowEndAndroid-Control                  
 bad: DataReductionProxyUseQuic-Enabled12                                       
good: DynamicExpectCT-DynamicExpectCTEnabled                                    
 bad: DisablePreferCompositingToLCDTextOnLowEndAndroid-DisablePreferCompositingToLCDTextOnLowEndAndroid         
good: NTPArticleSuggestions-Enabled-FCS-NewSchedule                             
 bad: NTPArticleSuggestions-Enabled-FCS                                         
good: NetDelayableH2AndQuicRequests-Control2                                    
 bad: NetDelayableH2AndQuicRequests-Yielding                                    
good: NewPhotoPicker-Enabled                                                    
 bad: NewPhotoPicker-Control                                                    
good: OmniboxBundledExperimentV1-Dev_Android_HQPUrlsIndexing_Control            
 bad: OmniboxBundledExperimentV1-Dev_Android_HighHQPUrlsIndexedAtStartup        
good: OmniboxPlaceholderExperiment-Unused                                       
 bad: OmniboxPlaceholderExperiment-Default                                      
good: PasswordGeneration-Disabled                                               
 bad: PasswordGeneration-Enabled                                                
good: PersistentHistograms-EnabledInMemory7                                     
 bad: PersistentHistograms-EnabledOnDisk7                                       
good: PreviewsClientLoFi-Enabled_2G_Replace_Server_Placeholders                 
 bad: PreviewsClientLoFi-Control_2G_Replace_Server_Placeholders                 
good: QUIC-EnabledPathMtuDiscoveryLow                                           
 bad: QUIC-EnabledMultipleVersions39and37                                       
good: ResourceLoadScheduler-Disabled                                            
 bad: ResourceLoadScheduler-Enabled_bg_limit_5                                  
good: SocketReadIfReady-Control                                                 
 bad: SocketReadIfReady-Enabled                                                 
good: SyncUSSAutocomplete-Control                                               
 bad: SyncUSSAutocomplete-Enabled                                               
good: TranslateRankerModel-OverrideControl2                                     
 bad: TranslateRankerModel-CKEnableTranslateRankerDisableUlp                    
good: UKM-Control2                                                              
 bad: UKM-Enabled2                                                              
good: UMA-Uniformity-Trial-10-Percent-group_02                                  
 bad: UMA-Uniformity-Trial-10-Percent-group_01                                  
good: UMA-Uniformity-Trial-20-Percent-group_01                                  
 bad: UMA-Uniformity-Trial-20-Percent-default                                   
good: VsyncAlignedInput-Enabled                                                 
 bad: VsyncAlignedInput-Control

I'd focus on forcing opt in for the QUIC/Net trials on your good build and see what happens

i've not had luck with this.  on my good build, setting this does nothing:

--force-fieldtrials=QUIC/EnabledMultipleVersions39and37/SocketReadIfReady/Enabled/NetDelayableH2AndQuicRequests/Yielding/DataReductionProxyUseQuic/Enabled12/UKM/Enabled2
--enable-features=Ukm<UKM
--force-fieldtrial-params=QUIC.EnabledMultipleVersions39and37:enable_quic/true/quic_version/QUIC_VERSION_39%2CQUIC_VERSION_37,UKM.Enabled2:RecordSessionId/true

setting the entire list of "bad" field trials also doesn't seem to help, though it's possible that i've simply done something wrong with such a big list.

since i re-installed the internal build, i can't get chrome-dev to fail, either.  i've tried uninstalling and re-installing chrome-dev many, many times.  it's now 61.0.3163.13.  the omnibox is sometimes at the bottom, so it's definitely getting new field trials.

> network folks: in the failed case, i'm seeing this error in the official dev build:
> Assert failed: XHR length is (32072) which should be (828645)

I am not sure if that's the fatal error here. Please follow instructions at https://sites.google.com/a/chromium.org/dev/for-testers/providing-network-details to obtain a chrome://net-export log. It should tell us what is happening in the //net stack. 

cmasso: TL;DR: without a repro, and even then only observed on O, may i remove ReleaseBlock-Beta?

still no luck.  i've tried re-building .3, with some of the finch-releated features hardwired in the code, since not all of the --force-fieldtrials seemed to actually show up in the variations.  i also tried clearing app data a bunch of times to get new variations.

no repro.

i'm not sure what else to do with this without a repro.  would be nice to know if clearing app data is sufficient to fix it once it's broken, since  we don't really know for sure that a finch experiment is involved.  just seems liklely to me.

Seems fine to remove RBB label if yining@ can no longer repro. Were we able to capture a net log?

net log: no, unfortunately.  the repro was gone by that point.

i'll return the pixel to yiningc@, and if still no repro, then i'll -RBB.

I couldn't repro it on the latest chrome dev (61.0.3163.13), canary (62.0.3172.0) and beta (60.0.3112.78) build anymore.

thanks, yiningc@.

marking WAI, removing ReleaseBlock-Beta.  please re-open if this comes back.