How can I repro this locally?
Does it make sense for it to be an SSL error? I thought it used a pre-recorded version of the page.

I ran tools/perf/run_benchmark webrtc.stress --device=<device_id> and it executed fine.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3af95a7a68224916175dd750aa8cff30af2e0f74

commit 3af95a7a68224916175dd750aa8cff30af2e0f74
Author: sullivan <sullivan@chromium.org>
Date: Wed Nov 09 19:26:31 2016

Disable failing webrtc.stress benchmark on android.

BUG= 663802 
TBR=ehmaldonado@chromium.org

Review-Url: https://codereview.chromium.org/2491593002
Cr-Commit-Position: refs/heads/master@{#431005}

[modify] https://crrev.com/3af95a7a68224916175dd750aa8cff30af2e0f74/tools/perf/benchmarks/webrtc.py

Hmm. Maybe github is correctly configured for https:// but test.webrtc.org is not? Try changing the URL to http:// instead in webrtc_cases, re-record the page set and try re-enabling the test?

You'd think this would repro locally though. You didn't pass the use-origin-sites flag in #2 right?

+nednguyen

This seems like it's probably some issue with WPR, since WPR effectively MITMs the device. We've had issues like this in the past: WPR used to self-sign its certs using SHA1, and then Chrome stopped accepting SHA1 certs by default, so that broke perf tests.

I'll do a bit of research this afternoon and see if I can repro this.

Also, as long as the test has already been disabled this should be P2.

Update: I can repro locally! I'll try to inspect the run to see what might be happening.

My research seems to confirm that this is a bug to do with the interaction between this particular site, web page replay, and the device forwarder.

What I now know:
0. I tried rerecording (using Python 2.7.11; used the system default for everything else since devil doesn't seem to support 2.7.11); it didn't fix the problem.
1. The replay plays back fine on desktop
2. It doesn't work on Android (same error as seen on the bots, when using a Nexus 5 running the same build that the perf bots do)
3. It works on Android if I --use-live-sites
4. Other sites play back fine on Android.

Ned, do you know of any known issue with WPR that may be causing this issue? It seems similar to https://github.com/chromium/web-page-replay/issues/73 but as I said the fix of rerecording did not help.

WebRTC folks: it does seem like the workaround of using HTTP for the recording may be a solution here.

Ethan: can you show the WPR log? 

I'm not sure we can do https. WebRTC getUserMedia is a "powerful feature", which means it needs --unsafely-treat-insecure-origin-as-secure="http://example.com" (see https://www.chromium.org/Home/chromium-security/deprecating-powerful-features-on-insecure-origins). In turn, that flags needs --user-data-dir to be specified. 

This is true for most Telemetry devices, but not for instance Android (or so we found in this bug: https://bugs.chromium.org/p/chromium/issues/detail?id=523517. At least that was the case back in Sep 2015. The effect then was that getUserMedia hard-requires https.

Edward, maybe give it a shot with adding --unsafely-treat-insecure-origin-as-secure="http://test.webrtc.org" to the browser flags and changing the URL to http://. It could be it works now.

To what should I set --user-data-dir?

Telemetry should already do that. Hopefully, at least. That's why I mentioned in  bug 523517 ; Telemetry didn't use to set --user-data-dir for everything but I hope it does now.

You can verify chrome's command line set by Telemetry in the log:

INFO:root:Android app/browser startup args: chrome --enable-features=AndroidSpellCheckerNonLowEnd,AutofillCreditCardSigninPromo,AutofillProfileCleanup,DocumentWriteEvaluator,DownloadsUi,ExpectCTReporting,ImeThread,ImportantSitesInCBD,NTPArticleSuggestions,NTPBookmarkSuggestions,NTPSnippets,PrioritySupportedRequestsDelayable,NetworkTimeServiceQuerying,NewAudioRenderingMixingStrategy,NonValidatingReloadOnNormalReload,NTPOfflinePages,ParseHTMLOnMainThread,PassiveDocumentEventListeners,PassiveEventListenersDueToFling,PersistentHistograms,PhysicalWeb,PointerEvent,PreconnectMore,RafAlignedMouseInput,RafAlignedTouchInput,SecurityWarningIconUpdate,SpeculativeLaunchServiceWorker,NegotiateTLS13,token-binding,TranslateRankerEnforcement,WebRTC-EnableWebRtcEcdsa,use-new-media-cache --disable-features=AndroidSpellChecker --use-fake-ui-for-media-stream --use-fake-device-for-media-stream --force-fieldtrial-params=AutofillCreditCardSigninPromo.EnabledFive:impression_limit/5,AutofillProfileOrderByFrecency.EnabledLimitTo3:limit/3,CSSExternalScanner.Enabled_ScanAndPreload:cssExternalScannerNoPreload/false/cssExternalScannerPreload/true,ContextualSearch.Expected:enable_quick_answers/true/enable_translation/true/recent_scroll_duration_ms/300/suppression_taps/10,DataCompressionProxyLoFi.Enabled:hysteresis_period_seconds/120/rtt_msec/2000,DataReductionProxyServerExperiments.Prefetch:exp/prefetch,DisallowFetchForDocWrittenScriptsInMainFrame.DocumentWriteEvaluatorGroup:disallowFetchForDocWrittenScriptsInMainFrame/false/disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections/false,LowPriorityIFrames.Enabled:lowPriorityIframes/true,NTPSnippets.Enabled:content_suggestions_backend/https%3A%2F%2Fchromecontentsuggestions-pa%2Egoogleapis%2Ecom%2Fv1%2Fsuggestions%2Ffetch/quota_SuggestionFetcher/2,NetworkQualityEstimator.Enabled:2G%2EDefaultMedianKbps/81/2G%2EDefaultMedianRTTMsec/1337/3G%2EDefaultMedianKbps/658/3G%2EDefaultMedianRTTMsec/297/4G%2EDefaultMedianKbps/1756/4G%2EDefaultMedianRTTMsec/159/Bluetooth%2EDefaultMedianKbps/449/Bluetooth%2EDefaultMedianRTTMsec/135/Ethernet%2EDefaultMedianKbps/3263/Ethernet%2EDefaultMedianRTTMsec/106/HalfLifeSeconds/60/None%2EDefaultMedianKbps/567/None%2EDefaultMedianRTTMsec/272/Slow2G%2EThresholdMedianHttpRTTMsec/2000/Unknown%2EDefaultMedianKbps/1916/Unknown%2EDefaultMedianRTTMsec/121/WiFi%2EDefaultMedianKbps/2736/WiFi%2EDefaultMedianRTTMsec/168/correlation_logging_probability/0%2E001,ParseHTMLOnMainThread.Enabled:parseHTMLOnMainThreadCoalesceChunks/false/parseHTMLOnMainThreadSyncTokenize/false,PersistentHistograms.EnabledInMemory:storage/LocalMemory,ProgressBarAnimationAndroid.SmoothIndeterminateAnimation:progress-bar-animation/smooth-indeterminate,ReportCertificateErrors.ShowAndPossiblySend:sendingThreshold/1%2E0,SafeBrowsingAndroid.Enabled:enabled/true/types_to_check/0%2C1%2C3%2C6%2C7%2C8%2C9%2C10%2C11%2C13%2C14%2C15,SiteEngagement.AggressiveAccumulation:bootstrap_points/24/decay_period_in_hours/2/decay_points/0/decay_proportion/0%2E984/first_daily_engagement_points/1%2E5/hidden_media_playing_points/0%2E01/high_engagement_boundary/50/last_engagement_grace_period_in_hours/1/max_decays_per_score/4/max_points_per_day/15/medium_engagement_boundary/15/navigation_points/1%2E5/score_cleanup_threshold/0%2E5/user_input_points/0%2E6/visible_media_playing_points/0%2E06/web_app_installed_points/5,UMA_EnableCellularLogUpload.Enabled_wo_sampling:Enabled/true/Optimize/true/Sample_Probability/100/Uma_Quota/204800/Uma_Ratio/0%2E05,UpdateMenuItem.Enabled:enable_update_menu_item/true,WebBluetoothBlacklist.TestGroup:blacklist_additions/00060000%3Ae%2Cfffd%3Ae%2Ced5f25a4%3Ae --force-fieldtrials=AndroidSessionNotifications/Disabled/AndroidSpellChecker/Enabled_NonLowEnd/AsyncDns/AsyncDnsA/AutodetectEncoding/Enabled/AutofillClassifier/Enabled/AutofillCreditCardSigninPromo/EnabledFive/AutofillFieldMetadata/Enabled/AutofillProfileCleanup/Enabled/AutofillProfileOrderByFrecency/EnabledLimitTo3/BrowserScheduler/Default/CSSExternalScanner/Enabled_ScanAndPreload/ChromotingQUIC/Enabled/ContextualSearch/Expected/DataCompressionProxyDevRollout/Control/DataCompressionProxyLoFi/Enabled/DataCompressionProxyPromoVisibility/Enabled/DataReductionProxyFREPromo/Enabled/DataReductionProxyServerExperiments/Prefetch/DataReductionProxyUseQuic/Enabled/DisallowFetchForDocWrittenScriptsInMainFrame/DocumentWriteEvaluatorGroup/DownloadsUi/Enabled/EnableMediaThreadForMediaPlayback/Enabled/ExpectCTReporting/ExpectCTReportingEnabled/ImeThread/Enabled/ImportantSitesInCBD/DialogEnabled/InvalidationsGCMUpstream/Enabled/LimitSyncPassphrasePrompt/Enabled/LowPriorityIFrames/Enabled/ModerateBindingOnBackgroundTabCreation/Enabled/MojoChannel/Enabled/NTPPopularSites/Enabled/NTPSnippets/Enabled/NetDelayableH2AndQuicRequests/Enabled/NetworkQualityEstimator/Enabled/NetworkTimeQueries/NetworkTimeQueriesEnabled/NewAudioRenderingMixingStrategy/Enabled/NoCreditCardAbort/Disabled/NonValidatingReloadOnNormalReload/Enabled/OfferUploadCreditCards/Enabled/OfflinePages/EnabledAsBookmarks/PageRevisitInstrumentation/Enabled/ParseHTMLOnMainThread/Enabled/PassiveDocumentEventListeners/Enabled/PassiveEventListenersDueToFling/Enabled/PasswordBranding/SmartLockBrandingSavePromptOnly/PasswordManagerSettingsMigration/Enable/PersistentHistograms/EnabledInMemory/PhysicalWeb/Enabled/PointerEvent/Enabled/Precache/Enabled/PreconnectMore/Enabled/ProgressBarAnimationAndroid/SmoothIndeterminateAnimation/QUIC/Enabled/RafAlignedMouseInput/Enabled/RafAlignedTouchInput/Enabled/ReaderModeUI/AdaBoost/ReaderModeUIFeedback/Show/ReportCertificateErrors/ShowAndPossiblySend/SafeBrowsingAndroid/Enabled/SafeBrowsingReportPhishingErrorLink/Enabled/SchedulerExpensiveTaskBlocking/Enabled/SdchPersistence/Enabled/SecurityWarningIconUpdate/Enabled/SiteEngagement/AggressiveAccumulation/SpeculativeLaunchServiceWorker/Enabled/StrictSecureCookies/Enabled/SyncHttpContentCompression/Enabled/TLS13Negotiation/Enabled/TabManagementExperiment/Elderberry/TabSyncByRecency/Enabled/TokenBinding/TokenBinding/TranslateRanker/EnforcementEnabled/UMA_EnableCellularLogUpload/Enabled_wo_sampling/UnifiedMediaPipelineTrial/Enabled/UpdateMenuItem/Enabled/V8CacheStrategiesForCacheStorage/default/WebBluetoothBlacklist/TestGroup/WebFontsInterventionV2/Enabled-slow2g/WebRTC-EnableWebRtcEcdsa/Enabled/WebRTC-LocalIPPermissionCheck/Enabled/use-new-media-cache/Enabled --enable-net-benchmarking --metrics-recording-only --no-default-browser-check --no-first-run --enable-gpu-benchmarking --disable-background-networking --proxy-server=socks://localhost:51336 --disable-component-extensions-with-background-pages --disable-default-apps --enable-remote-debugging --disable-fre --disable-external-intent-requests

(https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Nexus5%20Perf%20%283%29/builds/3717/steps/webrtc.stress/logs/stdio)

Looks like the --user-data-dir is not set currently. According to https://www.chromium.org/Home/chromium-security/deprecating-powerful-features-on-insecure-origins , it seems like you can just use any temp directory.

Ok. Umm. How do we make that work on android? If we call tempfile.mkdtemp() that will give a sane temp dir for desktop, but not on Android/CrOS (because telemetry runs on a host machine and not the actual device, right?)

Juan: any suggestion on how to use temp dir on Android?

https://github.com/catapult-project/catapult/blob/master/devil/devil/android/device_temp_file.py

I think we're fine with being disabled on Android until the WPR bug is fixed. I'll rephrase this as a WPR bug.

Edward investigated a workaround but I think it sounds hard to implement, we might as well wait for a root cause fix.

 Issue 668121  has been merged into this issue.

 Issue 702201  has been merged into this issue.

Does anyone plan to work on fixing this?

Our team don't have bandwidth to work on this. I can help guiding someone from webrtc team to the solution.

I could work on that.

Ned:
What about doing something like this:
https://cs.chromium.org/chromium/src/tools/perf/page_sets/mse_cases.py?q=mse_cases&dr=C&l=27

That is, save our pages locally instead? We could even use a single page.

If the webrtc pages can be run locally without require WPR, that would be great solution. In general, only use WPR for real world cases that cannot be replicated easily with local HTML server.

Yes, they can be run locally. Should I just download them, or would it better to create new ones?

#26: I am not sure I understand your question. Are you asking whether you should be downloading or creating the .html files yourself? Do those pages have open source license? If not, you may need to create your own version.

Yes, I'm asking whether I should be downloading or creating the .html files myself. What would be better?
And those pages are part of WebRTC, there should be no problem. I'll ask my team.

#28: if the pages is created by your team, it doesn't matter whether you download the code or just copy it yourself.

ehmaldonado@ what are the next steps here?

This is fixed now.