key_mobile_sites pageset failing on 4 builders |
||||||||||||||||||||||||
Issue descriptionblink_style.key_mobile_sites failing on 4 builders Builders failed on: - Android Nexus5 Perf (1): https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf%20%281%29 - Android Nexus6 Perf (1): https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus6%20Perf%20%281%29 - Android Nexus7v2 Perf (1): https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus7v2%20Perf%20%281%29 - Android One Perf (1): https://build.chromium.org/p/chromium.perf/builders/Android%20One%20Perf%20%281%29 Failing at 454725:454942 Sample log: https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5_Perf__1_%2F5389%2F%2B%2Frecipes%2Fsteps%2Fblink_style.key_mobile_sites%2F0%2Fstdout [ RUN ] LinkedIn ... Traceback (most recent call last): File "/b/c/b/Android_Nexus5_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 88, in _RunStoryAndProcessErrorIfNeeded state.RunStory(results) File "/b/c/b/Android_Nexus5_Perf__1_/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function return func(*args, **kwargs) File "/b/c/b/Android_Nexus5_Perf__1_/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 299, in RunStory self._current_page.Run(self) File "/b/c/b/Android_Nexus5_Perf__1_/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 108, in Run shared_state.page_test.RunNavigateSteps(self, current_tab) File "/b/c/b/Android_Nexus5_Perf__1_/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/c/b/Android_Nexus5_Perf__1_/src/third_party/catapult/telemetry/telemetry/page/legacy_page_test.py", line 195, in RunNavigateSteps page.RunNavigateSteps(action_runner) File "/b/c/b/Android_Nexus5_Perf__1_/src/tools/perf/page_sets/key_mobile_sites_pages.py", line 116, in RunNavigateSteps 'document.getElementById("profile-view-scroller") !== null') File "/b/c/b/Android_Nexus5_Perf__1_/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/c/b/Android_Nexus5_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 252, in WaitForJavaScriptCondition return self._tab.WaitForJavaScriptCondition(*args, **kwargs) File "/b/c/b/Android_Nexus5_Perf__1_/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/c/b/Android_Nexus5_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 191, in WaitForJavaScriptCondition return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs) File "/b/c/b/Android_Nexus5_Perf__1_/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/c/b/Android_Nexus5_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 294, in WaitForJavaScriptCondition e.message + '\n' + debug_message) TimeoutException: Timed out while waiting 60s for IsJavaScriptExpressionTrue Kicking off a return_code bisect.
,
Mar 16 2017
Actually this is for all the benchmarks using the pageset.
,
Mar 16 2017
cc-ing owners of key_mobile_sites benchmarks, going to disable them until it's fixed.
,
Mar 16 2017
,
Mar 16 2017
Hmph, I launched a bisect here: https://chromeperf.appspot.com/buildbucket_job_status/8984962893948144816 but it failed. Traceback (most recent call last): File "/b/c/b/android_nexus5_perf_bisect/src/build/android/provision_devices.py", line 553, in main return ProvisionDevices(args) File "/b/c/b/android_nexus5_perf_bisect/src/build/android/provision_devices.py", line 74, in ProvisionDevices raise device_errors.DeviceUnreachableError(args.device) I'll launch another.
,
Mar 16 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8984957026751775664
,
Mar 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/4d954851ba9c129b4529302f77862f6b812165d8 commit 4d954851ba9c129b4529302f77862f6b812165d8 Author: sullivan <sullivan@chromium.org> Date: Thu Mar 16 14:47:05 2017 Disable failing blink_style.key_mobile_sites BUG= 702194 Review-Url: https://codereview.chromium.org/2758533003 Cr-Commit-Position: refs/heads/master@{#457432} [modify] https://crrev.com/4d954851ba9c129b4529302f77862f6b812165d8/tools/perf/benchmarks/blink_style.py
,
Mar 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/2b9c05fb215e52b9b360eb11118e3ee5c3b657e6 commit 2b9c05fb215e52b9b360eb11118e3ee5c3b657e6 Author: sullivan <sullivan@chromium.org> Date: Thu Mar 16 15:05:13 2017 Disable key_mobile_sites benchmarks which are timing out. BUG= 702194 TBR=nednguyen@google.com Review-Url: https://codereview.chromium.org/2754053002 Cr-Commit-Position: refs/heads/master@{#457435} [modify] https://crrev.com/2b9c05fb215e52b9b360eb11118e3ee5c3b657e6/tools/perf/benchmarks/oilpan_gc_times.py [modify] https://crrev.com/2b9c05fb215e52b9b360eb11118e3ee5c3b657e6/tools/perf/benchmarks/smoothness.py [modify] https://crrev.com/2b9c05fb215e52b9b360eb11118e3ee5c3b657e6/tools/perf/benchmarks/v8.py
,
Mar 16 2017
=== Auto-CCing suspected CL author rsleevi@chromium.org === Hi rsleevi@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Test failure found with culprit Suspected Commit Author : rsleevi Commit : 0f9bfb00c432d594504502728b8a1405a0ff2cf1 Date : Sat Mar 04 03:07:20 2017 Subject: Disable commonName matching for certificates Bisect Details Configuration: android_nexus5_perf_bisect Benchmark : smoothness.sync_scroll.key_mobile_sites_smooth Metric : mean_input_event_latency/LinkedIn Revision Exit Code N chromium@454721 0 +- N/A 20 good chromium@454748 0 +- N/A 20 good chromium@454750 0 +- N/A 20 good chromium@454751 0 +- N/A 20 good chromium@454752 1 +- N/A 20 bad <-- chromium@454755 1 +- N/A 20 bad chromium@454762 1 +- N/A 20 bad chromium@454775 1 +- N/A 20 bad chromium@454822 1 +- N/A 20 bad chromium@454928 1 +- N/A 20 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=LinkedIn smoothness.sync_scroll.key_mobile_sites_smooth Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8984957026751775664 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5054365674504192 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Mar 16 2017
Removing auto-assignment. Is it possible these bots are using a testing environment that relies on deprecated (now disabled) interception configurations? For example, not properly constructing MITM certificates? Whomever owns those test tools would be in the best position to fix them.
,
Mar 16 2017
Ned, any ideas if WPR is relying on the above?
,
Mar 16 2017
Are you using WPR? If there's source somewhere checked in, happy to look at and suggest fix. The in-Chromium test harnesses should all have been fixed to add subjectAltNames as necessary
,
Mar 16 2017
The failing test here is linkedin, which use WPR. The source code of WPR is in https://github.com/chromium/web-page-replay The certificate installation is in https://github.com/chromium/web-page-replay/blob/master/adb_install_cert.py & https://github.com/chromium/web-page-replay/blob/master/certutils.py
,
Mar 16 2017
It sounds like the solution is a pretty simple fix then: https://github.com/chromium/web-page-replay/blob/master/certutils.py#L131 crypto.X509Extension('subjectAltName', False, "DNS:"+subject) That's assuming "subject" will always be a DNS name. If it might be an IP _or_ domain name, then something like https://chromium.googlesource.com/chromium/src/+/0f9bfb00c432d594504502728b8a1405a0ff2cf1%5E%21/#F0 probably needs to happen - if it's an IP, the prefix is "IP:" rather than "DNS:" Are there instructions how to run these tests locally? I don't have an Android device around me to run these tests, but it 'should' be that easy. Can the telemetry infrastructure force Prefs and/or Enterprise Flags? The Enterprise policy is "EnableCommonNameFallbackForLocalAnchors", which maps to setting ssl.common_name_fallback_enabled_for_local_anchors to true. That should give a short-term fix to WPR-based infrastructure until we can get the WPR fix upstreamed.
,
Mar 17 2017
=== BISECT JOB RESULTS === Bisect was unable to run to completion Error: INFRA_FAILURE The bisect was able to narrow the range, you can try running with: good_revision: 137b6c7fbf49d0920f5a709ca66166719795b4cf bad_revision : 6ebf780ebaa4e6bea494255675832c3e017997d3 If failures persist contact the team (see below) and report the error. Bisect Details Configuration: android_nexus5_perf_bisect Benchmark : blink_style.key_mobile_sites Metric : parse_css_enormous_tokens/parse_css_enormous_tokens Revision Exit Code N chromium@454721 0 +- N/A 20 good chromium@454832 1 +- N/A 20 bad chromium@454942 1 +- N/A 20 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests blink_style.key_mobile_sites Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8984959753752263648 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=6360959016239104 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Mar 21 2017
Issue 703625 has been merged into this issue.
,
Mar 21 2017
I will make the fix & have rsleevi review
,
Mar 21 2017
rsleevi@ what is the exact flags to use for forcing Prefs and/or Enterprise Flags? We can certainly set the flags for Chrome in https://github.com/catapult-project/catapult/blob/master/telemetry/telemetry/internal/backends/chrome/chrome_browser_backend.py#L78
,
Mar 21 2017
I'm not aware of any command-line flags to force prefs/enterprise flags. That's done in the Preferences file (the JSON file). There wasn't an explicit command-line flag added to the CommandLinePrefStore to map a command-line to pref.
,
Mar 21 2017
Hmhh, if there isn't an easy way to do the short term fix, I think we should just proceed with the right fix then. I have the instruction to set up the device for running telemetry test at go/telemetry-device-setup Can you make the WPR change? I will take care of update the WPR in telemetry.
,
Mar 21 2017
I need to update status; I'm in all-day meetings for the next three days, will not be near a Linux host machine nor do I have an Android device. Is there a way to run WPR w/o a device on macOS so I can at least test changes?
,
Mar 21 2017
You can run WPR locally with https://github.com/chromium/web-page-replay/blob/master/documentation/GettingStarted.md#replay
,
Mar 21 2017
,
Mar 21 2017
,
Mar 22 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9fd79ea4e3ae8261ce5522d095577efdf4f58bc9 commit 9fd79ea4e3ae8261ce5522d095577efdf4f58bc9 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Wed Mar 22 07:54:05 2017 Roll src/third_party/catapult/ 84775f4ca..5004585e0 (2 commits) https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/84775f4ca78a..5004585e04bd $ git log 84775f4ca..5004585e0 --date=short --no-merges --format='%ad %ae %s' 2017-03-21 weisert Disable component updater in perftests. 2017-03-21 nednguyen Roll WPR to the latest commit Created with: roll-dep src/third_party/catapult BUG= 702194 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2767863002 Cr-Commit-Position: refs/heads/master@{#458678} [modify] https://crrev.com/9fd79ea4e3ae8261ce5522d095577efdf4f58bc9/DEPS
,
Mar 22 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f96f72670ef6916a0be0b69e8be0424c10f2e019 commit f96f72670ef6916a0be0b69e8be0424c10f2e019 Author: nednguyen <nednguyen@google.com> Date: Wed Mar 22 13:21:14 2017 Revert of Disable key_mobile_sites benchmarks which are timing out. (patchset #1 id:1 of https://codereview.chromium.org/2754053002/ ) Reason for revert: Root cause should be fixed. Original issue's description: > Disable key_mobile_sites benchmarks which are timing out. > > BUG= 702194 > TBR=nednguyen@google.com > > Review-Url: https://codereview.chromium.org/2754053002 > Cr-Commit-Position: refs/heads/master@{#457435} > Committed: https://chromium.googlesource.com/chromium/src/+/2b9c05fb215e52b9b360eb11118e3ee5c3b657e6 TBR=sullivan@chromium.org # Not skipping CQ checks because original CL landed more than 1 days ago. BUG= 702194 Review-Url: https://codereview.chromium.org/2764323002 Cr-Commit-Position: refs/heads/master@{#458729} [modify] https://crrev.com/f96f72670ef6916a0be0b69e8be0424c10f2e019/tools/perf/benchmarks/oilpan_gc_times.py [modify] https://crrev.com/f96f72670ef6916a0be0b69e8be0424c10f2e019/tools/perf/benchmarks/smoothness.py [modify] https://crrev.com/f96f72670ef6916a0be0b69e8be0424c10f2e019/tools/perf/benchmarks/v8.py
,
Mar 22 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c4928c8fba4766c4fb25cbb0ecce574101b29aca commit c4928c8fba4766c4fb25cbb0ecce574101b29aca Author: nednguyen <nednguyen@google.com> Date: Wed Mar 22 13:24:32 2017 Revert of Disable failing blink_style.key_mobile_sites (patchset #1 id:1 of https://codereview.chromium.org/2758533003/ ) Reason for revert: Root cause should be fixed. Original issue's description: > Disable failing blink_style.key_mobile_sites > > BUG= 702194 > > Review-Url: https://codereview.chromium.org/2758533003 > Cr-Commit-Position: refs/heads/master@{#457432} > Committed: https://chromium.googlesource.com/chromium/src/+/4d954851ba9c129b4529302f77862f6b812165d8 TBR=sullivan@chromium.org # Not skipping CQ checks because original CL landed more than 1 days ago. BUG= 702194 Review-Url: https://codereview.chromium.org/2768683005 Cr-Commit-Position: refs/heads/master@{#458730} [modify] https://crrev.com/c4928c8fba4766c4fb25cbb0ecce574101b29aca/tools/perf/benchmarks/blink_style.py
,
Mar 27 2017
This test is still failing. See https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5_Perf__1_%2F5451%2F%2B%2Frecipes%2Fsteps%2Fblink_style.key_mobile_sites%2F0%2Fstdout for an example log. It appears to be failing with the same error. See https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/profiler-file-id_5-2017-03-27_12-37-4515759.png for what the screen looks like. Should we revert again?
,
Mar 27 2017
Err, disable the test again.
,
Mar 28 2017
I am going to go ahead and revert the revert since it is still failing.
,
Mar 29 2017
rsleevi@: problem isn't fixed, can you take a look?
,
Mar 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/908b7419b238a50d685634fa7844fd690830fa9c commit 908b7419b238a50d685634fa7844fd690830fa9c Author: rnephew <rnephew@chromium.org> Date: Wed Mar 29 13:12:41 2017 [Android][Telemetry] Disable failing key_mobile_sites benchmarks. BUG= 702194 TBR=nednguyen@google.com Review-Url: https://codereview.chromium.org/2781713005 Cr-Commit-Position: refs/heads/master@{#460362} [modify] https://crrev.com/908b7419b238a50d685634fa7844fd690830fa9c/tools/perf/benchmarks/blink_style.py [modify] https://crrev.com/908b7419b238a50d685634fa7844fd690830fa9c/tools/perf/benchmarks/oilpan_gc_times.py [modify] https://crrev.com/908b7419b238a50d685634fa7844fd690830fa9c/tools/perf/benchmarks/smoothness.py [modify] https://crrev.com/908b7419b238a50d685634fa7844fd690830fa9c/tools/perf/benchmarks/v8.py
,
Mar 29 2017
Do you have details for a recent log (on the road, but happy to do as much remote debugging as possible)
,
Mar 29 2017
Issue 704583 has been merged into this issue.
,
Mar 29 2017
Here is the last run before my disabling CL landed: https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf%20%283%29/builds/4840 STDIO Log: https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5_Perf__3_%2F4840%2F%2B%2Frecipes%2Fsteps%2Fsmoothness.sync_scroll.key_mobile_sites_smooth%2F0%2Fstdout Failure Message: Traceback (most recent call last): File "/b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 89, in _RunStoryAndProcessErrorIfNeeded state.RunStory(results) File "/b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function return func(*args, **kwargs) File "/b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 299, in RunStory self._current_page.Run(self) File "/b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 108, in Run shared_state.page_test.RunNavigateSteps(self, current_tab) File "/b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/telemetry/telemetry/page/legacy_page_test.py", line 195, in RunNavigateSteps page.RunNavigateSteps(action_runner) File "/b/c/b/Android_Nexus5_Perf__3_/src/tools/perf/page_sets/key_mobile_sites_smooth.py", line 50, in RunNavigateSteps super(LinkedInSmoothPage, self).RunNavigateSteps(action_runner) File "/b/c/b/Android_Nexus5_Perf__3_/src/tools/perf/page_sets/key_mobile_sites_pages.py", line 116, in RunNavigateSteps 'document.getElementById("profile-view-scroller") !== null') File "/b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 252, in WaitForJavaScriptCondition return self._tab.WaitForJavaScriptCondition(*args, **kwargs) File "/b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 191, in WaitForJavaScriptCondition return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs) File "/b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 294, in WaitForJavaScriptCondition e.message + '\n' + debug_message) TimeoutException: Timed out while waiting 60s for IsJavaScriptExpressionTrue. Console output: INFO:root:Chrome version: 3056 INFO:devil.utils.cmd_helper:[host]> /b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 03852c29003b9cb6 shell '( rm -f /data/local/chrome-trace-config.json );echo %$?' INFO:devil.utils.cmd_helper:[host]> /b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 03852c29003b9cb6 shell '( dumpsys SurfaceFlinger --latency SurfaceView );echo %$?' INFO:devil.utils.cmd_helper:[host]> /b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 03852c29003b9cb6 shell '( dumpsys SurfaceFlinger --latency SurfaceView );echo %$?' INFO:devil.utils.cmd_helper:[host]> /b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 03852c29003b9cb6 shell '( ps | grep -F surfaceflinger; echo "PIPESTATUS: ${PIPESTATUS[@]}" );echo %$?' INFO:devil.utils.cmd_helper:[host]> /b/c/b/Android_Nexus5_Perf__3_/src/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 03852c29003b9cb6 shell '( log -p i -t Telemetry '"'"'END LinkedIn'"'"' );echo %$?' [ FAILED ] LinkedIn (90931 ms) I have android devices at my desk, if you need me to run anything locally and send you the output just let me know.
,
Mar 30 2017
Are you sure that's related? Looking at the other tests, they seemed to pass, which suggests the certificate related errors were fixed, and this was some sort of other timeout. is /b/c/b/Android_Nexus5_Perf__3_/src/tools/perf/page_sets/data/key_mobile_sites_002.wpr the page that plays back the LinkedIn data? Can you send that over? The patch only added support for DNS names, because "surely" no one would be using direct IP addresses, right?
,
Mar 30 2017
Mikecase@ is the sheriff now, assigning to him. I cna help you look at this tomorrow if you need me to case. If you run the benchmark locally, it will automatically download the wpr archive to 'src/tools/perf/page_sets/data'
,
Mar 30 2017
I'm not sure I understand what the steps are to "run the benchmark locally" - are their further details beyond that in Comment #22?
,
Mar 30 2017
I think you will want to do something like.... # Build and install Chrome ninja -C out/Debug -j30 -l10 chrome_public_test_apk adb install out/Debug/apks/ChromePublicTest.apk # Run test tools/perf/run_benchmark -v smoothness.sync_scroll.key_mobile_sites_smooth --browser=android-chromium --output-dir=./output_dir --output-format=json --story-filter=LinkedIn ... to run the benchmark that is failing locally (I haven't read thru this entire bug, but this looks like the failing test.)
,
Mar 30 2017
Looks like you also need to add the arg --also-run-disabled-tests. So like... tools/perf/run_benchmark -v smoothness.sync_scroll.key_mobile_sites_smooth --browser=android-chromium --output-dir=./output_dir --output-format=json --story-filter=LinkedIn --also-run-disabled-tests
,
Mar 30 2017
Thats for key_mobile_sites_smooth which I'm not sure if thats different, he migh twant something more like: ./run_benchmark --browser=android-chromium blink_style.key_mobile_sites -v --also-run-disabled-tests --story-filter="any_filter_you_want_here" That will run it locally if you have an android phone attached.
,
Mar 30 2017
To #38: sorry for your confusion, Ryan. What we have in #22 is how to reply the web-page-replay's recording directly without relying on Telemetry (usually this is recommended for debugging issues only related to WPR). In #37, what Randy mean is the instruction to run the Telemetry benchmark, which uses almost the same mechanism underneath to automate browser test against the web-page-replay server. The slight difference is Telemetry use the ts_proxy server as the middle man between it & WPR server for traffic shaping, but my guess is this difference doesn't really matter in this case.
,
Mar 31 2017
So Comment #39 & #40 will be difficult because I'm on the road and without an Android device. I can see about that Monday. I'm currently looking through the wpr archive to see if there is an IP address related loading. If someone can run the tests and supply a "--log-net-log" directive to Chrome as part of it (to record a chrome://net-internals trace of the test run), that would help definitively identify if it's a networking error or something else/independent.
,
Apr 7 2017
,
Apr 7 2017
Apologies for the late reply here, got sucked up into other things. I'm more suspicious that the failure is related - the certs being used are not for IP addresses, and there's nothing to indicate it's a cert related failure. As I mentioned, I don't have any access to an Android device to further test this (nor would I know where to start), so definitely assistance from a sheriff would be useful in understanding what your tests are doing. But I don't think they're (any more) related to my change.
,
Apr 11 2017
Any updates or additional logs? I wanted to follow-up, because I don't want people think I'm not interested in helping on this, other than I can find no reason from the available information and steps taken so far to believe the remaining failures are related. But I would love if someone can sanity check that or offer any additional suggestions here for this piece of infrastructure.
,
Apr 11 2017
I run the benchmark again with "--log-net-log" & use "adb logcat" to capture Chrome log but that doesn't show anything special. Probably we need some other work to surface net log in Android logcat? Interestingly, there is nothing on the device screen showing that there is a credential issue with the browser, so I suspect this is a problem with WPR recording for LinkedIn page.
,
Apr 11 2017
--log-net-log requires specifying a filepath (e.g. https://dev.chromium.org/for-testers/providing-network-details ). Apologies for not being clearer. It does not log to logcat.
,
Apr 12 2017
Thanks Ned. Looks like I was testing wrong! https://github.com/chromium/web-page-replay/blob/841e12ec929a615bffeeec758668e9bd4686e952/sslproxy.py#L35 is set to call https://github.com/chromium/web-page-replay/blob/841e12ec929a615bffeeec758668e9bd4686e952/httpproxy.py#L403 for all the connections. https://github.com/chromium/web-page-replay/blob/841e12ec929a615bffeeec758668e9bd4686e952/httpproxy.py#L407 then gets the cert from the HAR using https://github.com/chromium/web-page-replay/blob/master/httparchive.py#L432 (if stored) or from the network using https://github.com/chromium/web-page-replay/blob/6cffdf1fb6c9a6d5dccbcc9cc18b8738a538eeba/certutils.py#L149 Once the cert is loaded, it creates a clone of the original cert using https://github.com/chromium/web-page-replay/blob/841e12ec929a615bffeeec758668e9bd4686e952/httpproxy.py#L413 , which copies the SANs from the original cert to the dummy cert using https://github.com/chromium/web-page-replay/blob/6cffdf1fb6c9a6d5dccbcc9cc18b8738a538eeba/certutils.py#L240 If the SANs are missing in the original cert (which should never be the case), or if fetching the host cert fails (which seems likely the case here on the bots, especially if they have no network access and the HAR doesn't store the cert), then https://github.com/chromium/web-page-replay/blob/6cffdf1fb6c9a6d5dccbcc9cc18b8738a538eeba/certutils.py#L240 isn't true, and the SANs aren't copied. As far as I can tell, it's "just" a matter of making sure that https://github.com/chromium/web-page-replay/blob/6cffdf1fb6c9a6d5dccbcc9cc18b8738a538eeba/certutils.py#L239 is populated with the SAN if https://github.com/chromium/web-page-replay/blob/6cffdf1fb6c9a6d5dccbcc9cc18b8738a538eeba/certutils.py#L240 is empty. Separately, I'm not sure if the original LinkedIn recording cert is from pre-2014; if so, then we'd also have to populate SANs regardless, but I'm in a poor position to evaluate whether that change is appropriate, because it means that WPR could "fix" a broken server (by adding the SAN when the original server lacks it), and that seems wrong. So what are next steps: 1) Have I been reading the HAR wrong? Do you have enough information to trace through to see if the original cert is stored in the LinkedIn case for "touch.licdn.com" (which is the host that's failing) 2) Assuming the original cert is _not_ in the HAR, then it's a matter of something like "if server_cert_str: .. existing logic else: reused_extensions = [ crypto.X509Extension('subjectAltName', False, 'DNS:' + server_host) ] " If you wanted to make it unconditionally 'fix' old HARs, then it's instead "if server_host_str: ... if (SAN isn't in reused_extensions): reused_extensions.append(...) "
,
Apr 12 2017
I can confirm that there is no SERVER_CERT in the WPR archive (we use Python pickle, not HAR to store the data, sadly :-() I also think unconditionally 'fix' old recording is wrong, as it make the tests more unrealistic. The best thing here is probably try to rerecord the WPR file for LinkedIn so that it contains the original cert & make error reporting clearer.
,
Apr 12 2017
,
Apr 12 2017
If there's no server cert at all, then adding the first part from #2 will fix WPR and not require a new recording (that is, the "else:" clause). It was just a path I missed with the previous fix.
,
Apr 12 2017
,
Apr 12 2017
I confirm that Ryan's fix in https://github.com/chromium/web-page-replay/pull/91 fix the issue for Linkedin page.
,
Apr 12 2017
Once https://codereview.chromium.org/2811373002/ is roll to Chromium, the test can be reenable. Set Ryan as owner since he did most of the work.
,
Apr 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6a99f7cdee4e9a37392b7e4683f6cce9645d2443 commit 6a99f7cdee4e9a37392b7e4683f6cce9645d2443 Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org> Date: Wed Apr 12 21:38:22 2017 Roll src/third_party/catapult/ 90b692aaf..41bda7361 (1 commit) https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/90b692aafa2d..41bda73617ed $ git log 90b692aaf..41bda7361 --date=short --no-merges --format='%ad %ae %s' 2017-04-12 nednguyen [web-page-replay] Roll WPR to the latest commit Created with: roll-dep src/third_party/catapult BUG= 702194 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=sullivan@chromium.org Change-Id: I5b8dc6073eb3f6b3b22799e62b06248d8d6e635c Reviewed-on: https://chromium-review.googlesource.com/476070 Reviewed-by: <catapult-deps-roller@chromium.org> Commit-Queue: <catapult-deps-roller@chromium.org> Cr-Commit-Position: refs/heads/master@{#464157} [modify] https://crrev.com/6a99f7cdee4e9a37392b7e4683f6cce9645d2443/DEPS
,
Apr 25 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/40ce519e1b5e2dce59f290f84bcfce20b2a3ba5c commit 40ce519e1b5e2dce59f290f84bcfce20b2a3ba5c Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org> Date: Tue Apr 25 00:50:04 2017 Roll src/third_party/catapult/ f0a161081..892a0b6a4 (1 commit) https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/f0a16108160d..892a0b6a45cc $ git log f0a161081..892a0b6a4 --date=short --no-merges --format='%ad %ae %s' 2017-04-24 nednguyen Roll WPR to the lastest commit Created with: roll-dep src/third_party/catapult BUG= 702194 ,711634 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=sullivan@chromium.org Change-Id: Ie143bf24f0c15a8694009dcad6f5291560c7fe15 Reviewed-on: https://chromium-review.googlesource.com/486164 Reviewed-by: <catapult-deps-roller@chromium.org> Commit-Queue: <catapult-deps-roller@chromium.org> Cr-Commit-Position: refs/heads/master@{#466844} [modify] https://crrev.com/40ce519e1b5e2dce59f290f84bcfce20b2a3ba5c/DEPS
,
Apr 25 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/bf332e3e4c1ccfcc5ae69e2a63ea34879c4e3dee commit bf332e3e4c1ccfcc5ae69e2a63ea34879c4e3dee Author: catapult-deps-roller@chromium.org <catapult-deps-roller@chromium.org> Date: Tue Apr 25 15:41:36 2017 Roll src/third_party/catapult/ b367d4648..bc8989ef1 (1 commit) https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/b367d46487c3..bc8989ef1a06 $ git log b367d4648..bc8989ef1 --date=short --no-merges --format='%ad %ae %s' 2017-04-25 nednguyen Revert of Roll WPR to the lastest commit (patchset #2 id:20001 of https://codereview.chromium.org/2841533003/ ) Created with: roll-dep src/third_party/catapult BUG= 702194 ,711634 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, see: http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel TBR=sullivan@chromium.org Change-Id: I042a4de7dc9dc70b9ce6014cf221fab771f70f1d Reviewed-on: https://chromium-review.googlesource.com/486206 Reviewed-by: <catapult-deps-roller@chromium.org> Commit-Queue: <catapult-deps-roller@chromium.org> Cr-Commit-Position: refs/heads/master@{#466998} [modify] https://crrev.com/bf332e3e4c1ccfcc5ae69e2a63ea34879c4e3dee/DEPS
,
Apr 25 2017
,
Apr 26 2017
Ryan: this is affecting so many benchmarks & the fixes to WPR keep getting revert because it broke more pages. I think we should just enable "--ignore-certificate-error" for all Android perf bots, then look into fixing the certificate in WPR once we have better CQ coverage?
,
Apr 26 2017
I know this issue is a bit longer-running than a P1 should be, but it's actively worked on, and affecting many benchmarks. I think P1 is the right priority.
,
Apr 26 2017
That works. Just understand that --ignore-certificate-errors will bypass/disable the cache and double the socket workload. Just making sure that it's clear that some degree of rebaselining will be needed.
,
May 1 2017
key_mobile_sites test are passing in https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf%20%281%29/builds/5601
,
Jan 16
,
Jan 16
|
||||||||||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||||||||||
Comment 1 by 42576172...@developer.gserviceaccount.com
, Mar 16 2017