load:games:spychase system health story failing on webview bots |
||
Issue description
A few stories, most notably load:games:spychase, have started failing on webview bots.
On that story the error is timing out after 60 seconds waiting for:
File "/b/swarming/w/ir/tools/perf/page_sets/system_health/loading_stories.py", line 385, in _DidLoadDocument
'document.querySelector("#game canvas").style.background !== ""')
Other stories started failing around that time
https://ci.chromium.org/buildbot/chromium.perf/Android%20Nexus6%20WebView%20Perf/
(most recent build 905 from the left to older builds on the right)
load:games:spychase FFFF?-?F--
browse:social:facebook_infinite_scroll ----F-?---
browse:shopping:avito ------F---
load:games:lazors ----?-?---
load:media:dailymotion ----?-?---
https://ci.chromium.org/buildbot/chromium.perf/Android%20Nexus5X%20WebView%20Perf/
(most recent build 863)
load:games:spychase FF--FFF---
On the internal webview-low-end-phone (most recent build 6286):
load:games:spychase 3-331331?-~
browse:social:facebook --12121??-~
browse:news:washingtonpost ----1?-??-~
browse:news:cnn ----?-11?-~
browse:shopping:amazon ----?-11?-~
browse:news:cricbuzz ----?-1??-~
browse:shopping:avito ----?-1??-~
background:search:google ----?-1??-~
Intersecting the first failure on all those bots throws up the following commit range:
http://test-results.appspot.com/revision_range?start=525532&end=525587
I'll see if a bisect can help narrow down the source of the issues.
,
Dec 22 2017
=== BISECT JOB RESULTS === NO Test failure found Bisect Details Configuration: android_webview_arm64_aosp_perf_bisect Benchmark : system_health.common_mobile Metric : cpuPercentage:all_processes:all_threads:Load:Successful_avg/load_games/load_games_spychase Revision Exit Code N chromium@525532 1 +- N/A 20 good chromium@525587 1 +- N/A 20 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.games.spychase system_health.common_mobile More information on addressing performance regressions: http://g.co/ChromePerformanceRegressions Debug information about this bisect: https://chromeperf.appspot.com/buildbucket_job_status/8959509590294625904 For feedback, file a bug with component Speed>Bisection
,
Dec 22 2017
Let's try again with a wider range, triggering bisect from: https://chromeperf.appspot.com/report?sid=1edfaaedda4af535ab9b15435f6e83efb98994791ec10da32a345bca3c223a78
,
Dec 22 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8959507737768807056
,
Dec 22 2017
=== BISECT JOB RESULTS === NO Test failure found Bisect Details Configuration: android_webview_arm64_aosp_perf_bisect Benchmark : system_health.common_mobile Metric : after_load:power_avg/load_games/load_games_spychase Revision Exit Code N chromium@525400 1 +- N/A 20 good chromium@525587 1 +- N/A 20 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.games.spychase system_health.common_mobile More information on addressing performance regressions: http://g.co/ChromePerformanceRegressions Debug information about this bisect: https://chromeperf.appspot.com/buildbucket_job_status/8959507737768807056 For feedback, file a bug with component Speed>Bisection
,
Dec 22 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8959500421456967728
,
Dec 22 2017
=== BISECT JOB RESULTS === NO Test failure found Bisect Details Configuration: android_webview_arm64_aosp_perf_bisect Benchmark : system_health.common_mobile Metric : cpuPercentage:all_processes:all_threads:all_stages:all_initiators_avg/load_games/load_games_spychase Revision Exit Code N chromium@525300 1 +- N/A 20 good chromium@525587 1 +- N/A 20 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.games.spychase system_health.common_mobile More information on addressing performance regressions: http://g.co/ChromePerformanceRegressions Debug information about this bisect: https://chromeperf.appspot.com/buildbucket_job_status/8959500421456967728 For feedback, file a bug with component Speed>Bisection
,
Jan 2 2018
Not sure what's going on, but I'll start by disabling the test. It's still failing quite consistently.
,
Jan 2 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/cd0be3816c008c2eb9b812382cfc77d123d5715f commit cd0be3816c008c2eb9b812382cfc77d123d5715f Author: Juan Antonio Navarro Perez <perezju@google.com> Date: Tue Jan 02 15:37:01 2018 [tools/perf] Disable load:games:spychase on webview Test has been consistently failing on webview bots. Bisect was unable to find culprit. TBR=nednguyen@google.com NOTRY=true Bug: 797261 Change-Id: Ia0bf67071777bc98e5aea77c729ce5d7f2971e2d Reviewed-on: https://chromium-review.googlesource.com/847575 Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org> Cr-Commit-Position: refs/heads/master@{#526482} [modify] https://crrev.com/cd0be3816c008c2eb9b812382cfc77d123d5715f/tools/perf/expectations.config
,
Jan 3 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/7cd3aa1df66f0df0256977c286b964488ffd9f07 commit 7cd3aa1df66f0df0256977c286b964488ffd9f07 Author: Juan Antonio Navarro Perez <perezju@google.com> Date: Wed Jan 03 12:03:17 2018 [tools/perf] Disable system_health.memory_mobile/load:games:spychase (for real this time) TBR=nednguyen@google.com NOTRY=true Bug: 797261 Change-Id: I68bf8771f4b3eb0b8e0cd662b87067b805b8a259 Reviewed-on: https://chromium-review.googlesource.com/848933 Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org> Cr-Commit-Position: refs/heads/master@{#526670} [modify] https://crrev.com/7cd3aa1df66f0df0256977c286b964488ffd9f07/tools/perf/expectations.config
,
Jul 19
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/13aa0015a40000
,
Jul 21
😿 Pinpoint job stopped with an error. https://pinpoint-dot-chromeperf.appspot.com/job/13aa0015a40000 All of the runs failed. The most common error (1/20 runs) was: IOError: [Errno 2] No such file or directory: '/b/s/w/itHBpt9c/tmp0F2vx1telemetry/histograms.json'
,
Jul 23
Ned, Simon, do you know if we already have a bug and/or fix for that error in #12?
,
Jul 23
These are failing because telemetry doesnt know what to run. Looks like the specified benchmark is "system_health.common_mobile/load:games:spychase", instead f just system_health.common_mobile
,
Jul 23
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/14e0dad9a40000
,
Jul 24
📍 Job complete. See results below. https://pinpoint-dot-chromeperf.appspot.com/job/14e0dad9a40000
,
Jul 25
Simon, where is the benchmark specified incorrectly?
,
Jul 25
Looks like whoever kicked off #c12 copy pasted both the benchmark name and part of the test path into the benchmark field when starting that pinpoint try job.
,
Jul 25
📍 Pinpoint job started. https://pinpoint-dot-chromeperf.appspot.com/job/16e1a9f7a40000
,
Jul 26
📍 Job complete. See results below. https://pinpoint-dot-chromeperf.appspot.com/job/16e1a9f7a40000
,
Jul 27
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/27548c03b3ecb23329b7952a891aaebc504243c4 commit 27548c03b3ecb23329b7952a891aaebc504243c4 Author: Liquan(Max) Gu <maxlg@chromium.org> Date: Fri Jul 27 14:49:15 2018 Enable load:games:spychase system health story This test is working correctly on other platforms. Try enabling the test to verify that it's no longer flaky. Bug: 797261 Change-Id: Id31748868eacb41fbc560a877eef22db3e6c29b1 Reviewed-on: https://chromium-review.googlesource.com/1143597 Reviewed-by: Annie Sullivan <sullivan@chromium.org> Commit-Queue: Liquan (Max) Gǔ <maxlg@chromium.org> Cr-Commit-Position: refs/heads/master@{#578638} [modify] https://crrev.com/27548c03b3ecb23329b7952a891aaebc504243c4/tools/perf/expectations.config |
||
►
Sign in to add a comment |
||
Comment 1 by 42576172...@developer.gserviceaccount.com
, Dec 22 2017