media.android.tough_video_cases benchmark failing on Android Nexus 5X bot |
||||||||||||
Issue descriptionhttps://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Nexus5X%20Perf/builds/24 Seems very probably that this CL is the cause the of the failure... https://codereview.chromium.org/2866703004/ here is the error message... (ERROR) 2017-05-09 23:56:22,147 android_browser_finder.Create:129 Failure while creating Android browser. Traceback (most recent call last): File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome/android_browser_finder.py", line 127, in Create browser_backend, self._platform_backend, self._credentials_path) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 58, in __init__ self._browser_backend.Start() File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function return func(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome/android_browser_backend.py", line 151, in Start self._WaitForBrowserToComeUp() File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function return func(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome/chrome_browser_backend.py", line 159, in _WaitForBrowserToComeUp raise exceptions.BrowserConnectionGoneException(self.browser, e)
,
May 10 2017
I mean, I can start a bisect, but Im almost certain its just going to return the CL where you modified the test (that I linked above). Could be wrong.
,
May 10 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8979949207542331168
,
May 10 2017
,
May 10 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8979946067435645072
,
May 10 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8979946012332595408
,
May 10 2017
Only one test failed: crowd720_vp9.webm&seek, which is called "Page35". It is really strange that this test is failing but all the other tests are succeeding because this test is very similar to the other tests. For example, the only difference between it and Page33 is the resolution of the media file. If this is really the only one failing, then we could comment that video out here https://cs.chromium.org/chromium/src/tools/perf/page_sets/tough_video_cases.py?l=433 and then investigate further. I would appreciate if we did that instead of turning the whole benchmark off. I'm OOO starting tomorrow and going for the next three weeks. +John is also OOO right now. watk@, could you please take a look at this bug? You should be able to repro locally using these instructions: https://sites.google.com/a/chromium.org/dev/developers/telemetry/run_locally The benchmark name is media.android.tough_video_cases. Make sure you run against ToT because Dev channel doesn't seem to have the bug.
,
May 11 2017
=== BISECT JOB RESULTS === NO Perf regression found, tests failed to produce values Bisect Details Configuration: android_nexus5X_perf_bisect Benchmark : media.android.tough_video_cases Metric : application_energy_consumption_mwh/video.html?src_crowd720_vp9.webm To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter='video.html?src=crowd720_vp9.webm&seek' media.android.tough_video_cases Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8979946067435645072 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5200351927468032 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
May 11 2017
=== BISECT JOB RESULTS === NO Test failure found Bisect Details Configuration: android_nexus5X_perf_bisect Benchmark : media.android.tough_video_cases Metric : application_energy_consumption_mwh/video.html?src_crowd720_vp9.webm Revision Exit Code N chromium@470346 0 +- N/A 8 good chromium@470430 0 +- N/A 8 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=video.html.src.crowd720.vp9.webm media.android.tough_video_cases Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8979949207542331168 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=6446804318552064 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
May 11 2017
=== BISECT JOB RESULTS === NO Test failure found Bisect Details Configuration: android_nexus5X_perf_bisect Benchmark : media.android.tough_video_cases Metric : application_energy_consumption_mwh/application_energy_consumption_mwh Revision Exit Code N chromium@470346 0 +- N/A 20 good chromium@470430 0 +- N/A 20 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter='video.html?src=crowd720_vp9.webm&seek' media.android.tough_video_cases Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8979946012332595408 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=6402074046103552 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
May 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8979933645068237232
,
May 11 2017
=== BISECT JOB RESULTS === NO Perf regression found, tests failed to produce values Bisect Details Configuration: android_nexus5X_perf_bisect Benchmark : media.android.tough_video_cases Metric : application_energy_consumption_mwh/video.html?src_crowd720_vp9.webm To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter='video.html?src=crowd720_vp9.webm&seek' media.android.tough_video_cases Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8979933645068237232 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5200351927468032 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
May 11 2017
I can't repro this locally on a Pixel C. All 16 tests succeed. mikecase: I don't know how to interpret these bisect results?
,
May 11 2017
#9 and #10 are saying the test passed (exit code 0) on both the "good" and "bad" revision. So that plus the fact that I think it's only failing on N5X make me think it's a device issue. But I think #8 and #12 are accidentally bisecting for the mean value to change. However, they output is saying that the test did not produce any values. So that's not good. It's not getting output for application_energy_consumption_mwh/video.html?src_crowd720_vp9.webm.
,
May 11 2017
The story filter might be wrong, there don't seem to be any tests running (which may explain why #9 and #10 dont find anything). Maybe we should try rerunning this without the story filter field filled in.
,
May 11 2017
Ah, thanks. Re 8 & 12: Potentially we don't expect to get that metric for the story filter that was used. A metric we should definitely get is seek time. Appears as "seek.crowd720_vp9.webm_seek_cold" in results.html.
,
Jun 1 2017
Chris, do you want to take this bug since you have already done some work on it? I assigned to you in case to want to take it. Otherwise please assign it back to me and I will look into it next week. +Vitali or +yiningc@ may have a Nexus 5X that we could borrow if we think the issue is specific to that model.
,
Jun 2 2017
Caleb, we have Nexus 5X that you could borrow.
,
Jun 2 2017
Don't have too much time to look at this soon. I'll let you try to repro it next week.
,
Jun 2 2017
,
Jun 8 2017
I reproduced this on the nexus 5x that Vitali kindly provided. Everything is the same except that the test that it failed on was different: [ PASSED ] 15 tests. [ FAILED ] 1 test, listed below: [ FAILED ] video.html?src=tulip2.mp4&seek The failed test happened in the middle of a bunch of passed tests. Then after it failed, the following tests passed again. So what I'm thinking is that something must be accumulating over test runs that eventually gets to a point that it resets, which fails the test, and then it starts over again. I'll continue looking into this.
,
Jun 8 2017
I also have a guess for why this started happening when I merged tough_video_cases_extra pages into tough_video_cases pages. I think that now that tough_video_cases has more pages it reaches the point where it fails. Before it didn't have enough pages in it to hit this bug.
,
Jun 8 2017
So the bug is that Chrome has failed to provide a devtools connection to us: BrowserConnectionGoneException. It is raised here: https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/chrome_browser_backend.py?sq=package:chromium&l=157 So it seems like this is a Telemetry bug. Ned, do you agree? I'm assigning this to you to find someone from your team to work on it.
,
Jun 9 2017
Hmhh, this is from the log: ******************************************************************************** UI dump - (no package): - (no id) - org.chromium.chrome: - (no id) - android:id/content - android:id/navigationBarBackground - android:id/statusBarBackground - org.chromium.chrome:id/action_bar_root - org.chromium.chrome:id/fre_content_wrapper - org.chromium.chrome:id/fre_image_and_content - org.chromium.chrome:id/fre_main_layout - org.chromium.chrome:id/fre_pager - org.chromium.chrome:id/image - org.chromium.chrome:id/send_report_checkbox['Help make Chrome better by sending usage statistics and crash reports to Google.'] - org.chromium.chrome:id/terms_accept['Accept & continue'] - org.chromium.chrome:id/title['Welcome to Chrome'] - org.chromium.chrome:id/tos_and_privacy[u'By using this application, you agree to Chrome\u2019s Terms of Service and Privacy Notice.'] So basically we are getting stuck at the Chrome Android consent screen. Not sure why this happen. Juan: thoughts?
,
Jun 9 2017
Hmm.. that's the odd flakiness we've seen when the devtools connection is not responding, possibly Chrome not picking up command line flags, but we're not sure. Are there any recent logs I can look at?
,
Jun 9 2017
Log is in: https://chromium-swarm.appspot.com/task?id=36a0aedbdc44c610&refresh=10&show_raw=1 It's very unclear why only "video.html?src=crowd720_vp9.webm&seek" story is failing if Chrome not picking up the command line flags is the problem here.
,
Jun 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8f243d9ff32782239f6d6c49c9c9fab2ca5021d2 commit 8f243d9ff32782239f6d6c49c9c9fab2ca5021d2 Author: crouleau <crouleau@chromium.org> Date: Fri Jun 09 18:26:36 2017 Remove Page35 "crowd720_vp9.webm&seek" since not needed. Also, removing this will help with crbug/720514 to show that the test case itself is not the root cause of the bug. (Or if it is the root cause, then this will fix that bug.) crowd720_vp9.webm&seek is not needed since we have tulip2.vp9.webm&seek and crowd1080_vp9.webm&seek. Those two page provide enough vp9 coverage to make up for this removal. BUG=720514, 710253 Review-Url: https://codereview.chromium.org/2929003003 Cr-Commit-Position: refs/heads/master@{#478341} [modify] https://crrev.com/8f243d9ff32782239f6d6c49c9c9fab2ca5021d2/tools/perf/page_sets/tough_video_cases.py
,
Jun 12 2017
,
Aug 2
,
Jan 16
,
Jan 16
|
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by crouleau@chromium.org
, May 10 2017Owner: mikec...@chromium.org