New issue
Advanced search Search tips

Issue 671643 link

Starred by 1 user
Status: Fixed
Owner: ----
Closed: Dec 2016
Cc:
simonhatch@chromium.org
charliea@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug-Regression

Blocked on:
issue 664765
issue 673440


Sign in to add a comment

media.mse_cases flaky failure on chromium.perf Android One (3) at 435222 onward

Project Member Reported by rnep...@chromium.org, Dec 6 2016 
Revision range first seen: 435222 - 435255
Link to failing step log:
https://build.chromium.org/p/chromium.perf/builders/Android%20One%20Perf%20%283%29/builds/4458/steps/media.mse_cases/logs/stdio


https://build.chromium.org/p/chromium.perf/builders/Android%20One%20Perf%20%283%29/builds/4436

Traceback (most recent call last):
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 87, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 299, in RunStory
    self._current_page.Run(self)
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 107, in Run
    shared_state.page_test.RunNavigateSteps(self, current_tab)
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/telemetry/telemetry/page/legacy_page_test.py", line 195, in RunNavigateSteps
    page.RunNavigateSteps(action_runner)
  File "/b/rr/tmpoTrj24/w/src/tools/perf/page_sets/mse_cases.py", line 14, in RunNavigateSteps
    super(MseCasesPage, self).RunNavigateSteps(action_runner)
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/telemetry/telemetry/page/__init__.py", line 116, in RunNavigateSteps
    url, script_to_evaluate_on_commit=self.script_to_evaluate_on_commit)
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 177, in Navigate
    timeout_in_seconds=timeout_in_seconds))
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 56, in _RunAction
    action.RunAction(self._tab)
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/telemetry/telemetry/internal/actions/navigate.py", line 30, in RunAction
    tab.WaitForFrameToBeDisplayed()
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 104, in WaitForFrameToBeDisplayed
    timeout)
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/rr/tmpoTrj24/w/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 142, in WaitForJavaScriptExpression
    e.message + '\n' + debug_message)
TimeoutException: Timed out while waiting 90s for IsJavaScriptExpressionTrue.
Console output:


[  FAILED  ] startup_test.html?testType=V (114139 ms)

If the test is disabled, please downgrade to Pri-2.

Comment 2 by rnep...@chromium.org, Dec 6 2016

Summary: media.mse_cases flaky failure on chromium.perf Android One (3) at 435222 onward (was: media.mse_cases flaky failure on chromium.perf at 435222 onward)
Project Member

Comment 3 by 42576172...@developer.gserviceaccount.com, Dec 6 2016 


===== BISECT JOB RESULTS =====
Status: started


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N   Good?
chromium@435212  0          N/A      20  good
chromium@435265  0          N/A      20  bad

Bisect job ran on: android_one_perf_bisect
Bug ID: 671643

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter='startup_test.html?testType=V' media.mse_cases
Test Metric: append/AV_buffer_post_load
Relative Change: 0.00%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_one_perf_bisect/builds/1850
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994009548718899792


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=4906031434432512

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 5 by 42576172...@developer.gserviceaccount.com, Dec 6 2016 


===== BISECT JOB RESULTS =====
Status: started


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N   Good?
chromium@435222  0          N/A      20  good
chromium@435255  0          N/A      20  bad

Bisect job ran on: android_one_perf_bisect
Bug ID: 671643

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=AV.buffer.post.load media.mse_cases
Test Metric: append/AV_buffer_post_load
Relative Change: 0.00%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_one_perf_bisect/builds/1851
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994007300598562816


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5199273816227840

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 6 by eyaich@chromium.org, Dec 7 2016

Cc: simonhatch@chromium.org
Can you confirm that if one of these 20 runs fails then the exit code shows a failure.

Comment 7 by simonhatch@chromium.org, Dec 7 2016 

Yeah any single failure should make the entire run a failure.

One thing I noticed was that I'm not sure if any tests are actually running:

All the "bad" revision outputs were short and ended with this:

@@@STEP_LOG_LINE@Captured Output@INFO:root:Chose browser: PossibleAndroidBrowser(browser_type=android-chromium)@@@
@@@STEP_LOG_LINE@Captured Output@[  PASSED  ] 0 tests.@@@


I'll try rekicking this without a story filter.
Project Member

Comment 9 by 42576172...@developer.gserviceaccount.com, Dec 7 2016 


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N   Good?
chromium@435222  0          N/A      20  good
chromium@435239  0          N/A      20  good
chromium@435247  0          N/A      20  good
chromium@435255  1          N/A      20  bad

Bisect job ran on: android_one_perf_bisect
Bug ID: 671643

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.mse_cases
Test Metric: append/AV_buffer_pre_load
Relative Change: 0.00%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_one_perf_bisect/builds/1857
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8993917167529370944


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6371941051531264

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 10 by simonhatch@chromium.org, Dec 7 2016

Blockedon: 664765
Project Member

Comment 12 by 42576172...@developer.gserviceaccount.com, Dec 7 2016 


===== BISECT JOB RESULTS =====
Status: started


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N   Good?
chromium@435247  0          N/A      30  good
chromium@435255  0          N/A      30  bad

Bisect job ran on: android_one_perf_bisect
Bug ID: 671643

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests media.mse_cases
Test Metric: append/AV_buffer_pre_load
Relative Change: 0.00%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_one_perf_bisect/builds/1858
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8993899590488922512


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5879232738099200

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 14 by 42576172...@developer.gserviceaccount.com, Dec 8 2016 

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_one_perf_bisect/builds/1859
Failure reason: the build has failed due to infrastructure failure.
Project Member

Comment 16 by 42576172...@developer.gserviceaccount.com, Dec 8 2016 


===== BISECT JOB RESULTS =====
Status: started


===== TESTED REVISIONS =====
Revision         Exit Code  Std Dev  N   Good?
chromium@435222  0          N/A      50  good
chromium@435255  0          N/A      50  bad

Bisect job ran on: android_one_perf_bisect
Bug ID: 671643

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter='startup_test.html?testType=V' media.mse_cases
Test Metric: test_total/AV_buffer_pre_load
Relative Change: 0.00%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_one_perf_bisect/builds/1863
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8993829944095682160


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5866602447241216

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 17 by simonhatch@chromium.org, Dec 12 2016

Blockedon: 673440
Logged  crbug.com/673440  for all the tests that show status "started" but are actually finished.

Comment 18 by eyaich@chromium.org, Dec 20 2016

Status: Fixed (was: Untriaged)
This was fixed in  crbug.com/673461 .
Project Member

Comment 20 by 42576172...@developer.gserviceaccount.com, Mar 23 2017 


=== BISECT JOB RESULTS ===
NO Test failure found

Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : media.mse_cases
  Metric       : append/AV_buffer_post_load

Revision             Exit Code      N
chromium@435212      0 +- N/A       20      good
chromium@435265      0 +- N/A       20      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter='startup_test.html?testType=V' media.mse_cases

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8984378634840558192

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=4906031434432512


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
Project Member

Comment 22 by 42576172...@developer.gserviceaccount.com, Apr 11 2017 


=== BISECT JOB RESULTS ===
NO Test failure found

Bisect Details
  Configuration: android_one_perf_bisect
  Benchmark    : media.mse_cases
  Metric       : append/AV_buffer_post_load

Revision             Exit Code      N
chromium@435222      0 +- N/A       20      good
chromium@435255      0 +- N/A       20      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=AV.buffer.post.load media.mse_cases

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982651882568536368

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5199273816227840


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
►

Sign in to add a comment