New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 703015 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Apr 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

video_PlaybackPerf failing since 9378.0.0

Project Member Reported by wuchengli@chromium.org, Mar 20 2017

Issue description

From wmatrix, video_PlaybackPerf started to fail since 9378.0.0. This is a new failure so please take a look soon.
 

Comment 1 by hywu@chromium.org, Mar 22 2017

Checking now.

Comment 2 by hywu@chromium.org, Mar 22 2017

According to log, it seems to be lab network issue.

File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/core/local_server.py", line 152, in StartServer
    assert self._platform_backend.network_controller_backend.is_initialized
AssertionError

Besides, I tested elm/R59-9378.0.0 with "test_that elm video_PlaybackPerf.power_vp9" for three times and all passed.
Cc: rohi...@chromium.org vsu...@chromium.org
+Vin for network issue.

Daniel. Can you also try "test_that elm video_PlaybackPerf.vp9"?

Comment 4 by hywu@chromium.org, Mar 22 2017

"test_that elm video_PlaybackPerf.vp9" passed as well.

I also tried the server side test with "test_that elm video_PowerConsumption.vp8" and it failed although I'm not sure whether the test needs a special environment setup. There is no error message in the log.

04:03:39 INFO | autoserv| Starting master ssh connection '/usr/bin/ssh -a -x   -N -o ControlMaster=yes -o ControlPath=/tmp/_autotmp_HvpQu2ssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 elm'
04:03:40 INFO | autoserv| get_network_stats: at-end RXbytes 59407475 TXbytes 2546651
04:03:43 INFO | autoserv| record_state_duration failed: job_or_task_id=None, hostname=elm, status=Running
------------------------------------------------------------------------------
/tmp/test_that_results_4kdQUO/results-1-video_PowerConsumption.vp8 [  FAILED  ]
------------------------------------------------------------------------------
Cc: dshi@chromium.org akes...@chromium.org
Owner: akes...@chromium.org
+dshi, akeshet. Have you seen errors like this?

File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/core/local_server.py", line 152, in StartServer
    assert self._platform_backend.network_controller_backend.is_initialized
AssertionError

Comment 6 by dshi@chromium.org, Mar 23 2017

I've never seen such error. How about try test_that with :lab: option to run it in lab?
Owner: hywu@chromium.org
Daniel. Please try lab option.

Comment 8 by hywu@chromium.org, Mar 24 2017

The lab is still closed. Will try when it comes back.
This is still failing in tot.

Comment 10 by hywu@chromium.org, Apr 20 2017

I got INFRA_FAILURE while trying to run it in lab.
$ test_that :lab: --max_runtime_mins 600 -b elm -i 'elm-release/R59-9378.0.0' video_PlaybackPerf.vp9
INFO:root:About to start lab suite with command ['/mnt/host/source/src/third_party/autotest/files/site_utils/run_suite.py', '--board=elm', '--build=elm-release/R59-9378.0.0', '--suite_name=test_that_wrapper', '--pool=suites', '--max_runtime_mins=600', '--suite_args=:lab: -b elm -i elm-release/R59-9378.0.0 video_PlaybackPerf.vp9'].
Autotest instance: localhost
04-16-2017 [20:08:00] Submitted create_suite_job rpc
Unhandled run_suite exception: retry exception (function="create_suite()"), timeout = 600s
Traceback (most recent call last):
  File "/mnt/host/source/src/third_party/autotest/files/site_utils/run_suite.py", line 1831, in main
    code, output_dict = main_without_exception_handling(options)
  File "/mnt/host/source/src/third_party/autotest/files/site_utils/run_suite.py", line 1635, in main_without_exception_handling
    job_id = create_suite(afe, options)
  File "/mnt/host/source/src/third_party/autotest/files/client/common_lib/cros/retry.py", line 233, in func_retry
    raise error.TimeoutException(exception_message)
TimeoutException: retry exception (function="create_suite()"), timeout = 600s
Will return from run_suite with status: INFRA_FAILURE
The failure stack:
04/14 07:59:10.109 WARNI|              test:0615| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 609, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/common_lib/test.py", line 823, in _call_test_function
    raise error.UnhandledTestFail(e)
UnhandledTestFail: Unhandled AssertionError: 
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 817, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/common_lib/test.py", line 470, in execute
    dargs)
  File "/usr/local/autotest/common_lib/test.py", line 347, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/common_lib/test.py", line 380, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/tests/video_PlaybackPerf/video_PlaybackPerf.py", line 122, in run_once
    keyvals = self.test_power(local_path)
  File "/usr/local/autotest/tests/video_PlaybackPerf/video_PlaybackPerf.py", line 220, in test_power
    return self.test_playback(local_path, get_power)
  File "/usr/local/autotest/tests/video_PlaybackPerf/video_PlaybackPerf.py", line 258, in test_playback
    self.start_playback(cr, local_path)
  File "/usr/local/autotest/tests/video_PlaybackPerf/video_PlaybackPerf.py", line 72, in start_playback
    cr.browser.platform.SetHTTPServerDirectories(self.bindir)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/core/platform.py", line 410, in SetHTTPServerDirectories
    self.StartLocalServer(server)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/core/platform.py", line 377, in StartLocalServer
    self._local_server_controller.StartServer(server)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/core/local_server.py", line 152, in StartServer
    assert self._platform_backend.network_controller_backend.is_initialized
AssertionError
Cc: hiroh@chromium.org
+hiroh@ who has also been looking at failures in this test.
video_WebRtcPerf has the same problem. Maybe all tests that use SetHTTPServerDirectories are broken.

R59-9461.0.0 auron_paine

09:16:01.899 WARNI|              test:0615| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 609, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/common_lib/test.py", line 823, in _call_test_function
    raise error.UnhandledTestFail(e)
UnhandledTestFail: Unhandled AssertionError: 
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 817, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/common_lib/test.py", line 470, in execute
    dargs)
  File "/usr/local/autotest/common_lib/test.py", line 347, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/common_lib/test.py", line 380, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/tests/video_WebRtcPerf/video_WebRtcPerf.py", line 163, in run_once
    keyvals = self.test_power(local_path)
  File "/usr/local/autotest/tests/video_WebRtcPerf/video_WebRtcPerf.py", line 291, in test_power
    return self.test_webrtc(local_path, get_power)
  File "/usr/local/autotest/tests/video_WebRtcPerf/video_WebRtcPerf.py", line 212, in test_webrtc
    self.start_loopback(cr)
  File "/usr/local/autotest/tests/video_WebRtcPerf/video_WebRtcPerf.py", line 104, in start_loopback
    cr.browser.platform.SetHTTPServerDirectories(self.bindir)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/core/platform.py", line 410, in SetHTTPServerDirectories
    self.StartLocalServer(server)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/core/platform.py", line 377, in StartLocalServer
    self._local_server_controller.StartServer(server)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/core/local_server.py", line 152, in StartServer
    assert self._platform_backend.network_controller_backend.is_initialized
AssertionError

09:16:02.533 ERROR|          parallel:0026| child process failed
video_WebRtcPerf also started to fail in 9378.0.0.
Re #10, Lab has some network issue in these two days due to lots of chrome crash. (See http://crbug.com/713004)
It is getting recovered, maybe you can try it again. 
We need to add init_network_controller=True.

See https://bugs.chromium.org/p/chromium/issues/detail?id=701629 for reference.
Please also help add init_network_controller=True for video_WebRtcPerf.
Project Member

Comment 18 by bugdroid1@chromium.org, Apr 22 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/646f5c8ea8a81489c5e22290033132e8be2137b3

commit 646f5c8ea8a81489c5e22290033132e8be2137b3
Author: Daniel Hung-yu Wu <hywu@chromium.org>
Date: Sat Apr 22 04:57:45 2017

video tests: set init_network_controller=True

Set init_network_controller=True while using SetHTTPServerDirectories.

BUG= chromium:703015 

Change-Id: I86b3a87d410f7b0ebdd3f2d932efa5f1c177ba46
Reviewed-on: https://chromium-review.googlesource.com/484203
Commit-Ready: Wu-cheng Li <wuchengli@chromium.org>
Tested-by: Wu-cheng Li <wuchengli@chromium.org>
Reviewed-by: Wu-cheng Li <wuchengli@chromium.org>

[modify] https://crrev.com/646f5c8ea8a81489c5e22290033132e8be2137b3/client/site_tests/video_WebRtcPerf/video_WebRtcPerf.py
[modify] https://crrev.com/646f5c8ea8a81489c5e22290033132e8be2137b3/client/site_tests/video_PlaybackPerf/video_PlaybackPerf.py

Status: Fixed (was: Assigned)

Comment 20 by hywu@chromium.org, May 5 2017

Labels: Merge-Request-59
Can you please mark which OS this is for?

Comment 22 by hywu@chromium.org, May 5 2017

Labels: OS-Chrome
Project Member

Comment 23 by sheriffbot@chromium.org, May 6 2017

Labels: -Merge-Request-59 Hotlist-Merge-Approved Merge-Approved-59
Your change meets the bar and is auto-approved for M59. Please go ahead and merge the CL to branch 3071 manually. Please contact milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), gkihumba@(ChromeOS), Abdul Syed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 24 by bugdroid1@chromium.org, May 8 2017

Labels: merge-merged-release-R59-9460.B
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c8f3593e710288d95b65f978bc0f3b41eacd741b

commit c8f3593e710288d95b65f978bc0f3b41eacd741b
Author: Daniel Hung-yu Wu <hywu@chromium.org>
Date: Mon May 08 06:09:44 2017

video tests: set init_network_controller=True

Set init_network_controller=True while using SetHTTPServerDirectories.

BUG= chromium:703015 

Change-Id: I86b3a87d410f7b0ebdd3f2d932efa5f1c177ba46
Reviewed-on: https://chromium-review.googlesource.com/484203
Commit-Ready: Wu-cheng Li <wuchengli@chromium.org>
Tested-by: Wu-cheng Li <wuchengli@chromium.org>
Reviewed-by: Wu-cheng Li <wuchengli@chromium.org>
(cherry picked from commit 646f5c8ea8a81489c5e22290033132e8be2137b3)
Reviewed-on: https://chromium-review.googlesource.com/497231
Commit-Queue: Hung-yu Wu <hywu@chromium.org>
Tested-by: Hung-yu Wu <hywu@chromium.org>
Trybot-Ready: Hung-yu Wu <hywu@chromium.org>
Reviewed-by: Kuang-che Wu <kcwu@chromium.org>

[modify] https://crrev.com/c8f3593e710288d95b65f978bc0f3b41eacd741b/client/site_tests/video_WebRtcPerf/video_WebRtcPerf.py
[modify] https://crrev.com/c8f3593e710288d95b65f978bc0f3b41eacd741b/client/site_tests/video_PlaybackPerf/video_PlaybackPerf.py

Project Member

Comment 25 by sheriffbot@chromium.org, May 10 2017

This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 26 by hywu@chromium.org, May 10 2017

Labels: -Hotlist-Merge-Approved -Merge-Approved-59
Cc: avkodipelli@chromium.org
Status: Verified (was: Fixed)
Verified on M59 using https://wmatrix.googleplex.com/unfiltered?hide_missing=True&tests=video_PlaybackPerf&releases=59&days_back=30

Sign in to add a comment