Purple Bot on chromium.perf Android Galaxy S5 Perf (2) |
|||||
Issue descriptionLink to buildbot status page: https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Galaxy%20S5%20Perf%20%282%29/builds/2908 It looks like it is having trouble booting the device. See log for most recent failing step: https://uberchromegw.corp.google.com/i/chromium.perf/builders/Android%20Galaxy%20S5%20Perf%20%282%29/builds/2908/steps/Sharded%20Perf%20Tests/logs/stdio Stack trace: Traceback (most recent call last): File "/b/rr/tmpW7LJFt/w/src/third_party/catapult/devil/devil/android/tools/device_recovery.py", line 90, in RecoverDevice device.WaitUntilFullyBooted(retries=0) File "/b/rr/tmpW7LJFt/w/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/rr/tmpW7LJFt/w/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 157, in Run error_log_func=error_log_func) File "/b/rr/tmpW7LJFt/w/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll self._JoinAll(watcher, timeout) File "/b/rr/tmpW7LJFt/w/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 158, in _JoinAll thread.ReraiseIfException() File "/b/rr/tmpW7LJFt/w/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/rr/tmpW7LJFt/w/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 150, in <lambda> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), File "/b/rr/tmpW7LJFt/w/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/rr/tmpW7LJFt/w/src/third_party/catapult/devil/devil/android/device_utils.py", line 575, in WaitUntilFullyBooted timeout_retry.WaitFor(boot_completed) File "/b/rr/tmpW7LJFt/w/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 111, in WaitFor msg='Timed out waiting for %r' % condition_name) File "/b/rr/tmpW7LJFt/w/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 54, in GetRemainingTime raise reraiser_thread.TimeoutError(msg) CommandTimeoutError: Timed out waiting for 'boot_completed', wait of 5.0 secs required but only 1.6 secs left
,
Jul 25 2016
This bot has continued being red, with occasional purpleness. smoothness.tough_pinch_zoom_cases started failing from build #2889 and has appeared in the failure list for every subsequent build. I think that there are probably real failures mixed in with some infra-structure issues. Adding in nednguyen@ as someone who knows about smoothness.tough_pinch_zoom_cases and eyaich@ who made some recent changes to smootheness.py
,
Jul 25 2016
I believe this is probably caused by change to test_runner.py?
,
Jul 25 2016
The purple runs were due to that switch, but rnephew reverted it in https://codereview.chromium.org/2179513002/
,
Jul 25 2016
As a side note, that device timeout is probably not what is causing it to turn purple, I will investigate that further though; But like John said my test_runner switch was reverted so all purple-ness related to that should be gone as of Friday evening.
,
Jul 25 2016
Yep, timed out do to lack of output: command timed out: 2400 seconds without output, attempting to kill process killed by signal 9 program finished with exit code -1 elapsedTime=14685.616930 I have a fix for this I am working on before landing again.
,
Jul 25 2016
,
Jul 25 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e8555d3785393f8bf04579226a9e39f4e574c599 commit e8555d3785393f8bf04579226a9e39f4e574c599 Author: rnephew <rnephew@chromium.org> Date: Mon Jul 25 21:17:46 2016 [Android] Add heartbeat of platform mode perf test runner. Extremely long running tests can cause timeouts due to lack of output. This adds a heartbeat message to stop that while tests are still running. BUG= 615157 , 630640 Review-Url: https://codereview.chromium.org/2182563002 Cr-Commit-Position: refs/heads/master@{#407576} [modify] https://crrev.com/e8555d3785393f8bf04579226a9e39f4e574c599/build/android/pylib/local/device/local_device_perf_test_run.py
,
Jul 25 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/772e098a3f629dfeadab58212623a07e54ee902a commit 772e098a3f629dfeadab58212623a07e54ee902a Author: rnephew <rnephew@chromium.org> Date: Mon Jul 25 22:04:39 2016 [Android] Reland Galaxy s5 perf bots to run using platform mode. BUG= 615157 , 630627 , 630640 Review-Url: https://codereview.chromium.org/2177863004 [modify] https://crrev.com/772e098a3f629dfeadab58212623a07e54ee902a/scripts/slave/recipes/android/perf.expected/full_chromium_perf_Android_Galaxy_S5_Perf__1_.json [modify] https://crrev.com/772e098a3f629dfeadab58212623a07e54ee902a/scripts/slave/recipes/android/perf.expected/full_chromium_perf_Android_Galaxy_S5_Perf__2_.json [modify] https://crrev.com/772e098a3f629dfeadab58212623a07e54ee902a/scripts/slave/recipes/android/perf.expected/full_chromium_perf_Android_Galaxy_S5_Perf__3_.json [modify] https://crrev.com/772e098a3f629dfeadab58212623a07e54ee902a/scripts/slave/recipes/android/perf.py
,
Jul 25 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/ce6b0ce17d5c82a02b3972492150a886a5e19dcf commit ce6b0ce17d5c82a02b3972492150a886a5e19dcf Author: recipe-roller <recipe-roller@chromium.org> Date: Mon Jul 25 22:07:59 2016
,
Jul 25 2016
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/81693cf3baf302312bc2e5b6185c7f2740b339b5 commit 81693cf3baf302312bc2e5b6185c7f2740b339b5 Author: recipe-roller <recipe-roller@chromium.org> Date: Mon Jul 25 22:27:51 2016 Roll recipe dependencies (trivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) build: https://crrev.com/772e098a3f629dfeadab58212623a07e54ee902a [Android] Reland Galaxy s5 perf bots to run using platform mode. (rnephew@chromium.org) R=rnephew@chromium.org BUG= 630627 , 630640 , 615157 TBR=martiniss@chromium.org,phajdan.jr@chromium.org Review-Url: https://codereview.chromium.org/2178073004 [modify] https://crrev.com/81693cf3baf302312bc2e5b6185c7f2740b339b5/infra/config/recipes.cfg
,
Jul 26 2016
This bot is still red, failure is now "Failed steps failed host_info failed sharded perf tests"
,
Jul 26 2016
Sharded perf tests appears to fail when one or more devices are blacklisted. Reverting the switch for now.
,
Jul 26 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/93b972af676e8098ebc020db5600b1197a761982 commit 93b972af676e8098ebc020db5600b1197a761982 Author: jbudorick <jbudorick@chromium.org> Date: Tue Jul 26 13:44:09 2016 Revert of [Android] Reland Galaxy s5 perf bots to run using platform mode. (patchset #1 id:1 of https://codereview.chromium.org/2177863004/ ) Reason for revert: the sharded perf tests step fails when one or more devices are blacklisted. Original issue's description: > [Android] Reland Galaxy s5 perf bots to run using platform mode. > > BUG= 615157 , 630627 , 630640 > > Committed: https://chromium.googlesource.com/chromium/tools/build/+/772e098a3f629dfeadab58212623a07e54ee902a TBR=stip@chromium.org,rnephew@chromium.org # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG= 615157 , 630627 , 630640 Review-Url: https://codereview.chromium.org/2187483002 [modify] https://crrev.com/93b972af676e8098ebc020db5600b1197a761982/scripts/slave/recipes/android/perf.expected/full_chromium_perf_Android_Galaxy_S5_Perf__1_.json [modify] https://crrev.com/93b972af676e8098ebc020db5600b1197a761982/scripts/slave/recipes/android/perf.expected/full_chromium_perf_Android_Galaxy_S5_Perf__2_.json [modify] https://crrev.com/93b972af676e8098ebc020db5600b1197a761982/scripts/slave/recipes/android/perf.expected/full_chromium_perf_Android_Galaxy_S5_Perf__3_.json [modify] https://crrev.com/93b972af676e8098ebc020db5600b1197a761982/scripts/slave/recipes/android/perf.py
,
Jul 26 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/1db1515ad68074bdb31c1aa1d4bf0784e8d9ce50 commit 1db1515ad68074bdb31c1aa1d4bf0784e8d9ce50 Author: recipe-roller <recipe-roller@chromium.org> Date: Tue Jul 26 13:48:27 2016
,
Jul 26 2016
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/5b9186bb318fad23806c4115489984ef1a58e828 commit 5b9186bb318fad23806c4115489984ef1a58e828 Author: recipe-roller <recipe-roller@chromium.org> Date: Tue Jul 26 14:08:05 2016 Roll recipe dependencies (trivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) build: https://crrev.com/93b972af676e8098ebc020db5600b1197a761982 Revert of [Android] Reland Galaxy s5 perf bots to run using platform mode. (patchset #1 id:1 of https://codereview.chromium.org/2177863004/ ) (jbudorick@chromium.org) R=jbudorick@chromium.org,rnephew@chromium.org,stip@chromium.org BUG= 630627 , 630640 , 615157 TBR=martiniss@chromium.org,phajdan.jr@chromium.org Review-Url: https://codereview.chromium.org/2178213004 [modify] https://crrev.com/5b9186bb318fad23806c4115489984ef1a58e828/infra/config/recipes.cfg
,
Jul 26 2016
This has been fixed. There is another issue with perf tests in platform mode failing that can be followed at crbug.com/630627
,
Jul 26 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/525217b0d1284f1b2a9ddd691b099f9a4f894349 commit 525217b0d1284f1b2a9ddd691b099f9a4f894349 Author: rnephew <rnephew@chromium.org> Date: Tue Jul 26 16:44:46 2016 Reland of [Android] Reland Galaxy s5 perf bots to run using platform mode. (patchset #1 id:1 of https://codereview.chromium.org/2187483002/ ) Reason for revert: Fixed with https://codereview.chromium.org/2187513002/ Original issue's description: > Revert of [Android] Reland Galaxy s5 perf bots to run using platform mode. (patchset #1 id:1 of https://codereview.chromium.org/2177863004/ ) > > Reason for revert: > the sharded perf tests step fails when one or more devices are blacklisted. > > Original issue's description: > > [Android] Reland Galaxy s5 perf bots to run using platform mode. > > > > BUG= 615157 , 630627 , 630640 > > > > Committed: https://chromium.googlesource.com/chromium/tools/build/+/772e098a3f629dfeadab58212623a07e54ee902a > > TBR=stip@chromium.org,rnephew@chromium.org > # Skipping CQ checks because original CL landed less than 1 days ago. > NOPRESUBMIT=true > NOTREECHECKS=true > NOTRY=true > BUG= 615157 , 630627 , 630640 > > Committed: https://chromium.googlesource.com/chromium/tools/build/+/93b972af676e8098ebc020db5600b1197a761982 TBR=stip@chromium.org,jbudorick@chromium.org # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG= 615157 , 630627 , 630640 Review-Url: https://codereview.chromium.org/2184803002 [modify] https://crrev.com/525217b0d1284f1b2a9ddd691b099f9a4f894349/scripts/slave/recipes/android/perf.expected/full_chromium_perf_Android_Galaxy_S5_Perf__1_.json [modify] https://crrev.com/525217b0d1284f1b2a9ddd691b099f9a4f894349/scripts/slave/recipes/android/perf.expected/full_chromium_perf_Android_Galaxy_S5_Perf__2_.json [modify] https://crrev.com/525217b0d1284f1b2a9ddd691b099f9a4f894349/scripts/slave/recipes/android/perf.expected/full_chromium_perf_Android_Galaxy_S5_Perf__3_.json [modify] https://crrev.com/525217b0d1284f1b2a9ddd691b099f9a4f894349/scripts/slave/recipes/android/perf.py
,
Jul 26 2016
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/8052d18580548dbcd820e90a638c873e9b283995 commit 8052d18580548dbcd820e90a638c873e9b283995 Author: recipe-roller <recipe-roller@chromium.org> Date: Tue Jul 26 16:48:26 2016
,
Jul 26 2016
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra.git/+/374c564fecdd069a6c1b0bf73b6541f21d6dee2f commit 374c564fecdd069a6c1b0bf73b6541f21d6dee2f Author: recipe-roller <recipe-roller@chromium.org> Date: Tue Jul 26 17:06:05 2016 Roll recipe dependencies (trivial). This is an automated CL created by the recipe roller. This CL rolls recipe changes from upstream projects (e.g. depot_tools) into downstream projects (e.g. tools/build). More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug (or complain) build: https://crrev.com/525217b0d1284f1b2a9ddd691b099f9a4f894349 Reland of [Android] Reland Galaxy s5 perf bots to run using platform mode. (patchset #1 id:1 of https://codereview.chromium.org/2187483002/ ) (rnephew@chromium.org) R=rnephew@chromium.org,jbudorick@chromium.org,stip@chromium.org BUG= 630627 , 630640 , 615157 TBR=martiniss@chromium.org,phajdan.jr@chromium.org Review-Url: https://codereview.chromium.org/2178353004 [modify] https://crrev.com/374c564fecdd069a6c1b0bf73b6541f21d6dee2f/infra/config/recipes.cfg |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by jo...@google.com
, Jul 22 2016Owner: jo...@chromium.org
Status: Assigned (was: Untriaged)