Issue metadata
Sign in to add a comment
|
system_health.common_desktop & battor.power_cases failure on Mac Retina Perf (1) |
||||||||||||||||||||||
Issue descriptionFirst seen at https://uberchromegw.corp.google.com/i/chromium.perf/builders/Mac%20Retina%20Perf%20%281%29/builds/4341 Revision range first seen: 422336 - 422340 It is a BattOr timeout error, see the log: Traceback (most recent call last): File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 88, in _RunStoryAndProcessErrorIfNeeded test.Measure(state.platform, results) File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py", line 287, in Measure trace_result = platform.tracing_controller.StopTracing() File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/core/tracing_controller.py", line 47, in StopTracing return self._tracing_controller_backend.StopTracing() File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 108, in StopTracing self._IssueClockSyncMarker() File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 203, in _IssueClockSyncMarker self._RecordIssuerClockSyncMarker) File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/battor_tracing_agent.py", line 103, in RecordClockSyncMarker self._battor.RecordClockSyncMarker(sync_id) File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py", line 201, in RecordClockSyncMarker self._SendBattorCommand('%s %s' % (self._RECORD_CLOCKSYNC_CMD, sync_id)) File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py", line 270, in _SendBattorCommand 'Outputted: %s' % (cmd, status)) BattorError: BattOr did not complete command 'RecordClockSyncMarker 6afdd628-a374-4c03-8aba-fffc5c67894b' correctly. Outputted: [1001/220714:FATAL:battor_agent_bin.cc(88)] Fatal error when communicating with the BattOr: TIMEOUT INFO:root:Chrome version: 2878 INFO:root:Try printing formatted exception: None None None Exception raised when cleaning story run: Traceback (most recent call last): _RunStoryAndProcessErrorIfNeeded at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:108 test.DidRunStory(state.platform) DidRunStory at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:309 platform.tracing_controller.StopTracing() StopTracing at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/core/tracing_controller.py:47 return self._tracing_controller_backend.StopTracing() StopTracing at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:108 self._IssueClockSyncMarker() _IssueClockSyncMarker at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:196 if agent.SupportsExplicitClockSync(): SupportsExplicitClockSync at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/battor_tracing_agent.py:89 return self._battor.SupportsExplicitClockSync() SupportsExplicitClockSync at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py:195 check_return=False))) _SendBattorCommand at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py:262 status = self._SendBattorCommandImpl(cmd) _SendBattorCommandImpl at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py:257 self._battor_shell.stdin.write('%s\n' % cmd) IOError: [Errno 32] Broken pipe Locals: cmd : 'SupportsExplicitClockSync' [ FAILED ] load:search:yahoo (23455 ms) INFO:root:Try printing formatted exception: None None None Traceback (most recent call last): RunBenchmark at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:336 benchmark.ShouldTearDownStateAfterEachStorySetRun()) Run at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:227 _RunStoryAndProcessErrorIfNeeded(story, results, state, test) _RunStoryAndProcessErrorIfNeeded at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:88 test.Measure(state.platform, results) Measure at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:287 trace_result = platform.tracing_controller.StopTracing() StopTracing at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/core/tracing_controller.py:47 return self._tracing_controller_backend.StopTracing() StopTracing at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:108 self._IssueClockSyncMarker() _IssueClockSyncMarker at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:203 self._RecordIssuerClockSyncMarker) RecordClockSyncMarker at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/battor_tracing_agent.py:103 self._battor.RecordClockSyncMarker(sync_id) RecordClockSyncMarker at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py:201 self._SendBattorCommand('%s %s' % (self._RECORD_CLOCKSYNC_CMD, sync_id)) _SendBattorCommand at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py:270 'Outputted: %s' % (cmd, status)) BattorError: BattOr did not complete command 'RecordClockSyncMarker 6afdd628-a374-4c03-8aba-fffc5c67894b' correctly. Outputted: [1001/220714:FATAL:battor_agent_bin.cc(88)] Fatal error when communicating with the BattOr: TIMEOUT Locals: check_return : True cmd : 'RecordClockSyncMarker 6afdd628-a374-4c03-8aba-fffc5c67894b' status : '[1001/220714:FATAL:battor_agent_bin.cc(88)] Fatal error when communicating with the BattOr: TIMEOUT\n'
,
Oct 3 2016
This appears to be the battor slow-write issue which we just got a fix for from Mellow. Hopefully we can get there go-ahead to roll this out to BattOrs today.
,
Oct 4 2016
Did you try updating the firmware on this BattOr with the slow writes firmware fix to see if it brought this bot back to life?
,
Oct 5 2016
Any updates on this?
,
Oct 6 2016
|
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by zh...@chromium.org
, Oct 3 2016Also seeing battor.power_cases failure on the same bot (also for the reference build). It is also BattOr error, though the first failure is one build run after system_health.common_desktop. Maybe it is due to the problem caused by the common_desktop one? Traceback (most recent call last): RunBenchmark at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:336 benchmark.ShouldTearDownStateAfterEachStorySetRun()) Run at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:227 _RunStoryAndProcessErrorIfNeeded(story, results, state, test) _RunStoryAndProcessErrorIfNeeded at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:88 test.Measure(state.platform, results) Measure at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:287 trace_result = platform.tracing_controller.StopTracing() StopTracing at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/core/tracing_controller.py:47 return self._tracing_controller_backend.StopTracing() StopTracing at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:108 self._IssueClockSyncMarker() _IssueClockSyncMarker at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:203 self._RecordIssuerClockSyncMarker) RecordClockSyncMarker at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/battor_tracing_agent.py:103 self._battor.RecordClockSyncMarker(sync_id) RecordClockSyncMarker at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py:201 self._SendBattorCommand('%s %s' % (self._RECORD_CLOCKSYNC_CMD, sync_id)) _SendBattorCommand at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py:270 'Outputted: %s' % (cmd, status)) BattorError: BattOr did not complete command 'RecordClockSyncMarker 93553775-b619-4678-814d-a3d2441ffc95' correctly. Outputted: [1003/071918:FATAL:battor_agent_bin.cc(88)] Fatal error when communicating with the BattOr: RECEIVE ERROR Alex, Charlie and Randy, can you take a look? The 5 CLs in the range seem to be unrelated. Is it the problem of the BattOr device (given the reference build also fail now)?