New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 652306 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 652384
Owner:
Last visit > 30 days ago
Closed: Oct 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 1
Type: Bug-Regression



Sign in to add a comment

system_health.common_desktop & battor.power_cases failure on Mac Retina Perf (1)

Project Member Reported by zh...@chromium.org, Oct 3 2016

Issue description

First seen at https://uberchromegw.corp.google.com/i/chromium.perf/builders/Mac%20Retina%20Perf%20%281%29/builds/4341

Revision range first seen: 422336 - 422340

It is a BattOr timeout error, see the log:


Traceback (most recent call last):
  File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 88, in _RunStoryAndProcessErrorIfNeeded
    test.Measure(state.platform, results)
  File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py", line 287, in Measure
    trace_result = platform.tracing_controller.StopTracing()
  File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/core/tracing_controller.py", line 47, in StopTracing
    return self._tracing_controller_backend.StopTracing()
  File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 108, in StopTracing
    self._IssueClockSyncMarker()
  File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 203, in _IssueClockSyncMarker
    self._RecordIssuerClockSyncMarker)
  File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/battor_tracing_agent.py", line 103, in RecordClockSyncMarker
    self._battor.RecordClockSyncMarker(sync_id)
  File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py", line 201, in RecordClockSyncMarker
    self._SendBattorCommand('%s %s' % (self._RECORD_CLOCKSYNC_CMD, sync_id))
  File "/b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py", line 270, in _SendBattorCommand
    'Outputted: %s' % (cmd, status))
BattorError: BattOr did not complete command 'RecordClockSyncMarker 6afdd628-a374-4c03-8aba-fffc5c67894b' correctly.
Outputted: [1001/220714:FATAL:battor_agent_bin.cc(88)] Fatal error when communicating with the BattOr: TIMEOUT


INFO:root:Chrome version: 2878
INFO:root:Try printing formatted exception: None None None

Exception raised when cleaning story run: 

Traceback (most recent call last):
  _RunStoryAndProcessErrorIfNeeded at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:108
    test.DidRunStory(state.platform)
  DidRunStory at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:309
    platform.tracing_controller.StopTracing()
  StopTracing at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/core/tracing_controller.py:47
    return self._tracing_controller_backend.StopTracing()
  StopTracing at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:108
    self._IssueClockSyncMarker()
  _IssueClockSyncMarker at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:196
    if agent.SupportsExplicitClockSync():
  SupportsExplicitClockSync at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/battor_tracing_agent.py:89
    return self._battor.SupportsExplicitClockSync()
  SupportsExplicitClockSync at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py:195
    check_return=False)))
  _SendBattorCommand at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py:262
    status = self._SendBattorCommandImpl(cmd)
  _SendBattorCommandImpl at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py:257
    self._battor_shell.stdin.write('%s\n' % cmd)
IOError: [Errno 32] Broken pipe

Locals:
  cmd : 'SupportsExplicitClockSync'

[  FAILED  ] load:search:yahoo (23455 ms)
INFO:root:Try printing formatted exception: None None None

Traceback (most recent call last):
  RunBenchmark at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:336
    benchmark.ShouldTearDownStateAfterEachStorySetRun())
  Run at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:227
    _RunStoryAndProcessErrorIfNeeded(story, results, state, test)
  _RunStoryAndProcessErrorIfNeeded at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:88
    test.Measure(state.platform, results)
  Measure at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:287
    trace_result = platform.tracing_controller.StopTracing()
  StopTracing at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/core/tracing_controller.py:47
    return self._tracing_controller_backend.StopTracing()
  StopTracing at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:108
    self._IssueClockSyncMarker()
  _IssueClockSyncMarker at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:203
    self._RecordIssuerClockSyncMarker)
  RecordClockSyncMarker at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/battor_tracing_agent.py:103
    self._battor.RecordClockSyncMarker(sync_id)
  RecordClockSyncMarker at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py:201
    self._SendBattorCommand('%s %s' % (self._RECORD_CLOCKSYNC_CMD, sync_id))
  _SendBattorCommand at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py:270
    'Outputted: %s' % (cmd, status))
BattorError: BattOr did not complete command 'RecordClockSyncMarker 6afdd628-a374-4c03-8aba-fffc5c67894b' correctly.
Outputted: [1001/220714:FATAL:battor_agent_bin.cc(88)] Fatal error when communicating with the BattOr: TIMEOUT

Locals:
  check_return : True
  cmd          : 'RecordClockSyncMarker 6afdd628-a374-4c03-8aba-fffc5c67894b'
  status       : '[1001/220714:FATAL:battor_agent_bin.cc(88)] Fatal error when communicating with the BattOr: TIMEOUT\n'
 

Comment 1 by zh...@chromium.org, Oct 3 2016

Summary: system_health.common_desktop & battor.power_cases failure on Mac Retina Perf (1) (was: system_health.common_desktop failure on Mac Retina Perf (1))
Also seeing battor.power_cases failure on the same bot (also for the reference build). It is also BattOr error, though the first failure is one build run after system_health.common_desktop. Maybe it is due to the problem caused by the common_desktop one?


Traceback (most recent call last):
  RunBenchmark at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:336
    benchmark.ShouldTearDownStateAfterEachStorySetRun())
  Run at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:227
    _RunStoryAndProcessErrorIfNeeded(story, results, state, test)
  _RunStoryAndProcessErrorIfNeeded at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:88
    test.Measure(state.platform, results)
  Measure at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:287
    trace_result = platform.tracing_controller.StopTracing()
  StopTracing at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/core/tracing_controller.py:47
    return self._tracing_controller_backend.StopTracing()
  StopTracing at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:108
    self._IssueClockSyncMarker()
  _IssueClockSyncMarker at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py:203
    self._RecordIssuerClockSyncMarker)
  RecordClockSyncMarker at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/battor_tracing_agent.py:103
    self._battor.RecordClockSyncMarker(sync_id)
  RecordClockSyncMarker at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py:201
    self._SendBattorCommand('%s %s' % (self._RECORD_CLOCKSYNC_CMD, sync_id))
  _SendBattorCommand at /b/c/b/Mac_Retina_Perf__1_/src/third_party/catapult/common/battor/battor/battor_wrapper.py:270
    'Outputted: %s' % (cmd, status))
BattorError: BattOr did not complete command 'RecordClockSyncMarker 93553775-b619-4678-814d-a3d2441ffc95' correctly.
Outputted: [1003/071918:FATAL:battor_agent_bin.cc(88)] Fatal error when communicating with the BattOr: RECEIVE ERROR


Alex, Charlie and Randy, can you take a look? The 5 CLs in the range seem to be unrelated. Is it the problem of the BattOr device (given the reference build also fail now)?
This appears to be the battor slow-write issue which we just got a fix for from Mellow. Hopefully we can get there go-ahead to roll this out to BattOrs today.
Did you try updating the firmware on this BattOr with the slow writes firmware fix to see if it brought this bot back to life?
Owner: rnep...@chromium.org
Status: Assigned (was: Untriaged)
Any updates on this? 
Mergedinto: 652384
Status: Duplicate (was: Assigned)
(See the main bug for an update)

Sign in to add a comment