system_health.common_desktop failing on chromium.perf/Mac Air 10.11 Perf
Builders failed on:
- Mac Air 10.11 Perf:
https://build.chromium.org/p/chromium.perf/builders/Mac%20Air%2010.11%20Perf
Looks like system_health.common_desktop has been really flaky recently
On Mac Air 10.11 Perf, it's failed 189 out of the last 200 runs
Looking at the failing tests, it seems to be a different set of failures each run:
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Mac%20Air%2010.11%20Perf/builds/1397
Unexpected Failures:
* system_health.common_desktop/browse:media:imgur
* system_health.common_desktop/browse:media:youtube
* system_health.common_desktop/browse:news:flipboard
* system_health.common_desktop/browse:social:twitter_infinite_scroll
* system_health.common_desktop/load:games:miniclip
* system_health.common_desktop/play:media:soundcloud
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Mac%20Air%2010.11%20Perf/builds/1396
Unexpected Failures:
* system_health.common_desktop/long_running:tools:gmail-background
* system_health.common_desktop/long_running:tools:gmail-foreground
* system_health.common_desktop/multitab:misc:typical24
* system_health.common_desktop/play:media:soundcloud
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Mac%20Air%2010.11%20Perf/builds/1395
ValueError: No JSON object could be decoded
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Mac%20Air%2010.11%20Perf/builds/1394
Unexpected Failures:
* system_health.common_desktop/browse:media:pinterest
* system_health.common_desktop/browse:social:facebook_infinite_scroll
* system_health.common_desktop/browse:social:twitter_infinite_scroll
* system_health.common_desktop/long_running:tools:gmail-foreground
* system_health.common_desktop/multitab:misc:typical24
https://uberchromegw.corp.google.com/i/chromium.perf/builders/Mac%20Air%2010.11%20Perf/builds/1393
Unexpected Failures:
* system_health.common_desktop/browse:media:flickr_infinite_scroll
* system_health.common_desktop/browse:media:youtube
* system_health.common_desktop/browse:news:reddit
* system_health.common_desktop/browse:social:tumblr_infinite_scroll
* system_health.common_desktop/browse:social:twitter_infinite_scroll
* system_health.common_desktop/browse:tech:discourse_infinite_scroll
* system_health.common_desktop/load:news:qq
* system_health.common_desktop/multitab:misc:typical24
Looks like all of the failing tests have a keystore failure and a battor failure
@charliea, could you take a look at the battor error below and let me know if that's expected?
https://chromium-swarm.appspot.com/task?id=391eaa83057efe10&refresh=10&show_raw=1
2017-10-10 06:49:47.857 Google Chrome[12116:168235] Errors logged by ksadmin: KSKeyedPersistentStore store directory does not exist. [com.google.UpdateEngine.CommonErrorDomain:501 - '/Library/Google/GoogleSoftwareUpdate/TicketStore' - 'KSKeyedPersistentStore.m:372']
KSPersistentTicketStore failed to load tickets. (productID: com.google.Chrome) [com.google.UpdateEngine.CoreErrorDomain:1051 - '/Library/Google/GoogleSoftwareUpdate/TicketStore/Keystone.ticketstore'] (KSKeyedPersistentStore store directory does not exist. - '/Library/Google/GoogleSoftwareUpdate/TicketStore' [com.google.UpdateEngine.CommonErrorDomain:501])
ksadmin cannot access the ticket store:<KSUpdateError:0x100602b60
domain="com.google.UpdateEngine.CoreErrorDomain"
code=1051
userInfo={
function = "-[KSProductKeyedStore(ProtectedMethods) errorForStoreError:productID:message:timeoutMessage:]";
date = 2017-10-10 13:49:47 +0000;
productids = {(
"com.google.Chrome"
)};
filename = "KSProductKeyedStore.m";
line = 91;
NSFilePath = "/Library/Google/GoogleSoftwareUpdate/TicketStore/Keystone.ticketstore";
NSUnderlyingError = <KSError:0x100602930
domain="com.google.UpdateEngine.CommonErrorDomain"
code=501
userInfo={
date = 2017-10-10 13:49:47 +0000;
line = 372;
filename = "KSKeyedPersistentStore.m";
function = "-[KSKeyedPersistentStore(PrivateMethods) validateStorePath]";
NSFilePath = "/Library/Google/GoogleSoftwareUpdate/TicketStore";
NSLocalizedDescription = "KSKeyedPersistentStore store directory does not exist.";
}
>;
NSLocalizedDescription = "KSPersistentTicketStore failed to load tickets.";
}
>
(INFO) 2017-10-10 06:50:31,851 browser.DumpStateUponFailure:396 *********** END OF BROWSER STANDARD OUTPUT ************
(INFO) 2017-10-10 06:50:31,851 browser.DumpStateUponFailure:398 ********************* BROWSER LOG *********************
(INFO) 2017-10-10 06:50:31,851 browser.DumpStateUponFailure:400 No log file
(INFO) 2017-10-10 06:50:31,851 browser.DumpStateUponFailure:403 ***************** END OF BROWSER LOG ******************
Traceback (most recent call last):
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 106, in _RunStoryAndProcessErrorIfNeeded
test.Measure(state.platform, results)
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py", line 281, in Measure
trace_result = platform.tracing_controller.StopTracing()
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/core/tracing_controller.py", line 47, in StopTracing
return self._tracing_controller_backend.StopTracing()
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 140, in StopTracing
'\n'.join(raised_exception_messages))
TracingException: Exceptions raised when trying to stop tracing:
Traceback (most recent call last):
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 118, in StopTracing
agent.StopAgentTracing()
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/battor_tracing_agent.py", line 83, in StopAgentTracing
self._battor.StopTracing()
File "/b/s/w/ir/third_party/catapult/common/battor/battor/battor_wrapper.py", line 247, in StopTracing
'%s %s' % (self._STOP_TRACING_CMD, self._trace_results_path))
File "/b/s/w/ir/third_party/catapult/common/battor/battor/battor_wrapper.py", line 359, in _SendBattOrCommand
'Outputted: %s' % (cmd, status))
BattOrError: BattOr did not complete command 'StopTracing /b/s/w/itmSnbPu/tmpea_Db3' correctly.
Outputted:
Traceback (most recent call last):
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 128, in StopTracing
agent.CollectAgentTraceData(builder)
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/battor_tracing_agent.py", line 112, in CollectAgentTraceData
data = self._battor.CollectTraceData(timeout=timeout)
File "/b/s/w/ir/third_party/catapult/common/battor/battor/battor_wrapper.py", line 262, in CollectTraceData
self._stop_tracing_time))
BattOrError: No start or stop time detected when collecting BattOr trace.
Start: 1507643382
Stop: None
(INFO) 2017-10-10 06:50:32,048 browser.Close:268 Closing browser (pid=12116) ...
(INFO) 2017-10-10 06:50:32,433 browser.Close:283 Browser is closed.
[ FAILED ] browse:news:flipboard@{'case': 'browse', 'group': 'news'} (57731 ms)
Comment 1 by charliea@chromium.org
, Oct 17 2017Status: Duplicate (was: Available)