system_health.memory_desktop/browse:news:nytimes flaky on few dekstop bots |
|||||
Issue descriptionwin-10-perf log: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/55a30f9e-8aba-11e8-9634-14b31f28cf07 mac-10_12_laptop_low_end-perf log: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/bdafda9c-8ad3-11e8-9d46-7c04d0d15d5c They both failed with this stack: File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 310, in RunStory self._current_page.Run(self) File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/__init__.py", line 99, in Run self.RunPageInteractions(action_runner) File "/b/s/w/ir/tools/perf/page_sets/system_health/system_health_story.py", line 110, in RunPageInteractions self._DidLoadDocument(action_runner) File "/b/s/w/ir/tools/perf/page_sets/system_health/browsing_stories.py", line 82, in _DidLoadDocument self._ReadNextArticle(action_runner) File "/b/s/w/ir/tools/perf/page_sets/system_health/browsing_stories.py", line 91, in _ReadNextArticle action_runner.tab.WaitForDocumentReadyStateToBeComplete() File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 91, in WaitForDocumentReadyStateToBeComplete 'document.readyState == "complete"', timeout=timeout) File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 239, in WaitForJavaScriptCondition return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs) File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 302, in WaitForJavaScriptCondition e.message + '\n' + debug_message) TimeoutException: Timed out while waiting 90s for IsJavaScriptExpressionTrue. Based on the log and the flaky pattern in https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=performance_test_suite&tests=system_health.common_desktop%2Fbrowse%3Anews%3Anytimes, I think we just need to increase the timeout
,
Jul 19
We also discovered today that the test may be flaky on Mac platform due to keychain popping up: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/4f34fa42-8af7-11e8-a35e-7c04d0d15d5e (this is from the run in https://chrome-swarming.appspot.com/task?id=3ec8fee22bcace10&refresh=10&show_raw=1) Lab folks, can you help disabling keychain on the Mac bots? The bot that has this problem is build53-a7
,
Jul 19
,
Jul 19
,
Jul 19
There is no "disabling the keychain". In the past this was due to permissions issues surrounding the "Chrome Safe Storage" and "Chromium Safe Storage" passwords. I have a feeling this is because these bots were all just moved to private swarming and hostnames changed without a reimage. I've re-deployed the entries for now and rebooted the hosts with popups.
,
Jul 20
dba@, can you also check 'build140-a7'? https://chrome-swarming.appspot.com/task?id=3eceabac54f68610&refresh=10&show_raw=1 just timed out recently & I believe it's also due to keychain
,
Jul 20
Bots this morning that that I remedied in #5 show some popups on the desktop regarding the keychain.
I have no idea how to fix this. I had fixed this the same way as we did in the past (issue 702215), and I don't know why this is magically an issue again as we've deployed all of these the same way since the previous issue and now it seems to be a problem again.
Documentation surrounding keychain partitions from Apple is lacking. The entries are there and Chrome shouldn't be prompting to access them as they're allowed by anything to use them.
From build140-a7:
keychain: "/Users/chrome-bot/Library/Keychains/login.keychain-db"
version: 512
class: "genp"
attributes:
0x00000007 <blob>="Chrome Safe Storage"
0x00000008 <blob>=<NULL>
"acct"<blob>="Chrome"
"cdat"<timedate>=0x32303138303232313233333830305A00 "20180221233800Z\000"
"crtr"<uint32>=<NULL>
"cusi"<sint32>=<NULL>
"desc"<blob>=<NULL>
"gena"<blob>=<NULL>
"icmt"<blob>=<NULL>
"invi"<sint32>=<NULL>
"mdat"<timedate>=0x32303138303232313233333830305A00 "20180221233800Z\000"
"nega"<sint32>=<NULL>
"prot"<blob>=<NULL>
"scrp"<sint32>=<NULL>
"svce"<blob>="Chrome Safe Storage"
"type"<uint32>=<NULL>
access: 5 entries
entry 0:
authorizations (6): decrypt derive export_clear export_wrapped mac sign
don't-require-password
description: Chrome Safe Storage
applications: <null>
entry 1:
authorizations (1): encrypt
don't-require-password
description: Chrome Safe Storage
applications: <null>
entry 2:
authorizations (1): integrity
don't-require-password
description: c78de6a1af50f63c5bb151fd4110ccf5d512fafc7f1357ab498ff9da160aa145
applications: <null>
entry 3:
authorizations (1): partition_id
don't-require-password
description: apple-tool:, unsigned:
applications: <null>
entry 4:
authorizations (1): change_acl
don't-require-password
description: Chrome Safe Storage
applications (0):
I've also attached a screenshot of the keychain utility on this bot as well which shows it shouldn't be prompting.
,
Jul 20
+erikchen who has helped with telemetry keychain issues in the past--any ideas on how to approach this?
,
Jul 20
Are these also the settings for Chromium Safe Storage? I believe telemetry uses unofficial builds.
,
Jul 20
#9 - telemetry runs official builds on the perf waterfall.
,
Jul 20
Hm, I wrote a bunch of code to deal with this in the past: https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/util/mac/keychain_helper.py?q=file:telemetry.*keychain&sq=package:chromium&dr And here's the setup instructions: https://www.chromium.org/developers/telemetry/telemetry-mac-keychain-setup Do those help? |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by bugdroid1@chromium.org
, Jul 19