Ocassionally, perf builder timed out after 10 hours |
|
Issue descriptionThis has happened in: https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/Win%207%20Nvidia%20GPU%20Perf/3183 https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win-10-perf/706 It's not immediately clear why the performance_test_suite took so long in those instance yet.
,
Sep 25
Execution timeout of the build is set to 7h. Swarming is trying to kill the build after 7h. "Failed to refresh the token" and vpython errors are just consequences of the buggy termination. But even without them the build would have been purple (due to running longer than 7h). One really annoying thing is that log and annotation streams stays open (because 'kitchen' could not close them when it was shutting down). It makes the build page in Milo look as if the last step is running forever.
,
Sep 25
|
|
►
Sign in to add a comment |
|
Comment 1 by bpastene@chromium.org
, Sep 25Pick a purple build -> click on swarming task -> scroll to bottom of log, which has: [W 2018-09-25 01:56:36] Failed to refresh the token: context canceled {"key":"luci_ctx/b762655e03441b280bef15c3eed37de17907dcc2", "scopes":"https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email"} [E 2018-09-25 01:56:36] Failed to send PubSub message. {"error":"rpc error: code = Unauthenticated desc = transport: context canceled", "pubsub":"pubsub(projects/luci-logdog/topics/logs)"} 3684 2018-09-25 08:57:01.588 E: Error while removing named cache u'C:\\b\\swarming\\w\\ir\\cache\\vpython' at u'vpython'. The cache will be lost. Traceback (most recent call last): File "C:\b\swarming\swarming_bot.1.zip\client\run_isolated.py", line 1357, in install_named_caches named_cache.uninstall(path, name) File "C:\b\swarming\swarming_bot.1.zip\client\local_caching.py", line 903, in uninstall name, path, ex)) NamedCacheError: cannot uninstall cache named u'vpython' at u'C:\\b\\swarming\\w\\ir\\cache\\vpython': [Error 5] Access is denied Looks like named cache/vpython errors.