New issue
Advanced search Search tips

Issue 888987 link

Starred by 1 user

Issue metadata

Status: Duplicate
Owner: ----
Closed: Sep 25
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Ocassionally, perf builder timed out after 10 hours

Project Member Reported by nednguyen@chromium.org, Sep 25

Issue description

This has happened in:
https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/Win%207%20Nvidia%20GPU%20Perf/3183
https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/win-10-perf/706


It's not immediately clear why the performance_test_suite took so long in those instance yet.
 
Labels: Foundation-Troopers
Pick a purple build -> click on swarming task -> scroll to bottom of log, which has:

[W 2018-09-25 01:56:36] Failed to refresh the token: context canceled {"key":"luci_ctx/b762655e03441b280bef15c3eed37de17907dcc2", "scopes":"https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email"}
[E 2018-09-25 01:56:36] Failed to send PubSub message.              {"error":"rpc error: code = Unauthenticated desc = transport: context canceled", "pubsub":"pubsub(projects/luci-logdog/topics/logs)"}
3684 2018-09-25 08:57:01.588 E: Error while removing named cache u'C:\\b\\swarming\\w\\ir\\cache\\vpython' at u'vpython'. The cache will be lost.
Traceback (most recent call last):
  File "C:\b\swarming\swarming_bot.1.zip\client\run_isolated.py", line 1357, in install_named_caches
    named_cache.uninstall(path, name)
  File "C:\b\swarming\swarming_bot.1.zip\client\local_caching.py", line 903, in uninstall
    name, path, ex))
NamedCacheError: cannot uninstall cache named u'vpython' at u'C:\\b\\swarming\\w\\ir\\cache\\vpython': [Error 5] Access is denied

Looks like named cache/vpython errors.
Execution timeout of the build is set to 7h. Swarming is trying to kill the build after 7h. "Failed to refresh the token" and vpython errors are just consequences of the buggy termination. But even without them the build would have been purple (due to running longer than 7h).

One really annoying thing is that log and annotation streams stays open (because 'kitchen' could not close them when it was shutting down). It makes the build page in Milo look as if the last step is running forever.
Mergedinto: 888913
Status: Duplicate (was: Untriaged)
I think it is ~= dup of issue 888913

Sign in to add a comment