New issue
Advanced search Search tips

Issue 861798 link

Starred by 1 user

Issue metadata

Status: Duplicate
Owner: ----
Closed: Jul 10
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

"Win 7 Perf" builder sometimes fails the remote_run_result step

Project Member Reported by nednguyen@chromium.org, Jul 9

Issue description

Components: Infra>Platform>LogDog
Owner: ----
These appear to be failures in logdog bootstrapping.
Mergedinto: 811232
Status: Duplicate (was: Untriaged)
INFO:remote_run:Bootstrapping through LogDog: ['E:\\b\\.recipe_cipd\\logdog_butler.exe', '-log-level', 'warning', '-project', 'chrome', '-prefix', 'bb/chromium.perf/Win_7_Perf/2730', '-coordinator-host', 'logs.chromium.org', '-output', 'logdog', '-tag', 'buildbot.master=chromium.perf', '-tag', 'buildbot.builder=Win_7_Perf', '-tag', 'buildbot.buildnumber=2730', '-tag', u'logdog.viewer_url=https://luci-milo.appspot.com/buildbot/chromium.perf/Win 7 Perf/2730', '-service-account-json', 'c:\\creds\\service_accounts\\service-account-luci-logdog-publisher.json', '-output-max-buffer-age', '30s', '-io-keepalive-stderr', '5m', 'run', '-stdout', 'tee=stdout', '-stderr', 'tee=stderr', '-streamserver-uri', 'net.pipe:LUCILogDogButler', '--', 'E:\\b\\.recipe_cipd\\logdog_annotee.exe', '-log-level', 'warning', '-name-base', 'recipes', '-print-summary', '-tee', 'annotations', '-json-args-path', 'E:\\b\\rr\\tmpikawyy\\logdog_annotee_cmd.json', '-result-path', 'E:\\b\\rr\\tmpikawyy\\bootstrap_result.json']
INFO:remote_run:Executing command: ['E:\\b\\.recipe_cipd\\logdog_butler.exe', '-log-level', 'warning', '-project', 'chrome', '-prefix', 'bb/chromium.perf/Win_7_Perf/2730', '-coordinator-host', 'logs.chromium.org', '-output', 'logdog', '-tag', 'buildbot.master=chromium.perf', '-tag', 'buildbot.builder=Win_7_Perf', '-tag', 'buildbot.buildnumber=2730', '-tag', u'logdog.viewer_url=https://luci-milo.appspot.com/buildbot/chromium.perf/Win 7 Perf/2730', '-service-account-json', 'c:\\creds\\service_accounts\\service-account-luci-logdog-publisher.json', '-output-max-buffer-age', '30s', '-io-keepalive-stderr', '5m', 'run', '-stdout', 'tee=stdout', '-stderr', 'tee=stderr', '-streamserver-uri', 'net.pipe:LUCILogDogButler', '--', 'E:\\b\\.recipe_cipd\\logdog_annotee.exe', '-log-level', 'warning', '-name-base', 'recipes', '-print-summary', '-tee', 'annotations', '-json-args-path', 'E:\\b\\rr\\tmpikawyy\\logdog_annotee_cmd.json', '-result-path', 'E:\\b\\rr\\tmpikawyy\\bootstrap_result.json']
[W2018-07-09T08:10:26.975048-07:00 4008 0 client.go:306] RPC failed transiently. Will retry in 1s    {"error":"failed to send request: Post https://logs.chromium.org/prpc/logdog.Registration/RegisterPrefix: net/http: TLS handshake timeout", "host":"logs.chromium.org", "method":"RegisterPrefix", "service":"logdog.Registration", "sleepTime":"1s"}
[W2018-07-09T08:10:57.996352-07:00 4008 0 client.go:306] RPC failed transiently. Will retry in 2s    {"error":"failed to send request: context deadline exceeded", "host":"logs.chromium.org", "method":"RegisterPrefix", "service":"logdog.Registration", "sleepTime":"2s"}
[W2018-07-09T08:11:30.004951-07:00 4008 0 client.go:306] RPC failed transiently. Will retry in 4s    {"error":"failed to send request: context deadline exceeded", "host":"logs.chromium.org", "method":"RegisterPrefix", "service":"logdog.Registration", "sleepTime":"4s"}
[W2018-07-09T08:12:04.014151-07:00 4008 0 client.go:306] RPC failed transiently. Will retry in 8s    {"error":"failed to send request: context deadline exceeded", "host":"logs.chromium.org", "method":"RegisterPrefix", "service":"logdog.Registration", "sleepTime":"8s"}
[W2018-07-09T08:12:42.024551-07:00 4008 0 client.go:306] RPC failed transiently. Will retry in 16s   {"error":"failed to send request: context deadline exceeded", "host":"logs.chromium.org", "method":"RegisterPrefix", "service":"logdog.Registration", "sleepTime":"16s"}

Good news: While we still don't know the root cause (My money is on OS socket issues), we believe we've found a reasonable workaround.

Sign in to add a comment