"Win 7 Perf" builder sometimes fails the remote_run_result step |
||
Issue descriptionSome recent failures: https://ci.chromium.org/buildbot/chromium.perf/Win%207%20Perf/2730 https://ci.chromium.org/buildbot/chromium.perf/Win%207%20Perf/2717 John: can you help triaging this issue?
,
Jul 10
These appear to be failures in logdog bootstrapping.
,
Jul 10
INFO:remote_run:Bootstrapping through LogDog: ['E:\\b\\.recipe_cipd\\logdog_butler.exe', '-log-level', 'warning', '-project', 'chrome', '-prefix', 'bb/chromium.perf/Win_7_Perf/2730', '-coordinator-host', 'logs.chromium.org', '-output', 'logdog', '-tag', 'buildbot.master=chromium.perf', '-tag', 'buildbot.builder=Win_7_Perf', '-tag', 'buildbot.buildnumber=2730', '-tag', u'logdog.viewer_url=https://luci-milo.appspot.com/buildbot/chromium.perf/Win 7 Perf/2730', '-service-account-json', 'c:\\creds\\service_accounts\\service-account-luci-logdog-publisher.json', '-output-max-buffer-age', '30s', '-io-keepalive-stderr', '5m', 'run', '-stdout', 'tee=stdout', '-stderr', 'tee=stderr', '-streamserver-uri', 'net.pipe:LUCILogDogButler', '--', 'E:\\b\\.recipe_cipd\\logdog_annotee.exe', '-log-level', 'warning', '-name-base', 'recipes', '-print-summary', '-tee', 'annotations', '-json-args-path', 'E:\\b\\rr\\tmpikawyy\\logdog_annotee_cmd.json', '-result-path', 'E:\\b\\rr\\tmpikawyy\\bootstrap_result.json'] INFO:remote_run:Executing command: ['E:\\b\\.recipe_cipd\\logdog_butler.exe', '-log-level', 'warning', '-project', 'chrome', '-prefix', 'bb/chromium.perf/Win_7_Perf/2730', '-coordinator-host', 'logs.chromium.org', '-output', 'logdog', '-tag', 'buildbot.master=chromium.perf', '-tag', 'buildbot.builder=Win_7_Perf', '-tag', 'buildbot.buildnumber=2730', '-tag', u'logdog.viewer_url=https://luci-milo.appspot.com/buildbot/chromium.perf/Win 7 Perf/2730', '-service-account-json', 'c:\\creds\\service_accounts\\service-account-luci-logdog-publisher.json', '-output-max-buffer-age', '30s', '-io-keepalive-stderr', '5m', 'run', '-stdout', 'tee=stdout', '-stderr', 'tee=stderr', '-streamserver-uri', 'net.pipe:LUCILogDogButler', '--', 'E:\\b\\.recipe_cipd\\logdog_annotee.exe', '-log-level', 'warning', '-name-base', 'recipes', '-print-summary', '-tee', 'annotations', '-json-args-path', 'E:\\b\\rr\\tmpikawyy\\logdog_annotee_cmd.json', '-result-path', 'E:\\b\\rr\\tmpikawyy\\bootstrap_result.json'] [W2018-07-09T08:10:26.975048-07:00 4008 0 client.go:306] RPC failed transiently. Will retry in 1s {"error":"failed to send request: Post https://logs.chromium.org/prpc/logdog.Registration/RegisterPrefix: net/http: TLS handshake timeout", "host":"logs.chromium.org", "method":"RegisterPrefix", "service":"logdog.Registration", "sleepTime":"1s"} [W2018-07-09T08:10:57.996352-07:00 4008 0 client.go:306] RPC failed transiently. Will retry in 2s {"error":"failed to send request: context deadline exceeded", "host":"logs.chromium.org", "method":"RegisterPrefix", "service":"logdog.Registration", "sleepTime":"2s"} [W2018-07-09T08:11:30.004951-07:00 4008 0 client.go:306] RPC failed transiently. Will retry in 4s {"error":"failed to send request: context deadline exceeded", "host":"logs.chromium.org", "method":"RegisterPrefix", "service":"logdog.Registration", "sleepTime":"4s"} [W2018-07-09T08:12:04.014151-07:00 4008 0 client.go:306] RPC failed transiently. Will retry in 8s {"error":"failed to send request: context deadline exceeded", "host":"logs.chromium.org", "method":"RegisterPrefix", "service":"logdog.Registration", "sleepTime":"8s"} [W2018-07-09T08:12:42.024551-07:00 4008 0 client.go:306] RPC failed transiently. Will retry in 16s {"error":"failed to send request: context deadline exceeded", "host":"logs.chromium.org", "method":"RegisterPrefix", "service":"logdog.Registration", "sleepTime":"16s"} Good news: While we still don't know the root cause (My money is on OS socket issues), we believe we've found a reasonable workaround. |
||
►
Sign in to add a comment |
||
Comment 1 by jbudorick@chromium.org
, Jul 10Owner: ----