Issue 835293  has been merged into this issue.

Goma related issue, change the label to Foundation-Troopers.

goma_compiler_proxy logs are not preserved (nor sent to Cloud Logging) in case goma fails to start, so it is hard to debug :-/ Trying to catch the bot in the middle of staring goma to grab logs via SSH.

There's a step "preprocess_for_goma.upload_goma_start_failed_logs" with something. But I can't find many details there: https://chromium-build-stats.appspot.com/compiler_proxy_log/2018/04/20/vm31-m1/compiler_proxy.vm31-m1.golo.chromium.org.chrome-bot.log.INFO.20180420-152409.68672.gz

This step itself also has errors from tsmon library:
Failed to retrieve access token: {
  "error" : "invalid_grant",
  "error_description" : "Invalid JWT: Token must be a short-lived token (60 minutes) and in a reasonable timeframe. Check your iat and exp values and use a clock with skew to account for clock differences between systems."
}

Which suggest issues with the clock. They probably affect goma client too. 

The clock on the machine in 5 min into the future. I think it is enough to screw up JWTs.

I ran "ntpdate -u time1.golo.chromium.org" to fix the time. Will look at the build to confirm it helped.

+ Labs, since I believe we are running ntpd automatically everywhere and this should not have happened.

Yeah, the problem was in the clock. Next build started goma successfully: https://ci.chromium.org/buildbot/chromium.webkit/WebKit%20Mac%20Builder/240394

The builder might not auto-reboot?
I heard that mac sometimes stops ntpd silently.  Bots with auto-reboot automatically restarts the job.  However, for bots without auto-reboot, ntpd is kept on dead and time would be unacceptable large difference for GoogleAPI.

If my memory is correct, we might have seen the same issue before, and we made the bot auto-reboot at that time.

already fixed?
or anything remain to do with this issue?