New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 645223 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Last visit > 30 days ago
Closed: Sep 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

parrot missing R53 builds

Project Member Reported by tienchang@chromium.org, Sep 8 2016

Issue description

parrot-release release-R53-8530.B builder appears to have failed for the last 10 builds. The latest payload found for parrot was 8350.74.0.

https://chromegw.corp.google.com/i/chromeos_release/builders/parrot-release%20release-R53-8530.B

build#70 failed paygen step with:
Failed in stage Paygen:

<class 'chromite.lib.gs.GSCommandError'>: return code: 1; command: /b/cbuild/shared_internal/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' stat -- gs://chromeos-releases/stable-channel/parrot/8530.81.0/payloads/signing/17001-139821285111552/1.payload.hash.update_signer.signed.bin
WARNING 0907 20:32:09.992029 oauth2_client.py] FileSystemTokenCache.PutToken: Failed to create cache file /tmp/cbuildbot-tmpYIibTN/cbuildbot-tmpRlSqXX/oauth2_client-tokencache.1001.534d7de05989f9d65394a17ce0e37362d9e78f83: [Errno 17] File exists: '/tmp/cbuildbot-tmpYIibTN/cbuildbot-tmpRlSqXX/oauth2_client-tokencache.1001.534d7de05989f9d65394a17ce0e37362d9e78f83'
No URLs matched: gs://chromeos-releases/stable-channel/parrot/8530.81.0/payloads/signing/17001-139821285111552/1.payload.hash.update_signer.signed.bin
cwd=None, extra env={'BOTO_CONFIG': '/b/build/site_config/.boto'}
Traceback (most recent call last):
  File "/b/cbuild/shared_internal/chromite/lib/parallel.py", line 601, in TaskRunner
    task(*x, **task_kwargs)
  File "/b/cbuild/shared_internal/chromite/lib/parallel.py", line 799, in <lambda>
    fn = lambda idx, task_args: out_queue.put((idx, task(*task_args)))
  File "/b/cbuild/shared_internal/chromite/lib/paygen/paygen_build_lib.py", line 265, in _GenerateSinglePayload
    dry_run=dry_run)
  File "/b/cbuild/shared_internal/chromite/lib/paygen/paygen_payload_lib.py", line 836, in CreateAndUploadPayload
    dry_run=dry_run).Run()
  File "/b/cbuild/shared_internal/chromite/lib/paygen/paygen_payload_lib.py", line 704, in Run
    self._drm(self._Create)
  File "/b/cbuild/shared_internal/chromite/lib/paygen/dryrun_lib.py", line 45, in __call__
    return self.Run(func, *args, **kwargs)
  File "/b/cbuild/shared_internal/chromite/lib/paygen/dryrun_lib.py", line 72, in Run
    return self._Call(func, *args, **kwargs)
  File "/b/cbuild/shared_internal/chromite/lib/paygen/dryrun_lib.py", line 76, in _Call
    return func(*args, **kwargs)
  File "/b/cbuild/shared_internal/chromite/lib/paygen/paygen_payload_lib.py", line 559, in _Create
    _, metadata_signatures = self._SignPayload()
  File "/b/cbuild/shared_internal/chromite/lib/paygen/paygen_payload_lib.py", line 526, in _SignPayload
    [payload_hash, metadata_hash])
  File "/b/cbuild/shared_internal/chromite/lib/paygen/paygen_payload_lib.py", line 377, in _SignHashes
    keysets=self.PAYLOAD_SIGNATURE_KEYSETS)
  File "/b/cbuild/shared_internal/chromite/lib/paygen/signer_payloads_client.py", line 378, in GetHashSignatures
    if not self._WaitForSignatures(all_signature_uris):
  File "/b/cbuild/shared_internal/chromite/lib/paygen/signer_payloads_client.py", line 282, in _WaitForSignatures
    while missing_signatures and self._ctx.Exists(missing_signatures[0]):
  File "/b/cbuild/shared_internal/chromite/lib/gs.py", line 909, in Exists
    self.Stat(path, **kwargs)
  File "/b/cbuild/shared_internal/chromite/lib/gs.py", line 963, in Stat
    res = self.DoCommand(['stat', '--', path], redirect_stdout=True, **kwargs)
  File "/b/cbuild/shared_internal/chromite/lib/gs.py", line 654, in DoCommand
    raise GSCommandError(e.msg, e.result, e.exception)
GSCommandError: return code: 1; command: /b/cbuild/shared_internal/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' stat -- gs://chromeos-releases/stable-channel/parrot/8530.81.0/payloads/signing/17001-139821285111552/1.payload.hash.update_signer.signed.bin
WARNING 0907 20:32:09.992029 oauth2_client.py] FileSystemTokenCache.PutToken: Failed to create cache file /tmp/cbuildbot-tmpYIibTN/cbuildbot-tmpRlSqXX/oauth2_client-tokencache.1001.534d7de05989f9d65394a17ce0e37362d9e78f83: [Errno 17] File exists: '/tmp/cbuildbot-tmpYIibTN/cbuildbot-tmpRlSqXX/oauth2_client-tokencache.1001.534d7de05989f9d65394a17ce0e37362d9e78f83'
No URLs matched: gs://chromeos-releases/stable-channel/parrot/8530.81.0/payloads/signing/17001-139821285111552/1.payload.hash.update_signer.signed.bin
cwd=None, extra env={'BOTO_CONFIG': '/b/build/site_config/.boto'}

 
Cc: xixuan@chromium.org
Labels: M-53
Cc: vapier@chromium.org
Vapier@ does this look like a signer issue and if yes can you please help investigate what could be going on?
Cc: dgarr...@chromium.org
looks like an internal gsutil error unrelated to signing
My comment in email earlier today:

That signing failure isn't a signing failure. It looks like a network/server flake for gsutil that is very unusual and was reported in a way that blocked our usual retries (which probably would have recovered).
So this could work the next time we run a build?
Wait... no, that was from looking at a TOT signing failure.

This looks like 'gsutil' wigged out. This may be fixed in newer releases of it, but we've had trouble updating. The fix for that is in progress.
I would not expect to see the same flake. ;>
If this is indeed a flake then why would the last 10 builds fail consistently?
All but the most recent were compile errors, looks like in Chrome.
This view seems relevant:

https://chromegw.corp.google.com/i/chromeos_release/builders/parrot-release%20release-R53-8530.B

Also, the build errors were NOT flake. R53 was broken for this board, I don't know how that came to be, but it has been fixed.
Yes there were chrome compile errors and other gyp gn related issues which were fixes recently. So the previous failures could be related. Let us  start another build and see if the issue persists.

Comment 12 by dchan@chromium.org, Sep 12 2016

payload generation issue, ketakid to follow up
Status: Fixed (was: Assigned)
The next two builds were successful.

Unless we want to follow up on the gsutil flake (which I don't think it very profitable unless that symptom starts recurring), I think it's appropriate to close this.
Status: Verified (was: Fixed)

Sign in to add a comment