parrot missing R53 builds |
|||||
Issue descriptionparrot-release release-R53-8530.B builder appears to have failed for the last 10 builds. The latest payload found for parrot was 8350.74.0. https://chromegw.corp.google.com/i/chromeos_release/builders/parrot-release%20release-R53-8530.B build#70 failed paygen step with: Failed in stage Paygen: <class 'chromite.lib.gs.GSCommandError'>: return code: 1; command: /b/cbuild/shared_internal/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' stat -- gs://chromeos-releases/stable-channel/parrot/8530.81.0/payloads/signing/17001-139821285111552/1.payload.hash.update_signer.signed.bin WARNING 0907 20:32:09.992029 oauth2_client.py] FileSystemTokenCache.PutToken: Failed to create cache file /tmp/cbuildbot-tmpYIibTN/cbuildbot-tmpRlSqXX/oauth2_client-tokencache.1001.534d7de05989f9d65394a17ce0e37362d9e78f83: [Errno 17] File exists: '/tmp/cbuildbot-tmpYIibTN/cbuildbot-tmpRlSqXX/oauth2_client-tokencache.1001.534d7de05989f9d65394a17ce0e37362d9e78f83' No URLs matched: gs://chromeos-releases/stable-channel/parrot/8530.81.0/payloads/signing/17001-139821285111552/1.payload.hash.update_signer.signed.bin cwd=None, extra env={'BOTO_CONFIG': '/b/build/site_config/.boto'} Traceback (most recent call last): File "/b/cbuild/shared_internal/chromite/lib/parallel.py", line 601, in TaskRunner task(*x, **task_kwargs) File "/b/cbuild/shared_internal/chromite/lib/parallel.py", line 799, in <lambda> fn = lambda idx, task_args: out_queue.put((idx, task(*task_args))) File "/b/cbuild/shared_internal/chromite/lib/paygen/paygen_build_lib.py", line 265, in _GenerateSinglePayload dry_run=dry_run) File "/b/cbuild/shared_internal/chromite/lib/paygen/paygen_payload_lib.py", line 836, in CreateAndUploadPayload dry_run=dry_run).Run() File "/b/cbuild/shared_internal/chromite/lib/paygen/paygen_payload_lib.py", line 704, in Run self._drm(self._Create) File "/b/cbuild/shared_internal/chromite/lib/paygen/dryrun_lib.py", line 45, in __call__ return self.Run(func, *args, **kwargs) File "/b/cbuild/shared_internal/chromite/lib/paygen/dryrun_lib.py", line 72, in Run return self._Call(func, *args, **kwargs) File "/b/cbuild/shared_internal/chromite/lib/paygen/dryrun_lib.py", line 76, in _Call return func(*args, **kwargs) File "/b/cbuild/shared_internal/chromite/lib/paygen/paygen_payload_lib.py", line 559, in _Create _, metadata_signatures = self._SignPayload() File "/b/cbuild/shared_internal/chromite/lib/paygen/paygen_payload_lib.py", line 526, in _SignPayload [payload_hash, metadata_hash]) File "/b/cbuild/shared_internal/chromite/lib/paygen/paygen_payload_lib.py", line 377, in _SignHashes keysets=self.PAYLOAD_SIGNATURE_KEYSETS) File "/b/cbuild/shared_internal/chromite/lib/paygen/signer_payloads_client.py", line 378, in GetHashSignatures if not self._WaitForSignatures(all_signature_uris): File "/b/cbuild/shared_internal/chromite/lib/paygen/signer_payloads_client.py", line 282, in _WaitForSignatures while missing_signatures and self._ctx.Exists(missing_signatures[0]): File "/b/cbuild/shared_internal/chromite/lib/gs.py", line 909, in Exists self.Stat(path, **kwargs) File "/b/cbuild/shared_internal/chromite/lib/gs.py", line 963, in Stat res = self.DoCommand(['stat', '--', path], redirect_stdout=True, **kwargs) File "/b/cbuild/shared_internal/chromite/lib/gs.py", line 654, in DoCommand raise GSCommandError(e.msg, e.result, e.exception) GSCommandError: return code: 1; command: /b/cbuild/shared_internal/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' stat -- gs://chromeos-releases/stable-channel/parrot/8530.81.0/payloads/signing/17001-139821285111552/1.payload.hash.update_signer.signed.bin WARNING 0907 20:32:09.992029 oauth2_client.py] FileSystemTokenCache.PutToken: Failed to create cache file /tmp/cbuildbot-tmpYIibTN/cbuildbot-tmpRlSqXX/oauth2_client-tokencache.1001.534d7de05989f9d65394a17ce0e37362d9e78f83: [Errno 17] File exists: '/tmp/cbuildbot-tmpYIibTN/cbuildbot-tmpRlSqXX/oauth2_client-tokencache.1001.534d7de05989f9d65394a17ce0e37362d9e78f83' No URLs matched: gs://chromeos-releases/stable-channel/parrot/8530.81.0/payloads/signing/17001-139821285111552/1.payload.hash.update_signer.signed.bin cwd=None, extra env={'BOTO_CONFIG': '/b/build/site_config/.boto'}
,
Sep 9 2016
Vapier@ does this look like a signer issue and if yes can you please help investigate what could be going on?
,
Sep 9 2016
looks like an internal gsutil error unrelated to signing
,
Sep 9 2016
My comment in email earlier today: That signing failure isn't a signing failure. It looks like a network/server flake for gsutil that is very unusual and was reported in a way that blocked our usual retries (which probably would have recovered).
,
Sep 9 2016
So this could work the next time we run a build?
,
Sep 9 2016
Wait... no, that was from looking at a TOT signing failure. This looks like 'gsutil' wigged out. This may be fixed in newer releases of it, but we've had trouble updating. The fix for that is in progress.
,
Sep 9 2016
I would not expect to see the same flake. ;>
,
Sep 9 2016
If this is indeed a flake then why would the last 10 builds fail consistently?
,
Sep 9 2016
All but the most recent were compile errors, looks like in Chrome.
,
Sep 9 2016
This view seems relevant: https://chromegw.corp.google.com/i/chromeos_release/builders/parrot-release%20release-R53-8530.B Also, the build errors were NOT flake. R53 was broken for this board, I don't know how that came to be, but it has been fixed.
,
Sep 9 2016
Yes there were chrome compile errors and other gyp gn related issues which were fixes recently. So the previous failures could be related. Let us start another build and see if the issue persists.
,
Sep 12 2016
payload generation issue, ketakid to follow up
,
Sep 12 2016
The next two builds were successful. Unless we want to follow up on the gsutil flake (which I don't think it very profitable unless that symptom starts recurring), I think it's appropriate to close this.
,
Oct 5 2016
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by tienchang@chromium.org
, Sep 9 2016Labels: M-53