New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 752679 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Aug 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

GS error during payload signing

Project Member Reported by jkwang@google.com, Aug 4 2017

Issue description

This paygen error happened a few times:
12:49:56: WARNING: GS_ERROR: No URLs matched: gs://chromeos-releases/dev-channel/veyron-fievel/9808.0.0/payloads/signing/6166-140527908542272/1.payload.hash.update_signer.signed.bin 
...

@@@STEP_FAILURE@@@
13:09:03: ERROR: <class 'chromite.lib.gs.GSCommandError'>: return code: 1; command: /b/c/cbuild/repository/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' stat -- gs://chromeos-releases/dev-channel/veyron-fievel/9808.0.0/payloads/signing/6166-140527908542272/1.payload.hash.update_signer.signed.bin
WARNING 0804 12:51:21.005610 oauth2_client.py] FileSystemTokenCache.PutToken: Failed to create cache file /tmp/cbuildbot-tmprrL2BY/oauth2_client-tokencache.1002.534d7de05989f9d65394a17ce0e37362d9e78f83: [Errno 17] File exists: '/tmp/cbuildbot-tmprrL2BY/oauth2_client-tokencache.1002.534d7de05989f9d65394a17ce0e37362d9e78f83'
No URLs matched: gs://chromeos-releases/dev-channel/veyron-fievel/9808.0.0/payloads/signing/6166-140527908542272/1.payload.hash.update_signer.signed.bin
cmd=['/b/c/cbuild/repository/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil', '-o', 'Boto:num_retries=10', 'stat', '--', 'gs://chromeos-releases/dev-channel/veyron-fievel/9808.0.0/payloads/signing/6166-140527908542272/1.payload.hash.update_signer.signed.bin'], extra env={'BOTO_CONFIG': '/b/build/site_config/.boto'}
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/lib/parallel.py", line 602, in TaskRunner
    task(*x, **task_kwargs)
  File "/b/c/cbuild/repository/chromite/lib/parallel.py", line 800, in <lambda>
    fn = lambda idx, task_args: out_queue.put((idx, task(*task_args)))
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_build_lib.py", line 251, in _GenerateSinglePayload
    dry_run=dry_run)
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 827, in CreateAndUploadPayload
    dry_run=dry_run).Run()
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 705, in Run
    self._drm(self._Create)
  File "/b/c/cbuild/repository/chromite/lib/paygen/dryrun_lib.py", line 45, in __call__
    return self.Run(func, *args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/paygen/dryrun_lib.py", line 82, in Run
    return self._Call(func, *args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/paygen/dryrun_lib.py", line 86, in _Call
    return func(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 560, in _Create
    _, metadata_signatures = self._SignPayload()
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 527, in _SignPayload
    [payload_hash, metadata_hash])
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 378, in _SignHashes
    keysets=self.PAYLOAD_SIGNATURE_KEYSETS)
  File "/b/c/cbuild/repository/chromite/lib/paygen/signer_payloads_client.py", line 376, in GetHashSignatures
    if not self._WaitForSignatures(all_signature_uris):
  File "/b/c/cbuild/repository/chromite/lib/paygen/signer_payloads_client.py", line 280, in _WaitForSignatures
    while missing_signatures and self._ctx.Exists(missing_signatures[0]):
  File "/b/c/cbuild/repository/chromite/lib/gs.py", line 1039, in Exists
    self.Stat(path, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/gs.py", line 1093, in Stat
    res = self.DoCommand(['stat', '--', path], redirect_stdout=True, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/gs.py", line 783, in DoCommand
    raise GSCommandError(e.msg, e.result, e.exception)
GSCommandError: return code: 1; command: /b/c/cbuild/repository/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' stat -- gs://chromeos-releases/dev-channel/veyron-fievel/9808.0.0/payloads/signing/6166-140527908542272/1.payload.hash.update_signer.signed.bin
WARNING 0804 12:51:21.005610 oauth2_client.py] FileSystemTokenCache.PutToken: Failed to create cache file /tmp/cbuildbot-tmprrL2BY/oauth2_client-tokencache.1002.534d7de05989f9d65394a17ce0e37362d9e78f83: [Errno 17] File exists: '/tmp/cbuildbot-tmprrL2BY/oauth2_client-tokencache.1002.534d7de05989f9d65394a17ce0e37362d9e78f83'
No URLs matched: gs://chromeos-releases/dev-channel/veyron-fievel/9808.0.0/payloads/signing/6166-140527908542272/1.payload.hash.update_signer.signed.bin
cmd=['/b/c/cbuild/repository/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil', '-o', 'Boto:num_retries=10', 'stat', '--', 'gs://chromeos-releases/dev-channel/veyron-fievel/9808.0.0/payloads/signing/6166-140527908542272/1.payload.hash.update_signer.signed.bin'], extra env={'BOTO_CONFIG': '/b/build/site_config/.boto'}

Link to builder: https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_fievel-release/builds/1368
 
Summary: GS error during payload signing (was: Paygen error:)
Components: Infra>Client>ChromeOS
Labels: -Restrict-View-Google
I believe this is the important error:

WARNING 0804 12:51:21.005610 oauth2_client.py] FileSystemTokenCache.PutToken: Failed to create cache file /tmp/cbuildbot-tmprrL2BY/oauth2_client-tokencache.1002.534d7de05989f9d65394a17ce0e37362d9e78f83: [Errno 17] File exists: '/tmp/cbuildbot-tmprrL2BY/oauth2_client-tokencache.1002.534d7de05989f9d65394a17ce0e37362d9e78f83'

I don't know why we're seeing it now though.
That was a gsutil bug seen a long time ago, but we haven't updated our version in a very long time.
I reinstanced the builder, just be paranoid.

Comment 7 by nxia@chromium.org, Aug 9 2017

Owner: dgarr...@chromium.org
is this solved by re-initiating the builder?
Status: Fixed (was: Untriaged)
It seems to be fixed, but I'm not sure if I fixed it.

Sign in to add a comment