peach group canary failed in Paygen with LockNotAcquired error
Reported by
bfreed@chromium.org,
Mar 2 2016
|
||||||
Issue descriptionThis looks similar to https://bugs.chromium.org/p/chromium/issues/detail?id=487858 of 10 months ago. In this case, the peach group canary failed in paygen: https://uberchromegw.corp.google.com/i/chromeos/builders/peach%20group%20canary/builds/2120/steps/Paygen%20%5Bpeach_pit%5D/logs/stdio @@@STEP_FAILURE@@@ 14:15:14: ERROR: <class 'chromite.lib.paygen.gslock.LockNotAcquired'>: Lock: gs://chromeos-releases/tobesigned/45,canary-channel,peach-pit,7993.0.0,payloads,signing,30669-140238617470720,update_signer.payload.signer.instructions.lock held by: PROG=//cros/signer/signing_poller.py USER=root HOSTNAME=chromeos-signing13.hot.corp.google.com PID=12824 TIMESTAMP=Tue, 01 Mar 2016 21:35:11 +0000 (UTC) RANDOM_JUNK=b0a5da582ef913cdbe525f69e79778a8 Traceback (most recent call last): File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 601, in TaskRunner task(*x, **task_kwargs) File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 799, in <lambda> fn = lambda idx, task_args: out_queue.put((idx, task(*task_args))) File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_build_lib.py", line 274, in _GenerateSinglePayload dry_run=dry_run) File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 836, in CreateAndUploadPayload dry_run=dry_run).Run() File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 704, in Run self._drm(self._Create) File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 45, in __call__ return self.Run(func, *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 72, in Run return self._Call(func, *args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 76, in _Call return func(*args, **kwargs) File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 559, in _Create _, metadata_signatures = self._SignPayload() File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 526, in _SignPayload [payload_hash, metadata_hash]) File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 377, in _SignHashes keysets=self.PAYLOAD_SIGNATURE_KEYSETS) File "/b/cbuild/internal_master/chromite/lib/paygen/signer_payloads_client.py", line 387, in GetHashSignatures self._CleanSignerFiles(hashes, keysets) File "/b/cbuild/internal_master/chromite/lib/paygen/signer_payloads_client.py", line 122, in _CleanSignerFiles self._CleanSignerFilesByKeyset(hashes, keyset) File "/b/cbuild/internal_master/chromite/lib/paygen/signer_payloads_client.py", line 96, in _CleanSignerFilesByKeyset with gslock.Lock(request_uri + '.lock'): File "/b/cbuild/internal_master/chromite/lib/paygen/gslock.py", line 205, in __enter__ self.Acquire() File "/b/cbuild/internal_master/chromite/lib/paygen/gslock.py", line 169, in Acquire self._AcquireLock() File "/b/cbuild/internal_master/chromite/lib/paygen/gslock.py", line 158, in _AcquireLock raise LockNotAcquired('Lock: %s held by: %s' % (self._gs_path, contents)) LockNotAcquired: Lock: gs://chromeos-releases/tobesigned/45,canary-channel,peach-pit,7993.0.0,payloads,signing,30669-140238617470720,update_signer.payload.signer.instructions.lock held by: PROG=//cros/signer/signing_poller.py USER=root HOSTNAME=chromeos-signing13.hot.corp.google.com PID=12824 TIMESTAMP=Tue, 01 Mar 2016 21:35:11 +0000 (UTC) RANDOM_JUNK=b0a5da582ef913cdbe525f69e79778a8
,
Mar 2 2016
The link you provided in the initial comment returns 404 for me. +troopers
,
Mar 2 2016
Builders were renamed yesterday. The new link is: https://uberchromegw.corp.google.com/i/chromeos/builders/peach-release-group/builds/2120/steps/Paygen%20%5Bpeach_pit%5D/logs/stdio
,
Mar 2 2016
Renamed to "peach-release-group"?!? The new URL is (currently) https://uberchromegw.corp.google.com/i/chromeos/builders/peach-release-group/builds/2120/steps/Paygen%20%5Bpeach_pit%5D/logs/stdio The next build (2121) succeeded, and the one after that failed for a different reason.
,
Mar 2 2016
Chrome Trooper here... How can I help?
,
Mar 2 2016
Mike explains the cause in #1. The lock will have timed out by now, and payloads can be regenerated by tryjob if they are critical. Otherwise, there is nothing to do.
,
Mar 2 2016
,
Apr 27 2016
,
Aug 12 2016
Closing. please reopen if its not fixed. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by vapier@chromium.org
, Mar 2 2016