New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 591256 link

Starred by 1 user

Issue metadata

Status: Verified
Owner: ----
Closed: Mar 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

peach group canary failed in Paygen with LockNotAcquired error

Reported by bfreed@chromium.org, Mar 2 2016

Issue description

This looks similar to https://bugs.chromium.org/p/chromium/issues/detail?id=487858 of 10 months ago.

In this case, the peach group canary failed in paygen:
https://uberchromegw.corp.google.com/i/chromeos/builders/peach%20group%20canary/builds/2120/steps/Paygen%20%5Bpeach_pit%5D/logs/stdio

@@@STEP_FAILURE@@@
14:15:14: ERROR: <class 'chromite.lib.paygen.gslock.LockNotAcquired'>: Lock: gs://chromeos-releases/tobesigned/45,canary-channel,peach-pit,7993.0.0,payloads,signing,30669-140238617470720,update_signer.payload.signer.instructions.lock held by: PROG=//cros/signer/signing_poller.py
USER=root
HOSTNAME=chromeos-signing13.hot.corp.google.com
PID=12824
TIMESTAMP=Tue, 01 Mar 2016 21:35:11 +0000 (UTC)
RANDOM_JUNK=b0a5da582ef913cdbe525f69e79778a8

Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 601, in TaskRunner
    task(*x, **task_kwargs)
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 799, in <lambda>
    fn = lambda idx, task_args: out_queue.put((idx, task(*task_args)))
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_build_lib.py", line 274, in _GenerateSinglePayload
    dry_run=dry_run)
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 836, in CreateAndUploadPayload
    dry_run=dry_run).Run()
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 704, in Run
    self._drm(self._Create)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 45, in __call__
    return self.Run(func, *args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 72, in Run
    return self._Call(func, *args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/dryrun_lib.py", line 76, in _Call
    return func(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 559, in _Create
    _, metadata_signatures = self._SignPayload()
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 526, in _SignPayload
    [payload_hash, metadata_hash])
  File "/b/cbuild/internal_master/chromite/lib/paygen/paygen_payload_lib.py", line 377, in _SignHashes
    keysets=self.PAYLOAD_SIGNATURE_KEYSETS)
  File "/b/cbuild/internal_master/chromite/lib/paygen/signer_payloads_client.py", line 387, in GetHashSignatures
    self._CleanSignerFiles(hashes, keysets)
  File "/b/cbuild/internal_master/chromite/lib/paygen/signer_payloads_client.py", line 122, in _CleanSignerFiles
    self._CleanSignerFilesByKeyset(hashes, keyset)
  File "/b/cbuild/internal_master/chromite/lib/paygen/signer_payloads_client.py", line 96, in _CleanSignerFilesByKeyset
    with gslock.Lock(request_uri + '.lock'):
  File "/b/cbuild/internal_master/chromite/lib/paygen/gslock.py", line 205, in __enter__
    self.Acquire()
  File "/b/cbuild/internal_master/chromite/lib/paygen/gslock.py", line 169, in Acquire
    self._AcquireLock()
  File "/b/cbuild/internal_master/chromite/lib/paygen/gslock.py", line 158, in _AcquireLock
    raise LockNotAcquired('Lock: %s held by: %s' % (self._gs_path, contents))
LockNotAcquired: Lock: gs://chromeos-releases/tobesigned/45,canary-channel,peach-pit,7993.0.0,payloads,signing,30669-140238617470720,update_signer.payload.signer.instructions.lock held by: PROG=//cros/signer/signing_poller.py
USER=root
HOSTNAME=chromeos-signing13.hot.corp.google.com
PID=12824
TIMESTAMP=Tue, 01 Mar 2016 21:35:11 +0000 (UTC)
RANDOM_JUNK=b0a5da582ef913cdbe525f69e79778a8

 
i might have caused that by inadvertently rebooting 13.hot in the middle of its run
Labels: -Infra Infra-Troopers
The link you provided in the initial comment returns 404 for me.

+troopers
Renamed to "peach-release-group"?!?

The new URL is (currently)
https://uberchromegw.corp.google.com/i/chromeos/builders/peach-release-group/builds/2120/steps/Paygen%20%5Bpeach_pit%5D/logs/stdio

The next build (2121) succeeded, and the one after that failed for a different reason.
Chrome Trooper here... How can I help? 
Cc: msartori@chromium.org
Mike explains the cause in #1. The lock will have timed out by now, and payloads can be regenerated by tryjob if they are critical.

Otherwise, there is nothing to do.
Status: Fixed (was: Untriaged)

Comment 8 by benhenry@google.com, Apr 27 2016

Components: Infra>Client>ChromeOS
Labels: -Infra-ChromeOS
Status: Verified (was: Fixed)
Closing. please reopen if its not fixed.

Sign in to add a comment