New issue
Advanced search Search tips
Starred by 2 users

Issue metadata

Status: Verified
Owner:
Closed: Feb 7
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Paladins sometimes can't mount chroot

Project Member Reported by bmgordon@chromium.org, Feb 6 Back to list

Issue description

WARNING: Failed to activate VG on try 1.

ERROR: <class 'chromite.lib.cros_build_lib.RunCommandError'>: return code: 32; command: sudo -n 'CROS_CACHEDIR=/b/c/cbuild/repository/.cache' 'CROS_SUDO_KEEP_ALIVE=unknown' -- mount -text4 -onoatime /dev/cros_b+c+cbuild+repository+chroot_000/chroot /b/c/cbuild/repository/chroot

mount: special device /dev/cros_b+c+cbuild+repository+chroot_000/chroot does not exist

https://luci-milo.appspot.com/buildbot/chromeos/edgar-paladin/2113
 
Looks like the VG setup didn't complete quickly enough.  The previous two runs (and all the other paladins) worked fine, so this sounds like a race condition in LVM.  I'll look into adding some retry logic.
Summary: Paladins sometimes can't mount chroot (was: edgar-paladin can't mount chroot)
This has sporadically hit a couple of other paladins:

https://luci-milo.appspot.com/buildbot/chromeos/veyron_jaq-paladin/2602
https://luci-milo.appspot.com/buildbot/chromeos/rainier-paladin/383
https://luci-milo.appspot.com/buildbot/chromeos/nyan-full-compile-paladin/11777

Looking to see if there's any commonality about when it happens.
Cc: cra...@chromium.org adlr@chromium.org
Project Member

Comment 4 by bugdroid1@chromium.org, Feb 7

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/31cf629aad061dbe68a77ed49240c3353eba8d23

commit 31cf629aad061dbe68a77ed49240c3353eba8d23
Author: Benjamin Gordon <bmgordon@chromium.org>
Date: Wed Feb 07 02:19:03 2018

cros_build_lib: Wait for LV to appear before mounting

Sometimes after running lvchange, the device takes a few seconds to
appear.  Check for the device and retry a couple of times as needed to
wait for it.  This is similar to the logic that was previously added to
wait for vgchange.

BUG= chromium:809570 
TEST=cros tryjob edgar-paladin-tryjob

Change-Id: I77648232a7af36285253f5267ae2f90c4dfad063
Reviewed-on: https://chromium-review.googlesource.com/904731
Commit-Ready: Benjamin Gordon <bmgordon@chromium.org>
Tested-by: Benjamin Gordon <bmgordon@chromium.org>
Reviewed-by: Benjamin Gordon <bmgordon@chromium.org>

[modify] https://crrev.com/31cf629aad061dbe68a77ed49240c3353eba8d23/lib/cros_build_lib.py

Are recent failures of betty-arc64-paladin related to this? I filed Issue 809957.

Issue 809957 has been merged into this issue.
Status: Verified
The underlying race was triggered on one build in the latest CQ run (https://luci-milo.appspot.com/buildbot/chromeos/link-paladin/30993).  The change in https://crrev.com/c/904731 handled it correctly, so I'm calling this fixed.

Sign in to add a comment