lakitu_next pre-cqs running on wrong builders |
|||||
Issue descriptionpre-cq tests are failing for lakitu_next board. See the attempts on https://chrome-internal-review.googlesource.com/#/c/259660/ Ex: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/pre-cq/builds/30747 The VMTests fail with error: Starting a KVM instance Could not access KVM kernel module: No such file or directory The trybot run for the same patch passes: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/incremental/builds/165 The theory is that pre-cq is running on 'cros-standard26-c2' builder which cannot run VMtests whereas trybot correctly runs on 'build275-m2' builders. Can chromeos-infra deputy please confirm if this is indeed the root cause and help us address this? This is currently blocking commits to lakitu_next. FYI, change https://chrome-internal-review.googlesource.com/#/c/258963/ recently enabled running pre-cq for lakitu_next board.
,
May 26 2016
Yes, that test build clearly ran on a GCE instance which won't work with VM Tests enabled. I'm going to try and track down how the trybot waterfall decides which builds go where.
,
May 26 2016
Has the trybot waterfall been restarted since the lakitu config changes went in? Reading through the trybot logic, it should do the right thing because it examines the build configs to decide what goes where. However, it only re-reads config changes when the waterfall is restarted.
,
May 26 2016
I've restarted the waterfall restart that I think will fix this. Blocking this bug on the restart bug.
,
May 26 2016
In the Chrome Infra codebase, see:
build/masters/master.chromiumos.tryserver/chromiumos_tryserver_util.py.
This is the important bit of code:
precq_builders = set(
v['_template'] or k for k, v in configs.iteritems() if v.IsPreCqBuilder())
precq_novmtest_builders = set(
v['_template'] or k for k, v in configs.iteritems()
if v.IsPreCqBuilder() and not v.HasVmTests() and not v.HasHwTests())
,
May 26 2016
Thanks for looking into this. lakitu changes were done in CL:344288 and CL:*258963. I don't know if waterfall was restarted since then. Any idea of ETA for this?
,
May 26 2016
,
May 26 2016
Hopefully, this now fixed. Can someone run a tryjob to confirm?
,
May 26 2016
Retrying committing https://chrome-internal-review.googlesource.com/#/c/259660
,
May 26 2016
The latest pre-cq run once again got scheduled on "cros-standard18-c2" machine: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/pre-cq/builds/31073 Probably going to fail again?
,
May 26 2016
Yep. I'm asking questions on the blocking bug. Maybe the restart hasn't really happened, only been scheduled. Or maybe the chromite ping wasn't bumped up before the restart.
,
May 26 2016
The pin wasn't bumped (yes, confusing terminology). Nodir@ will bump it and restart that waterfall again. After that's done (and it may take a while), I expect this to be fixed.
,
May 27 2016
The trybot waterfall was pin bumped and restarted, and it looks like the CL above passed PreCQ testing. I think this is fixed!
,
May 27 2016
Thanks! The pre-cq seems to be fixed. Though it seems the lakitu_next-release builder started failing https://uberchromegw.corp.google.com/i/chromeos/builders/lakitu_next-release and the reason is unclear. Actually, only the 'cbuildbot' step is marked as failed, but the image build and test and all other steps are successful. Not sure if its related to the restart.
,
May 27 2016
I'm certain those issues are 100% unrelated, but looking.
,
Jun 27 2016
Closing... please feel free to reopen if its not fixed.
,
Jun 27 2016
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by kevcheng@chromium.org
, May 26 2016