Failed master run: https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/17945

The CL in question depends on https://crrev.com/c/936141.

Looking at the history for both CLs, I seem them failing in build_packages repeatedly while in the PreCQ. They were resubmitted until they passed.

Why did they eventually pass? Does the CL cause build time flake, instead of consistent build time failure?

 Issue 819318  has been merged into this issue.

From duped bug:

We lost a lot of CQ runs in the last day to BuildPackages failing on eve-arcnext-paladin: 

https://uberchromegw.corp.google.com/i/chromeos/builders/eve-arcnext-paladin/builds/154
https://uberchromegw.corp.google.com/i/chromeos/builders/eve-arcnext-paladin/builds/156
(Same failure twice)

Blamed CL: https://chrome-internal-review.googlesource.com/c/chromeos/overlays/project-cheets-private/+/581565


https://uberchromegw.corp.google.com/i/chromeos/builders/eve-arcnext-paladin/builds/160
(This one hit some other builders as well)
https://uberchromegw.corp.google.com/i/chromeos/builders/poppy-paladin/builds/4260
https://uberchromegw.corp.google.com/i/chromeos/builders/rainier-paladin/builds/626
https://uberchromegw.corp.google.com/i/chromeos/builders/scarlet-paladin/builds/3417

Blamed CL: ... maybe...
https://chrome-internal-review.googlesource.com/c/chromeos/overlays/project-cheets-private/+/567098

To examine this week.

I'm confused by this, we have both arc and arcnext boards in the PreCQ:

    # Betty is the designated board to run vmtest on N.
    'betty-pre-cq',                   # vm board                       vmtest
    'cyan-no-vmtest-pre-cq',          # braswell     kernel 3.18
    'daisy_spring-no-vmtest-pre-cq',  # arm32        kernel 3.8
    'kevin-arcnext-no-vmtest-pre-cq', # arm64        kernel 4.4        arcnext
    'nyan_blaze-no-vmtest-pre-cq',    # arm32        kernel 3.10
    'reef-no-vmtest-pre-cq',          # apollolake   kernel 4.4        vulkan
    'samus-no-vmtest-pre-cq',         # broadwell    kernel 3.14
    'whirlwind-no-vmtest-pre-cq',     # brillo
    'zako-no-vmtest-pre-cq',          # haswell      kernel 3.8

Asking the current ARC Constable for advice.

What are the substantial differences between the boards used by our PreCQ builders, and the paladin builders mentioned above, ie:

  poppy-paladin
  eve-arcnext-paladin
  rainier-paladin
  scarlet-paladin

For example, are all of the paladins ARM and non of the PreCQ builders ARM?

domlaskowski@ has promised feedback tomorrow.

Edited from a conversation with domlaskowski:

i don't see any obvious differences, e.g. the paladins are a mix of x86/ARM boards with N and P containers
taking a closer look at the compile failures...
hmm, looking at the blamed CL, we might be building different image types for containers, i.e. user/userdebug/eng

looks like we're building conditionally on the cheets_{user,userdebug} USE flags

Do you know where those flags are defined? IE: Is it per build config? Or per board? Or something else?

not entierly sure, but it looks like the ebuilds for the N and P containers use cheets_user by default, and some boards override that

e.g. boards using 64-bit container:
https://chrome-internal.git.corp.google.com/chromeos/overlays/project-cheets-private/+/master/profiles/android/amd64/make.defaults#10
or some explicit board variants:
https://chrome-internal.git.corp.google.com/chromeos/overlays/overlay-caroline-userdebug-private/+/c0cdb14b5ad4f8096b70f23350aa91c09e43cea4/profiles/base/make.defaults#10

After reading through ebuilds, I believe thse are the boards currently using cheets_userdebug, and which so need extra coverage.

novato-compile-only-pre-cq
caroline-compile-only-pre-cq
eve-compile-only-pre-cq

I'm currently running tryjobs against the compile-only pre-cq builders for them to confirm they build, and to try to confirm the useflag usage.

http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8950776505193609120
http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8950776504184376208
http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8950776502411987840

Build logs are showing different use flags than I was predicting from reading ebuilds, based on examining BuildPackages logs.

Existing PreCQ:
  samus:          cheets_user
  kevin-arcnext:  cheets_user

Candidates:
  novato:      cheets_userdebug
  caronline:   cheets_user
  eve:         cheets_user_64
  eve-arcnext: cheets_user_64
  poppy:       cheets_user
  rainier:     cheets_user
  scarlet:     cheets_user

Novato isn't on the CQ at all, so I believe the missing coverage is for "cheets_uesr_64".

My proposal is to remove "kevin-arcnext-novmtest-pre-cq" and add "eve-arcnext-no-vmtest-pre-cq" to the default list.

That doesn't cover all four arc/arcnext and cheets_user/cheets_user_64 combinations, but will also keep our builder load consistent with today.

Running a question sanity build before adding something to the list of default PreCQ builders.

http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8950766498151516784

That proposal would leave us with no arm64 in the PreCQ.

So, changing the proposal to leaving kevin-archnext alone and adding "eve-no-vmtest-pre-cq". 

Better coverage, but heavier load.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7017df438ed3652f4095a51536b11f715bd04fcb

commit 7017df438ed3652f4095a51536b11f715bd04fcb
Author: chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri Mar 30 06:23:43 2018

Roll src/third_party/chromite/ 33146a204..36a9c890b (5 commits)

https://chromium.googlesource.com/chromiumos/chromite.git/+log/33146a20418a..36a9c890b979

$ git log 33146a204..36a9c890b --date=short --no-merges --format='%ad %ae %s'
2018-03-29 yunlian Add USE=thinlto for AMD64 release buidlers only.
2018-03-29 vapier gs: update to 4.30
2018-03-29 yunlian chromeos-config: remove USE=thinlto from terra and caroline-release
2018-03-27 smbarber chromeos_config: add tael to external waterfall
2018-03-28 dgarrett precq: Add eve-no-vmtest-pre-cq.

Created with:
  roll-dep src/third_party/chromite
BUG= chromium:827217 ,chromium:None,chromium:707803,chromium:824615,chromium:820252


The AutoRoll server is located here: https://chromite-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.


TBR=chrome-os-gardeners@chromium.org

Change-Id: I83683cdbd852c5097195bb0689107322912eeb7f
Reviewed-on: https://chromium-review.googlesource.com/987452
Reviewed-by: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: Chromite Chromium Autoroll <chromite-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#547117}
[modify] https://crrev.com/7017df438ed3652f4095a51536b11f715bd04fcb/DEPS