We have had issues in the past due to architecture differences so I think it would be good to do both tatl and tael and an x86 and arm host.

i'm wondering if we should add tatl/tael (compile-only? unittests-only?) to the default pre-cq since they do use a number of system packages with different USE settings.  have we seen breakage along those lines in the past in the CQ to warrant it ?

or should we just go with {eve,kevin,tael,tatl}-precq for vm_tools/ and call it a day ?

I think that would be useful. We've had breakage from LXC uprevs ( issue 859895 ). I also recall some USE flag changes with other ebuilds being caught by tatl-paladin before.

most recently i saw a wayland update only being caught in the CQ

ok, lets start with adding tael/tatl unittest precq bots (running in incremental mode) to the default precq config and see how well this performs

Yep, sounds like a good idea. Lann as current oncall can you do this? You can use CL https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1055251 as a guide. The next time the PreCQ relaunches (every 4 hours) it should get this configuration change.

crrev.com/c/1220626

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/4aefea51330c8ae9009ae8da736888ee8cc88695

commit 4aefea51330c8ae9009ae8da736888ee8cc88695
Author: Lann Martin <lannm@chromium.org>
Date: Fri Sep 14 01:58:45 2018

Add tael and tatl to default PreCQ

BUG= chromium:882965 
TEST=none

Change-Id: I5f22713599691fc48a21a3f085d83f16489af6fa
Reviewed-on: https://chromium-review.googlesource.com/1220626
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Lann Martin <lannm@chromium.org>
Reviewed-by: Jason Clinton <jclinton@chromium.org>

[modify] https://crrev.com/4aefea51330c8ae9009ae8da736888ee8cc88695/lib/constants.py

i think something is off.  these pre-cq configs are dying on internal CL:
  https://chrome-internal-review.googlesource.com/679457

if these configs normally only build the public manifest, we'll need to adjust their configs so they don't run for internal-only repos, or they always run with an internal manifest (just the precq configs).

Over to current oncall to fix.

Ack. Although I haven't the firs idea what tatl/tael is or what is being discussed here?

Is the TL;DR to remove those from PreCQ for now?

That would immediately make it not a P0; the actual fix is what vapier@ said in #8.

crrev.com/1226173

Woops, crrev.com/c/1226173. Chump it?

Chumped. Now to try and figure out what Vapier is talking about :s

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/7437f9dd69ca59a63e0aca91378c2a011164c5a2

commit 7437f9dd69ca59a63e0aca91378c2a011164c5a2
Author: Alec Thilenius <athilenius@google.com>
Date: Fri Sep 14 18:21:05 2018

Revert "Add tael and tatl to default PreCQ"

This reverts commit 4aefea51330c8ae9009ae8da736888ee8cc88695.

Reason for revert: See https://bugs.chromium.org/p/chromium/issues/detail?id=882965#c8

Original change's description:
> Add tael and tatl to default PreCQ
> 
> BUG= chromium:882965 
> TEST=none
> 
> Change-Id: I5f22713599691fc48a21a3f085d83f16489af6fa
> Reviewed-on: https://chromium-review.googlesource.com/1220626
> Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
> Tested-by: Lann Martin <lannm@chromium.org>
> Reviewed-by: Jason Clinton <jclinton@chromium.org>

Bug:  chromium:882965 
Change-Id: I42972b5700ee091e3ebcb213e9f9187ae974c83c
Reviewed-on: https://chromium-review.googlesource.com/1226173
Reviewed-by: Alec Thilenius <athilenius@google.com>
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Tested-by: Jason Clinton <jclinton@chromium.org>

[modify] https://crrev.com/7437f9dd69ca59a63e0aca91378c2a011164c5a2/lib/constants.py

i don't think we currently have the ability to filter by internal manifests, otherwise that'd be ideal ... we'd have certain configs only run for CLs in the public manifest.

which means that all pre-cq configs probably should be marked as internal builders, not just tael/tatl.  which means updating the 'pre_cq' template to inherit site_config.templates.internal.

is my guess.

Filter what by internal manifests? CLs? Are there internal and external PreCQs? And what does it mean to 'mark a pre-cq config as an internal builder'?

And meta question, why is this a CI Bobby role? This seems like an incremental improvement (CI sure, but not on-call).

This just hit https://chrome-internal-review.googlesource.com/c/chromeos/chromeos-hwid/+/678868 FYI.

He means we cannot have a public build target testing internal CLs, so the PreCQ selection would need to differentiate between internal and public CLs (based on where they exist in the manifests).

So we could make tael an internal builder, then reland the CL here, or the PreCQ logic needs to be modified. The CL at https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1228265 takes the easy way out, not sure how critical it is for tael to be built as a public target.

Why are tael/tatle external builders, if they have internal overlays?

It looks like it was a workaround to https://bugs.chromium.org/p/chromium/issues/detail?id=712542

I think this is something that smbarber may need to own?

For now you either get PreCQ coverage and the board has to be set up like all the rest, or you get to be public. 

they don't have internal overlays

having the pre-cq configs for these boards include internal overlays is OK for now.  if we can do it only for the pre-cq configs and not the full, that'd be better.

As it happens, we are shutting down the external waterfall soon (due to BuildBot shutdown), so we should just move this to internal now. That solves the immediate problem. Stephen, do you actually need a public GS bucket for build artifacts?

The reason for the public bucket and builder is just that we didn't see any reason to make it private. See https://bugs.chromium.org/p/chromium/issues/detail?id=712542#c2

I don't need the public bucket. I was going to need access to the bucket for VM image testing, but I can add kokoro to the access list for the artifact bucket later.

What's the transition plan for the other external build configs, like amd64-generic-paladin? There's no internal bucket that I can find for that, so I have no idea where these artifacts should go.

the generic public bots (like amd64-generic & arm-generic) must stay public.  not in the sense that they have to be on a public waterfall (although that would be nice), but they must build public manifests & publish their artifacts for people to use.  so in that regard, i don't think the location of the waterfall matters.

keeping tael/tatl working on public manifests is prob the right thing since it doesn't use any internal stuff at all.

Internal and external artifacts all end up in the same bucket, but we apply different ACLs to the artifacts.

Don volunteered to hack up a solution to the PreCQ Launcher/Builders selection of the internal manifest.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/274a6922da390684d70ff9aeb321146f02c7df3b

commit 274a6922da390684d70ff9aeb321146f02c7df3b
Author: Don Garrett <dgarrett@google.com>
Date: Wed Sep 19 01:16:05 2018

chromeos_config: Make external PreCQ builds internal.

The PreCQ Launcher doesn't distinguish between internal and external
CLs when launching PreCQ build configs. This can cause external PreCQ
boards to fail to fail, because they can't apply an internal CL.

This change hacks all external PreCQ configs to use internal
checkouts, so that they can apply (and hopefully ignore) internal CLs.

It would be better to make the PreCQ launcher smarter, but this is a
quick way to allow tael/tatle to be added as PreCQ builders.

BUG= chromium:882965 
TEST=chromeos_config_unittest

Change-Id: Ibcc6c0199e10fafb60b90f25a8419e79924363ac
Reviewed-on: https://chromium-review.googlesource.com/1232555
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Mike Nichols <mikenichols@chromium.org>

[modify] https://crrev.com/274a6922da390684d70ff9aeb321146f02c7df3b/config/chromeos_config.py
[modify] https://crrev.com/274a6922da390684d70ff9aeb321146f02c7df3b/config/config_dump.json

Running two test builds here:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8934993745693074992

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8934993836048569664

They both passed. Calling this fixed.