this looks like a general/external problem.  the gs://android-build-chromeos/builds/git_mnc-dr-arc-dev-linux-static_sdk_tools/ directory was straight up deleted which means any builds relying on those paths are now broken.

Nicolas: can you find out what's going on with that bucket ?  and communicate with whoever is managing it to not delete artifacts from it ?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/02d31fde79f3e36d1f7c356e3d4f31349ba46a10

commit 02d31fde79f3e36d1f7c356e3d4f31349ba46a10
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat Jul 14 17:04:41 2018

do not uprev android on firmware branches

We don't care about Android when building firmware.

BUG=chromium:863649
TEST=None

Change-Id: I7670192707c1a65d9a0ec4870d59eed337ad8836
Reviewed-on: https://chromium-review.googlesource.com/1137379
Reviewed-by: Jason Clinton <jclinton@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/02d31fde79f3e36d1f7c356e3d4f31349ba46a10/cbuildbot/config_dump.json
[modify] https://crrev.com/02d31fde79f3e36d1f7c356e3d4f31349ba46a10/cbuildbot/chromeos_config.py

the bot is green since we turned off Android stuff, but the underlying issue of Android artifacts going missing still needs addressing

It looks like the branch is no longer built: https://buganizer.corp.google.com/issues/65130006

Supposedly we had moved all boards to NYC and didn't care that much about MNC anymore, that's probably why the artifacts were removed at some point.
Assigning to Bernie, he might know more.

the artifacts cannot be removed once a builder relies on it.  doing so breaks historical manifests/branches.

who owns this bucket so we can get them to stop deleting things ?

i've at least turned on versioning on the bucket so any further deletions will be archived automatically ...

We certainly turned down the builders on the Android side, but deletion of GS artifacts is new to me. I suspect this is just how Android handles old binaries, the bucket in question is an Android owned bucket, they may not be interested in keeping around our unused artifacts forever (which is why we copy our production artifacts into our own bucket which we don't delete).

The place this is breaking is not the build itself, Chrome OS has all the stuff it needs to build still, rather this is the logic in the builder that checks the Android side to see if a newer version is available for uprev.

The fix in comment 2 seems perfectly reasonable for this case, we will not ever be uprevving on this branch.

A more comprehensive long term fix might be to catch the exception and go ahead anyway with a warning saying that we did not find a new Android version to uprev.

I think this falls under CI jurisdiction as part of the uprev logic (as opposed to an actual failure to build an OS image), if we want to implement something like that? I defer to Jason if we want to do something here or close as WAI.

i assumed gs://android-build-chromeos was own by us.  if not, then OK.

if our builds have their own copies in our own buckets, and that's only what our bots use to build, then that sounds fine too.

i think we should add a unittest to make sure factory/firmware configs never have android uprev enabled, and double check that our factory branches don't have it enabled anymore now.

i don't think we want missing paths exceptions to be swallowed if uprev is enabled.  that's an easy way for real problems to get missed.

Don: can you handle the unittest part in ToT ?  you've been doing a lot of config work here recently.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/f3a0038005ebdf2f3a50b1a9ddf57978ef69caea

commit f3a0038005ebdf2f3a50b1a9ddf57978ef69caea
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat Jul 21 09:16:29 2018

do not uprev android on factory/firmware branches

We don't care about Android when building factory/firmware.

BUG=chromium:863649
TEST=None

Change-Id: I7670192707c1a65d9a0ec4870d59eed337ad8836
Reviewed-on: https://chromium-review.googlesource.com/1146044
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/f3a0038005ebdf2f3a50b1a9ddf57978ef69caea/cbuildbot/config_dump.json
[modify] https://crrev.com/f3a0038005ebdf2f3a50b1a9ddf57978ef69caea/cbuildbot/chromeos_config.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/f592970915f33997a99e7112626f97589afd683e

commit f592970915f33997a99e7112626f97589afd683e
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat Jul 21 09:16:52 2018

do not uprev android on factory/firmware branches

We don't care about Android when building factory/firmware.

BUG=chromium:863649
TEST=None

Change-Id: I7670192707c1a65d9a0ec4870d59eed337ad8836
Reviewed-on: https://chromium-review.googlesource.com/1146047
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/f592970915f33997a99e7112626f97589afd683e/cbuildbot/config_dump.json
[modify] https://crrev.com/f592970915f33997a99e7112626f97589afd683e/cbuildbot/chromeos_config.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/ed3ec1b4d97a4e436b213429529e2b748998bf96

commit ed3ec1b4d97a4e436b213429529e2b748998bf96
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat Jul 21 09:16:57 2018

do not uprev android on factory/firmware branches

We don't care about Android when building factory/firmware.

BUG=chromium:863649
TEST=None

Change-Id: I7670192707c1a65d9a0ec4870d59eed337ad8836
Reviewed-on: https://chromium-review.googlesource.com/1146046
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/ed3ec1b4d97a4e436b213429529e2b748998bf96/cbuildbot/config_dump.json
[modify] https://crrev.com/ed3ec1b4d97a4e436b213429529e2b748998bf96/cbuildbot/chromeos_config.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/8bfa0444ae2e53637a7c3082fdff55aa65cd27ed

commit 8bfa0444ae2e53637a7c3082fdff55aa65cd27ed
Author: Mike Frysinger <vapier@chromium.org>
Date: Sat Jul 21 09:17:02 2018

do not uprev android on factory/firmware branches

We don't care about Android when building factory/firmware.

BUG=chromium:863649
TEST=None

Change-Id: I7670192707c1a65d9a0ec4870d59eed337ad8836
Reviewed-on: https://chromium-review.googlesource.com/1146045
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/8bfa0444ae2e53637a7c3082fdff55aa65cd27ed/cbuildbot/config_dump.json
[modify] https://crrev.com/8bfa0444ae2e53637a7c3082fdff55aa65cd27ed/cbuildbot/chromeos_config.py

Right I believe gs://android-build-chromeos is Android owned, and we copy to our own bucket gs://chromeos-arc-images. 

I guess ignoring a missing folder could mask some problems, but I am not sure what, given this is only looking for 'is a new version available' and a negative answer to that is acceptable, no folder seems like it should be ok to infer a negative answer in the cases I can think of. The typical problem I could see is never having a new version available, but that could happen today and no one would notice. 

OTOH I agree that disabling it is the technically correct thing to do for these, if we never intend to uprev, why bother wasting cycles checking for an update that will never be there. Is there a clean way to decide we are a firmware/factory branch and disable this automatically (parsing the branch name seems hacky...)?

I guess getting rid of the preflight here entirely would also solve it.

Would prefer disabling android in the factory/firmware branches (which is definitely fine) to getting rid of the preflight entirely. 

It may be that including android disable when creating the branch is the right thing?

i'm thinking of typos or other underlying breakage that would go unnoticed.  you're right that, at some point, hopefully people would notice the issue (like ARC++ growing stale), but i'd rather have these failures be upfront rather than dragging along.  at least on the Chrome side, the more stale the Chrome uprev gets, the harder it is to get back in sync as we let changes on both sides fall out and possibly introduce more regressions.

lets have Don look at the unittest side for feasibility and see if we can report back a different solution.

disabling ARC++ entirely is a sep and much larger topic.  lets not fork this bug further.

gs://android-build-chromeos is owned the chromeos.bot project that owns most of our buckets. The android build system writes directly to it whenever it builds a new ARC container for us. Many more ARC containers are written out than we never use, so all files in it have a limited lifespan.

We copy files to a second bucket with no lifespan on uprev. This is supposed to preserve the artifacts we need to make all existing ebuilds valid, without paying for the stuff we don't need.

If I understand correctly, the PFQs on the branches failed because there were not only no new builds to ingest, but old builds weren't even available to consider?

In that case, I believe there is a bug in the PFQ logic. We should probably fix that bug in TOT, and turn off the PFQs on the branches that can never get a new container.

s/never/ever/

if the artifacts are auto-expiring, and our bots auto-copy them to another bucket for the long term, and it's only the android uprev stage that cares about this, and we don't want/need that on firmware/factory, then all those pieces sound fine.

that just leaves us with unittest coverage.  i think in ToT chromite, we can add a test that requires android_rev to be set to False for all factory/firmware configs ?

TOT has now knowledge of branch build configs. It can't even checkout the branch and parse the existing config because there is no compatibility.

We currently have goofy mechanism by which GE tells us that a given branch is a release branch, but it's release branch specific, and depends on the "config_updater" builder checking in an update json file to the branch.

If we run the tests on the branch, the branched chromite could assert things, but don't currently know if they are on a firmware/factory branch, except by parsing the branch name. Doing so is feasible, and probably not quite as evil as it feels, just because so much stuff is already parsing branch names.

i'm not suggesting the unittest checkout every branch and somehow parse/ingest the data there.  we keep firmware/factory configs in ToT so people can do some early testing before branching, and to serve as the basis for the branch when it happens.  having a unittest at the very least keep those configs sane should be a step in the right direction.

once it's in ToT, the unittests flow back into branches as we create new ones.  i don't think we need to spend time backporting any of that as we can do a one-off search (which i've done now) and just focus on new branches moving forward.

as for how we detect "is this config a factory config", i'm guessing you don't want to leverage display_label anymore ;), so we could assert based on sign_types=factory or factory_toolkit=True (as i don't think we set that to non-factory configs anymore).

Ah... yeah, testing the TOT configs should be easy. And both of those values would be trivial to check.

Here is one of many examples:
  https://cs.corp.google.com/chromeos_public/chromite/config/chromeos_config_unittest.py?rcl=5389d34c833dae162248c0c8ddb1b71d31851924&l=923