I think it's a bug that these artifacts would ever be missing
from a beta build.  We might want to see to why that's happening.

The build in question failed.

https://uberchromegw.corp.google.com/i/chromeos_release/builders/gandof-release%20release-R55-8872.B/builds/51

However, hwtest failures are the only reported failures. They might have been retried, or just ignored. I'm not sure.

There is a conceptual disconnect. When writing code, we generally assume that the output of a failed build is 'unsafe' and to be ignored. TPMs (quite reasonably) look at the failures and ignore the ones they think are safe to ignore.

This suggests that we need an explicit way to report "failed and invalid" versus "failed but recoverable", and make sure our code always keeps going with producing and uploading artifacts when it's "recoverable".

OK, then, it feels like the main fix here is to fix our
release processes so that broken builds like this can't
be released on any channel.  I don't think that's my bug
to fix.

Agreed. Longer term, the Centaur work should address this by breaking the build into fully distinct chunks.

The missing au_control.tar.bz2 are happening on a few platforms on M59. Below is an examples:

The error looks like

DownloaderException: Could not find au_control.tar.bz2 in Google Storage at gs://chromeos-image-archive/chell-release/R59-9413.0.0

Here are the list of platforms

Chell:http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=110066910
quawks: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=110066911
reks: http://cautotest.corp.google.com/afe/#tab_id=view_job&object_id=110066914

This is preventing build being push for M59 dev channel. 

+pprabhu current sheriff 

Re #6: That problem is unrelated to this bug.

Looking at just the Chell case:
It looks like the test was kicked off manually by someone.

The build that generated the artifacts being used was: https://uberchromegw.corp.google.com/i/chromeos/builders/chell-release/builds/964
This build had failed for multiple reasons.


As the failing (manually kicked) test claims the artifacts uploaded do not include au_control.tar.bz2 -- https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/chell-release/R59-9413.0.0

Compare this to the preceding build which does have full artifacts: https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/chell-release/R59-9412.0.0

-------------------
So, if a build failed, it is not completely surprising that some artifacts are missing.

One main claim that the au_control.tar.bz2 artifact should be generated whether or not HWTesting in all its forms succeeds (which is what had failed in that build), but that is unrelated to stable version builds missing certain artifacts (which this bug is about).

A bit more detail about this specific case: That tarball is created and archived by the AUTestStage. We didn't even run that stage in the failing build because HWTests stage failed.
As such, it is not possible to run autests later for that build.

Please file a separate FR if this behaviour is not what we would like. Returning this bug to its old state.

We shouldn't be bumping the 'stable-version' to a failed build version.

> We shouldn't be bumping the 'stable-version' to a failed build version.

Agreed.  However, in this case, the build in question was pushed to
the beta channel.  So, the statement above quickly degenerates to
"We shouldn't trust builds merely because they've been released on
Beta channel."  Down that path lies madness.