The tast-local-tests-cros package should be installing the missing file. It installs it under /usr/libexec, but as I understand it, the file is diverted to /usr/local/libexec by virtue of this being a test-specific package.

One potential cause of the failures could be tast-local-tests-cros not being installed, but I see it in the log file at https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8927772215955200080/+/steps/BuildPackages/0/stdout (from the link-paladin build at http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8927772215955200080), so that seems unlikely.

Alternately, maybe the file is getting installed to /usr/libexec instead of /usr/local/libexec for some reason.

Or maybe /usr/local is getting messed up or wiped for some other reason.

I don't think there have been any changes related to this on Tast's side for a while. Does anyone on the infra side know of any changes that could've caused this?

From http://stainless/browse/chromeos-autotest-results/264753286-chromeos-test/ :

12/07 13:19:16.933 INFO |        server_job:0217| START	tast	tast	timestamp=1544217556	localtime=Dec 07 13:19:16	
12/07 13:19:17.042 INFO |             utils:0287| [stdout] tast version tast-cmd-0.0.1-r219
12/07 13:19:17.045 INFO |              tast:0263| Getting list of tests that will be run
12/07 13:19:17.046 INFO |              tast:0233| Running '/usr/local/tast/tast' -verbose=true -logtime=false list -build=false '-remotebundledir=/usr/local/tast/bundles/remote' '-remotedatadir=/usr/local/tast/data' '-remoterunner=/usr/local/tast/remote_test_runner' -json=true 'chromeos2-row8-rack6-host4:22' '(!disabled && !"group:*" && !informational && !"dep:android" && !"dep:chrome" && !"dep:chrome_login")'
12/07 13:19:17.340 ERROR|             utils:0287| [stderr] 2018/12/07 13:19:17 Connecting to chromeos2-row8-rack6-host4:22
12/07 13:19:17.341 ERROR|             utils:0287| [stderr] 2018/12/07 13:19:17 Failed to run tests: Process exited with status 3: no bundles matched by "/usr/local/libexec/tast/bundles/local/*"

This is pretty weird, because I think it implies that tast-local-test-runner was installed to /usr/local on the DUT (since I think it's what generates the "no bundles matched by" error here), but that tast-local-tests-cros wasn't.

Hmm. I just downloaded and extracted https://storage.cloud.google.com/chromeos-image-archive/cave-paladin/R73-11365.0.0-rc1/stateful.tgz, an artifact from the stale build at http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8927772276154627344. When I decompress it, it looks like the "cros" bundle file is in an extra "cros" subdirectory:

dev_image_new/libexec/tast/bundles/local/cros/cros

Looking in https://storage.cloud.google.com/chromeos-image-archive/cave-paladin/R73-11363.0.0-rc2/stateful.tgz from the earlier successful run at http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8927793095171517168, the file is at the expected location, without the extra subdir:

dev_image_new/libexec/tast/bundles/local/cros

Aha, I'm able to repro the problem locally after applying https://crrev.com/c/1359026 (which was in that CQ run). The system works.

(Feel free to WontFix this after fixing the change.)

Wait, why did that change make it through the pre-CQ? The TastVMTest stage failed in the betty-pre-cq run at http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8928000219296528576, but then we got "The Pre-Commit Queue has successfully verified your change."

Prathmesh, do you know why that was reported as a success? The builder doesn't appear to be experimental. Should I file a separate bug about that?

Re #6: Hmm. Is there anything special that needs to be done to make the pre-CQ punt changes in response to failures in the TastVMTest stage? I'd assumed that validation would fail if any of the triggered *-pre-cq builds failed, but perhaps we only look at specific stages in those builds.

Thank you and sorry for the breakage.  I had tested other builds that imported the cros-go.eclass, but not this one.

The problem was here:

               installdir="${override%/}"

should have been

               installdir="${override%/*}"

#8 also, had I tested that package by itself, it would have passed.  I gather it was dependent packages that failed.

A unit test would have caught this.  There is an eclass/tests subdirectory.  Do we run those tests?

CL is resubmitted with fix.

Marking as "fixed" rather than "won't fix" because I had to debug and fix the CL and in any case it makes me feel happier.

I didn't know that eclass tests were a thing. I won't argue against increased unit test coverage, but the pre-CQ already runs Tast tests and should've caught this. I filed  issue 913159  to figure out why it didn't.