The question is how to get vpython on a chromeos test image. Someone on chromium-os-dev might know?

The standard mechanism would be to fetch depot_tools (https://chromium.googlesource.com/chromium/tools/depot_tools/+/master) then execute the vpython shell script in there which does all the network magic to download the actual vpython binary the first time you run it. I just tried all that myself in a VM and got pretty far, but got stuck on a vpython bug. Still not sure if this is the best route to go on, but I filed bug 843291 for that, cause it'd be generally useful.

 +vadimsh@
vpython is actually deployed by CIPD. depot_tools is used to deploy CIPD client && pin vpython version to **developers**. On bots, depot_tools isn't used.

Thus, a more fundamental question is does your VM have a CIPD client? If not, I think it's time to deploy it there.

For Swarming tasks, vpython package is a part of the task definition.

E.g. here's how Chromium isolated tests get it: https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/swarming/api.py?q=swarming/api.py&sq=package:chromium&dr&l=34

How do tests in cros VM initialize themselves (where do they get binaries from, etc)? Would regular linux-amd64 go binaries work on cros VM? Is it possible maybe just to mount a directory with binaries from the host machine into the VM, so cros VM environment inherits all task environment binaries?

The VM gets initialized before and torn-down after a swarming task. The swarming bot isn't running in the VM, and the VM's arch isn't necessarily going to match the host's arch. Hence why we can't just use the cipd/vpython the task inherits. That said, the VMs we're currently using do match the linux-amd64 distro, and those versions of various cipd packages seem to work fine on the VM (minus the vpython bug I mentioned above). But we should still look for a solution that would work even if the VM is on a different arch.

How do VMs get binaries under test? Can we use same mechanism to deploy files from CIDP, e.g. use host's CIPD client (and cipd cache) to fetch CIPD packages for the VM (targeting VM's arch, if necessary), and then just transfer them to the VM.

That way we avoid another place where we need to bootstrap CIPD client, there are too many already.

That way we can setup CIPD client and vpython inside the VM. Then VM be able to fetch more stuff using them, if it wants too (e.g. vpython wheels). We'll need a way to preserve CIPD cache on VM between tests though.

> How do VMs get binaries under test?

Reading GN's runtime deps file and scp'ing them over:
https://chromium.googlesource.com/chromium/src/+/master/tools/gn/docs/reference.md#write_runtime_deps

CIPD/vpython doesn't fit it in at all with that, and I'm reluctant to make our data-deps mechanism/pre-test setup any more bespoke. But I may have to. Let me sit on this for a bit and think it through. Whatever we come up with should also resolve bug 814068, which is slightly different.

> we avoid another place where we need to bootstrap CIPD client, there are too many already.

It follows that the number of ways we bootstrap CIPD is proportional to the number of different places we run CIPD, no? But I agree that we should try to avoid that if possible. I remember the trouble Robbie had when trying to bump the umpteen different depot_tools pins that live in cros-land during some vpython outage.

this sounds like something the test mechanisms would resolve on the fly rather than trying to bake depot_tools/etc... into the test image, so i'd defer to Prathmesh/Aviv

I think a reasonable solution to this is to attach a cipd'ed vpython to our swarming tasks the same way we add logdog and friends today:
https://chromium.googlesource.com/chromium/src/+/f02506d60db6768f8924d9659b58f664658cdcb6/tools/mb/mb.py#382

Only we wouldn't use "${platform}", but instead hard-code the arch of the board. This would prob need to go somewhere in the bot's test-spec definition:
https://chromium.googlesource.com/chromium/src/+/f02506d60db6768f8924d9659b58f664658cdcb6/testing/buildbot/waterfalls.pyl#532 (those files prob don't support arbitrary cipd packages, but I can add that)

Then I'd have to add the location of that package to the list of files pushed to the VM. (It'll make that logic a bit more hairy, but whatcanyado...) Then at test runtime, I'll have to prepend that target location of vpython onto the VM's PATH. I think this is all do-able, but I might run into trouble with either scp/rsync screwing with the symlinks that cipd loves to use or bash/ssh prematurely eval'ing the PATH env var we're trying to modify.

Current design won't be persisting the cache of wheels vpython will fetch on the VM, but I don't think that's much of an issue. The test binary itself is 200+ MB, what's a few MB of python wheels to that?

How do we handle the python issue for net_unittests on fuchsia/android/ios ? Shouldn't they have the same problem?

fuchsia: Doesn't seem to run net_unittests? https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.ci/Fuchsia%20x64/9259

android: Runs the server on the host and uses adb to port-forward from host to device: https://codesearch.chromium.org/chromium/src/build/android/pylib/local/local_test_server_spawner.py

ios: not a clue; I'd have to image the whole iOS/WebKit thing would make testing of net libs a wholly different story. (They even have their own special net_unitests: https://chromium.googlesource.com/chromium/src/+/master/ios/net/BUILD.gn#100) But I could be totally wrong there...

Looks like I was wrong wrt fuchsia in #11. I found https://ci.chromium.org/buildbot/chromium.fyi/Fuchsia/ which indeed does run net_unittests, a product of  bug 731302 .

Looking through the code, they do run the server on the host and communicate to it from the fuchsia VM (presumably via an ssh tunnel). Given that there's two different approaches here for cros, lemme break this off to a net-dev@ thread and see what the folks over there think.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8f2d48dcff54fd25a4e653534fd7007a53b97563

commit 8f2d48dcff54fd25a4e653534fd7007a53b97563
Author: Ben Pastene <bpastene@chromium.org>
Date: Fri Jun 01 17:59:16 2018

Add option to cros test runner to deploy vpython to VM.

This will:
- copy a dir containing vpython over onto the VM
- put the dest dir onto the VM's PATH env var
- initialize the vpython cache prior to running the test

And utilize this option on the fyi bot by adding a vpython cipd
package to net_unittests.

Bug:  843279 
Change-Id: I19d8c68e048aed03e6424119d08a3d13ef490def
Reviewed-on: https://chromium-review.googlesource.com/1063043
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#563717}
[modify] https://crrev.com/8f2d48dcff54fd25a4e653534fd7007a53b97563/build/chromeos/run_vm_test.py
[modify] https://crrev.com/8f2d48dcff54fd25a4e653534fd7007a53b97563/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/8f2d48dcff54fd25a4e653534fd7007a53b97563/testing/buildbot/test_suites.pyl

With #13, vpython is now available in the VM. The suite that depended on it (net_unittests) went from ~200 test failures:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests/2689
To only 4:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel-vm-tests/2690

And I don't think those remaining 4 are python related. Going to close this out.