This looks like the failing line?

07:33:18: INFO: RunCommand: /bin/bash -c 'ssh -p 9222 -oConnectionAttempts=4 -oUserKnownHostsFile=/dev/null -oProtocol=2 -oConnectTimeout=30 -oServerAliveCountMax=3 -oStrictHostKeyChecking=no -oServerAliveInterval=10 -oNumberOfPasswordPrompts=0 -oIdentitiesOnly=yes -i /b/s/w/itQqh5Uf/ssh-tmp_NAHiU/testing_rsa chronos@localhost -- "cd /b/s/w/ir/out_amd64-generic/Release && ./tmp3qeY5k.sh"'
Warning: Permanently added '[localhost]:9222' (ED25519) to the list of known hosts.
Warning: Permanently added '[localhost]:9222' (ED25519) to the list of known hosts.
bash: line 0: cd: /b/s/w/ir/out_amd64-generic/Release: No such file or directory

Full log:
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8930963136090196112/+/steps/midi_unittests__with_patch_/0/stdout

Likely due to the arg path change.

The command that fails looks like the following w/o the change:
08:00:56: INFO: RunCommand: /bin/bash -c 'ssh -p 9222 -oConnectionAttempts=4 -oUserKnownHostsFile=/dev/null -oProtocol=2 -oConnectTimeout=30 -oServerAliveCountMax=3 -oStrictHostKeyChecking=no -oServerAliveInterval=10 -oNumberOfPasswordPrompts=0 -oIdentitiesOnly=yes -i /b/s/w/itKp5l24/ssh-tmpsGW09E/testing_rsa chronos@localhost -- "cd /usr/local/vm_test/out_amd64-generic/Release && ./tmpX6lY3q.sh"'

With the change, it looks like:
07:33:18: INFO: RunCommand: /bin/bash -c 'ssh -p 9222 -oConnectionAttempts=4 -oUserKnownHostsFile=/dev/null -oProtocol=2 -oConnectTimeout=30 -oServerAliveCountMax=3 -oStrictHostKeyChecking=no -oServerAliveInterval=10 -oNumberOfPasswordPrompts=0 -oIdentitiesOnly=yes -i /b/s/w/itQqh5Uf/ssh-tmp_NAHiU/testing_rsa chronos@localhost -- "cd /b/s/w/ir/out_amd64-generic/Release && ./tmp3qeY5k.sh"'



"cd /usr/local/vm_test/out_amd64-generic/Release" vs "cd /b/s/w/ir/out_amd64-generic/Release"

The 2nd arg is the absolute path on the host, and doesn't exist on the VM. cros_run_vm_test is likely doing something different with cwd now... not sure what though.

I'm having a hard time finding the source code for 'path', but it seems like the argparse type are functions which are implicitly run, ie, you can have it automatically create files for you.

So my guess is that 'path' is getting resolved on the host machine and not the VM?

Owner to achuith@ to revert if this sounds reasonable

From the docs at https://docs.python.org/3.4/library/argparse.html#type

>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('bar', type=argparse.FileType('w'))
>>> parser.parse_args(['out.txt'])
Namespace(bar=<_io.TextIOWrapper name='out.txt' encoding='UTF-8'>)

^ the snippet above shows the 'out.txt' file getting created as part of the 'parse_args' call

Yeah, looks like the problem is the "--cwd" arg. Since it's type='path' now, passing in a relative path (which we do on the chrome bots) makes it automatically translated to an abs path on the host, like jdufault pointed out above.

An easy fix for this is just to remove type='path' for the cwd arg I think.

Testing locally adding `type='path'` to the cwd argument specification causes this failure.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/3d2886849216faf07789db8590c707c24c4e5a9a

commit 3d2886849216faf07789db8590c707c24c4e5a9a
Author: Achuith Bhandarkar <achuith@chromium.org>
Date: Mon Nov 05 15:49:38 2018

cros_run_vm_test: cwd should not be of type 'path'

Passing in a relative path for cwd, which we do on the chrome bots,
makes it automatically translated to an abs path on the host, which
we don't want.

BUG= chromium:901376 
TEST=manual, bots.

Change-Id: I806c53f85a81fb0b12fe6cbebd00c890d8ef8b62
Reviewed-on: https://chromium-review.googlesource.com/c/1315774
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Tested-by: Achuith Bhandarkar <achuith@chromium.org>
Commit-Queue: Achuith Bhandarkar <achuith@chromium.org>
Trybot-Ready: Achuith Bhandarkar <achuith@chromium.org>

[modify] https://crrev.com/3d2886849216faf07789db8590c707c24c4e5a9a/scripts/cros_run_vm_test.py

Hmm, chromite roll still failing:
https://chromium-review.googlesource.com/c/chromium/src/+/1317169

I can't tell what's wrong from the logs :/

Passing to current gardener

Re #10: That roll didn't pick up your fix in #9. That latest roll attempt does, and it looks like it's passing:
https://chromium-review.googlesource.com/c/chromium/src/+/1318132

Should get committed ~shortly.

Roll succeeded. Thanks!