It doesn't seems like a software issue to me? Can you elaborate more about this?

On a VM the boot script is starting before the USB device is being mounted.

I do not see this on physical devices, I know very little about how VM's treat external USB disks, I see the disk being created here: https://cs.corp.google.com/chromeos_public/chromite/lib/moblab_vm.py?rcl=051b1952ca4b29bde5f5b1f07ce0e1fff20d9fce&l=496

But I am not sure how exactly a VM "mounts" that disk and if it behaves in exactly the same way as a real device, the boot script waits for cros_disks to be started, if the VM mounting is not using the same method we might need some other signal to know a VM has completed all external device mounting.

The udev script that handles the mounting of the file system could emit a "MOBLAB-MOUNTED" event to upstart somehow, no?

Or just entirely move the script from upstart to udev control scripts? does this have to use upstart at all?

Re #3, the disk is provided on the qemu commandline. This _should_ boot the VM such that disk is present from the beginning. 

Also, the failed build in the link in OP has a different failure reason: 

/tmp/cbuildbotYqCXm8/results/results-1-moblab_DummyServerNoSspSuite/moblab_RunSuite   03/22 11:23:09.501 ERROR|             utils:0282| [stderr] @@@STEP_LINK@[Test-Logs]: provision: FAIL: Failed to update 192.168.231.100 to build R67-10509.0.0-b2404993; found build 10509.0.2018_03_22_0824 instead@http://localhost/tko/retrieve_logs.cgi?job=/results/2-moblab/@@@


That is the same as  issue 824956 

Did you include the wrong link?

No - I was aware that the failure was because of the provision issue, I found this as part of the investigation and filed it since it happens long before the provision starts and some parts of the init script will not be run in this condition and so things are not being tested / perhaps will not work.

I have not found the root cause of the failure.

Another build link
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/pre_cq/builds/91251

So I might have got a bit confused it seem this log is part of the provisioning of the sub dut - are we putting moblab on the sub dut as well ?  I thought it was a betty image we were using.  Sorry if this is just my mix up.

https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/moblab-generic-vm-pre-cq/R67-10508.0.0-b2403238/moblab_vm_test_results/results-1-moblab_DummyServerNoSspSuite/sysinfo/mnt/moblab/results/hosts/192.168.231.100/2-provision/sysinfo/var/spool/crash/

I think this is fixed - I put some logic in to delay before trying to detect the external usb

Also I made some changes where the moblab will still correctly boot if the readlink fails, but assume no external disk