moblab-generic-vm-paladin is failing due to this error several times in a row.
https://cros-goldeneye.corp.google.com/chromeos/legoland/builderHistory?buildConfig=moblab-generic-vm-paladin&buildBranch=master
Let me raise the priority.

This is failing consistently.  Please treat as P0.

Looks like a test failure to me?

It looks like a bug in the test.

If this isn't the right component and/or you're not the right assignee, please help me find the right owner for this.

Already assigned to akeshet :)

The stacktrace looks like it is masking some other root failure.

-> haddowk any insight on moblab failure?

Nothing recently has been checked in for moblab ( that I am aware of )

Autotest is creating another host type for the test, not moblab_host - I need to trace back to how that decision is make in the code.   If something in lsb-release has changed it might cause that.

My guess is the the VM is not booting correctly, I see this log

12/13 15:07:15.686 INFO | test_runner_utils:0259| autoserv| Failed to verify connectivity to host. Skipping host auto detection logic.

https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/server/hosts/factory.py?rcl=43ff6a9e662219e91c474c279cb931971ff7f978&l=219

So it is just assigning it a default host type rather than a moblab host

So I can bring up the VM locally - moblab seems to be booting correctly, there are some issues with provisioning the betty DUT but that is not what this error is about.

Honestly I think this is a generic VM issue - it just happens that the default host works for the other VM's not moblab, I have not checked for the SSH error in other VM runs though.

At this point I see it as an infra problem :

12/13 15:07:15.686 INFO | test_runner_utils:0259| autoserv| Failed to verify connectivity to host. Skipping host auto detection 

However I will keep working on the DUT provisioning inside the moblab VM

Hi, anything could be done from infra side?  How can we find why "VM is not booting correctly"? 

Just to be clear the VM is booting correctly when copied onto my workstation.

Since it works locally ( exact same VM image ) it has to be something environmental, like the VM takes longer to boot in the lab infrastructure, any suggestion on how to test / debug that ?

Re: #11-#12 - It looks like that INFO line comes from a subprocess which originally outputs it as a WARNING:

https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/server/hosts/factory.py?rcl=5a0ec3144726f22695024865b42e209726067396&l=219-220

Since host auto detection seems to be essential, how about we make the test hard fail at that point, to not obfuscate the problem?  Note the TODO from pprabhu@ to make exactly that change.

If we stop trying to continue running after such a failure, the error.AutoservRunError traceback might provide some useful information for troubleshooting, or at least get us closer to finding the original source of the error.

Can anyone on this bug craft and test a CL for that?



I do see the comment saying "We need to make sure stopping here does not block repair flows."  I'm not very familiar with lab DUT repair flows.  Is anyone here familiar enough with the lab DUT repair process to comment on whether that would really be a problem?

If that will be a problem, perhaps we can stop relying on auto-detection in this case by passing in the correct host class?  (For that matter, how about doing that always?  Are there any scenarios where we don't already know what kind of DUT / VM we are running tests against?)

Taking myself off this bug.  For the moment the test is back to green, leaving open in case someone want to take up the suggestions in #14 which I think is very reasonable.

I do not think at this point there is anything moblab specific breaking, it is just that the default value in host detection works for all other VM's but not moblab.