Moblab is not setting up correctly in to be able to run the HWtest since the devserver is not running.

10/02 22:09:57.678 WARNI|              test:0612| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/test.py", line 573, in _exec
    _cherry_pick_call(self.initialize, *args, **dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 721, in _cherry_pick_call
    return func(*p_args, **p_dargs)
  File "/usr/local/autotest/server/cros/moblab_test.py", line 35, in initialize
    self._host.verify_moblab_services()
  File "/usr/local/autotest/server/hosts/moblab_host.py", line 267, in verify_moblab_services
    % service)
AutoservError: Moblab service: moblab-devserver-init is not running.
10/02 22:09:57.678 DEBUG|              test:0617| Running cleanup for test.

From the logs in the machine dev server is not running because of the change to update_payload imports

CL https://chromium-review.googlesource.com/#/c/chromiumos/platform/dev-util/+/598434/

Is suspected

Can you confirm if this is or not your change that is causing this please,  I am considering a rollback as I need to get the moblab build back to green and I see no easy fix forward, but I am not very familiar with this code area.

This could be the cause of the problem, but not necessarily. There was a crbug.com/769538 that needed a push for the healthy devserver labs. The devservers need to have update_engine repo checked out, so it can find the update_payload. I think it might be because, that push was not done. Reverting only this patch might cause other problems like file duplications.

dgarrett@ can you confirm that that push has been necessary or not?

Just to be clear - the problem is that the devserver on moblab will not start, pushes to prod will not make any changes.

Moblab is the full lab running on a guado chromebox

Any other suggestions as to what might be the fix,  I have created a revert and running a trybot job right now to see if it makes the build green.

I'm not familiar with moblab. How the devserver on moblab is installed? Is it installed through ebuild/etc or the devserver starts from the source?

The image is build via the ebuilds but there is a upstart script that starts the devserver on boot

How the server is started
https://cs.corp.google.com/chromeos_public/src/overlays/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-devserver-init.conf?q=package:chromeos_.*+moblab-dev&l=1

This CL (614389), changes the use of devserver to update_payload ebuild instead of the update_payload in the devserver itself. Can you check if this CL is present in the moblab build?

I believe so - the tests are running at ToT moblab depends on devserver ebuild here
https://cs.corp.google.com/chromeos_public/src/overlays/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.5.ebuild?rcl=0cbeb1f9a193eaf6f54f41effa1c93ef97f12af0&l=35

The thing that I don't understand is that this CLs were pushed a few days ago and had no problem on other builds. I'm wondering if there is some different configuration on the moblab build that causing this effect. 

On the moblab the directory mentioned on that CL is present

moblab@localhost ~ $ cd /build/guado_moblab/usr/lib64/python2.7/site-packages/update_payload
moblab@localhost /build/guado_moblab/usr/lib64/python2.7/site-packages/update_payload $ ls
__init__.py  applier.py  block_tracer.py  checker.py  common.py  error.py  format_utils.py  histogram.py  payload.py  test_utils.py  update-payload-key.pub.pem  update_metadata_pb2.py
moblab@localhost /build/guado_moblab/usr/lib64/python2.7/site-packages/update_payload $ 

If the update_payload is present, then devserver should have no problem finding it. Maybe it is using a different python version and update_payload is not installed there?

Moblab build has been down for some other reasons (rootfs was full), so as I slowly fix the issues I come across new issues that have been checked in in the mean time.

Default python that is installed

moblab@localhost / $ python -V
Python 2.7.10
moblab@localhost / $ which python
/usr/bin/python

So when I print out the sys.path before the import I get

['/usr/lib64/python2.7/site-packages/chromite/third_party', '/usr/lib64/python2.7/site-packages/chromite/third_party/dpkt', '/usr/lib64/python2.7/site-packages/chromite/third_party/google', '/usr/lib64/python2.7/site-packages/chromite/third_party/pyelftools', '/', '/usr/lib/devserver', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/PIL', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/local/lib64/python2.7/site-packages', '/usr/local/telemetry/src/third_party/catapult/telemetry']

So /build/guado_moblab/usr/lib64/python2.7/site-packages/update_payload is not on the include list - any ideas on what should be putting it on sys.path ?

I see the /usr/lib64/python2.7/site-packages is in the list, and we are importing update_payload which is a directory in site-packages. Isn't that should be enough?

There shouldn't be anything installed on the moblab under "/build/guado_moblab/"

That's a directory for the chroot, everything after that is relative to the root on the final Chrome OS image.

Like "/build/reef/<root of actually reef image>"

Whatever is installing update_payload is putting it in the wrong place.

update_payload used to be  installed in site-packages/dev/host/lib/update_payload, we only changed its location to site-packages/update_payload and modified imports thereafter. 

So probably the changes to the ebuilds in https://chromium-review.googlesource.com/#/c/chromiumos/overlays/chromiumos-overlay/+/614389/ are the problem

whats the difference between inherit python vs inherit python-r1 ?

I'm not totally sure about the difference, Maybe vapier@ can help?

If you want to revert, you have to revert 3 CLs. Please don't do that until we find a solution specific to the moblab.

FYI, moblab went back to red again!

I have not submitted the reverts I am just trying to get a trybot to run to prove the reverts will do what we want, but so far no luck.

i don't really know how moblab comes together.  does the package that installs the python libs need to be added to some moblab dep packages ?

I do not think that there is a missing dep, the issue seems to be that the ebuild file installs the package in the incorrect place on moblab.

The update_payload package is installed in /build/guado_moblab/usr/lib64/python2.7/site-packages/ on moblab which is incorrect.

It would appear that in this file

https://chromium-review.googlesource.com/#/c/chromiumos/overlays/chromiumos-overlay/+/614389/14/chromeos-base/update_payload/update_payload-9999.ebuild

$(python_get_sitedir) is not returning the correct thing when building moblab.

you've lost me.  what is wrong with the path you cited ?  that looks like the right one to me.

I also don't think the path is wrong! It seems the path is right, the update_payload dir is in the path (#12), but the devserver.py cannot find it. We have to figure out why!

Ok - just going by what sbasi said in #18 - and the fact that the only package to be installed in 

/build/guado_moblab/usr/lib64/python2.7/site-packages/

is update_payload

Other packages are installed in /usr/lib64/python2.7/site-packages

Happy to be wrong, just want moblab to be back to green, no great preference on where things are installed on the device.

I'm sorry, I didn't pay attention to the prefix: /build/guado_moblab. Then that is the problem.

ok, so the /build path is being used even on the DUT ... i thought you were quoting the path as seen in the sdk.  but in that case, it's actually /build/xxx/build/xxx/usr/lib64/....

Keith where are the files located in your chroot?

looks like a bug in the core python logic.  i'll upload a hack to get the bots green while we figure out the right answer.

update: Workaround cl by vapier@

https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/698947

Mikes fix solves the issue but is not submitted ( I had a green run when the CL was in the CQ ) when it gets submitted we should be back to green.

Is it safe to just chump the CL ?

it should be safe, so i'll chump it

Thanks appreciate the help.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/fd4d1832dd528f7883caaac5b25be7d3b0c4b9dc

commit fd4d1832dd528f7883caaac5b25be7d3b0c4b9dc
Author: Mike Frysinger <vapier@chromium.org>
Date: Wed Oct 04 15:36:38 2017

update_payload: hack around python bug

There seems to be a bug in the core python logic where we end up
installing under the SYSROOT path when it should be anchored to
the root dir.  Hack that to get moblab green while we figure out
the best way to address this in the python code.

BUG= chromium:771085 
TEST=`emerge-guado_moblab update_payload` installs into the right path

Change-Id: Ibc630ce2cb34bb8da82c999fdd4492236f054e8f
Reviewed-on: https://chromium-review.googlesource.com/698947
Reviewed-by: Amin Hassani <ahassani@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/fd4d1832dd528f7883caaac5b25be7d3b0c4b9dc/chromeos-base/update_payload/update_payload-9999.ebuild

Is this believed to be fixed now?

Yes the latest CQ run for moblab should come back green ( the HW tests have all passed ) when it does I will mark as fixed and take moblab off experimental

Thanks!

Is it safe to say that we shouldn't push a new moblab image until that's true?

Not sure what you mean by push, I do the moblab pushes and we push from the stable channel so the current problems are just with the CQ.  Sorry if I misunderstand.  There technically is a beta and dev channel for moblab but they are not used.

I'm sorry, I think I just confused moblab with labstation. Sigh.