File "/home/chromeos-test/chromiumos/chromite/lib/workqueue/throttle.py", line 11, in <module>
    from chromite.lib.workqueue import service
  File "/home/chromeos-test/chromiumos/chromite/lib/workqueue/service.py", line 24, in <module>
    from chromite.lib import metrics

That's my new code, so probably my fault.

I've seen this problem before. This sounds like you're (partially) importing the system protobuf library instead of using the one bundled with chromite.

chromite now has some hackery to deal with this. See  issue 674760 .

This should have "just worked" for you because chromite's __init__ patches up the path for you.

Also, looks like we've already fixed some flavor of this problem for devserver: issue 708890

Right, that solution involved switching the script to use virtualenv. Is their an easier quick fix involving changing the import order?

Did you introduce a new entry point? Is the throttling service itself chocking in this case?

Import order is not the problem here. There is a hacky fix (see chromite issue I pointed above), but given that devserver is already running off of virtualenv, best (and even easiest) way forward seems like virtual_env.

That is also the blessed way forward :)

> Did you introduce a new entry point? Is the throttling service itself chocking in this case?

More or less, yes.  The new code adds a new import path that is
being imported from code in src/platform/dev, rather than imported
from pure chromite code.  I believe that that's why the existing
fixes in chromite didn't work.

The fix should be to separate the workqueue client and server code
into their own separate modules; the failing import path only needs
the client code, and the client code doesn't need ts_mon.

The fix is also easy.  The only problem I've got is I don't know how
to reproduce the failure locally in order to prove out the fix.  In
principle, I think what's required is to install the protobuf package
on the local host.  However, I've hit a snag, in that the "local host"
has to be a chrome OS build chroot, and the chroot already has a
different version of the protobuf package - one which doesn't exhibit
the failure.

I've dug a bit more.
  * The devservers install protobuf version 2.5.0
  * The chroot installs protobuf version 2.6.1
  * chromite installs protobuf version 3.0.0

As best I can tell, 2.5.0 and 2.6.1 are both similar, so the chroot
_ought_ to be able to exhibit the failure; I just have to figure out
how to persuade the python interpreter of that...

OK.  In the process of digging around, I learned a bit more about
how to reproduce the problem on a devserver, and how to avoid it.

If you go to the source repo, under src/platform/dev, and start
an interactive python interpreter, then the following will reproduce
the failure:
>>> import devserver
>>> from chromite.lib import metrics

The following avoids the failure:
>>> import setup_chromite
>>> import devserver
>>> from chromite.lib import metrics

It happens that the 'devserver' module code already includes
'import setup_chromite', but it happens too late in the file to
avoid the problem.  Simply moving that magic import earlier would
seem to be a different fix.

I've also learned enough about the problem to be satisfied that my
more involved fix to change the workqueue code is also adequate.

This CL will fix the problem from the devserver end:
    https://chromium-review.googlesource.com/#/c/chromiumos/platform/dev-util/+/621670/

This has the added advantage that we should be able to add
metrics to either the devserver or provisioning code, should
the time come.

 Issue 758329  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/dev-util/+/df35c32a5d297385b51ad1b0f96cb9ca3f75e3d2

commit df35c32a5d297385b51ad1b0f96cb9ca3f75e3d2
Author: Richard Barnette <jrbarnette@chromium.org>
Date: Wed Aug 23 19:19:09 2017

Make sure chromite is properly imported at devserver startup.

At startup, before importing any modules from chromite, it's
necessary to import the 'chromite' package proper, in order to force
chromite/__init__.py to run.  Because Python.

This adjusts the import order in devserver.py to honor the
requirement, and adds a "here be dragons" comment for the unwary.

BUG= chromium:757003 
TEST=start the devserver, see it still work

Change-Id: I5f1d00c76a71fb985db4e1b11a65ed6461b1d149
Reviewed-on: https://chromium-review.googlesource.com/621670
Commit-Ready: Richard Barnette <jrbarnette@chromium.org>
Tested-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Paul Hobbs <phobbs@google.com>
Reviewed-by: Xixuan Wu <xixuan@chromium.org>

[modify] https://crrev.com/df35c32a5d297385b51ad1b0f96cb9ca3f75e3d2/devserver.py

We're ready to try pushing to devserver again.

Passing to the secondary deputy for action.

Actually, the push to devserver is a different bug.  This bug is fixed.