This is happening intermittently on the Chrome PFQ builders as well and fouled up last night's uprev. It's killing dummy_PassServer.sanity_SERVER_JOB in HWTest [sanity] there, e.g. http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8941946317082646224.

Might be related to the recent upgrade at 853401.

( issue 853401 , just to make that auto-link)

This looks like it probably started a bit before 2018-07-03 22:43:00 PDT.

Possibly related:
https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1123026

Also observed this issue on moblabs, b/111153934.

This is a Test issue, not CI. Over to the Infra Deputy.

Reverted https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1123026
anyway.

Re #4 and #7, I don't think it's related to that CL because it hasn't been pushed to prod.

Re: #8, in a server-side test, the code is packaged and tested within the server-side container. It doesn't have to be pushed to prod.

Looking at the waterfall, it looks like every paladin that runs a server test was failing with this.

paladins are turning green. Down from 30+ red paladins to 14 now.

master paladin is green.

Re #9, one question, which branch the service side testing check out? In our production, the autotest server code is on branch 'prod' other than 'master'. It would be better to make server side testing also consistent with this.

-> pprabhu for more investigation as he is the CL owner.

Re #13: As far as I know, the server-side tests from the same revision as the tested device should be used (if a 'prod' server is trying to test a R69-xxxxx.0.0 device, R69-xxxxx.0.0 test code is used, not from 'prod' or 'master'):
https://docs.google.com/document/d/1oGuXg3gz0X1VzGIlEyycvF7uDnt8i7pflPalegpT8jE/edit#

https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1123026 landed in the CQ because it was tested only by moblab-pre-cq, and committed from the pre-cq.

This is the second instance of this. Autotest code is clearly not ready to be submitted after being tested only on moblab-pre-cq :(

I'm removing the land-from-pre-cq config for this project.

Root cause of failure: Before CL:1123026, the MySQLdb module was imported only when db.db() function was called. After the CL, this module is imported unconditionally.

MySQLdb was never importable inside the SSP container. But autoserv imports server.cros.dynamic_suite.suite, which imports tko.db. But db.db() was never called in the context of the test.

This is what we get for being too smart with our imports.

The fix is to factor out the functions being called from server.cros.dynamic_suite.suite module, which are really unrelated to suites entirely...

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/25288075f399ac91d72ebb752043e310b7e9bf53

commit 25288075f399ac91d72ebb752043e310b7e9bf53
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Wed Jul 11 18:26:47 2018

autotest: Do not submit autotest infrastructure code from pre-cq.

Moblab-pre-cq has proved insufficient for vetting CLs to the existing
autotest infrastructure code. In particular, moblab-vm does not run any
ssp tests, which creates a large hole in coverage.

lucifer and skylab related directories continue to be commited by pre-cq
directly.

BUG= chromium:860166 
TEST=None

Change-Id: Ia26ab526487e1aa115e63c78293dfb6fe82e81a9
Reviewed-on: https://chromium-review.googlesource.com/1129378
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[delete] https://crrev.com/ed74cc7a9cb43ac38a9e7bc8666105e37df909d7/tko/COMMIT-QUEUE.ini
[delete] https://crrev.com/ed74cc7a9cb43ac38a9e7bc8666105e37df909d7/scheduler/COMMIT-QUEUE.ini
[delete] https://crrev.com/ed74cc7a9cb43ac38a9e7bc8666105e37df909d7/database/COMMIT-QUEUE.ini
[delete] https://crrev.com/ed74cc7a9cb43ac38a9e7bc8666105e37df909d7/frontend/COMMIT-QUEUE.ini

Aviv ended up writing a functionally equivalent CL for this, so letting him merge his (with a hack to avoid the issue seen above).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/7f4071c3e10e819b7facfcc74691e3586fa9a7ff

commit 7f4071c3e10e819b7facfcc74691e3586fa9a7ff
Author: Aviv Keshet <akeshet@chromium.org>
Date: Tue Jul 17 09:26:17 2018

autotest: remove unneccesary indirection in tko db handle creation

We only support a mysql based tko db, yet we have indirection to fork
between different db backends that adds confusion and complexity.

 - delete db_mysql.py and move its connect method back to tko/db.py
 - delete _get_error_class frok tko/db.py

BUG=chromium:852083,  chromium:860166 
TEST=grep for db_mysql and get_error_class reveals no other callers;
examination of calls to |driver| at import time to ensure that they are
satisfied by the fake if necessary

Change-Id: I1c2d0e0daae6fc6b23c4e6c084c37c6602e777ed
Reviewed-on: https://chromium-review.googlesource.com/1135814
Commit-Ready: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/7f4071c3e10e819b7facfcc74691e3586fa9a7ff/tko/db.py
[delete] https://crrev.com/1929fb3f1e58320857c6495d4927e3ffbca91ed2/tko/db_mysql.py