kenobi@, can you confirm if this is still happening on TOT?

Yes, this is still happening on ToT (at least, for me).  Are you not able to repro it?

dshi@, ideas on why the container test failures?

git bisect shows the first bad CL:

0248035d3d970e9d32334610aeba11f853390858 is the first bad commit
commit 0248035d3d970e9d32334610aeba11f853390858
Author: Allen Li <ayatane@chromium.org>
Date:   Mon Jul 17 12:38:26 2017 -0700

    [autotest] Remove puppylab code
    
    grep returns no results
    
    BUG=None
    TEST=Run local dummy test
    
    Change-Id: I5fc78f750e0eaaedd5a06ddb1d730c587520b25f
    Reviewed-on: https://chromium-review.googlesource.com/576875
    Commit-Ready: Allen Li <ayatane@chromium.org>
    Tested-by: Allen Li <ayatane@chromium.org>
    Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

:100644 100644 b40f5c738c8266cf35a2a039c827fb950dd0da8a 09b824999586f82aa040a6315a3a29420b03fb49 M	.gitignore
:040000 040000 b4ed10e24e1c6c55aedc9b0f524448908d953c69 02ab8b128d3056eced34b1b9b580cc663801ca6c M	client
:040000 040000 455bf335bf65fff0ce3f447de308567219e64c44 767368682ac41ccd85efe76c8598acc774ff7e00 M	frontend
:040000 000000 50857bc88ccf739e2c53b473e9be7b675fc82179 0000000000000000000000000000000000000000 D	puppylab
:040000 040000 f5c25e3e68213bb276cc7db9f072be3b01f03729 29a4fab5adad29c4e04641539f731b50025c0770 M	server
:040000 040000 d61b6f1cd69236cedead68748388c578d29fb089 264691fff8890d0eb40d7efd018a96ab1935820d M	site_utils

Odd, I get a different error

autotest_lib.client.common_lib.error.CmdError: Command <sudo lxc-attach -P /usr/local/autotest/containers/container_test_aE0Cn_ -n test_123_1502391368_137363 -- bash -c "python /usr/local/autotest/results/123-debug_user/create_fake_key.py"> failed, rc=1, Command returned non-zero exit status
* Command: 
    sudo lxc-attach -P /usr/local/autotest/containers/container_test_aE0Cn_
    -n test_123_1502391368_137363 -- bash -c "python
    /usr/local/autotest/results/123-debug_user/create_fake_key.py"
Exit status: 1
Duration: 0.453724145889

stderr:
Traceback (most recent call last):
  File "/usr/local/autotest/results/123-debug_user/create_fake_key.py", line 3, in <module>
    import rsa
ImportError: No module named rsa

You need to do some setup before you can actually run this test.

IIRC, minimally you have to run /usr/local/autotest/utils/build_externals.py.  This is assuming you already have your autotest source dir bind-mounted on /usr/local/autotest.

Note that running build_externals.py produces a bunch of python syntax errors.  I have been told it's safe to ignore those.

Ping.  Was the CL that caused this just a cleanup?  If so I propose to revert it until a proper cleanup can be done that doesn't break things.

If I don't hear back by EOD tomorrow I'll go ahead and revert, because this is blocking stuff I'm working on.

Thanks.

I'll take a look and post by EOD tomorrow.

A few things:

The error I'm running into is inside the container.  I can import rsa locally (i.e., I have already run build_externals.py).

You're getting a different image than me.

Mine is http://storage.googleapis.com/abci-ssp/autotest-containers/base.tar.xz
Yours is http://storage.googleapis.com/abci-ssp/autotest-containers/base_09.tar.xz

Oh derp, it's probably because we're using the Puppylab code to make fake results for autoserv in lxc tests.

Here's the fix: https://chromium-review.googlesource.com/c/616254

kenobi, can you cherry pick it and try it out?

Six Stages of Debugging

1. That can’t happen.
2. That doesn’t happen on my machine.
3. That shouldn’t happen.
4. Why does that happen?
5. Oh, I see.
6. How did that ever work?

https://chromium-review.googlesource.com/c/616254 doesn't appear to fix the issue.  I cherry-picked it and ran the test, and I'm still seeing the same error.

- regarding the import error - yes, the build_externals script is the correct thing to do.  Part of the container setup mounts certain directories (e.g. /usr/local/autotest/site-packages) into the container, and build_externals puts the right packages into those locations.

- regarding the wrong base image, I attached a zip containing a number of config files.  These should go in your /usr/local/autotest dir.  Hopefully that'll enable you to run the test with the correct config.

thanks.

Deleted: export.zip 7.8 KB

export.zip 7.8 KB Download

PS I did verify (this shouldn't be a surprise) that reverting all of 0248035d3, fixes the problem.  I guess the hard part is figuring out exactly what parts of that code are needed for the test to run.

lxc_functional_test does not use mock result in autoserv.
It actually run the code against a dut if it's provided.

For the base image, you should update the shadow config file to use the base image in lab (base_09).

I commented https://chromium-review.googlesource.com/c/616254

test_mode is needed for lxc_functional_test.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/f146e879007f326d2d92953a43f01cfd01931e84

commit f146e879007f326d2d92953a43f01cfd01931e84
Author: Allen Li <ayatane@chromium.org>
Date: Thu Aug 17 08:56:56 2017

[autotest] Restore results_mocker code

So this is actually used.

In fact, Im quite surprised that nothing broke beside
lxc_functional_test.py.

That probably means that most of this code is unneeded.

Figuring out what exactly this does and what small subset of that is
actually needed is not something I want to do, so Ive partially
reverted the offending commit.

BUG= chromium:753267 
TEST=Pre-CQ
TEST=CQ

Change-Id: I92d9e339ef23e3cd290767329c686c267853cadb
Reviewed-on: https://chromium-review.googlesource.com/616254
Commit-Ready: Allen Li <ayatane@chromium.org>
Tested-by: Allen Li <ayatane@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[add] https://crrev.com/f146e879007f326d2d92953a43f01cfd01931e84/server/results_mocker.py
[modify] https://crrev.com/f146e879007f326d2d92953a43f01cfd01931e84/server/autoserv

I've update the autotest server package used by the functional test, it's working now.