Issue 774349  has been merged into this issue.

pool_unittest failed in CQ:

https://uberchromegw.corp.google.com/i/chromeos/builders/betty-arc64-paladin/builds/704/steps/UnitTest/logs/stdio

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/a1b3b1e76448f5c968dbe17ece4e524f70ae0e72

commit a1b3b1e76448f5c968dbe17ece4e524f70ae0e72
Author: Ben Kwa <kenobi@google.com>
Date: Tue Oct 17 04:03:17 2017

Temporarily disable a flakey test.

BUG= chromium:774534 
TEST=pool_unittest.py PoolTests.testCleanup_timeout (should skip the test)

Change-Id: I5293ebae524729c7f5f641e1a0c45cefcb9f0add
Reviewed-on: https://chromium-review.googlesource.com/719217
Reviewed-by: Ben Kwa <kenobi@chromium.org>
Tested-by: Ben Kwa <kenobi@chromium.org>
Commit-Queue: Ben Kwa <kenobi@chromium.org>

[modify] https://crrev.com/a1b3b1e76448f5c968dbe17ece4e524f70ae0e72/site_utils/lxc/container_pool/pool_unittest.py

 Issue 775355  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/785edabb9a4fe2c4c23d5fe2f53f44ce0640a0db

commit 785edabb9a4fe2c4c23d5fe2f53f44ce0640a0db
Author: Ben Kwa <kenobi@google.com>
Date: Wed Oct 25 05:47:57 2017

Re-enable a de-flaked unittest.

Add a timeout to the test to wait for all worker threads to.. time out.

BUG= chromium:774534 
TEST=pool_unittest.py PoolTests.testCleanup_timeout
TEST=Run the unit-test repeatedly with and without the fix.  Without the
fix, the test fails on repeated runs.  With the fix, it ran and passed
200x with no flakes.

Change-Id: Ib5500ea68bb3844bad113079accef22ee9d0239a
Reviewed-on: https://chromium-review.googlesource.com/736773
Commit-Ready: Ben Kwa <kenobi@chromium.org>
Tested-by: Ben Kwa <kenobi@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/785edabb9a4fe2c4c23d5fe2f53f44ce0640a0db/site_utils/lxc/container_pool/pool_unittest.py

https://uberchromegw.corp.google.com/i/chromeos/builders/tricky-paladin/builds/10014

https://chromium-review.googlesource.com/#/c/chromiumos/third_party/autotest/+/742382 

test_CleanupTimeout is no longer flaking.... but (another one is).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/607e3b75aab7c6416449e50469a6abf9217219ed

commit 607e3b75aab7c6416449e50469a6abf9217219ed
Author: Ben Kwa <kenobi@google.com>
Date: Thu Nov 16 00:07:28 2017

[autotest] Fix a flakey test.

- Instead of fiddling with the error queue, use the new worker count to wait for
  workers to time out.

- Remove the intermediate assert which was unnecessary (and flakey).  The
  problem was that workers starting up in the interim period before the pool
  fully shuts down, causes this assert to be incorrect.  What we really care
  about is that (1) something timed out, and (2) the final container creation
  and destruction counts are equal in spite of the timeout.

BUG= chromium:774534 
TEST=pool_unittest.py -v PoolTests.testCleanup_timeout
TEST=Repeatedly run test under load.  Observe flakes.  After fix, observe no
     more flakes.
TEST=Insert a time.sleep(1) in the top of Pool.cleanup to exacerbate the race
     condition.  Observe now consistent failures.  Apply fix, observe no more
     failures (don't keep the sleep in there as it is a waste of time under
     non-testing conditions).

Change-Id: Ia35fd6a1709369729178122772880cdf6f944b11
Reviewed-on: https://chromium-review.googlesource.com/772291
Commit-Ready: Ben Kwa <kenobi@chromium.org>
Tested-by: Ben Kwa <kenobi@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>
Reviewed-by: Ben Kwa <kenobi@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/607e3b75aab7c6416449e50469a6abf9217219ed/site_utils/lxc/container_pool/pool_unittest.py
[modify] https://crrev.com/607e3b75aab7c6416449e50469a6abf9217219ed/site_utils/lxc/container_pool/error.py
[modify] https://crrev.com/607e3b75aab7c6416449e50469a6abf9217219ed/site_utils/lxc/unittest_setup.py

 Issue 786129  has been merged into this issue.

Issue 779686 has been merged into this issue.

It looks like the bug is still present in the R63 branch:

Failed builds:
https://luci-milo.appspot.com/buildbot/chromeos_release/novato-arc64-release%20release-R63-10032.B/49
https://luci-milo.appspot.com/buildbot/chromeos_release/guado_moblab-release%20release-R63-10032.B/53

Flakey:
https://luci-milo.appspot.com/buildbot/chromeos_release/reef-android-nyc-pre-flight-branch%20release-R63-10032.B/210
https://luci-milo.appspot.com/buildbot/chromeos_release/reef-android-nyc-pre-flight-branch%20release-R63-10032.B/215