New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 833478 link

Starred by 2 users
Status: Fixed
Owner:
█ jkop@chromium.org
Last visit > 30 days ago
Closed: May 2018
Cc:
█ agicsaki@chromium.org
henryhsu@chromium.org
coconutruben@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: ----



Sign in to add a comment

some autotest_lib.site_utils.lxc.* unittests are flaky.

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, Apr 16 2018

Comment 1 by lepton@chromium.org, Apr 16 2018

Components: OS>Systems>Containers
Owner: dgreid@chromium.org
This is a unit test failed for container, could you take a look?

https://luci-logdog.appspot.com/v/?s=chromeos%2Fbb%2Fchromeos%2Fcave-release%2F2112%2F%2B%2Frecipes%2Fsteps%2FUnitTest%2F0%2Fstdout


autotest-0.0.2-r9301: DEBUG:root:qsize:0 shortfall:5 workers:1
autotest-0.0.2-r9301: ERROR:root:Worker error: Traceback (most recent call last):
autotest-0.0.2-r9301:   File "/build/cave/tmp/portage/chromeos-base/autotest-0.0.2-r9301/work/autotest-0.0.2/site_utils/lxc/container_pool/pool.py", line 392, in run
autotest-0.0.2-r9301:     container = self._factory.create_container()
autotest-0.0.2-r9301:   File "/build/cave/tmp/portage/chromeos-base/autotest-0.0.2-r9301/work/autotest-0.0.2/site_utils/lxc/container_pool/pool_unittest.py", line 514, in create_container
autotest-0.0.2-r9301:     raise TestException()
autotest-0.0.2-r9301: TestException: test error
autotest-0.0.2-r9301: 
autotest-0.0.2-r9301: ERROR:root:[140101244950272] Worker error: test error
autotest-0.0.2-r9301: DEBUG:root:qsize:0 shortfall:5 workers:1
autotest-0.0.2-r9301: ERROR:root:Worker error: Traceback (most recent call last):
autotest-0.0.2-r9301:   File "/build/cave/tmp/portage/chromeos-base/autotest-0.0.2-r9301/work/autotest-0.0.2/site_utils/lxc/container_pool/pool.py", line 392, in run
autotest-0.0.2-r9301:     container = self._factory.create_container()
autotest-0.0.2-r9301:   File "/build/cave/tmp/portage/chromeos-base/autotest-0.0.2-r9301/work/autotest-0.0.2/site_utils/lxc/container_pool/pool_unittest.py", line 514, in create_container
autotest-0.0.2-r9301:     raise TestException()
autotest-0.0.2-r9301: TestException: test error
autotest-0.0.2-r9301: 
autotest-0.0.2-r9301: ERROR:root:[140101244950272] Worker error: test error
autotest-0.0.2-r9301: DEBUG:root:qsize:0 shortfall:5 workers:1
autotest-0.0.2-r9301: ERROR:root:Worker error: Traceback (most recent call last):
autotest-0.0.2-r9301:   File "/build/cave/tmp/portage/chromeos-base/autotest-0.0.2-r9301/work/autotest-0.0.2/site_utils/lxc/container_pool/pool.py", line 392, in run
autotest-0.0.2-r9301:     container = self._factory.create_container()
autotest-0.0.2-r9301:   File "/build/cave/tmp/portage/chromeos-base/autotest-0.0.2-r9301/work/autotest-0.0.2/site_utils/lxc/container_pool/pool_unittest.py", line 514, in create_container
autotest-0.0.2-r9301:     raise TestException()
autotest-0.0.2-r9301: TestException: test error
autotest-0.0.2-r9301: 
autotest-0.0.2-r9301: ERROR:root:[140101244950272] Worker error: test error
autotest-0.0.2-r9301: DEBUG:root:qsize:0 shortfall:5 workers:1
autotest-0.0.2-r9301: ERROR:root:Worker error: Traceback (most recent call last):
autotest-0.0.2-r9301:   File "/build/cave/tmp/portage/chromeos-base/autotest-0.0.2-r9301/work/autotest-0.0.2/site_utils/lxc/container_pool/pool.py", line 392, in run
autotest-0.0.2-r9301:     container = self._factory.create_container()
autotest-0.0.2-r9301:   File "/build/cave/tmp/portage/chromeos-base/autotest-0.0.2-r9301/work/autotest-0.0.2/site_utils/lxc/container_pool/pool_unittest.py", line 514, in create_container
autotest-0.0.2-r9301:     raise TestException()
autotest-0.0.2-r9301: TestException: test error

Comment 2 by dgreid@chromium.org, Apr 16 2018

Owner: lepton@chromium.org
I think this is the autotest unittest failing. Maybe it uses a containter to run the test.

We did upgrad eLXC at some point so if it's using the system version there might have been a behavior change.

Comment 3 by lepton@chromium.org, Apr 16 2018

Owner: jkop@chromium.org
So it's this test failing, could it be a flaky tests?  I've seen there is some sleep in the test:

FAIL: testCleanup_timeout (autotest_lib.site_utils.lxc.container_pool.pool_unittest.PoolTests)

Comment 4 by lepton@chromium.org, Apr 16 2018 

Is that possible this recent change make that test more flaky?

https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/967221

Comment 5 by lepton@chromium.org, Apr 17 2018 

It passed for later runs (although finally failed for some other reason)

So it looks like a flaky test.

Comment 6 by lepton@chromium.org, Apr 17 2018 

Hit another flaky lxc test:

https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Freef-paladin%2F5612%2F%2B%2Frecipes%2Fsteps%2FUnitTest%2F0%2Fstdout

autotest_lib.site_utils.lxc.shared_host_dir_unittest failed in first run then
passed in 2nd run.

Comment 7 by lepton@chromium.org, Apr 17 2018

Summary: some autotest_lib.site_utils.lxc.* unittests are flaky. (was: cave-release:2112 failed)

Comment 8 by jkop@chromium.org, Apr 17 2018

Status: Assigned (was: Available)
That CL does seem like a likely culprit. I'll investigate.
Project Member

Comment 9 by bugdroid1@chromium.org, Apr 24 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/b28c7628ad691e591023ab9abd3c0d41a3fece81

commit b28c7628ad691e591023ab9abd3c0d41a3fece81
Author: Jacob Kopczynski <jkop@google.com>
Date: Tue Apr 24 07:30:15 2018

autotest: lxc: remove setup-run-before check

Speculative, may fix test flake.

BUG= chromium:833478 
TEST=unittests and tryjob

Change-Id: Ib650fd98573037b948f1a140e26db92266f63db2
Reviewed-on: https://chromium-review.googlesource.com/1016018
Commit-Ready: Jacob Kopczynski <jkop@chromium.org>
Tested-by: Jacob Kopczynski <jkop@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/b28c7628ad691e591023ab9abd3c0d41a3fece81/site_utils/lxc/unittest_setup.py

Comment 10 by jkop@chromium.org, May 3 2018

Cc: dgreid@chromium.org
dgreid@, lepton@, has this recurred since April 24th?

Comment 11 by jkop@chromium.org, May 4 2018

Cc: -dgreid@chromium.org -lepton@google.com coconutruben@chromium.org agicsaki@chromium.org henryhsu@chromium.org
Status: Fixed (was: Assigned)
No sign of recurrence, marking fixed.
►

Sign in to add a comment