SYNC_COUNT > 1 tests fail to run |
|||||||||||||||||||
Issue description
Version: 8172.27.0 guado_moblab
I have some tests that require multiple DUTs.
Using the SYNC_COUNT parameter in the control file to specify the
number of DUTs needed by the test, I am seeing the following
behavior:
- the requested number of DUTs are successfully provisioned
- as soon as all DUTs are provisioned, the job aborts
- the AFE shows the job status as 'Stopped'
Attached is a simple example server-based test, autotest_SyncCount, that
uses SYNC_COUNT = 2 to demonstrate the problem.
To reproduce:
1) connect 2 chromeos devices as moblab clients and give them a pool
label. This example assumes panther clients.
2) add the autotest_SyncCount test to autotest/files/server/site_tests/
3) cros_workon --board=panther autotest-server-tests
4) add the test to the autotest-server-tests ebuild
5) build_packages --board=panther autotest-all
6) cros stage gs://chromeos-image-archive/panther-release/R53-8460.0.0 <moblab-ip>
7) run the test:
test_that --board=panther --build=panther-custom/R53-8460.0.0 --web=<moblab-ip> --pool=<pool> :lab: autotest_SyncCount
Observed:
- DUTs are provisioned successfully
- The test then fails immediately with:
autotest_SyncCount ABORT: Timed out, did not run.
- The AFE shows the job in state '2 Stopped'
Expected:
- The test should execute
,
Jun 18 2016
I have had success with this patch to scheduler_models Job.run_if_ready:
*** /usr/local/autotest/scheduler/scheduler_models.py.orig
--- /usr/local/autotest/scheduler/scheduler_models.py
***************
*** 1459,1465 ****
ready to run.
"""
if not self.is_ready():
! self.stop_if_necessary()
elif queue_entry.atomic_group:
self.run_with_ready_delay(queue_entry)
else:
--- 1459,1465 ----
ready to run.
"""
if not self.is_ready():
! logging.info('Job not ready: %s', self)
elif queue_entry.atomic_group:
self.run_with_ready_delay(queue_entry)
else:
but don't know if it is the right solution.
,
Jun 21 2016
-> kevcheng@ since you'll have to fix this eventually anyway (I think) for servov4
,
Jul 1 2016
I did some investigation of this and have a fix to propose. Could you take a look to see if it is on the right track? Doc at go/autotest-sync-count-fix
,
Jul 1 2016
Uploaded https://chromium-review.googlesource.com/#/c/358092/ for comment.
,
Aug 10 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/303d2669b78eb5a2ebd6c41dce80ca18d06a19cf commit 303d2669b78eb5a2ebd6c41dce80ca18d06a19cf Author: Laurence Goodby <lgoodby@google.com> Date: Fri Jul 01 22:57:04 2016 [autotest] Scheduler fix for SYNC_COUNT > 1. Details in go/autotest-sync-count-fix BUG= chromium:621257 TEST=Run included tests. Change-Id: If08df8fb04771a321dbdf2122b885935e7ef3b41 Reviewed-on: https://chromium-review.googlesource.com/358092 Commit-Ready: Laurence Goodby <lgoodby@chromium.org> Tested-by: Laurence Goodby <lgoodby@chromium.org> Reviewed-by: Richard Barnette <jrbarnette@chromium.org> Reviewed-by: Richard Barnette <jrbarnette@google.com> Reviewed-by: Dan Shi <dshi@google.com> [add] https://crrev.com/303d2669b78eb5a2ebd6c41dce80ca18d06a19cf/server/site_tests/autotest_SyncCount/control [modify] https://crrev.com/303d2669b78eb5a2ebd6c41dce80ca18d06a19cf/scheduler/monitor_db_functional_test.py [modify] https://crrev.com/303d2669b78eb5a2ebd6c41dce80ca18d06a19cf/scheduler/scheduler_models.py [add] https://crrev.com/303d2669b78eb5a2ebd6c41dce80ca18d06a19cf/server/site_tests/autotest_SyncCount/autotest_SyncCount.py [modify] https://crrev.com/303d2669b78eb5a2ebd6c41dce80ca18d06a19cf/client/common_lib/host_queue_entry_states.py [modify] https://crrev.com/303d2669b78eb5a2ebd6c41dce80ca18d06a19cf/frontend/afe/models.py
,
Aug 10 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/303d2669b78eb5a2ebd6c41dce80ca18d06a19cf commit 303d2669b78eb5a2ebd6c41dce80ca18d06a19cf Author: Laurence Goodby <lgoodby@google.com> Date: Fri Jul 01 22:57:04 2016 [autotest] Scheduler fix for SYNC_COUNT > 1. Details in go/autotest-sync-count-fix BUG= chromium:621257 TEST=Run included tests. Change-Id: If08df8fb04771a321dbdf2122b885935e7ef3b41 Reviewed-on: https://chromium-review.googlesource.com/358092 Commit-Ready: Laurence Goodby <lgoodby@chromium.org> Tested-by: Laurence Goodby <lgoodby@chromium.org> Reviewed-by: Richard Barnette <jrbarnette@chromium.org> Reviewed-by: Richard Barnette <jrbarnette@google.com> Reviewed-by: Dan Shi <dshi@google.com> [add] https://crrev.com/303d2669b78eb5a2ebd6c41dce80ca18d06a19cf/server/site_tests/autotest_SyncCount/control [modify] https://crrev.com/303d2669b78eb5a2ebd6c41dce80ca18d06a19cf/scheduler/monitor_db_functional_test.py [modify] https://crrev.com/303d2669b78eb5a2ebd6c41dce80ca18d06a19cf/scheduler/scheduler_models.py [add] https://crrev.com/303d2669b78eb5a2ebd6c41dce80ca18d06a19cf/server/site_tests/autotest_SyncCount/autotest_SyncCount.py [modify] https://crrev.com/303d2669b78eb5a2ebd6c41dce80ca18d06a19cf/client/common_lib/host_queue_entry_states.py [modify] https://crrev.com/303d2669b78eb5a2ebd6c41dce80ca18d06a19cf/frontend/afe/models.py
,
Aug 10 2016
,
Aug 29 2016
,
Oct 7 2016
,
Oct 10 2016
,
Nov 19 2016
,
Jan 21 2017
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Jun 1 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/483287116e6b9607ef3415f527116e88a3e70471 commit 483287116e6b9607ef3415f527116e88a3e70471 Author: Aviv Keshet <akeshet@chromium.org> Date: Thu Jun 01 04:14:31 2017 autotest-server-tests: build autotest_SyncCount CQ-DEPEND=CL:519465 BUG=chromium:726490, chromium:621257 TEST=Attempt to run autotest_SyncCount in lab, see that it in fact runs. Change-Id: Ie844d6c0956fd9b7eb7434cd07585c083658a723 Reviewed-on: https://chromium-review.googlesource.com/518982 Commit-Ready: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Aviv Keshet <akeshet@chromium.org> [modify] https://crrev.com/483287116e6b9607ef3415f527116e88a3e70471/chromeos-base/autotest-server-tests/autotest-server-tests-9999.ebuild
,
Jun 1 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/828e78005dbda76677072d88c465d7304c8230ae commit 828e78005dbda76677072d88c465d7304c8230ae Author: Aviv Keshet <akeshet@chromium.org> Date: Thu Jun 01 04:14:31 2017 autotest: temporarily remove autotest_SyncControl from push_to_prod BUG=chromium:726490, chromium:621257 TEST=None Change-Id: Ib9dec32b0f8880b27dec2c683f60ec7e7f8e07b3 Reviewed-on: https://chromium-review.googlesource.com/519465 Commit-Ready: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/828e78005dbda76677072d88c465d7304c8230ae/server/site_tests/autotest_SyncCount/control
,
Jun 2 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/deadecfbe39473b474fa542ab65786e138bc28ad commit deadecfbe39473b474fa542ab65786e138bc28ad Author: Aviv Keshet <akeshet@chromium.org> Date: Fri Jun 02 21:44:45 2017 autotest-server-tests: build autotest_SyncCount CQ-DEPEND=CL:519465 BUG=chromium:726490, chromium:621257 TEST=Attempt to run autotest_SyncCount in lab, see that it in fact runs. Change-Id: Ie844d6c0956fd9b7eb7434cd07585c083658a723 Reviewed-on: https://chromium-review.googlesource.com/518982 Commit-Ready: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Aviv Keshet <akeshet@chromium.org> (cherry picked from commit 483287116e6b9607ef3415f527116e88a3e70471) Reviewed-on: https://chromium-review.googlesource.com/523265 Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/deadecfbe39473b474fa542ab65786e138bc28ad/chromeos-base/autotest-server-tests/autotest-server-tests-9999.ebuild
,
Jun 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/9f917d77714f7e6c12bbf699a4ecfe507ecd999f commit 9f917d77714f7e6c12bbf699a4ecfe507ecd999f Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Sat Jun 03 01:39:40 2017 Revert "autotest: temporarily remove autotest_SyncControl from push_to_prod" This reverts commit 828e78005dbda76677072d88c465d7304c8230ae. Reason for revert: autotest_SyncCount has been shown to be stable (famous last words) Original change's description: > autotest: temporarily remove autotest_SyncControl from push_to_prod > > BUG=chromium:726490, chromium:621257 > TEST=None > > Change-Id: Ib9dec32b0f8880b27dec2c683f60ec7e7f8e07b3 > Reviewed-on: https://chromium-review.googlesource.com/519465 > Commit-Ready: Aviv Keshet <akeshet@chromium.org> > Tested-by: Aviv Keshet <akeshet@chromium.org> > Reviewed-by: Dan Shi <dshi@google.com> > BUG=chromium:726490, chromium:621257 Change-Id: I39a364998f68691974932f352cc0544f9fc700df Reviewed-on: https://chromium-review.googlesource.com/522932 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/9f917d77714f7e6c12bbf699a4ecfe507ecd999f/server/site_tests/autotest_SyncCount/control
,
Jun 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/6036317cf1e24db61a077186e23a37b4770c269b commit 6036317cf1e24db61a077186e23a37b4770c269b Author: Aviv Keshet <akeshet@chromium.org> Date: Mon Jun 05 17:01:22 2017 autotest-server-tests: build autotest_SyncCount CQ-DEPEND=CL:519465 BUG=chromium:726490, chromium:621257 TEST=Attempt to run autotest_SyncCount in lab, see that it in fact runs. Change-Id: Ie844d6c0956fd9b7eb7434cd07585c083658a723 Reviewed-on: https://chromium-review.googlesource.com/518982 Commit-Ready: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Aviv Keshet <akeshet@chromium.org> (cherry picked from commit 483287116e6b9607ef3415f527116e88a3e70471) Reviewed-on: https://chromium-review.googlesource.com/523263 Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Commit-Queue: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/6036317cf1e24db61a077186e23a37b4770c269b/chromeos-base/autotest-server-tests/autotest-server-tests-9999.ebuild
,
Aug 1 2017
,
Aug 3 2017
Closing. Please reopen it if its not fixed. Thanks! |
|||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||
Comment 1 by jrbarnette@chromium.org
, Jun 18 2016Components: Infra>Client>ChromeOS