Both M60 and M61 release branch failed due to AUTest failure |
||||||||||||||||||||||||||||
Issue description
One example of the failures:
DEBUG:root:Searching for pattern *_delta_* from url gs://chromeos-image-archive/bob-release/R61-9765.61.0
CRITICAL:root:no test configurations generated, nothing to do
[1;31m15:13:30: ERROR: Traceback (most recent call last):
File "/b/c/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 700, in Run
self.PerformStage()
File "/b/c/cbuild/repository/chromite/cbuildbot/stages/test_stages.py", line 527, in PerformStage
self.version, self.upload_url)
File "/b/c/cbuild/repository/chromite/cbuildbot/commands.py", line 2195, in BuildAUTestTarball
cros_build_lib.RunCommand(cmd, env=run_env, cwd=cwd)
File "/b/c/cbuild/repository/chromite/lib/cros_build_lib.py", line 645, in RunCommand
raise RunCommandError(msg, cmd_result)
RunCommandError: return code: 1; command: site_utils/autoupdate/full_release_test.py --npo --nmo --dump --dump_dir /tmp/cbuildbot-tmp47nDME/tmpSAjd8a/autotest/au_control_files --archive_url gs://chromeos-image-archive/bob-release/R61-9765.61.0 9765.61.0 bob '--log=debug'
cmd=['site_utils/autoupdate/full_release_test.py', '--npo', '--nmo', '--dump', '--dump_dir', '/tmp/cbuildbot-tmp47nDME/tmpSAjd8a/autotest/au_control_files', '--archive_url', u'gs://chromeos-image-archive/bob-release/R61-9765.61.0', '9765.61.0', u'bob', '--log=debug'], cwd=/b/c/cbuild/repository/src/third_party/autotest/files
[0m
15:13:30: INFO: Translating result return code: 1; command: site_utils/autoupdate/full_release_test.py --npo --nmo --dump --dump_dir /tmp/cbuildbot-tmp47nDME/tmpSAjd8a/autotest/au_control_files --archive_url gs://chromeos-image-archive/bob-release/R61-9765.61.0 9765.61.0 bob '--log=debug'
cmd=['site_utils/autoupdate/full_release_test.py', '--npo', '--nmo', '--dump', '--dump_dir', '/tmp/cbuildbot-tmp47nDME/tmpSAjd8a/autotest/au_control_files', '--archive_url', u'gs://chromeos-image-archive/bob-release/R61-9765.61.0', '9765.61.0', u'bob', '--log=debug'], cwd=/b/c/cbuild/repository/src/third_party/autotest/files to fail.
15:13:30: INFO: Running cidb query on pid 15124, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7faeeaa9ed90>
15:13:30: INFO: Running cidb query on pid 15124, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7faeeaa9ef50>
************************************************************
** Finished Stage AUTest [au] - Sun, 10 Sep 2017 15:13:30 -0700 (PDT)
************************************************************
[1;31m15:13:30: ERROR: BaseException in _RunParallelStages <class 'chromite.lib.failures_lib.StepFailure'>:
Traceback (most recent call last):
File "/b/c/cbuild/repository/chromite/lib/parallel.py", line 441, in _Run
self._task(*self._task_args, **self._task_kwargs)
File "/b/c/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 720, in Run
raise failures_lib.StepFailure()
StepFailure
Traceback (most recent call last):
File "/b/c/cbuild/repository/chromite/cbuildbot/builders/generic_builders.py", line 119, in _RunParallelStages
parallel.RunParallelSteps(steps)
File "/b/c/cbuild/repository/chromite/lib/parallel.py", line 678, in RunParallelSteps
return [queue.get_nowait() for queue in queues]
File "/b/c/cbuild/repository/chromite/lib/parallel.py", line 675, in RunParallelSteps
pass
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/b/c/cbuild/repository/chromite/lib/parallel.py", line 561, in ParallelTasks
raise BackgroundFailure(exc_infos=errors)
BackgroundFailure: <class 'chromite.lib.failures_lib.StepFailure'>:
Traceback (most recent call last):
File "/b/c/cbuild/repository/chromite/lib/parallel.py", line 441, in _Run
self._task(*self._task_args, **self._task_kwargs)
File "/b/c/cbuild/repository/chromite/cbuildbot/stages/generic_stages.py", line 720, in Run
raise failures_lib.StepFailure()
StepFailure
[0m
There is a critical error saying that:
CRITICAL:root:no test configurations generated, nothing to do
Don, ningning, do you know what it means?
,
Sep 11 2017
I checked one example, the file was missing when it looked for it, but was uploaded afterward. The auto_update code is in autotest http://shortn/_t1h0bxGtqz and I don't have knowledge about it, you may need to find a right owner. https://uberchromegw.corp.google.com/i/chromeos_release/builders/edgar-release%20release-R61-9765.B/builds/48 nxia@nxia:~/chromiumos/src/third_party/autotest/files/site_utils$ gsutil stat gs://chromeos-image-archive/edgar-release/R61-9765.61.0/chromeos_R61-9765.61.0_R61-9765.61.0_edgar_delta_dev.bin gs://chromeos-image-archive/edgar-release/R61-9765.61.0/chromeos_R61-9765.61.0_R61-9765.61.0_edgar_delta_dev.bin: Creation time: Sun, 10 Sep 2017 21:09:59 GMT Update time: Sun, 10 Sep 2017 21:09:59 GMT
,
Sep 11 2017
Sounds like there is an interstage dependency that is wrong. What stage generates the file, and what stage consumes it?
,
Sep 11 2017
Is this stage upload the au_test tarball to the gs, and the tarball will be used by HWTest stage later? > #3 which file do you mean? benzh@, can you take a look of the 'no test configuration' error?
,
Sep 12 2017
+bernie as fyi. Thanks.
,
Sep 12 2017
Last time we had this issue was due to gsutil upgrade and it wasn't clear what was the root cause of gsutil failures. We just reverted back to 4.19 crbug.com/639314#c33 Looks like we had a recent gsutil upgrade on 2017-09-08 from 4.19 to 4.27 https://chromium-review.googlesource.com/653370 which may be related. Let me take a closer look.
,
Sep 12 2017
master branch is now on 4.27, but M61 is still using 4.19. Can you give some sample GS URLs related to the problem?
,
Sep 13 2017
It seems like this issue also affect M60 release branch. Raise the priority.
,
Sep 13 2017
,
Sep 13 2017
,
Sep 13 2017
,
Sep 13 2017
For M61, the last successful AUTest for most boards was 2017-09-05 11:30 9765.53.0, and the first failed AUTest was 2017-09-07 19:21 9765.58.0 https://crosland.corp.google.com/log/9765.53.0..9765.58.0 I don't see any suspect patch that could affect AUTest on so many boards. Lumpy is interesting. The Sep 07 19:18 build failed AUTest, but later builds were fine. https://uberchromegw.corp.google.com/i/chromeos_release/builders/lumpy-release%20release-R61-9765.B
,
Sep 13 2017
I planned to make some changes to the AUTest phase/au suite so this might be a good time to do it. (See details issue 758307 ). Quick summary: The AUTest phase on the builder generates a control file that is used to run autoupdate_EndToEndTest with an N+1 (npo) delta payload. This is used to ensure we can autoupdate away from the current build. But we already get this coverage with the N2N test in paygen_au* suites. So this test in the au suite is redundant. Another reason I wanted to get rid of this test is that when it is not generated on the builder it never will be (see issue 709663 ) and then the test team need to manually run the other tests in this suite: powerwash and rollback
,
Sep 13 2017
This CL should stop the generation of the control files on the builder (stopping the problem in this bug): https://chromium-review.googlesource.com/#/c/chromiumos/chromite/+/665170/ This CL will stop the au suite from looking for the control file: https://chromium-review.googlesource.com/#/c/chromiumos/third_party/autotest/+/665366/ I am not sure how to test builder phase changes though...
,
Sep 14 2017
While we are debating the CL in c#14, is there anything we can do to have AU test running again ? Is this a infra failure ?
,
Sep 14 2017
It is an infra failure in that the artifact needed is not generated. It has the same outcome as issue 709663 in that rerunning will not get the suite running (because the artifact is needed and trying to stage it fails)
,
Sep 14 2017
+shuqianz, infra deputy to help from infra angle.
,
Sep 14 2017
,
Sep 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/729ca887199523d0c0d6dbd8c94d25850124874c commit 729ca887199523d0c0d6dbd8c94d25850124874c Author: David Haddock <dhaddock@chromium.org> Date: Fri Sep 15 07:24:02 2017 [chromite] Remove the AUTestStage from builders. We want to stop generating npo control files during AUTest stage. Generating N to N+1 (npo) control file is not necessary as we already have sufficient coverage with N to N test as part of the paygen_au* suites. So we don't need this stage and its assoicated code. BUG= chromium:764038 BUG= chromium:758307 TEST=Not sure how best to test this change CQ-DEPEND=CL:665366 Change-Id: Iadc9904c4e81770896cf02756f0025f87f5b5eb0 Reviewed-on: https://chromium-review.googlesource.com/665170 Commit-Ready: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/729ca887199523d0c0d6dbd8c94d25850124874c/cbuildbot/stages/test_stages.py [modify] https://crrev.com/729ca887199523d0c0d6dbd8c94d25850124874c/cbuildbot/commands_unittest.py [modify] https://crrev.com/729ca887199523d0c0d6dbd8c94d25850124874c/cbuildbot/stages/test_stages_unittest.py [modify] https://crrev.com/729ca887199523d0c0d6dbd8c94d25850124874c/cbuildbot/commands.py [modify] https://crrev.com/729ca887199523d0c0d6dbd8c94d25850124874c/cbuildbot/builders/simple_builders.py
,
Sep 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/8f9b1e6bfd3bda3fc922034cbd2f3c89ba7bc485 commit 8f9b1e6bfd3bda3fc922034cbd2f3c89ba7bc485 Author: David Haddock <dhaddock@chromium.org> Date: Fri Sep 15 07:24:02 2017 Remove staging of control files for au suite. We no longer need to stage control files as part of the au suite. The npo test case that this test suite controls is covered as part of the N2N tests in paygen_au* suites. This change will remove a lot of manual work required by the test team to run powerwash tests manually when the sanity stage on the builder fails and the control file is not generated. BUG= chromium:758307 BUG= chromium:764038 BUG= chromium:709663 TEST=None Change-Id: I88a9bf7266c5615deb221d4adb50e5fbf4977d25 Reviewed-on: https://chromium-review.googlesource.com/665366 Commit-Ready: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> Reviewed-by: David Haddock <dhaddock@chromium.org> [modify] https://crrev.com/8f9b1e6bfd3bda3fc922034cbd2f3c89ba7bc485/test_suites/control.au
,
Sep 15 2017
,
Sep 15 2017
This worked on ToT Build 4666 has the AUTestStage and the au suite has 3 tests: https://uberchromegw.corp.google.com/i/chromeos/builders/samus-release/builds/4666 Build 4667 does not have the AUTestStage and the au suite has 2 tests: https://uberchromegw.corp.google.com/i/chromeos/builders/samus-release/builds/4667 Requesting merge to the broken branches +ketaki + josafat
,
Sep 15 2017
,
Sep 15 2017
,
Sep 15 2017
We are probably done with M60 now actually. So merge request to 62 and 61
,
Sep 15 2017
This bug requires manual review: M62 has already been promoted to the beta branch, so this requires manual review Please contact the milestone owner if you have questions. Owners: amineer@(Android), cmasso@(iOS), bhthompson@(ChromeOS), abdulsyed@(Desktop) For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Sep 15 2017
Merge approved for 62.
,
Sep 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/4366f7cce5bd1220e13c266a02c9802e0e687266 commit 4366f7cce5bd1220e13c266a02c9802e0e687266 Author: David Haddock <dhaddock@chromium.org> Date: Fri Sep 15 23:09:09 2017 [chromite] Remove the AUTestStage from builders. We want to stop generating npo control files during AUTest stage. Generating N to N+1 (npo) control file is not necessary as we already have sufficient coverage with N to N test as part of the paygen_au* suites. So we don't need this stage and its assoicated code. BUG= chromium:764038 BUG= chromium:758307 TEST=Not sure how best to test this change CQ-DEPEND=CL:665366 Change-Id: Iadc9904c4e81770896cf02756f0025f87f5b5eb0 Reviewed-on: https://chromium-review.googlesource.com/665170 Commit-Ready: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> (cherry picked from commit 729ca887199523d0c0d6dbd8c94d25850124874c) Reviewed-on: https://chromium-review.googlesource.com/669701 Reviewed-by: David Haddock <dhaddock@chromium.org> Trybot-Ready: David Haddock <dhaddock@chromium.org> [modify] https://crrev.com/4366f7cce5bd1220e13c266a02c9802e0e687266/cbuildbot/stages/test_stages.py [modify] https://crrev.com/4366f7cce5bd1220e13c266a02c9802e0e687266/cbuildbot/commands_unittest.py [modify] https://crrev.com/4366f7cce5bd1220e13c266a02c9802e0e687266/cbuildbot/stages/test_stages_unittest.py [modify] https://crrev.com/4366f7cce5bd1220e13c266a02c9802e0e687266/cbuildbot/commands.py [modify] https://crrev.com/4366f7cce5bd1220e13c266a02c9802e0e687266/cbuildbot/builders/simple_builders.py
,
Sep 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/24874293ea89dcce5af71f7d5da74d483e4f4dc4 commit 24874293ea89dcce5af71f7d5da74d483e4f4dc4 Author: David Haddock <dhaddock@chromium.org> Date: Fri Sep 15 23:09:34 2017 Remove staging of control files for au suite. We no longer need to stage control files as part of the au suite. The npo test case that this test suite controls is covered as part of the N2N tests in paygen_au* suites. This change will remove a lot of manual work required by the test team to run powerwash tests manually when the sanity stage on the builder fails and the control file is not generated. BUG= chromium:758307 BUG= chromium:764038 BUG= chromium:709663 TEST=None Change-Id: I88a9bf7266c5615deb221d4adb50e5fbf4977d25 Reviewed-on: https://chromium-review.googlesource.com/665366 Commit-Ready: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> Reviewed-by: David Haddock <dhaddock@chromium.org> (cherry picked from commit 8f9b1e6bfd3bda3fc922034cbd2f3c89ba7bc485) Reviewed-on: https://chromium-review.googlesource.com/669901 Trybot-Ready: David Haddock <dhaddock@chromium.org> [modify] https://crrev.com/24874293ea89dcce5af71f7d5da74d483e4f4dc4/test_suites/control.au
,
Sep 18 2017
Approving merge to M61. Please merge and validate on M62 prior to merge on M61.
,
Sep 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/8028677de916a3172a53611e3e57c74f183637c8 commit 8028677de916a3172a53611e3e57c74f183637c8 Author: David Haddock <dhaddock@chromium.org> Date: Mon Sep 18 18:27:01 2017 [chromite] Remove the AUTestStage from builders. We want to stop generating npo control files during AUTest stage. Generating N to N+1 (npo) control file is not necessary as we already have sufficient coverage with N to N test as part of the paygen_au* suites. So we don't need this stage and its assoicated code. BUG= chromium:764038 BUG= chromium:758307 TEST=Not sure how best to test this change CQ-DEPEND=CL:665366 Change-Id: Iadc9904c4e81770896cf02756f0025f87f5b5eb0 Reviewed-on: https://chromium-review.googlesource.com/665170 Commit-Ready: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/671455 Reviewed-by: David Haddock <dhaddock@chromium.org> Trybot-Ready: David Haddock <dhaddock@chromium.org> [modify] https://crrev.com/8028677de916a3172a53611e3e57c74f183637c8/cbuildbot/stages/test_stages.py [modify] https://crrev.com/8028677de916a3172a53611e3e57c74f183637c8/cbuildbot/commands_unittest.py [modify] https://crrev.com/8028677de916a3172a53611e3e57c74f183637c8/cbuildbot/stages/test_stages_unittest.py [modify] https://crrev.com/8028677de916a3172a53611e3e57c74f183637c8/cbuildbot/commands.py [modify] https://crrev.com/8028677de916a3172a53611e3e57c74f183637c8/cbuildbot/builders/simple_builders.py
,
Sep 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/eb06fa4395f1cf3bd8d3b930308d8f977a7bdd9a commit eb06fa4395f1cf3bd8d3b930308d8f977a7bdd9a Author: David Haddock <dhaddock@chromium.org> Date: Mon Sep 18 18:27:41 2017 Remove staging of control files for au suite. We no longer need to stage control files as part of the au suite. The npo test case that this test suite controls is covered as part of the N2N tests in paygen_au* suites. This change will remove a lot of manual work required by the test team to run powerwash tests manually when the sanity stage on the builder fails and the control file is not generated. BUG= chromium:758307 BUG= chromium:764038 BUG= chromium:709663 TEST=None Change-Id: I88a9bf7266c5615deb221d4adb50e5fbf4977d25 Reviewed-on: https://chromium-review.googlesource.com/665366 Commit-Ready: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> Reviewed-by: David Haddock <dhaddock@chromium.org> (cherry picked from commit 8f9b1e6bfd3bda3fc922034cbd2f3c89ba7bc485) Reviewed-on: https://chromium-review.googlesource.com/671186 Trybot-Ready: David Haddock <dhaddock@chromium.org> [modify] https://crrev.com/eb06fa4395f1cf3bd8d3b930308d8f977a7bdd9a/test_suites/control.au
,
Sep 18 2017
Checked on 62 and it is working. I've merged to M61 too. Let me know if M60 still needs it. Marking fixed now
,
Sep 18 2017
,
Sep 18 2017
,
Sep 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/78a645b0df7702f8b82ffca216b6cffda17cf1ea commit 78a645b0df7702f8b82ffca216b6cffda17cf1ea Author: David Haddock <dhaddock@chromium.org> Date: Tue Sep 19 01:37:57 2017 [chromite] Remove the AUTestStage from builders. We want to stop generating npo control files during AUTest stage. Generating N to N+1 (npo) control file is not necessary as we already have sufficient coverage with N to N test as part of the paygen_au* suites. So we don't need this stage and its assoicated code. BUG= chromium:764038 BUG= chromium:758307 TEST=Not sure how best to test this change CQ-DEPEND=CL:665366 Change-Id: Iadc9904c4e81770896cf02756f0025f87f5b5eb0 Reviewed-on: https://chromium-review.googlesource.com/665170 Commit-Ready: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/671685 Reviewed-by: David Haddock <dhaddock@chromium.org> Trybot-Ready: David Haddock <dhaddock@chromium.org> [modify] https://crrev.com/78a645b0df7702f8b82ffca216b6cffda17cf1ea/cbuildbot/stages/test_stages.py [modify] https://crrev.com/78a645b0df7702f8b82ffca216b6cffda17cf1ea/cbuildbot/commands_unittest.py [modify] https://crrev.com/78a645b0df7702f8b82ffca216b6cffda17cf1ea/cbuildbot/stages/test_stages_unittest.py [modify] https://crrev.com/78a645b0df7702f8b82ffca216b6cffda17cf1ea/cbuildbot/commands.py [modify] https://crrev.com/78a645b0df7702f8b82ffca216b6cffda17cf1ea/cbuildbot/builders/simple_builders.py
,
Sep 19 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/cc274348acbbad5bde4480bca36fca6ea8c77981 commit cc274348acbbad5bde4480bca36fca6ea8c77981 Author: David Haddock <dhaddock@chromium.org> Date: Tue Sep 19 01:38:10 2017 Remove staging of control files for au suite. We no longer need to stage control files as part of the au suite. The npo test case that this test suite controls is covered as part of the N2N tests in paygen_au* suites. This change will remove a lot of manual work required by the test team to run powerwash tests manually when the sanity stage on the builder fails and the control file is not generated. BUG= chromium:758307 BUG= chromium:764038 BUG= chromium:709663 TEST=None Change-Id: I88a9bf7266c5615deb221d4adb50e5fbf4977d25 Reviewed-on: https://chromium-review.googlesource.com/665366 Commit-Ready: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> Reviewed-by: David Haddock <dhaddock@chromium.org> (cherry picked from commit 8f9b1e6bfd3bda3fc922034cbd2f3c89ba7bc485) Reviewed-on: https://chromium-review.googlesource.com/671153 Trybot-Ready: David Haddock <dhaddock@chromium.org> [modify] https://crrev.com/cc274348acbbad5bde4480bca36fca6ea8c77981/test_suites/control.au
,
Sep 19 2017
Merge to 61 worked: https://uberchromegw.corp.google.com/i/chromeos_release/builders/eve-release%20release-R61-9765.B/builds/58 Merged to 60 too
,
Sep 19 2017
,
Sep 19 2017
M60 gone green now too: https://uberchromegw.corp.google.com/i/chromeos_release/builders/nyan_big-release%20release-R60-9592.B
,
Sep 22 2017
This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible! If all merges have been completed, please remove any remaining Merge-Approved labels from this issue. Thanks for your time! To disable nags, add the Disable-Nags label. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Sep 25 2017
This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible! If all merges have been completed, please remove any remaining Merge-Approved labels from this issue. Thanks for your time! To disable nags, add the Disable-Nags label. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Sep 26 2017
stopping the nags |
||||||||||||||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||||||||||||||
Comment 1 by dgarr...@chromium.org
, Sep 11 2017