LXC changes broke staging test |
||||||
Issue description... in a weird way. (it's failed for the past ~ 7 days) The latest failed test suite: http://chromeos-staging-master2.hot.corp.google.com/afe/#tab_id=view_job&object_id=12571 Logs are not correctly linked from this master, suite job logs: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/12571-chromeos-test/hostless/debug/ The failing test logs: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/12577-chromeos-test/chromeos2-row1-rack2-host1 Notice that the test actually passed, but I don't see any .parse.log file. Is job_reporter not parsing correctly?
,
Feb 28 2018
nvm,
error is in ssp_logs:
02/28 13:22:47.320 DEBUG| utils:0214| Running 'sudo bash -c 'umount '/usr/local/autotest/containers/host/container.53k5bm/usr/local/autotest/results/shared' && cd '('/usr/local/autotest/containers/host/container.53k5bm', u'usr/local/autotest/results/shared')'; rmdir -p '('/usr/local/autotest/containers/host/container.53k5bm', u'usr/local/autotest/results/shared')' || true''
02/28 13:22:47.340 DEBUG| utils:0214| Running 'sudo -n chown -R 99158 "/usr/local/autotest/results/12577-chromeos-test/chromeos2-row1-rack2-host1"'
02/28 13:22:47.647 DEBUG| utils:0214| Running 'sudo -n chgrp -R 5000 "/usr/local/autotest/results/12577-chromeos-test/chromeos2-row1-rack2-host1"'
02/28 13:22:47.956 ERROR| traceback:0013| Traceback (most recent call last):
02/28 13:22:47.956 ERROR| traceback:0013| File "/usr/local/autotest/server/autoserv", line 507, in run_autoserv
02/28 13:22:47.956 ERROR| traceback:0013| machines)
02/28 13:22:47.956 ERROR| traceback:0013| File "/usr/local/autotest/server/autoserv", line 233, in _run_with_ssp
02/28 13:22:47.956 ERROR| traceback:0013| test_container.destroy()
02/28 13:22:47.957 ERROR| traceback:0013| File "/usr/local/autotest/site_utils/lxc/zygote.py", line 111, in destroy
02/28 13:22:47.957 ERROR| traceback:0013| self._cleanup_host_mount()
02/28 13:22:47.957 ERROR| traceback:0013| File "/usr/local/autotest/site_utils/lxc/zygote.py", line 221, in _cleanup_host_mount
02/28 13:22:47.957 ERROR| traceback:0013| mount.cleanup()
02/28 13:22:47.957 ERROR| traceback:0013| File "/usr/local/autotest/site_utils/lxc/utils.py", line 186, in cleanup
02/28 13:22:47.957 ERROR| traceback:0013| sudo_commands(commands)
02/28 13:22:47.958 ERROR| traceback:0013| File "/usr/local/autotest/site_utils/lxc/utils.py", line 253, in sudo_commands
02/28 13:22:47.958 ERROR| traceback:0013| return utils.run(command)
02/28 13:22:47.958 ERROR| traceback:0013| File "/usr/local/autotest/client/common_lib/utils.py", line 748, in run
02/28 13:22:47.958 ERROR| traceback:0013| "Command returned non-zero exit status")
02/28 13:22:47.959 ERROR| traceback:0013| CmdError: Command <sudo bash -c 'umount '/usr/local/autotest/containers/host/container.53k5bm/usr/local/autotest/results/shared' && cd '('/usr/local/autotest/containers/host/container.53k5bm', u'usr/local/autotest/results/shared')'; rmdir -p '('/usr/local/autotest/containers/host/container.53k5bm', u'usr/local/autotest/results/shared')' || true'> failed, rc=1, Command returned non-zero exit status
02/28 13:22:47.960 ERROR| traceback:0013| * Command:
02/28 13:22:47.960 ERROR| traceback:0013| sudo bash -c 'umount '/usr/local/autotest/containers/host/container.53k5b
02/28 13:22:47.960 ERROR| traceback:0013| m/usr/local/autotest/results/shared' && cd
02/28 13:22:47.960 ERROR| traceback:0013| '('/usr/local/autotest/containers/host/container.53k5bm',
02/28 13:22:47.960 ERROR| traceback:0013| u'usr/local/autotest/results/shared')'; rmdir -p
02/28 13:22:47.960 ERROR| traceback:0013| '('/usr/local/autotest/containers/host/container.53k5bm',
02/28 13:22:47.960 ERROR| traceback:0013| u'usr/local/autotest/results/shared')' || true'
02/28 13:22:47.960 ERROR| traceback:0013| Exit status: 1
02/28 13:22:47.960 ERROR| traceback:0013| Duration: 6.41345977783e-05
02/28 13:22:47.960 ERROR| traceback:0013|
02/28 13:22:47.960 ERROR| traceback:0013| stderr:
02/28 13:22:47.960 ERROR| traceback:0013| /bin/bash: -c: line 0: syntax error near unexpected token `('
02/28 13:22:47.961 ERROR| traceback:0013| /bin/bash: -c: line 0: `sudo bash -c 'umount '/usr/local/autotest/containers/host/container.53k5bm/usr/local/autotest/results/shared' && cd '('/usr/local/autotest/containers/host/container.53k5bm', u'usr/local/autotest/results/shared')'; rmdir -p '('/usr/local/autotest/containers/host/container.53k5bm', u'usr/local/autotest/results/shared')' || true''
02/28 13:22:47.967 INFO | client:0570| Attempting refresh to obtain initial access_token
02/28 13:22:48.004 INFO | client:0872| Refreshing access_token
02/28 13:22:48.125 ERROR| autoserv:0759| Uncaught SystemExit with code 1
Traceback (most recent call last):
File "/usr/local/autotest/server/autoserv", line 755, in main
use_ssp)
File "/usr/local/autotest/server/autoserv", line 562, in run_autoserv
sys.exit(exit_code)
SystemExit: 1
02/28 13:22:48.185 DEBUG| ssh_multiplex:0122| Nuking ssh master_job
,
Feb 28 2018
Revert: https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/942099
,
Mar 1 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/23ad40061c34b1fa897ced4b3928f7f64ef2fefe commit 23ad40061c34b1fa897ced4b3928f7f64ef2fefe Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu Mar 01 17:45:07 2018 Revert "lxc: consolidate utils.py sudo calls for speed" This reverts commit 4adb9c72e49d4dc8365cea6235d61ede8f187b64. Reason for revert: Broke staging lab. Original change's description: > lxc: consolidate utils.py sudo calls for speed > > To aid this, add a helper function to combine commands into a single > `sudo bash -c` call. > > BUG=chromium:809306 > TEST=tryjob > > Change-Id: Ie678521a8801c072e5d208f17385a431ee78eb10 > Reviewed-on: https://chromium-review.googlesource.com/920925 > Commit-Ready: Jacob Kopczynski <jkop@chromium.org> > Tested-by: Ilja H. Friedel <ihf@chromium.org> > Reviewed-by: Jacob Kopczynski <jkop@chromium.org> BUG=chromium:809306 BUG= chromium:817569 Change-Id: I5c09baaaf349aec7e0648e4306e866651190564a Reviewed-on: https://chromium-review.googlesource.com/942099 Commit-Queue: Jacob Kopczynski <jkop@chromium.org> Tested-by: Jacob Kopczynski <jkop@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Jacob Kopczynski <jkop@chromium.org> [modify] https://crrev.com/23ad40061c34b1fa897ced4b3928f7f64ef2fefe/site_utils/lxc/utils.py
,
Mar 1 2018
Rolling LXC changes back out without causing this issue again is in progress.
,
Mar 2 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/6bee5a306df90f3a827b5c915a044d826486381a commit 6bee5a306df90f3a827b5c915a044d826486381a Author: Jacob Kopczynski <jkop@google.com> Date: Fri Mar 02 03:13:47 2018 lxc: Ignore rmdir failures at the command level Prerequisite to consolidating them for chromium:809306 without causing chromium:817569 to recur. BUG= chromium:817569 TEST=unittests and tryjob Change-Id: I318ff7d8218f49d5e293e4feba5d75c2c7f3d4ea Reviewed-on: https://chromium-review.googlesource.com/942601 Commit-Ready: Jacob Kopczynski <jkop@chromium.org> Tested-by: Jacob Kopczynski <jkop@chromium.org> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/6bee5a306df90f3a827b5c915a044d826486381a/site_utils/lxc/utils.py
,
Mar 6 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/0ddf0918cc925826ad3aab4bd1dec06621a02615 commit 0ddf0918cc925826ad3aab4bd1dec06621a02615 Author: Jacob Kopczynski <jkop@google.com> Date: Tue Mar 06 02:34:42 2018 lxc: consolidate and factor out sudos in utils Create a helper function to combine commands into a single `sudo bash -c` call. This change is relanding a reverted attempt, with less moving pieces. BUG=chromium:809306 BUG= chromium:817569 TEST=tryjob Change-Id: I0b9d36bbe3732bfb0ca2291ecd6f1e64c6438cee Reviewed-on: https://chromium-review.googlesource.com/947192 Commit-Ready: Jacob Kopczynski <jkop@chromium.org> Tested-by: Jacob Kopczynski <jkop@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/0ddf0918cc925826ad3aab4bd1dec06621a02615/site_utils/lxc/utils.py
,
Mar 8 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ebd19060fcfa21c983fba9d2a39e88a2819aebc5 commit ebd19060fcfa21c983fba9d2a39e88a2819aebc5 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu Mar 08 17:31:54 2018 Revert "lxc: consolidate and factor out sudos in utils" This reverts commit 0ddf0918cc925826ad3aab4bd1dec06621a02615. Reason for revert: Broke staging. Original change's description: > lxc: consolidate and factor out sudos in utils > > Create a helper function to combine commands into a single `sudo bash -c` call. > This change is relanding a reverted attempt, with less moving pieces. > > BUG=chromium:809306 > BUG= chromium:817569 > TEST=tryjob > > Change-Id: I0b9d36bbe3732bfb0ca2291ecd6f1e64c6438cee > Reviewed-on: https://chromium-review.googlesource.com/947192 > Commit-Ready: Jacob Kopczynski <jkop@chromium.org> > Tested-by: Jacob Kopczynski <jkop@chromium.org> > Reviewed-by: Ilja H. Friedel <ihf@chromium.org> > Reviewed-by: Dan Shi <dshi@google.com> BUG=chromium:809306, chromium:817569 BUG= chromium:820112 Change-Id: I671e64d6e38d68305d997298e320543533800fab Reviewed-on: https://chromium-review.googlesource.com/955938 Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/ebd19060fcfa21c983fba9d2a39e88a2819aebc5/site_utils/lxc/utils.py
,
Mar 8 2018
,
Mar 13 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/0d3cb3657dee5c652fcfc91c198e0350261fb969 commit 0d3cb3657dee5c652fcfc91c198e0350261fb969 Author: Jacob Kopczynski <jkop@google.com> Date: Tue Mar 13 14:37:08 2018 lxc: consolidate and factor out sudos in utils Create a helper function to combine commands into a single `sudo bash -c` call. This change is relanding a reverted attempt for the second time. BUG=chromium:809306 BUG= chromium:817569 TEST=tryjob Change-Id: Id5fcfe4379b9b325e127e04c0781bfffc510d06a Reviewed-on: https://chromium-review.googlesource.com/955888 Commit-Ready: Jacob Kopczynski <jkop@chromium.org> Tested-by: Jacob Kopczynski <jkop@chromium.org> Reviewed-by: Dan Shi <dshi@google.com> [modify] https://crrev.com/0d3cb3657dee5c652fcfc91c198e0350261fb969/site_utils/lxc/utils.py
,
Mar 13 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ec3a7e0588b83d92969a9aa742c0f0efec59038c commit ec3a7e0588b83d92969a9aa742c0f0efec59038c Author: Xixuan Wu <xixuan@chromium.org> Date: Tue Mar 13 19:26:40 2018 Revert "lxc: consolidate and factor out sudos in utils" This reverts commit 0d3cb3657dee5c652fcfc91c198e0350261fb969. Reason for revert: breaks staging lab ssp tests: crbug.com/821527 Original change's description: > lxc: consolidate and factor out sudos in utils > > Create a helper function to combine commands into a single `sudo bash -c` call. > This change is relanding a reverted attempt for the second time. > > BUG=chromium:809306 > BUG= chromium:817569 > TEST=tryjob > > Change-Id: Id5fcfe4379b9b325e127e04c0781bfffc510d06a > Reviewed-on: https://chromium-review.googlesource.com/955888 > Commit-Ready: Jacob Kopczynski <jkop@chromium.org> > Tested-by: Jacob Kopczynski <jkop@chromium.org> > Reviewed-by: Dan Shi <dshi@google.com> Bug: chromium:809306, chromium:817569 Change-Id: I47ff1f79b107c7964652411d03c14b67f198c991 Reviewed-on: https://chromium-review.googlesource.com/961342 Reviewed-by: Xixuan Wu <xixuan@chromium.org> Commit-Queue: Xixuan Wu <xixuan@chromium.org> Tested-by: Xixuan Wu <xixuan@chromium.org> [modify] https://crrev.com/ec3a7e0588b83d92969a9aa742c0f0efec59038c/site_utils/lxc/utils.py
,
Mar 15 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/52a8d3f292a77ae9bed502274fd59eca24983d58 commit 52a8d3f292a77ae9bed502274fd59eca24983d58 Author: Jacob Kopczynski <jkop@google.com> Date: Thu Mar 15 22:27:15 2018 autotest: lxc: factor out sudos in utils Create a helper function to combine commands into a single `sudo bash -c` call. This change is relanding a reverted attempt for the third time, with still fewer moving parts and a feature flag. Unlike previous attempts, this does not include any calls to the new method. BUG=chromium:809306 BUG= chromium:817569 BUG= chromium:821527 TEST=tryjob Change-Id: Icbc1632ec9a9898fd234c7261421dda2ef10bc89 Reviewed-on: https://chromium-review.googlesource.com/963515 Commit-Ready: Jacob Kopczynski <jkop@chromium.org> Tested-by: Jacob Kopczynski <jkop@chromium.org> Reviewed-by: Jacob Kopczynski <jkop@chromium.org> [modify] https://crrev.com/52a8d3f292a77ae9bed502274fd59eca24983d58/site_utils/lxc/utils.py [modify] https://crrev.com/52a8d3f292a77ae9bed502274fd59eca24983d58/global_config.ini
,
Apr 20 2018
,
Jun 6 2018
|
||||||
►
Sign in to add a comment |
||||||
Comment 1 by pprabhu@chromium.org
, Feb 28 2018