Provisioning broken with 'Cannot run an array command with a shell' |
|||||
Issue description
IMPORTANT: Infra deputies: Do not do a devserver update until this issue is resolved or provisioning will likely break.
The following change in chromite seems to break devserver autoupdates:
2415126 remote_access: enforce shell=True on shell commands
The error shown (Exception modified for debugging purposes) when trying to do an update to cros_au devserver RPC call is:
DEBUG:root:Error happens in CrOS auto-update: Exception('Cannot run an array command with a shell: [\'ssh\', \'-p\', \'22\', \'-oConnectionAttempts=4\', \'-oUserKnownHostsFile=/dev/null\', \'-oProtocol=2\', \'-oConnectTimeout=30\', \'-oServerAliveCountMax=3\', \'-oStrictHostKeyChecking=no\', \'-oServerAliveInterval=10\', \'-oNumberOfPasswordPrompts=0\', \'-oIdentitiesOnly=yes\', \'-i\', \'/tmp/ssh-tmpspeCc4/testing_rsa\', \'root@192.168.86.118\', \'--\', \'FILE="/mnt/stateful_partition/.labmachine" ; [ -f "$FILE" ] || ( touch "$FILE" ; start autoreboot )\']',)
Traceback (most recent call last):
File "/mnt/host/source/src/platform/dev/cros_update.py", line 329, in <module>
main()
File "/mnt/host/source/src/platform/dev/cros_update.py", line 321, in main
cros_updater_trigger.TriggerAU()
File "/mnt/host/source/src/platform/dev/cros_update.py", line 266, in TriggerAU
chromeos_AU.PostCheckCrOSUpdate()
File "/mnt/host/source/chromite/lib/auto_updater.py", line 1359, in PostCheckCrOSUpdate
shell=True, **self._cmd_kwargs)
File "/mnt/host/source/chromite/lib/auto_updater.py", line 1193, in _RetryCommand
cmd, delay_sec=DELAY_SEC_FOR_RETRY, **kwargs)
File "/mnt/host/source/chromite/lib/retry_util.py", line 272, in RetryException
return _run()
File "/mnt/host/source/chromite/lib/retry_util.py", line 177, in _Wrapper
ret = func(*args, **kwargs)
File "/mnt/host/source/chromite/lib/retry_util.py", line 271, in _run
return functor(*args, **kwargs)
File "/mnt/host/source/chromite/lib/remote_access.py", line 1115, in RunCommand
return super(ChromiumOSDevice, self).RunCommand(cmd, **kwargs)
File "/mnt/host/source/chromite/lib/remote_access.py", line 959, in RunCommand
return self.BaseRunCommand(cmd, **kwargs)
File "/mnt/host/source/chromite/lib/remote_access.py", line 906, in BaseRunCommand
return self.GetAgent().RemoteSh(cmd, **kwargs)
File "/mnt/host/source/chromite/lib/remote_access.py", line 341, in RemoteSh
return cros_build_lib.RunCommand(ssh_cmd, **kwargs)
File "/mnt/host/source/chromite/lib/cros_build_lib.py", line 560, in RunCommand
raise Exception('Cannot run an array command with a shell: %s' % repr(cmd))
Exception: Cannot run an array command with a shell: ['ssh', '-p', '22', '-oConnectionAttempts=4', '-oUserKnownHostsFile=/dev/null', '-oProtocol=2', '-oConnectTimeout=30', '-oServerAliveCountMax=3', '-oStrictHostKeyChecking=no', '-oServerAliveInterval=10', '-oNumberOfPasswordPrompts=0', '-oIdentitiesOnly=yes', '-i', '/tmp/ssh-tmpspeCc4/testing_rsa', 'root@192.168.86.118', '--', 'FILE="/mnt/stateful_partition/.labmachine" ; [ -f "$FILE" ] || ( touch "$FILE" ; start autoreboot )']
::ffff:192.168.86.3 - - [18/Oct/2017:13:02:48] "GET /get_au_status?full_update=False&force_update=True&build_name=cyan-paladin/R64-10040.0.0-rc3&host_name=192.168.86.118&async=True&clobber_stateful=True&quick_provision=True&pid=25641 HTTP/1.1" 200 2034 "" "curl/7.35.0"
Reverting the following changes in chromite seems to fix the issue: e7dc3d57cbebdd9866e9d488990c70f7b13ccd59 2415126c2943b21af0d9a25e2e640b4359406bfd
Reproduction:
cros_sdk
cd ~/trunk/src/platform/dev
./devserver.py --production --port 8080 --static_dir /var/lib/devserver/static/
ds=okaybye2.mtv.corp.google.com:8080
build=cyan-paladin/R64-10040.0.0-rc1
dut=XYZ
curl "http://${ds}/stage?archive_url=gs://chromeos-image-archive/${build}&artifacts=full_payload,stateful,autotest_packages"
curl "http://${ds}/cros_au?full_update=False&force_update=True&build_name=${build}&host_name=${dut}&async=False&clobber_stateful=True"
,
Oct 18 2017
i think https://chromium-review.googlesource.com/725287 includes a fix for this, but we prob want to split that out by itself
,
Oct 20 2017
^ looks like it isn't going to land in a hurry. We are unable to push new devserver code until this is fixed. Can you split ^ and land the smaller fix now?
,
Oct 23 2017
,
Oct 23 2017
We have a devserver push waiting behind this.
,
Oct 23 2017
Can we just revert https://chromium-review.googlesource.com/c/chromiumos/chromite/+/697065 ? does that actually fix this?
,
Oct 23 2017
I tested locally and you need to revert this one too: https://chromium-review.googlesource.com/#/c/chromiumos/chromite/+/714024/
,
Oct 23 2017
dhaddock since you are able to reproduce and test locally, can you just upload the reverts. I'll happily +2.
,
Oct 23 2017
Yeah as I stated in the original you need both to actually cleanly revert.
,
Oct 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/c7cecc736c92a2f8ce2570df44a0157f0dad6ba6 commit c7cecc736c92a2f8ce2570df44a0157f0dad6ba6 Author: David Haddock <dhaddock@chromium.org> Date: Tue Oct 24 16:17:12 2017 Revert "deploy_chrome: convert umount logic away from shell=True" This reverts commit e7dc3d57cbebdd9866e9d488990c70f7b13ccd59. It has broken the devserver provisioning. 1/2 BUG= chromium:776099 Change-Id: I7fc910c63d774779a0116b78a924bbc0aa3756e3 Reviewed-on: https://chromium-review.googlesource.com/734901 Reviewed-by: Richard Barnette <jrbarnette@google.com> Commit-Queue: David Haddock <dhaddock@chromium.org> Tested-by: David Haddock <dhaddock@chromium.org> [modify] https://crrev.com/c7cecc736c92a2f8ce2570df44a0157f0dad6ba6/scripts/deploy_chrome.py
,
Oct 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/3151d91dd5d0ca5e834be96f825191e251775434 commit 3151d91dd5d0ca5e834be96f825191e251775434 Author: David Haddock <dhaddock@chromium.org> Date: Tue Oct 24 16:23:16 2017 Revert "remote_access: enforce shell=True on shell commands" This reverts commit 2415126c2943b21af0d9a25e2e640b4359406bfd. 2/2 BUG= chromium:776099 Change-Id: I7c13bc56398cc204afbc35eef898ce7f6f36f90d Reviewed-on: https://chromium-review.googlesource.com/734981 Reviewed-by: Richard Barnette <jrbarnette@google.com> Tested-by: David Haddock <dhaddock@chromium.org> Trybot-Ready: David Haddock <dhaddock@chromium.org> [modify] https://crrev.com/3151d91dd5d0ca5e834be96f825191e251775434/scripts/deploy_chrome_unittest.py [modify] https://crrev.com/3151d91dd5d0ca5e834be96f825191e251775434/lib/remote_access.py [modify] https://crrev.com/3151d91dd5d0ca5e834be96f825191e251775434/lib/remote_access_unittest.py [modify] https://crrev.com/3151d91dd5d0ca5e834be96f825191e251775434/lib/auto_updater.py [modify] https://crrev.com/3151d91dd5d0ca5e834be96f825191e251775434/scripts/deploy_chrome.py
,
Oct 24 2017
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by akes...@chromium.org
, Oct 18 2017