New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 776099 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Oct 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

Provisioning broken with 'Cannot run an array command with a shell'

Project Member Reported by davidri...@chromium.org, Oct 18 2017

Issue description

IMPORTANT: Infra deputies: Do not do a devserver update until this issue is resolved or provisioning will likely break.

The following change in chromite seems to break devserver autoupdates:
2415126 remote_access: enforce shell=True on shell commands

The error shown (Exception modified for debugging purposes) when trying to do an update to cros_au devserver RPC call is:
DEBUG:root:Error happens in CrOS auto-update: Exception('Cannot run an array command with a shell: [\'ssh\', \'-p\', \'22\', \'-oConnectionAttempts=4\', \'-oUserKnownHostsFile=/dev/null\', \'-oProtocol=2\', \'-oConnectTimeout=30\', \'-oServerAliveCountMax=3\', \'-oStrictHostKeyChecking=no\', \'-oServerAliveInterval=10\', \'-oNumberOfPasswordPrompts=0\', \'-oIdentitiesOnly=yes\', \'-i\', \'/tmp/ssh-tmpspeCc4/testing_rsa\', \'root@192.168.86.118\', \'--\', \'FILE="/mnt/stateful_partition/.labmachine" ; [ -f "$FILE" ] || ( touch "$FILE" ; start autoreboot )\']',)
Traceback (most recent call last):
  File "/mnt/host/source/src/platform/dev/cros_update.py", line 329, in <module>
    main()
  File "/mnt/host/source/src/platform/dev/cros_update.py", line 321, in main
    cros_updater_trigger.TriggerAU()
  File "/mnt/host/source/src/platform/dev/cros_update.py", line 266, in TriggerAU
    chromeos_AU.PostCheckCrOSUpdate()
  File "/mnt/host/source/chromite/lib/auto_updater.py", line 1359, in PostCheckCrOSUpdate
    shell=True, **self._cmd_kwargs)
  File "/mnt/host/source/chromite/lib/auto_updater.py", line 1193, in _RetryCommand
    cmd, delay_sec=DELAY_SEC_FOR_RETRY, **kwargs)
  File "/mnt/host/source/chromite/lib/retry_util.py", line 272, in RetryException
    return _run()
  File "/mnt/host/source/chromite/lib/retry_util.py", line 177, in _Wrapper
    ret = func(*args, **kwargs)
  File "/mnt/host/source/chromite/lib/retry_util.py", line 271, in _run
    return functor(*args, **kwargs)
  File "/mnt/host/source/chromite/lib/remote_access.py", line 1115, in RunCommand
    return super(ChromiumOSDevice, self).RunCommand(cmd, **kwargs)
  File "/mnt/host/source/chromite/lib/remote_access.py", line 959, in RunCommand
    return self.BaseRunCommand(cmd, **kwargs)
  File "/mnt/host/source/chromite/lib/remote_access.py", line 906, in BaseRunCommand
    return self.GetAgent().RemoteSh(cmd, **kwargs)
  File "/mnt/host/source/chromite/lib/remote_access.py", line 341, in RemoteSh
    return cros_build_lib.RunCommand(ssh_cmd, **kwargs)
  File "/mnt/host/source/chromite/lib/cros_build_lib.py", line 560, in RunCommand
    raise Exception('Cannot run an array command with a shell: %s' % repr(cmd))
Exception: Cannot run an array command with a shell: ['ssh', '-p', '22', '-oConnectionAttempts=4', '-oUserKnownHostsFile=/dev/null', '-oProtocol=2', '-oConnectTimeout=30', '-oServerAliveCountMax=3', '-oStrictHostKeyChecking=no', '-oServerAliveInterval=10', '-oNumberOfPasswordPrompts=0', '-oIdentitiesOnly=yes', '-i', '/tmp/ssh-tmpspeCc4/testing_rsa', 'root@192.168.86.118', '--', 'FILE="/mnt/stateful_partition/.labmachine" ; [ -f "$FILE" ] || ( touch "$FILE" ; start autoreboot )']
::ffff:192.168.86.3 - - [18/Oct/2017:13:02:48] "GET /get_au_status?full_update=False&force_update=True&build_name=cyan-paladin/R64-10040.0.0-rc3&host_name=192.168.86.118&async=True&clobber_stateful=True&quick_provision=True&pid=25641 HTTP/1.1" 200 2034 "" "curl/7.35.0"

Reverting the following changes in chromite seems to fix the issue: e7dc3d57cbebdd9866e9d488990c70f7b13ccd59 2415126c2943b21af0d9a25e2e640b4359406bfd


Reproduction:
cros_sdk
cd ~/trunk/src/platform/dev
./devserver.py --production --port 8080 --static_dir /var/lib/devserver/static/
ds=okaybye2.mtv.corp.google.com:8080
build=cyan-paladin/R64-10040.0.0-rc1
dut=XYZ
curl "http://${ds}/stage?archive_url=gs://chromeos-image-archive/${build}&artifacts=full_payload,stateful,autotest_packages"
curl "http://${ds}/cros_au?full_update=False&force_update=True&build_name=${build}&host_name=${dut}&async=False&clobber_stateful=True"

 
Cc: shuqianz@chromium.org
+shuqianz did staging test catch this? Are we testing staging devservers yet?

Comment 2 by vapier@chromium.org, Oct 18 2017

i think https://chromium-review.googlesource.com/725287 includes a fix for this, but we prob want to split that out by itself
^ looks like it isn't going to land in a hurry. We are unable to push new devserver code until this is fixed. Can you split ^ and land the smaller fix now?
Cc: dhadd...@chromium.org
Cc: ayatane@chromium.org jrbarnette@chromium.org
We have a devserver push waiting behind this.
Can we just revert https://chromium-review.googlesource.com/c/chromiumos/chromite/+/697065 ?

does that actually fix this?
I tested locally and you need to revert this one too:
https://chromium-review.googlesource.com/#/c/chromiumos/chromite/+/714024/

Owner: dhadd...@chromium.org
Status: Assigned (was: Untriaged)
dhaddock since you are able to reproduce and test locally, can you just upload the reverts. I'll happily +2.
Yeah as I stated in the original you need both to actually cleanly revert.
Project Member

Comment 11 by bugdroid1@chromium.org, Oct 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/c7cecc736c92a2f8ce2570df44a0157f0dad6ba6

commit c7cecc736c92a2f8ce2570df44a0157f0dad6ba6
Author: David Haddock <dhaddock@chromium.org>
Date: Tue Oct 24 16:17:12 2017

Revert "deploy_chrome: convert umount logic away from shell=True"

This reverts commit e7dc3d57cbebdd9866e9d488990c70f7b13ccd59.

It has broken the devserver provisioning.

1/2 

BUG= chromium:776099 

Change-Id: I7fc910c63d774779a0116b78a924bbc0aa3756e3
Reviewed-on: https://chromium-review.googlesource.com/734901
Reviewed-by: Richard Barnette <jrbarnette@google.com>
Commit-Queue: David Haddock <dhaddock@chromium.org>
Tested-by: David Haddock <dhaddock@chromium.org>

[modify] https://crrev.com/c7cecc736c92a2f8ce2570df44a0157f0dad6ba6/scripts/deploy_chrome.py

Status: Fixed (was: Assigned)

Sign in to add a comment