Can you paste the commit# of your fix? 
xixuan@, can you take a look at this?

https://crrev.com/1bc79b267dd32dc69a0f6f4dc873de333841a50c

Also available in the linked bug at #11.

I cannot log into any servers. Due to log https://storage.cloud.google.com/chromeos-autotest-results/94501594-chromeos-test/chromeos4-row12-rack8-host3/debug/autoserv.DEBUG?_ga=1.21001784.2018250139.1482890129, It's devserver 100.115.219.134 to execute this failed provision. Deputy could check this devserver to see whether it's updated. If it's not (probably so), deputy could update all devservers with the newest chromite.

So reassign back to deputy :)

According to Kevin's latest push_to_prod, the change should have been pushed. 

chromite:
git log --oneline c2e9734..c559824
c559824 Use explicit virtualenv in virtualenv_wrapper


xixuan@, please paste the right steps to check the chromite version on devserver? I can help with checking.

Push-to-prod won't update devserver I think, only update drone&shard.

Log into the devserver and check the chromite directory, I don't remember where it exactly exists, but it should be at some very obvious path like ~/chromiumos/chromite. Check chromite/lib/remote_access.py, if it's not like the one in https://chromium-review.googlesource.com/#/c/413632/ and still has 'touch /tmp/awaiting_reboot && reboot' in it, this devserver's chromite is not updated well.

ok, I'll check.

The devserver chromite is behind the commit#, will update the devservers.

Updated the devservers, expect the following ones. 


"description": "Failed to update following devservers ['100.115.24.253', '172.25.65.106', '172.27.215.248', '172.27.215.252']",


shuqianz@, please advice how to update individual devservers.

The feature to support update individual devservers hasn't been added yet. CL https://chrome-internal-review.googlesource.com/#/c/310302/ is under review. For now, what you can do is to first debug why these devservers were fail to update, fix that and kick off another devserver update, which will update all devservers again. Most of the time, a given devserver failed to update because it was offline.  

Pass to current deputy, '100.115.24.253', '172.27.215.248', '172.27.215.252' are not ssh-able.  '172.25.65.106' failed to update because the virtualenv didn't setup properly on this server, and it fail to update virtualenv. 

Here is the error log of '172.25.65.106':
[172.25.65.106] run: git stash
[172.25.65.106] out: /bin/bash: line 0: cd: /usr/local/google/chromeos/infra_virtualenv: No such file or directory
[172.25.65.106] out: 


Fatal error: run() received nonzero return code 1 while executing!

Requested: git stash
Executed: /bin/bash -l -c "cd /usr/local/google/chromeos/infra_virtualenv >/dev/null && git stash"

Aborting.
ERROR:root:Traceback (most recent call last):
  File "/usr/local/google/home/shuqianz/chromiumos/chromeos-admin/server-management-lib/server_management_lib/tasks/atomic_common.py", line 113, in decorated_func
    func(self)
  File "/usr/local/google/home/shuqianz/chromiumos/chromeos-admin/server-management-lib/server_management_lib/tasks/atomic_devserver.py", line 163, in run
    self._update_all_devservers()
  File "/usr/local/google/home/shuqianz/chromiumos/chromeos-admin/server-management-lib/server_management_lib/tasks/atomic_devserver.py", line 157, in _update_all_devservers
    'Failed to update following devservers %s' % fail_devservers)
TaskRunFailure: Failed to update following devservers ['172.25.65.106']

ERROR:root:Failed to update following devservers ['172.25.65.106']
INFO:root:Printing out task report.
{
  "sub_reports": [],
  "exception": "TaskRunFailure(\"Failed to update following devservers ['172.25.65.106']\",)",
  "is_successful": false,
  "description": "Failed to update following devservers ['172.25.65.106']",
  "arguments_used": {
    "update_devserver_list": [
      "172.25.65.106"
    ]
  },
  "task_name": "DevserverPushTask"

cc Allen to take a look of this virualenv problem.

172.27.215.248 is the only machine from above which is still offline. I'll follow up on why.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/chromeos-admin/+/6641de3fec1d6ce188683d8293b30737d1d043f2

commit 6641de3fec1d6ce188683d8293b30737d1d043f2
Author: Allen Li <ayatane@chromium.org>
Date: Wed Jan 11 20:59:32 2017

I filed b/34225314 to cover the machine that is still down.

I'm going to assume that these fixes have been properly pushed by now.