Follow up. Based on the process list, I determined which was the hung process

$ ps -Af | grep deploy
akeshet   81232  46073  0 15:42 pts/25   00:00:00 /usr/bin/python ./automated_deploy.py --skip_autotest --skip_chromite
akeshet   81237  81232  0 15:42 pts/25   00:00:33 /usr/bin/python /usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site_utils/deploy_server.py --afe=cautotest
akeshet   84014  81237  0 15:47 pts/25   00:00:00 /bin/bash /usr/bin/googlesh -s -uchromeos-test -mchromeos-server71.cbf.corp.google.com /usr/local/autotest/site_utils/deploy_server_local.py 
akeshet  110388  80514  0 17:34 pts/18   00:00:00 grep --color=auto deploy


I went ahead and killed it
$ kill 84014

However, I don't see that a retry has been launched for it, and the automated_deploy script is still hung.
$ ps -Af | grep deploy
akeshet   81232  46073  0 15:42 pts/25   00:00:00 /usr/bin/python ./automated_deploy.py --skip_autotest --skip_chromite
akeshet   81237  81232  0 15:42 pts/25   00:00:34 /usr/bin/python /usr/local/google/home/akeshet/chromiumos/src/third_party/autotest/files/site_utils/deploy_server.py --afe=cautotest
akeshet  110689  80514  0 17:37 pts/18   00:00:00 grep --color=auto deploy

Also, strange that we have a huge quantity of un-reaped deploy_server processes.

deploy_server.p─┬─googlesh
                └─254*[{deploy_server.p}]



$ pstree 81232 -p
automated_deplo(81232)───deploy_server.p(81237)─┬─googlesh(84014)
                                                ├─{deploy_server.p}(81661)
                                                ├─{deploy_server.p}(81662)
                                                ├─{deploy_server.p}(81663)
                                                ├─{deploy_server.p}(81664)
                                                ├─{deploy_server.p}(81665)

Shameless plug: See a laundry list of deploy_server usability improvements at  issue 666101  from my deputy shift.

shuqianz@ writes:
I can't open my bug tracker because of some credential issue. So, I just reply in this email. 

When I retried the push, it got stuck at updating chromeos-server71.cbf again. So, I also canceled the push, and manually ran deploy on the skipped servers, which I found from the nagios alert. 

Then I ssh into the chromeos-server71.cbf, and manually ran deploy_server_local.py on this server. It hanged at running test_importer.py. I broke the deploy and tried to only run test_importer.py on that server.
$  /usr/local/autotest/utils/test_importer.py
It is running >1h30mins, and it is still running (not freeze) when I am sending this email. 

So, I think this issue has nothing to do with the deploy process. It is caused by the test_importer.py script taking too long. I've noticed that there are other servers taking about >30mins to update, I think it may be caused by the same issue.

Charlene

From the email sent by me about the fix on chromeos-server71.cbf:

For the chromeos-server71 server, I had tested deploy_server_local.py directly on the server and found out that the test_importer.py script was running super slow. That script took 2 hours to finish. I checked the code, it seemed that it was caused by the shard database was slow on that server. Therefore, I removed the shard from prod, cleaned up the shard database, and also cleaned up logs/ and results/ folder on that server. After all of these have been done, I kicked off another test_importer, this time it only took 5mins. 

Is it possible to add a timeout to the deployment script? Seems like <1hr should be a reasonably expectation.

Will add a timeput

After adding logging, and all other improvement. I think this issue is not applied to current deployment flow any more.