Also, deploy_server is hanging repeatedly in updating several servers. Currently, there is no easy way to say which servers are still being processed. If we separate out the logs from deploy_server_local, we should start pinging the status of "still processing" servers in the main script to know what servers we're waiting for still.

We should also leave a log of deploy_server_local.py on the server where it ran. So that we can go look at what happened for that server, **on that server**.

This seems like a very good fixit candidate for fixit week.

Summarize a discussion IRL:

- We would like to add a required argument --log-dir to deploy_server. Logs from the remote deploy_server_local.py scripts will be collected into files in this directory *on the server where deploy_server is run*. This matches current behaviour, and avoids having to SSH into servers to find out what's wrong (e.g., you want to grep *all logs* from certain error message)

- Once the logs are separated out, the main deploy_server log will become quite empty (yay!). Sometimes, deploy_server_local can take hours (see  issue 673585 ), and it's not good to have the user stare at a blank screen for >1 hour.

Instead, deploy_server should use multiprocessing.Pool's apply_async method to parallelize the deploy_server_local runs, and poll for completion every 1 minute, logging the servers which are still running every minute. This is a far more useful UX.

good summary :)

Erm.... Did you fix it? I don't see any CLs here :)

I just added the label, the CL is waiting for your review

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/9febd45bbbcaba56b66cd4b1febcb85f8b5bb591

commit 9febd45bbbcaba56b66cd4b1febcb85f8b5bb591
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Wed Feb 08 04:14:32 2017

[autotest] Separate the logs deployment logs

Instead flush all the logs into console, write the logs of every server
into a log file in a single thread. In this way, we can save the log for
debug purpose and all the logs will not be mixed up.

BUG= chromium:666101 
TEST=unittest and test in a test server.

Change-Id: Iaf7b303f0c3b0c9346c5346052ba07c79d2fb990
Reviewed-on: https://chromium-review.googlesource.com/435503
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/9febd45bbbcaba56b66cd4b1febcb85f8b5bb591/site_utils/lib/infra.py
[modify] https://crrev.com/9febd45bbbcaba56b66cd4b1febcb85f8b5bb591/site_utils/deploy_server.py
[modify] https://crrev.com/9febd45bbbcaba56b66cd4b1febcb85f8b5bb591/site_utils/deploy_server_local.py
[modify] https://crrev.com/9febd45bbbcaba56b66cd4b1febcb85f8b5bb591/site_utils/deploy_server_unittest.py
[modify] https://crrev.com/9febd45bbbcaba56b66cd4b1febcb85f8b5bb591/site_utils/deploy_server_local_unittest.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/fad50674321c90591cbaf35a3565398d4184d45d

commit fad50674321c90591cbaf35a3565398d4184d45d
Author: Shuqian Zhao <shuqianz@chromium.org>
Date: Thu Feb 09 00:24:28 2017

[autotest] Check deployment progress every 1 min to print unfinished servers

Poll the deployment progress every one minute and logging the servers
which are still running.

BUG= chromium:666101 
TEST=unittest and test on testing servers

Change-Id: Ibc1194507da45c8073ee0a2d383a4b3324ee8560
Reviewed-on: https://chromium-review.googlesource.com/436350
Commit-Ready: Shuqian Zhao <shuqianz@chromium.org>
Tested-by: Shuqian Zhao <shuqianz@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/fad50674321c90591cbaf35a3565398d4184d45d/site_utils/deploy_server.py