That CL is listed in every case other than the first one, which suggests that CL is innocent.

And it's not in the most recent run (which hung), which proves that CL is innocent.

So far, there is not clear sign of what is hanging.

Actually, the most recent run has not hung, it was just waiting. I'm going to unblock that CL to see what happens.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/5dcabd7b96608ea7fb6f66c5b0323bfeaf217bee

commit 5dcabd7b96608ea7fb6f66c5b0323bfeaf217bee
Author: Ningning Xia <nxia@chromium.org>
Date: Thu Nov 03 00:32:31 2016

Add info logs to debug pre-cq-launcher hangs.

Will remove the temp log when the issue is identified.

BUG= chromium:661704 
TEST=run_tests

Change-Id: If2df6a19ca7f0250aecc60b1ed08db1577a9d4d5
Reviewed-on: https://chromium-review.googlesource.com/406504
Reviewed-by: Don Garrett <dgarrett@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>

[modify] https://crrev.com/5dcabd7b96608ea7fb6f66c5b0323bfeaf217bee/cbuildbot/validation_pool.py

Based on the new logging, it's not hanging where nxia@ and I thought, but happening after we finish filtering changes.

23:24:00: INFO: Checking for change zhihongyu:*302116:*79be8abf
23:24:00: INFO: Found manifest change zhihongyu:*302116:*79be8abf
23:24:00: INFO: Checking for change zhihongyu:*302155:*4c07918e
23:24:00: INFO: Found manifest change zhihongyu:*302155:*4c07918e
01:49:10: WARNING: Killing tasks: [<_BackgroundTask(_BackgroundTask-9662, started)>]
01:49:10: WARNING: Killing 15687 (sig=24 SIGXCPU)
01:49:10: WARNING: RunCommand: pstree -Apals 15687

basically it hung when it tried to connect to the internal-CROS GOB server. 

  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 232, in FetchUrlJson
    fh = FetchUrl(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 224, in FetchUrl
    _FetchUrlHelper, sleep=SLEEP)
  File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 155, in RetryException
    return GenericRetry(exc_retry, max_retry, functor, *args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/retry_util.py", line 105, in GenericRetry
    ret = functor(*args, **kwargs)
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 166, in _FetchUrlHelper
    body=body)
  File "/b/cbuild/internal_master/chromite/lib/gob_util.py", line 137, in CreateHttpConn
    conn.request(**conn.req_params)

probably should add a timeout wrapper around the retry

Yep.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/66fab2d49030c9744fe26f03dbfadd95225a2ca5

commit 66fab2d49030c9744fe26f03dbfadd95225a2ca5
Author: Don Garrett <dgarrett@google.com>
Date: Mon Feb 27 20:03:27 2017

gob_util: Add 2 minute timeout to GOB requests.

If Gerrit On Borg is having issues, then web requests can hang. Add a
timeout to make sure we don't wait forever in these cases.

Also, fix timeout_util.TimeoutDecorator, and add unittests to verify
that fix.

BUG= chromium:661704 
TEST=run_tests

Change-Id: I6ba3c1d3d525b78a1e097ca2505f56115eaf7bd6
Reviewed-on: https://chromium-review.googlesource.com/446035
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/66fab2d49030c9744fe26f03dbfadd95225a2ca5/lib/timeout_util_unittest.py
[modify] https://crrev.com/66fab2d49030c9744fe26f03dbfadd95225a2ca5/lib/gob_util_unittest.py
[modify] https://crrev.com/66fab2d49030c9744fe26f03dbfadd95225a2ca5/lib/timeout_util.py
[modify] https://crrev.com/66fab2d49030c9744fe26f03dbfadd95225a2ca5/lib/gob_util.py