CommitQueueSync occasionally taking >2 hours |
||
Issue descriptionThis happened a few months ago (can't find the other bug just now) but sometimes the CommitQueueSync stage takes so long (>2 hours) that it ultimately causes the entire build to failure due to 4 hours timeout: https://luci-milo.appspot.com/buildbot/chromeos/cyan-paladin/6880 . This kills the entire run. The stage should normally take just <10 minutes. The log lines that are relevant are: 03:50:38: INFO: Applying via cherry-pick. Trying simple merge. [patch_branch 2f99c9086afa] CHROMIUM: net: xfrm: expose xfrm_acq_expires sysctl to namespaces Author: Hugo Benichi <hugobenichi@google.com> Date: Wed Aug 29 16:29:50 2018 +0900 1 file changed, 7 insertions(+), 3 deletions(-) 03:52:47: INFO: Attempting to apply change chromiumos/third_party/hdctools:refs/changes/48/1117748/14:bc37b24b "servod: introduce servodutil to manage servod instances easier." There's a 2 minute gap between the CL's getting downloaded. We need to dig in to the log messages and find out what is running between these lines.
,
Sep 20
,
Nov 7
Another instance: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8930564371845147008 This only happened on elm-paladin. Other paladin builds were fine at the same time.
,
Nov 9
More instances that eventually timed out and failed the CQ run. https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8930396941086306544 https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8930396924640861808 Also in the same CQ run, but didn't fail (due to no hwtests taking up time): https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8930396931562378832
,
Nov 19
Looks like this is happening with the current CQ run, for example these: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8929399710314515776 https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8929399641446341632 Have CommitQueueSync stages lasting nearly 3 hours (2 hrs 49 mins) while most of the others have long since finished and passed. |
||
►
Sign in to add a comment |
||
Comment 1 by jclinton@chromium.org
, Sep 20