Identify PFQ failures due to master timeouts |
||||||||||
Issue descriptionIf a PFQ run is delayed or takes an unusually long time, it may reach a hard deadline imposed by the PFQ master. The symptom for this is simply "ERROR: Timoeout occured- waited X seconds, failing" in whichever stage(s) are running. We should identify the actual cause of failure and make sure that is what gets reported to the PFQ master as the cause.
,
Mar 8 2016
Thanks, beat me to it! Note: that master run had several such failures because it appears to have been started while the previous run was in progress, so several builders had to complete the previous run before starting the new run (I think). We have seen this symptom in the past, but not recently. I will keep an eye out for better examples.
,
Mar 8 2016
,
Mar 9 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/446f07f5f4affde7a683127f7c6bef2b9208e697 commit 446f07f5f4affde7a683127f7c6bef2b9208e697 Author: Aviv Keshet <akeshet@chromium.org> Date: Tue Mar 08 19:32:31 2016 timeout_util: log when timeouts were due to master deadline BUG= chromium:593089 TEST=unit tests Change-Id: I753ea798fec1e5d1a60044f10834e26123a54b61 Reviewed-on: https://chromium-review.googlesource.com/331680 Commit-Ready: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Steven Bennetts <stevenjb@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/446f07f5f4affde7a683127f7c6bef2b9208e697/scripts/cbuildbot.py [modify] https://crrev.com/446f07f5f4affde7a683127f7c6bef2b9208e697/lib/timeout_util.py
,
Apr 23 2016
Can we consider this fixed after #4? Or further work has to be done?
,
Apr 25 2016
Feel free to resolve this as fixed if it is currently reasonably addressed. If there is more work that we should do, add a comment here to that effect. If there is more work that we -could- do but that is lower priority, we should file that separately and resolve this.
,
May 10 2016
,
Nov 14 2016
,
Jan 21 2017
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Aug 3 2017
Closing. Please reopen it if its not fixed. Thanks! |
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by akes...@chromium.org
, Mar 8 2016