recipes failure on multiple bots |
|||||||
Issue descriptionNot sure what's going on. Seeing output like this: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.linux%2Flinux_chromium_chromeos_ozone_rel_ng%2F317276%2F%2B%2Frecipes%2Fstderr on multiply try jobs for this CL: https://codereview.chromium.org/2668153003 This has been happening for > 24 hours
,
Feb 8 2017
,
Feb 8 2017
probably not. recipe_engine.stream.StreamEngineInvariants.StepStream.write_line breaks on strings that have a newline character. https://cs.chromium.org/chromium/infra/recipes-py/recipe_engine/stream.py?l=296 but recipes.py calls write_line with failure human_reason without escaping https://chromium.googlesource.com/external/github.com/luci/recipes-py/+blame/9456147555cc76adec014709e02928b620d7d8d9/recipes.py#107
,
Feb 8 2017
fix landed 23min ago https://codereview.chromium.org/2688563002/ and got rolled into build.git https://chromium.googlesource.com/chromium/tools/build/+/master/infra/config/recipes.cfg#14 please verify
,
Feb 8 2017
Well, the error message seems to have disappeared: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.linux%2Flinux_chromium_chromeos_rel_ng%2F361033%2F%2B%2Frecipes%2Fstderr but there still seems to be a related failure here: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_rel_ng/builds/361033 That build was started after comment #4 was posted and appears to have used build rev b3326718 which includes the recipes roll. Unsure if it's actually the same bug. Thoughts?
,
Feb 8 2017
This needs more attention. Still seeing failures for unclear reasons, on many CLs (not just mine). Another example: https://build.chromium.org/p/tryserver.chromium.linux/builders/chromeos_amd64-generic_chromium_compile_only_ng/builds/278845
,
Feb 8 2017
,
Feb 8 2017
Actually in all cases I think perhaps the new problem is: "Failing build because ninja reported work to do." So perhaps there's a separate bug in that errors are not being surfaced in any visible way, so they all look like the same generic "steps" failure and you have to be lucky enough to find the meaningful cause in thousands upon thousands of log lines. Either way, sounds like that's a separate issue, and the original bug here has been fixed.
,
Feb 9 2017
There's a real issue here with how the "Failing build because ninja reported work to do" problem is being handled. I'll follow up on that in bog 690269.
,
Feb 9 2017
bug 690269 :) |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by no...@chromium.org
, Feb 8 2017Owner: no...@chromium.org
Status: Started (was: Untriaged)