Create a tool for bisecting with swarming/isolate |
||||||
Issue descriptionIt would be awesome if we can have a tool to bisect a test failure using swarming and existing build artifacts from a given waterfall builder. i.e. pass it a waterfall link, starting & ending build number, and optionally extra build flags. I did this today to see when a failure with PlzNavigate started happening to one browser_test on Mac. I manually did this, and even though it was manual it took me only about 10 minutes which is amazing. Once it figures out which waterfall build the failure started it in, it can also drill down into all the revisions that went into it and see which caused it.
,
Mar 7 2017
,
Mar 7 2017
We were discussing this a while ago, and I think this is a great idea. I believe one of the concerns we had was how to keep the intermediate build artifacts (to gain the time advantage of incremental compilation) but I IIUC jam@'s post says that we can do that with isolate. I'll add this to my queue as I am looking for some work to do and this is right down my alley, but I'll discuss this with stgao@ and chanli@ to see if they have different opinions on this.
,
Mar 7 2017
To be clear, is this tool using swarming and existing build artifacts to find the first failed build and then drill down to a single revision as culprit? We have similar approaches in Findit (and Flake Analyzer). It's just we are using try jobs at revision level. To find the first failed build, 1. for reliable failures, Findit goes backwards one build by another to check if the failure happened, and it'll stop immediately if the failure didn't happen. 2. for flaky tests, Findit generates flakiness trend and identify the first build algorithmically.
,
Mar 8 2017
+jam@ Did you use swarming to both build the revisions after the last pass AND run the test?
,
Mar 16 2017
I used existing builds from the main waterfall builder (i.e. by specifying its isolate)
,
Apr 26 2017
,
Apr 26 2018
Issue has not been modified or commented on in the last 365 days, please re-open or file a new bug if this is still an issue. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by dpranke@chromium.org
, Mar 7 2017