integrate more trees into sheriff-o-matic
Project Member Reported by email@example.com, Sep 1 2014
V8 and memory teams want to use sheriff-o-matic. I think the steps involved for both of them are the same: 1. Change builder_alerts to crawl the appropriate waterfalls. Right now it crawls the ones in gatekeeper.json. We'll need to figure out what the right way to list the extra waterfalls is. Should we just hard-code into builder_alerts for now? Or we could add them to the gatekeeper.json, but I'm not sure that makes sense if gatekeeper isn't actually closing the tree for these waterfalls. Stip, WDYT? 2. Change the tree dropdown in the top-right corner of sheriff-o-matic to have the appropriate value for the tree in question. See https://codereview.chromium.org/508873005/, which will make it so that the tree name will be inferred from the master url for cases where the master isn't in gatekeeper_trees.json. If these waterfalls run any non-standard test suites or don't upload their test results to to test-results.appspot, then there will be more work needed in order to figure out failure reasons for a given step. But, even in those cases, it should correctly identify failures in those steps as whole step failures and get you regression ranges, etc correctly. builder_alerts code to modify: https://chromium.googlesource.com/infra/infra.git/+/master/infra/tools/builder_alerts/__main__.py sheriff-o-matic code to modify: https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/Tools/GardeningServer/model/ct-tree-list.html&sq=package:chromium&type=cs
Sep 2 2014,
is it an issue that the v8 waterfall (client.v8) runs on master4 (and not master1 like chromium and blink). Michael, does the layout test results upload work meanwhile, or is that still an issue (and is there a bug on file for this)?
Sep 3 2014,
Sep 3 2014,
The v8 layout tests are trybot like. In a discussion elsewhere, we agreed that it won't make much sense for them to be uploaded to the test results server (it might be different in the sheriff-o-matic context). As with all stand-alone v8 tests, we should upload those to the results server (added a sub-issue).
Sep 10 2014,
test-results.appspot is how builder_alerts breaks down step failures into lists of tests. FWIW, the landscape has changed a little here. We've started uploading some trybot results to test-results.appspot and plan on probably doing all of them. You should be able to get the layout tests uploading to work just by passing the right flags to run-webkit-tests. @jochen: I don't understand your question about masters. That all said, you can get sheriff-o-matic working without knowing which tests fail. It'll still work fine, just at the step level (i.e. it'll tell you when the step started failing and give you a regression range for that).
Sep 19 2014,
Oct 20 2014,
Skia would also like to use sheriff-o-matic: https://build.chromium.org/p/client.skia https://build.chromium.org/p/client.skia.android https://build.chromium.org/p/client.skia.compile https://build.chromium.org/p/client.skia.fyi https://chromegw.corp.google.com/i/client.skia.internal (if possible) We don't use gtest or any of the chromium test wrappers - at the moment we'd be happy just to get a link to the build page or log for the failed step.
Oct 22 2014,
Mar 19 2016,
This issue has been available for more than 365 days, and should be re-evaluated. Hotlist-Recharge-Cold label is added for tracking. Please re-triage this issue. For more details visit https://sites.google.com/a/chromium.org/dev/issue-tracking/autotriage - Your friendly Sheriffbot
Apr 27 2016,
Jun 29 2016,
Jun 29 2016,
Aug 3 2016,
Feb 10 2017,
Mar 7 2017,
May 1 2018,
This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue. Sorry for the inconvenience if the bug really should have been left as Available. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
May 1 2018,
Sign in to add a comment