I think in pinpoint something like #1 would be really hard, but we'd be able to have a nicer web interface for selecting benchmarks so maybe it's okay to go without?
Annie, what I meant for 1 is that it's manually specified by the code reviewers/author. Basically the work flow is s.t like:
Code author add modify benchmarks U & V
Code reviewer edit the description to add: CQ_EXTRA_TRYBOTS=benchmarkU:benchmarkV
Code reviewer can be sure that the change do not breaks benchmark U & V because now the CQ will run those two.
AFAIK that's what will be difficult, we won't have cq perf bots on the waterfall anymore. We'd need some new recipe or something that could communicate with Pinpoint to retain that functionality.
A couple of problems here:
1) CQ_EXTRA_TRYBOTS is a special format supported by infra, IIUC it only allows you to specify bot names of the trybots you want, not config like benchmark names.
2) When we migrate to pinpoint, there won't be "bots" in the same way, it'll be an app engine service, so the existing CQ_EXTRA_TRYBOTS functionality supported by infra won't work.
I might be misunderstanding something.
Currently, the cq perf trybots work by specifying CQ_EXTRA_TRYBOTS in the cl, which in turn sends the patch to the specified bots. These bots are on tryserver.chromium.perf waterfall, and are running the bisect recipe which can change functionality based on the incoming parameters/patch. From there its relatively straightforward, they request a build, analyze the patch a bit to get a list of benchmarks to run, and then run a straightforward perf try job.
But in the future Pinpoint world, there won't be bisect bots on the waterfall anymore. There will be an Appengine service communicating directly with swarming.
Ah got it. So it seems like we currently don't have the ability to parameterize the tasks of the bots from the infra.
It's fine to remove CQ_EXTRA_TRYBOTS for now as the false negative rate is too high.
+Dirk: is it possible to do things that I describe in #3?
No, it's not currently possible to do something like what you're asking for in comment #3.
I think it would be good if we could, but we're a ways away from being able to do that.
Just to confirm, from #c8 it sounds like you're on board then for just removing CQ_EXTRA_TRYBOTS then since it doesn't sound feasible to do the suggestions in #c3?
Owner: simonhatch@chromium.org Status: Started (was: Untriaged)
So I think the parts here are:
* Remove the code from PRESUBMIT that adds CQ_EXTRA_TRYBOTS
* Remove the *_cq bots from tryserver.chromium.perf and reclaim the hardware.
* Restart tryserver.chromium.perf
* Tear out the cq recipe code.
* Cleanup references to *_cq bots (such as in https://cs.chromium.org/chromium/src/tools/perf/core/trybot_command.py?l=43)
Sorry maybe that original description isn't quite right. Going forward, reviewers would request that they run a perf try job and link the successful run.
./tools/perf/run_benchmark tryjob android_nexus9_perf_bisect foo.benchmark
The *_cq bots wouldn't be needed, only the existing bisect bots.
It looks like tryserver.chromium.perf contains bisect bots and cq bots; if you remove the cq bots, does that mean you'll kick off jobs using the bisect bots?
(Sorry, just trying to understand the pieces here).
Comment 1 by nedngu...@google.com
, Jan 20 2017