New issue
Advanced search Search tips

Issue 923016 link

Starred by 1 user

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Consider being smarter about DEPS-roll changes when it comes to analyze

Project Member Reported by bpastene@chromium.org, Jan 17 (5 days ago)

Issue description

For most CLs, we try to limit what targets get compiled and what tests get run to only those affected by the change in question. For changes that touch DEPS however, we skip that analyze logic and instead run *all* the things:
https://chromium.googlesource.com/chromium/src/+/master/testing/buildbot/trybot_analyze_config.json#25

Consider all the autorollers that are continuously updating pins in chromium:
https://chromium.googlesource.com/chromium/src/+log/master/DEPS

Each one of those updates is (at the very least) one entire CQ run. Given that we're consistently running into capacity problems on the CQ (goma & test & even builder pool), I wonder how much unneeded drain on our resources these rolls are.

Gclient has a "diff" command that will theoretically provide the list of files changed in a sub-repo during a DEPS-roll. We could pass this list onto our analyze logic and remove the exclusion of DEPS (at least for rolls... any change to hooks will likely have to stay as exclusions). That way, DEPS rolls build and test only the targets that it would affect.

First I gotta get a good feel for how much of a drain deps rolls are on our capacity. Might have to plug into  gerrit's rest API for the full list of roll patches, since it doesn't appear possible to correlate the contents of a change with our CQ/buildbucket/swarming tables.
 

Comment 1 by bpastene@chromium.org, Jan 18 (4 days ago)

Ran some numbers and it appears DEPS-roll CLs from the two autoroller users (gerrit query is "o:chromium-autoroll@skia-public.iam.gserviceaccount.com AND file:DEPS OR o:chromium-internal-autoroll@skia-corp.google.com.iam.gserviceaccount.com AND file:DEPS") account for about 5-10% of builder-time per day. That spikes to about 20-30% on weekends (prob because ppl stop using the CQ on weekends, but rollers keep rolling).

I think that warrants further investigation. I'll play around with some rolls in the past few days, and see what their true diff would have triggered on the CQ if DEPS wasn't analyze-excluded.

Sign in to add a comment