Issue metadata
Sign in to add a comment
|
Extend Gitiles trigger to support subpaths |
||||||||||||||||||||
Issue description
We would like to extend the Gitiles trigger to support paths, to only trigger a job when the specified path within the repository changes. This is useful for very large repositories containing number of subprojects where individual subprojects have their own builders. Example use case of this feature is:
trigger {
id: "gn-build"
acl_sets: "internal"
gitiles: {
repo: "https://chromium.googlesource.com/chromium/src"
refs: "refs/heads/master"
path: "tools/gn"
}
triggers: "gn-builder-linux"
triggers: "gn-builder-win"
triggers: "gn-builder-mac"
}
The Gitiles trigger uses Gitiles /+refs/$ref endpoint to find out if any of the refs have changed. When the path is specified, we'd use /+log/$old..$new/$path to find out if the path has changed ($old is stored in the datastore, $new is the result of /+refs/$ref).
,
Mar 28 2018
SGTM.
,
Mar 29 2018
We also have a use case for an "any path but" blacklist, maybe call it "ignore_path" if the whitelist is called "path", or make them "include"/"exclude". That is, if the only paths touched in the $old..$new range match an ignore_path pattern, then don't trigger. (This is for a repo that has useless mechanical commits more often than real changes.)
,
Jan 14
+1 I'm also running into an instance where a filepath filter in the trigger would be useful. (Right now my recipe is a cron that just no-ops if the filepath hasn't changed since the last time it ran. Would be nice to instead trigger on changes to that filepath.) Not actively blocking anything tho.
,
Jan 15
This is now critical to Chrome OS’s ParallelCQ effort because they need to watch a single file
,
Jan 15
,
Jan 15
OK, assigning my way.
,
Jan 15
I've briefly looked at what's easiest way to implement it, and there are two approaches: 1. What Petr wrote in #0, namely let gitiles do filter by using /+log/$old..$new/$path - requires checking corner cases of gitiles to ensure code around corner cases can stay the same (e.g., what happens on force push? merge commit (see also internal http://b/122868467). Still, with luck doable anyway, plug gitiles bugs (if any) should be fixed anyway ++ very efficient on scheduler side -- not flexible (but can't be abused!) - e.g., no blacklists. 2. Ask gitiles to return tree diff structure alongside each commit. ++ easy to implement ++ supports arbitrary filepath filtering - more load on gitiles, more network IO. - tree diff structure will increase memory usage, however given current scheduler limits of how many commits to look back, this is minor. == I'm leaning towards 2. Vadim et al, WDYT?
,
Jan 15
also, nodir@ re #c5: where/whom do I learn from exact requirement of CrOS? Is it just a "path: not-a-regexp" sufficient?
,
Jan 15
Just a path would be sufficient. the context is go/converge-browser-os-infra where a poller must watch changes of one manifest file. Note that they keep multiple manifests in one repo
,
Jan 15
Considering 2 is "easy to implement" and potentially more flexible, we can start with it and then reconsider if we see any performance issues.
,
Jan 18
(4 days ago)
I've thought through implementation and there are some limitations, so I documented them in our config proto and sent for review. PTAL && comment if you care about this feature: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/1423099 |
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by no...@chromium.org
, Mar 28 2018