Need protocol for user task to understand its environment in swarming |
||||
Issue descriptionUse cases: * Copy CIPD package specs (i.e. subdir+pkg+tag) and envvar manipulations (add to PATH) in order to trigger a sub-task in swarming. * Copy CIPD package specs to set up matching environment in QEMU/VM/chroot environment. The first case will be somewhat (but not entirely) solved by the swarming task template stuff (when I finally finish that.......) The second case we have no solution for. Currently ChromeOS pins an (old) copy of depot_tools and relies on that to get vpython and cipd (!). Their chroot git and python versions are pinned separately from the fleet environment (which doesn't matter for python 2, but will probably start mattering again for python 3. Git MIGHT matter, but less likely to). Note that asking a tool (e.g. vpython) for its version will only tell the instance ID; if we need to deploy across architectures, this doesn't help. It's also pretty unsatisfying as supported APIs go. I'm thinking like writing out the input CIPD spec to a file somewhere and putting it in the env. This bug may depend on finishing issue 794657
,
Oct 19
Basically, we have a bunch of packages (The Environment) which buildbucket sets and passes to swarming. The job runs, and as part of its execution, it needs to pass The Environment to another entity (i.e. another swarming task, VM, QEMU machine, Docker container, etc.). Currently there's no API for the task code to figure pass the pertinent details on (i.e. "what CIPD tag of `git` should I use?"). Currently user tasks hard code these in one way or another, meaning that when we e.g. update git 'everywhere', it's really 'everywhere except where the user task hard-codes something else'.
,
Oct 19
ah ok good point.
I think about a formalized way to pass information, there's currently:
- ${SWARMING_BOT_FILE}; it's know to be used on Android
- LUCI_CONTEXT
- Hacks in on_before_task that changes the command line (rare but I know at least one)
This would potentially be a new way to pass information.
Do we want to generalize these into a single communication channel? I'm not sure, diverging to potentially find a disruptive solution here (You can see I'm back from a UX sprint :). LUCI_CONTEXT is special because it's dynamic, while others are static after the task started. So probably better to leave LUCI_CONTEXT alone. Still, the others could benefit (?)
cc Ben in case he has opinions.
Refs:
https://chromium.googlesource.com/infra/luci/luci-py.git/+/master/appengine/swarming/doc/Magic-Values.md
https://chromium.googlesource.com/infra/luci/luci-py.git/+/master/client/LUCI_CONTEXT.md
,
Oct 20
,
Oct 22
A general mechanism for bot-to-task communication would certainly be nice. That's kinda what I was going for w/ SWARMING_BOT_FILE, though we (chrome's tests) don't use that file anymore (some other work obviated it), so it's not an immediate need of ours.
,
Nov 1
|
||||
►
Sign in to add a comment |
||||
Comment 1 by mar...@chromium.org
, Oct 19