Yep, and as part of this it would be great to have support from recipe engine/MILO for showing concurrent steps (and their streaming logs) as well.

Having the engine manage these directly would also result in a lot less required machinery for client recipes (i.e. daemon controllers).

We'd also want to establish some basic protocol so that the script has the opportunity to return some information to the engine (probably as json) before going into the background. Something like --startup-output <fd|HANDLE>, which the engine will wait for the program to write to and close before returning control to the recipe.

I'd imagine an user-facing API something like:

  with api.step.background([normal step(...) args]) as output:
    # output is json written to the --startup-output handle.

+dnj for logdog awareness
+hinoka for milo awareness

This would also be useful for running things like the crash collector on windows and adb for android builds. I think vadimsh was also wanting something like this for authutil.

Probably the hardest part about this will be adding an internal registry of running background processes that the engine is aware of.

Some other potentially tricky things:
  * hooking up background script output to logdog
  * properly displaying steps in annotations/milo (including streaming logs from background processes)

We'll probably want to add a new BackgroundStepClient which is the raw engine hook for this (and wouldn't require the lexical scoping for more advanced use cases). I don't think it would be possible to implement cleanly in the current implementation of the engine without the use of daemonization tricks (double forking, pid files, etc.). One of the primary benefits of doing this in the engine is that it could be a normal subprocess fork that the engine can clean up; no need for pid files, etc.

LogDog output should be trivial, especially if we switch to recipe engine as the log provider and remove Annotee. Even if we don't, we can always just STEP_CURSOR over to the background task to dump its logs.

I don't think milo needs to be changed, just emit a few nested steps and mark multiples steps as running at the same time.  Unless you had a different UI in mind?

This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot