New issue
Advanced search Search tips

Issue 868590 link

Starred by 1 user

Issue metadata

Status: Untriaged
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

goma should ensure the server is started when trying to build

Project Member Reported by dpranke@chromium.org, Jul 27

Issue description

If goma is enabled, and you try to do a build but the server isn't running, then you will get inscrutable errors from ninja.

It seems like we should be able to figure out a way to have goma reliably auto-start itself.

One obvious way would be to do this in autoninja.py, by calling `goma_ctl.py ensure_start`, but we don't require everyone to use autoninja, and it's not obvious how we could require that (or if it would really be desirable).

A second (probably preferable) way would be for the goma client wrapper itself to call ensure_start. I think we've tried this in the past and seen issues, but I'm not sure if we know for sure that we *can't* make this work reliably.

Any other thoughts / options?
 
Yes, this would be great. Currently if you fail to run the server then you may get about a thousand errors.

Having the goma client start the server would be great. There would need to be an extremely quick way to see if the server was running and then a robust way to find out where goma is installed, although I guess gomacc can infer that from its own install path.

The trick is that we need just one of the gomacc.exe processes to start the goma server (and perhaps just one to check for whether it is running, given the cost of that). I think that a global named event could be used for that - the first process would create it and the others would get a handle to it and would then wait on it.

Goma used to do that at some point. I think it printed "goma not running; starting" from all processes and made them block until startup was done. That wasn't an awesome experience.

I don't have an opinion on putting it in autoninja, but I'm reasonably confident that most people don't use that.

imho nice to have, but not super important, given that we tried to do it, it didn't reallyw ork out, and things seem to be ok. *shrug*
I think you can try it with GOMA_START_COMPILER_PROXY=true for gomacc.
it was not so great experience as thakis@ stated.

I think it would be nice to put it in autoninja (but not so many people use it...), or add support in ninja to run some command at startup?
#3
I think it can be done to add such function in current ninja wrapper.
https://cs.chromium.org/chromium/tools/depot_tools/ninja
why we don't put autoninja feature in the wrapper, then?
From my point of view, I enforce autoninja's feature including NINJA_SUMMARIZE_BUILD to users and let them know how much time is taken in the build.
But that is too radical?

I don't think we should do this in ninja or ninja's wrapper. In the limit that leads to running build/install-build-deps.sh there too -- it seems like confusion of responsibilities to me.
> imho nice to have, but not super important, given that we tried to do it,
> it didn't reallyw ork out, and things seem to be ok. *shrug*

I'd probably describe this as "maybe tolerable", not "ok". This causes really confusing errors for the user, and I believe lots of users hit this.

I'd like to understand the "didn't really work out" part better. This doesn't seem like an unsolvable problem, and I'm not sure what the downside of blocking the individual goma processes is, assuming the check to see if goma is started or not can be done fast enough (I'm guessing this could be done on the order of time it takes to stat() something).

I agree we shouldn't try to push this into ninja itself. 


I think it would not be good idea to put such feature in ninja wrapper in depot_tools. it would be confusing there is difference between native ninja vs depot_tools ninja.

maybe, we should promote using autoninja in chromium (update docs to use autoninja instead of ninja, etc) and fix autoninja to start compiler_proxy if it is not started, or add goma specific feature?

for "didn't really work out", https://chrome-internal-review.googlesource.com/c/goma/client/+/260255
hmm no BUG= in CL description.  
we should invoke goma_ctl.py instead of directly run compiler_proxy?
I understand that we might be starting 10s or 1000s of goma processes in a small amount of time, all of which would then try to stat the file (or whatever) to start things up, but I still feel like this should be a relatively solvable problem?

I guess I don't know exactly what we saw before, but it looks like --start_compiler_proxy should still work, so I could test it out and see?
Re: #9
Just FYI, I searched our mailing list and found something related.
PSA on stop supporting automatic start compiler_proxy from gomacc.
https://goto.google.com/ziueb

Several reasons why we recommend to start compiler_proxy with goma_ctl.
https://goto.google.com/zhdgx

How past gomacc behavior was ugly?
https://goto.google.com/wijxq

One of the issues we decided to stop auto start feature.
b/28142555

List of things what other people do:
- ChromeOS simple chrome automatically download & starts compiler_proxy when cros command is executed.
- Android make automatically execute "goma_ctl.py ensure_start" if USE_GOMA is true.
- fuchsia silently execute "goma_ctl.py ensure_start" when fx set is executed.

Executing "goma_ctl.py ensure_start" in autoninja seems to be something similar to what other folks do, and I believe it works well if people don't try to execute autoninja from two or more different locations at once.

Thanks for the context in #c11. I still don't really understand why this seems hard to make work properly, but I'm happy to change autoninja.py for now and assign this as a low priority thing for myself to look at some day, rather than taking up any more of the goma team's time on it.

Sign in to add a comment