Is your experience of this bug that it's nondeterministic? I just built and ran content_shell several times on 7771ddcf9 and it started fine. I don't see any obvious missing manifest entries.

Yes, definitely non deterministic. Usually just running it again immediately afterwards works fine. But I've definitely ran into it at least once a day for a while noe.

I've been unable to repro this failure. AFAICT the wiring up is correct:

- Device Service exposes this here: https://cs.chromium.org/chromium/src/services/device/manifest.json?type=cs&q=device::mojom::PowerMonitor&sq=package:chromium&l=17
- GPU service requires it here:
https://cs.chromium.org/chromium/src/content/public/app/mojo/content_gpu_manifest.json?q=device:power_monitor&sq=package:chromium&dr=C&l=28

Note that content_shell doesn't have an overlay for the GPU service.

The only thing that I can think of is a race condition between the request being received and the Service Manager having fully initialized all of the relevant state from the manifests; however, I assume that the setup process would be designed to eliminate any such race conditions.

The GPU Service makes the request via PowerMonitorBroadcastSource here:
https://cs.chromium.org/chromium/src/content/child/child_thread_impl.cc?q=child_thread_impl.cc&sq=package:chromium&dr&l=526

Ken, can the idea of a race condition be ruled out? If so, any other ideas on what could be happening?

Race is unlikely. If there were a race it would have to be such that two ServiceManagers get created, with one of them not loading the manifest data. That probably isn't happening.

There is definitely no race between SM getting its manifest data and resolving incoming requests. All manifest data is loaded completely by the time this line[1] is reached. The ServiceManager isn't even created until the line after that.

I have a local script spam starting/killing content_shell and can reliably repro within a few minutes. I'll do some digging to see how this is possible.

[1] https://cs.chromium.org/chromium/src/content/browser/service_manager/service_manager_context.cc?rcl=f8589e89ed37bc1a807d3e88556a8bab4172859a&l=344

Looks like a bug in how the Service Manager decides where to get the interface spec from.

For some reason it does this[1], meaning that if an interface request reaches the service manager after the requesting service instance has terminated, the SM will still attempt to broker the request using an empty interface spec, which will in turn always fail.

Easy to fix.

[1] https://cs.chromium.org/chromium/src/services/service_manager/service_manager.cc?rcl=519cd7b8de887663b849cfaff1144900cd0de953&l=263

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6e9c25680f8898e1530b62d6c84b1cb536a24c9f

commit 6e9c25680f8898e1530b62d6c84b1cb536a24c9f
Author: Ken Rockot <rockot@chromium.org>
Date: Tue Jan 16 18:00:55 2018

Fix interface brokering for dead source instances

When the Service Manager processes an incoming interface request it gets
the source instance's (i.e. the requestor's) interface provider spec from
the corresponding running Instance object. If said Instance happens to
have become unreachable (i.e. it still exists but has begun to coordinate
clean shutdown) by the time one of its requests is dispatched and
processed, the Service Manager -- for some reason -- barrels ahead using
an empty interface spec which is guaranteed to fail all checks.

Instead the Service Manager should simply terminate the operation early,
reply to the dying instance who will ignore it anyway, and let the
request pipe be cast into oblivion.

This CL does that, and also updates some Bind callsites for extra credit.

Note that while this fixes flaky crashes in DCHECK builds, the effects
of the bug are otherwise innocuous (extra line of log spam) in
production.

Bug:  801618 
Change-Id: I3ac7280577d569e7bedc55723a974c79d161996f
Reviewed-on: https://chromium-review.googlesource.com/868033
Reviewed-by: Jay Civelli <jcivelli@chromium.org>
Commit-Queue: Ken Rockot <rockot@chromium.org>
Cr-Commit-Position: refs/heads/master@{#529459}
[modify] https://crrev.com/6e9c25680f8898e1530b62d6c84b1cb536a24c9f/services/service_manager/runner/host/service_process_launcher.cc
[modify] https://crrev.com/6e9c25680f8898e1530b62d6c84b1cb536a24c9f/services/service_manager/runner/host/service_process_launcher.h
[modify] https://crrev.com/6e9c25680f8898e1530b62d6c84b1cb536a24c9f/services/service_manager/service_manager.cc

\o/