Nothing elm specific here, I see this on many other devices as well, and on many versions, all the way back to 5500.100.7.

But the only three instances on elm were all from the same device on version 8872.22.0.

This probably means that two copies of powerd are running at the same time. It's unrelated to  issue 493726  (note that that one is a dbus-daemon segfault, not a powerd abort -- the powerd abort there is intentional and caused by dbus-daemon dying).

powerd starts on "started boot-services" and stops on "stopping boot-services", so it should never restart unless it crashes. Maybe there can be a race (under heavy load or on certain hardware?) where the new powerd process tries to register itself with dbus-daemon before dbus-daemon has realized the old one is gone. hashimoto@ or satorux@, have you ever seen this happen?

The logs that you linked support this theory. In 566ba40700000000, the last entry from the previous log was written just two seconds before the crashing process started:

===powerd.20161025-102602 (tail)===
...
[1025/102753:INFO:daemon.cc(1319)] Saw user activity
[1025/102758:INFO:daemon.cc(1319)] Saw user activity
EOF
===powerd.20161025-102800 (tail)===
[1025/102800:INFO:main.cc(310)] vcsid 0.0.2-r1563-05dfd21b41959276abfbf1a01439b078b95b6336
[1025/102800:ERROR:bus.cc(576)] Failed to get the ownership of org.chromium.PowerManager:
[1025/102800:FATAL:daemon.cc(932)] Check failed: dbus_wrapper_->PublishService(). Failed to publish D-Bus service
EOF

These crashes are harmless, assuming that Upstart doesn't give up on starting powerd before the D-Bus name becomes available, but it'd probably be cleaner to make powerd retry taking ownership of the name a few times before aborting.

Alternately, powerd could use REQUIRE_PRIMARY_ALLOW_REPLACEMENT rather than REQUIRE_PRIMARY when first acquiring the name. I believe that this would allow another powerd process to steal ownership of the name.

We definitely don't want to be running two functional copies of powerd at the same time, but if this just being triggered by dbus-daemon not having released an old, dead process's name yet, it'd probably be safe. org.chromium.PowerManager.conf restricts ownership of the "org.chromium.PowerManager" name to the "power" user, so I don't think there are security concerns around doing this.

libbrillo/brillo/daemons/dbus_daemon.cc also uses REQUIRE_PRIMARY, so I'm a bit surprised if we aren't seeing this with other daemons.

I ran "while true; do killall -9 powerd; sleep 4; done" for ~15 minutes on a samus device but wasn't able to cause any SIGABRTs. (A shorter delay is likely to trigger the Upstart job's respawn limit.)

Just as an aside, searching for the stable signature ("raise-01af9528") is probably more accurate, since it ought to just match PublishService() failures rather than all crashes within InitDBus(). The latter search also includes  issue 493726 , which powerd can't work around (the system is unusable if dbus-daemon isn't running). The number of powerd ABRTs with raise-01af9528 is much lower.

> Maybe there can be a race (under heavy load or on certain hardware?) where the new powerd process tries to register itself with dbus-daemon before dbus-daemon has realized the old one is gone. hashimoto@ or satorux@, have you ever seen this happen?

No I've never seen that.

I'm uneasy about using REQUIRE_PRIMARY_ALLOW_REPLACEMENT, since we really don't want to run two copies of powerd simultaneously and it feels like that would give up one (maybe our only?) guarantee of atomicity.

Another option might be giving the powerd job a post-stop stanza with a loop that waits for a command like the following to fail:

dbus-send --system --type=method_call --print-reply \
  --dest=org.freedesktop.DBus \
  /org/freedesktop/DBus \
  org.freedesktop.DBus.GetNameOwner \
  string:org.chromium.PowerManager

That should guarantee that dbus-daemon no longer believes the name is owned.

I'm still unable to repro this. With the powerd Upstart job modified locally to check whether the name is still owned in post-stop, I've still never seen this happen. I've tried both on a very fast device (samus) and a very slow one (cyan image running in a VM).

So I guess I could check in any of a variety of potential workarounds, but I'm not convinced it'll help. Could these crashes just be caused by people messing around on devices that are in dev mode?

@#8 ... quite possibly!  I just confirmed that the three crashes on elm are from the same manual tester device.

That same device also generated this feedback report, about 2 days after the crashes:

https://feedback.corp.google.com/product/208/neutron?lView=rd&lReport=14809536349

Maybe kalin remembers what was happening on the device when powerd was crashing on 10/25.

Please reopen if/when there's data suggesting that this isn't due to multiple copies of powerd being manually run.