Note: I suspect this is a regression caused by one of the recent changes from  Issue 705051 .

Follow-up: my analysis is probably wrong -- it looks like those changes really were CrOS-specific -- but need help from people who work on dbus to understand what changes might have introduced this flakiness.

Don't see any changes in our infrastructure around dbus. cc-ing a sys admin who made puppet changes recently.

I don't think this was an Infra-related change; martiniss@, cc'd you mainly because your CL was affected by this flakiness. I'm 99% sure a recent Chromium code change caused these intermittent failures.

Yes, Ben's changes were Chrome-OS-specific (and also to a system daemon rather than Chrome itself).

I don't see any recent changes in //dbus. I don't know much about D-Bus use on desktop Linux, but by grepping for "CallMethod" in *_unittest.cc, I see code in //chrome/browser/password_manager, //components/os_crypt, //device/battery, and //device/geolocation. Most of those don't have recent changes either, but password_manager has a few.

Maybe there's some change to the scheduler that made the task be posted to an unexpected thread? I can't see from the backtrace what sort of thread it's actually run on.

To confirm, does the above code expect to run on a named BrowserThread?

In either case, the code in question is expecting a MessageLoop. The TaskScheduler does not provide one (since you're supposed to post to the TaskScheduler or getting the current task runner handle instead of getting the loop).

We did enable redirection recently, but that would only matter if the above code expects to be running in a named Browser Thread. I'm attempting to chase it down, but it seems like most callers create their own DBus thread to handle this.

Mostly it runs on a specified dbus thread, but it looks like it sometimes just runs on the message loop of whatever created the object. For example with PowerSaveBlocker::Delegate::ApplyBlock I think it tries to create it on the FILE thread?

That would certainly do it. I'll start a revert.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f07732ec0e39ad7e9f1ad279f2624dbdf261803d

commit f07732ec0e39ad7e9f1ad279f2624dbdf261803d
Author: robliao <robliao@chromium.org>
Date: Tue Apr 04 00:33:33 2017

Revert of RedirectNonUINonIOBrowserThreads to TaskScheduler by default on trunk. (patchset #3 id:40001 of https://codereview.chromium.org/2690183002/ )

Reason for revert:
Breaks some DBUS code that requests for a message loop

Original issue's description:
> RedirectNonUINonIOBrowserThreads to TaskScheduler by default on trunk.
>
> The addition of AssertWaitAllowed in BrowserMainLoop::ShutdownThreadsAndCleanUp()
> is required because joining is a "wait" operation. It wasn't required
> before because PlatformThread::Join (platform_thread_posix.cc) actually still
> does AssertIOAllowed() instead of AssertWaitAllowed() for legacy reasons
> (and I/O is allowed in shutdown already). But this CL enables the redirection
> which uses a WaitableEvent to mimic thread join (ref. BrowserThreadImpl::StopRedirectionOfThreadID():
> waiting on the exact same set of tasks it did prior to redirection).
>
> BUG= 653916 
>
> Review-Url: https://codereview.chromium.org/2690183002
> Cr-Commit-Position: refs/heads/master@{#461481}
> Committed: https://chromium.googlesource.com/chromium/src/+/b6d0c9a06ab573af760763fcd7ecf76ae2c87695

TBR=jam@chromium.org,rkaplow@chromium.org,gab@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 653916 ,  707986 

Review-Url: https://codereview.chromium.org/2791933004
Cr-Commit-Position: refs/heads/master@{#461588}

[modify] https://crrev.com/f07732ec0e39ad7e9f1ad279f2624dbdf261803d/base/threading/thread_restrictions.h
[modify] https://crrev.com/f07732ec0e39ad7e9f1ad279f2624dbdf261803d/content/browser/browser_main_loop.cc
[modify] https://crrev.com/f07732ec0e39ad7e9f1ad279f2624dbdf261803d/testing/variations/fieldtrial_testing_config.json

Thanks John for the essential diagnosis uncovering the problem and Rob for your quick action!

Thanks, any chance a test can be added to catch this use case on CQ?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3e27cb6e501ae55d614b56d566487590518ec223

commit 3e27cb6e501ae55d614b56d566487590518ec223
Author: robliao <robliao@chromium.org>
Date: Mon May 01 18:57:05 2017

Reland RedirectNonUINonIOBrowserThreads to TaskScheduler by default on trunk

This reverts commit f07732ec0e39ad7e9f1ad279f2624dbdf261803d
(https://codereview.chromium.org/2791933004)
and reapplies b6d0c9a06ab573af760763fcd7ecf76ae2c87695
(https://codereview.chromium.org/2690183002/).

BUG= 653916 ,  707986 
TBR=jam@chromium.org, rkaplow@chromium.org
Previously reviewed at https://codereview.chromium.org/2690183002/

Review-Url: https://codereview.chromium.org/2856583002
Cr-Commit-Position: refs/heads/master@{#468376}

[modify] https://crrev.com/3e27cb6e501ae55d614b56d566487590518ec223/base/threading/thread_restrictions.h
[modify] https://crrev.com/3e27cb6e501ae55d614b56d566487590518ec223/content/browser/browser_main_loop.cc
[modify] https://crrev.com/3e27cb6e501ae55d614b56d566487590518ec223/testing/variations/fieldtrial_testing_config.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d4be08b81e51bbf512152939dce9955133f74864

commit d4be08b81e51bbf512152939dce9955133f74864
Author: robliao <robliao@chromium.org>
Date: Mon May 01 23:32:37 2017

Revert of Reland RedirectNonUINonIOBrowserThreads to TaskScheduler by default on trunk (patchset #1 id:1 of https://codereview.chromium.org/2856583002/ )

Reason for revert:
Possible failure: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.chromiumos%2FLinux_ChromiumOS_Tests__dbg__1_%2F25521%2F%2B%2Frecipes%2Fsteps%2Fbrowser_tests%2F0%2Flogs%2FCaptivePortalWindowTest.OnRedirected%2F0

Original issue's description:
> Reland RedirectNonUINonIOBrowserThreads to TaskScheduler by default on trunk
>
> This reverts commit f07732ec0e39ad7e9f1ad279f2624dbdf261803d
> (https://codereview.chromium.org/2791933004)
> and reapplies b6d0c9a06ab573af760763fcd7ecf76ae2c87695
> (https://codereview.chromium.org/2690183002/).
>
> BUG= 653916 ,  707986 
> TBR=jam@chromium.org, rkaplow@chromium.org
> Previously reviewed at https://codereview.chromium.org/2690183002/
>
> Review-Url: https://codereview.chromium.org/2856583002
> Cr-Commit-Position: refs/heads/master@{#468376}
> Committed: https://chromium.googlesource.com/chromium/src/+/3e27cb6e501ae55d614b56d566487590518ec223

TBR=jam@chromium.org,rkaplow@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 653916 ,  707986 

Review-Url: https://codereview.chromium.org/2848413002
Cr-Commit-Position: refs/heads/master@{#468481}

[modify] https://crrev.com/d4be08b81e51bbf512152939dce9955133f74864/base/threading/thread_restrictions.h
[modify] https://crrev.com/d4be08b81e51bbf512152939dce9955133f74864/content/browser/browser_main_loop.cc
[modify] https://crrev.com/d4be08b81e51bbf512152939dce9955133f74864/testing/variations/fieldtrial_testing_config.json

Fixed by virtue of http://crbug.com/708142