New issue
Advanced search Search tips

Issue 888745 link

Starred by 2 users

Issue metadata

Status: Verified
Owner:
Closed: Sep 26
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Race condition in D-Bus / gRPC with cicerone/garcon

Project Member Reported by jkardatzke@chromium.org, Sep 24

Issue description

I've discovered a race condition in the existing design whereby calls can timeout due to a deadlock between threads (it does recover after the timeout, so the deadlock isn't permanent).

What happens is the D-Bus thread in cicerone gets a request from Chrome and then makes a gRPC call into garcon which then may require use of the D-Bus thread in garcon (the current example is getting the info for a Linux package file). From the garcon side, it will make gRPC calls back into cicerone when it notices certain changes in the filesystem, and these originate from the D-Bus thread in garcon which then does a gRPC call into cicerone which then will usually make a D-Bus call back to Chrome on the D-Bus thread....and if there is a call coming in the other direction at the same time we can deadlock there until timeouts occur.

After looking through the code more, the only case where this actually can happen is with the PackageInfo call. It's the only gRPC call that goes into garcon which blocks its return on something being executed on the D-Bus thread.

I think what may be the cleanest solution is to just have another thread in garcon so that the D-Bus thread isn't using the same message loop as the one which makes gRPC calls back to cicerone; then there won't be contention there.
 
Project Member

Comment 1 by bugdroid1@chromium.org, Sep 26

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/67c1c30d9a25a6c4aa79ddf6b1cdf88edd3e1bc1

commit 67c1c30d9a25a6c4aa79ddf6b1cdf88edd3e1bc1
Author: Jeffrey Kardatzke <jkardatzke@google.com>
Date: Wed Sep 26 17:32:35 2018

vm_tools: garcon: Add independent thread for D-Bus communication

This fixes the threading model in garcon to avoid deadlocks relating to
communication with cicerone. We add a 3rd thread which is for D-Bus
communciation with PackageKit. This prevents contention with blocking
operations from the host over gRPC which required use of the D-Bus
thread (that was also being used for file system watching, and could
then cause invocations back to the host, resulting in outbound
communication from the host timing out).

BUG= chromium:888745 
TEST=tast run vm.CrostiniStartEverything, manually verified as well

Change-Id: I24cd72cf541c86355a46c15eee9df54f8e3ed18d
Reviewed-on: https://chromium-review.googlesource.com/1244167
Commit-Ready: Jeffrey Kardatzke <jkardatzke@google.com>
Tested-by: Jeffrey Kardatzke <jkardatzke@google.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>

[modify] https://crrev.com/67c1c30d9a25a6c4aa79ddf6b1cdf88edd3e1bc1/vm_tools/garcon/package_kit_proxy.h
[modify] https://crrev.com/67c1c30d9a25a6c4aa79ddf6b1cdf88edd3e1bc1/vm_tools/garcon/host_notifier.cc
[modify] https://crrev.com/67c1c30d9a25a6c4aa79ddf6b1cdf88edd3e1bc1/vm_tools/garcon/main.cc
[modify] https://crrev.com/67c1c30d9a25a6c4aa79ddf6b1cdf88edd3e1bc1/vm_tools/garcon/package_kit_proxy.cc
[modify] https://crrev.com/67c1c30d9a25a6c4aa79ddf6b1cdf88edd3e1bc1/vm_tools/garcon/host_notifier.h

Status: Fixed (was: Assigned)

Sign in to add a comment