New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 713947 link

Starred by 5 users

Issue metadata

Status: Fixed
Owner:
Closed: May 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 1
Type: Bug


Sign in to add a comment

Occasional DBUS hang running Chrome on Linux

Project Member Reported by rbyers@chromium.org, Apr 21 2017

Issue description

We have a persistent issue using Chrome on Linux for automated testing with Chrome hanging on startup.  

See  issue 309093  and https://bugs.chromium.org/p/chromedriver/issues/detail?id=1699 for examples and detailed investigation pointing at this glib bug: https://bugzilla.gnome.org/show_bug.cgi?id=738620

We (devtools / headless Chrome team?) should drive getting some permanent fix for Chrome (fixing GLib and/or landing a workaround into chromium) so this stops biting people every time someone tries to automate Chrome in some new environment.

 

Comment 1 by rbyers@chromium.org, Apr 21 2017

Blocking: chromedriver:1699

Comment 2 by kbr@chromium.org, Apr 21 2017

Blockedon: 309093
Cc: thomasanderson@chromium.org
Can you grab a stack trace of the UI thread during the hang?

Comment 5 by caseq@chromium.org, Apr 23 2017

Components: -Platform>DevTools
Removing devtools component as I fail to see how it is related to DevTools. Rick, in case this wasn't accidental, please feel free to bring it back.

Comment 6 by m...@bocoup.com, Apr 24 2017

> Removing devtools component as I fail to see how it is related to DevTools.
> Rick, in case this wasn't accidental, please feel free to bring it back.

Rick was likely acting on my based on my input. I was interpreting the logs at
https://bugs.chromium.org/p/chromedriver/issues/detail?id=1699#c2 to indicate
a problem with the communication between ChromeDriver and DevTools. Sounds like
I may have presumed too much, though.

Comment 7 by kbr@chromium.org, Apr 26 2017

Blockedon: 695643

Comment 8 by kbr@chromium.org, Apr 26 2017

Blocking: 715658

Comment 9 by rbyers@chromium.org, Apr 26 2017

Blocking: chromedriver:1620
Project Member

Comment 10 by bugdroid1@chromium.org, May 8 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8511820ec8280caacbd4f81f3ecd13b6c61681b0

commit 8511820ec8280caacbd4f81f3ecd13b6c61681b0
Author: primiano <primiano@chromium.org>
Date: Mon May 08 15:09:51 2017

Linux: Disable DBus auto-launch

This is a workaround (ETA ~ 2-3 years) for libdbus not being multi-threading
friendly and causing random hangs when running chrome outside of Linux
desktop environments.

Background:
-----------
Typically, Linux desktop environments set the DBUS_SESSION_BUS_ADDRESS
environment variable. This variable allows the dbus client library to
directly connect to the existing bus, which is started by the desktop
environment or systemd.
When this variable is missing, the dbus client library will fallback
to auto-launch mode [1], which causes 4 nested fork() + exec() calls.
Doing this has two problems: (i) slows down startup; (ii) can hang
the browser if the fork() happens while another thread is in a malloc()
(Chrome's tcmalloc has no at-fork handlers).
This situation (no env variable) is very common in test scenarios
(browsertests, chromedriver, etc).

Change introduced by this CL:
-----------------------------
This CL sets the bus address env variable to "disabled:" if not set.
This effectively shuts down the dbus auto-launch. If necessary, this
behavior can be restored by setting, before launching chrome,
DBUS_SESSION_BUS_ADDRESS="autolaunch:" .
This workaround will be necessary until libdbus and gspawn are fixed
to be multi-threading friendly [2,3] and that fix rolls into the
various distributions.
The change is introduced in the main embedder rather than in the
google-chrome wrapper, as several binaries can be affected by this,
for instance:
- browser tests (http://crbug.com/693668)
- chrome --headless
- webdriver/selenium which seem to directly invoke "chrome"
   see https://github.com/SeleniumHQ/docker-selenium/issues/87

[1] https://dbus.freedesktop.org/doc/dbus-launch.1.html
[2] https://bugs.freedesktop.org/show_bug.cgi?id=100843
[3] https://bugs.chromium.org/p/chromedriver/issues/detail?id=1699

BUG= 715658 , 695643 , 713947 
TEST=strace -ff -o trace chrome; grep dbus-launch trace*

Review-Url: https://codereview.chromium.org/2861163002
Cr-Commit-Position: refs/heads/master@{#469987}

[modify] https://crrev.com/8511820ec8280caacbd4f81f3ecd13b6c61681b0/services/service_manager/embedder/main.cc

Project Member

Comment 11 by bugdroid1@chromium.org, May 8 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1e78cb7863da28bb3411286cdbcc4fb4510ce173

commit 1e78cb7863da28bb3411286cdbcc4fb4510ce173
Author: iclelland <iclelland@chromium.org>
Date: Mon May 08 18:41:31 2017

Revert of Linux: Disable DBus auto-launch (patchset #1 id:1 of https://codereview.chromium.org/2861163002/ )

Reason for revert:
Speculative revert -- the TSan bots have been reporting a data race when setting Envvars (in this case, appending to the python path to start a websocket server). The race appeared immediately after this patch landed, so it may be legitimate. Reverting this to see if it clears the failures up; if so, we'll probably just have to serialize the calls to setenv.

Filed  crbug.com/719633  for this as well.

Original issue's description:
> Linux: Disable DBus auto-launch
>
> This is a workaround (ETA ~ 2-3 years) for libdbus not being multi-threading
> friendly and causing random hangs when running chrome outside of Linux
> desktop environments.
>
> Background:
> -----------
> Typically, Linux desktop environments set the DBUS_SESSION_BUS_ADDRESS
> environment variable. This variable allows the dbus client library to
> directly connect to the existing bus, which is started by the desktop
> environment or systemd.
> When this variable is missing, the dbus client library will fallback
> to auto-launch mode [1], which causes 4 nested fork() + exec() calls.
> Doing this has two problems: (i) slows down startup; (ii) can hang
> the browser if the fork() happens while another thread is in a malloc()
> (Chrome's tcmalloc has no at-fork handlers).
> This situation (no env variable) is very common in test scenarios
> (browsertests, chromedriver, etc).
>
> Change introduced by this CL:
> -----------------------------
> This CL sets the bus address env variable to "disabled:" if not set.
> This effectively shuts down the dbus auto-launch. If necessary, this
> behavior can be restored by setting, before launching chrome,
> DBUS_SESSION_BUS_ADDRESS="autolaunch:" .
> This workaround will be necessary until libdbus and gspawn are fixed
> to be multi-threading friendly [2,3] and that fix rolls into the
> various distributions.
> The change is introduced in the main embedder rather than in the
> google-chrome wrapper, as several binaries can be affected by this,
> for instance:
> - browser tests (http://crbug.com/693668)
> - chrome --headless
> - webdriver/selenium which seem to directly invoke "chrome"
>    see https://github.com/SeleniumHQ/docker-selenium/issues/87
>
> [1] https://dbus.freedesktop.org/doc/dbus-launch.1.html
> [2] https://bugs.freedesktop.org/show_bug.cgi?id=100843
> [3] https://bugs.chromium.org/p/chromedriver/issues/detail?id=1699
>
> BUG= 715658 , 695643 , 713947 
> TEST=strace -ff -o trace chrome; grep dbus-launch trace*
>
> Review-Url: https://codereview.chromium.org/2861163002
> Cr-Commit-Position: refs/heads/master@{#469987}
> Committed: https://chromium.googlesource.com/chromium/src/+/8511820ec8280caacbd4f81f3ecd13b6c61681b0

TBR=satorux@google.com,thestig@chromium.org,jam@chromium.org,satorux@chromium.org,primiano@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG= 715658 , 695643 , 713947 

Review-Url: https://codereview.chromium.org/2869843003
Cr-Commit-Position: refs/heads/master@{#470059}

[modify] https://crrev.com/1e78cb7863da28bb3411286cdbcc4fb4510ce173/services/service_manager/embedder/main.cc

Project Member

Comment 12 by bugdroid1@chromium.org, May 9 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2fc330d0b93d4bfd7bd04b9fdd3102e529901f91

commit 2fc330d0b93d4bfd7bd04b9fdd3102e529901f91
Author: primiano <primiano@chromium.org>
Date: Tue May 09 13:07:12 2017

Reland of Linux: Disable DBus auto-launch (patchset #1 id:1 of https://codereview.chromium.org/2869843003/ )

Reason for reland:
Adding TSan suppression. The race is independent of this CL, see  crbug.com/719633 

Original issue's description:
> Revert of Linux: Disable DBus auto-launch (patchset #1 id:1 of https://codereview.chromium.org/2861163002/ )
>
> Reason for revert:
> Speculative revert -- the TSan bots have been reporting a data race when setting Envvars (in this case, appending to the python path to start a websocket server). The race appeared immediately after this patch landed, so it may be legitimate. Reverting this to see if it clears the failures up; if so, we'll probably just have to serialize the calls to setenv.
>
> Filed  crbug.com/719633  for this as well.
>
> Original issue's description:
> > Linux: Disable DBus auto-launch
> >
> > This is a workaround (ETA ~ 2-3 years) for libdbus not being multi-threading
> > friendly and causing random hangs when running chrome outside of Linux
> > desktop environments.
> >
> > Background:
> > -----------
> > Typically, Linux desktop environments set the DBUS_SESSION_BUS_ADDRESS
> > environment variable. This variable allows the dbus client library to
> > directly connect to the existing bus, which is started by the desktop
> > environment or systemd.
> > When this variable is missing, the dbus client library will fallback
> > to auto-launch mode [1], which causes 4 nested fork() + exec() calls.
> > Doing this has two problems: (i) slows down startup; (ii) can hang
> > the browser if the fork() happens while another thread is in a malloc()
> > (Chrome's tcmalloc has no at-fork handlers).
> > This situation (no env variable) is very common in test scenarios
> > (browsertests, chromedriver, etc).
> >
> > Change introduced by this CL:
> > -----------------------------
> > This CL sets the bus address env variable to "disabled:" if not set.
> > This effectively shuts down the dbus auto-launch. If necessary, this
> > behavior can be restored by setting, before launching chrome,
> > DBUS_SESSION_BUS_ADDRESS="autolaunch:" .
> > This workaround will be necessary until libdbus and gspawn are fixed
> > to be multi-threading friendly [2,3] and that fix rolls into the
> > various distributions.
> > The change is introduced in the main embedder rather than in the
> > google-chrome wrapper, as several binaries can be affected by this,
> > for instance:
> > - browser tests (http://crbug.com/693668)
> > - chrome --headless
> > - webdriver/selenium which seem to directly invoke "chrome"
> >    see https://github.com/SeleniumHQ/docker-selenium/issues/87
> >
> > [1] https://dbus.freedesktop.org/doc/dbus-launch.1.html
> > [2] https://bugs.freedesktop.org/show_bug.cgi?id=100843
> > [3] https://bugs.chromium.org/p/chromedriver/issues/detail?id=1699
> >
> > BUG= 715658 , 695643 , 713947 
> > TEST=strace -ff -o trace chrome; grep dbus-launch trace*
> >
> > Review-Url: https://codereview.chromium.org/2861163002
> > Cr-Commit-Position: refs/heads/master@{#469987}
> > Committed: https://chromium.googlesource.com/chromium/src/+/8511820ec8280caacbd4f81f3ecd13b6c61681b0

> Review-Url: https://codereview.chromium.org/2869843003
> Cr-Commit-Position: refs/heads/master@{#470059}
> Committed: https://chromium.googlesource.com/chromium/src/+/1e78cb7863da28bb3411286cdbcc4fb4510ce173

BUG= 715658 , 695643 , 713947 , 719633 
TBR=satorux@google.com,thestig@chromium.org,jam@chromium.org

Review-Url: https://codereview.chromium.org/2865283002
Cr-Commit-Position: refs/heads/master@{#470301}

[modify] https://crrev.com/2fc330d0b93d4bfd7bd04b9fdd3102e529901f91/build/sanitizers/tsan_suppressions.cc
[modify] https://crrev.com/2fc330d0b93d4bfd7bd04b9fdd3102e529901f91/services/service_manager/embedder/main.cc

Owner: primiano@chromium.org
Status: Fixed (was: Untriaged)
Workaround landed in #12. please reopen if this happens again.

Sign in to add a comment