Occasional DBUS hang running Chrome on Linux |
||||||||
Issue descriptionWe have a persistent issue using Chrome on Linux for automated testing with Chrome hanging on startup. See issue 309093 and https://bugs.chromium.org/p/chromedriver/issues/detail?id=1699 for examples and detailed investigation pointing at this glib bug: https://bugzilla.gnome.org/show_bug.cgi?id=738620 We (devtools / headless Chrome team?) should drive getting some permanent fix for Chrome (fixing GLib and/or landing a workaround into chromium) so this stops biting people every time someone tries to automate Chrome in some new environment.
,
Apr 21 2017
,
Apr 21 2017
Can you grab a stack trace of the UI thread during the hang?
,
Apr 21 2017
There's a suitable one in https://bugs.chromium.org/p/chromium/issues/detail?id=309093#c51 .
,
Apr 23 2017
Removing devtools component as I fail to see how it is related to DevTools. Rick, in case this wasn't accidental, please feel free to bring it back.
,
Apr 24 2017
> Removing devtools component as I fail to see how it is related to DevTools. > Rick, in case this wasn't accidental, please feel free to bring it back. Rick was likely acting on my based on my input. I was interpreting the logs at https://bugs.chromium.org/p/chromedriver/issues/detail?id=1699#c2 to indicate a problem with the communication between ChromeDriver and DevTools. Sounds like I may have presumed too much, though.
,
Apr 26 2017
,
Apr 26 2017
,
Apr 26 2017
,
May 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8511820ec8280caacbd4f81f3ecd13b6c61681b0 commit 8511820ec8280caacbd4f81f3ecd13b6c61681b0 Author: primiano <primiano@chromium.org> Date: Mon May 08 15:09:51 2017 Linux: Disable DBus auto-launch This is a workaround (ETA ~ 2-3 years) for libdbus not being multi-threading friendly and causing random hangs when running chrome outside of Linux desktop environments. Background: ----------- Typically, Linux desktop environments set the DBUS_SESSION_BUS_ADDRESS environment variable. This variable allows the dbus client library to directly connect to the existing bus, which is started by the desktop environment or systemd. When this variable is missing, the dbus client library will fallback to auto-launch mode [1], which causes 4 nested fork() + exec() calls. Doing this has two problems: (i) slows down startup; (ii) can hang the browser if the fork() happens while another thread is in a malloc() (Chrome's tcmalloc has no at-fork handlers). This situation (no env variable) is very common in test scenarios (browsertests, chromedriver, etc). Change introduced by this CL: ----------------------------- This CL sets the bus address env variable to "disabled:" if not set. This effectively shuts down the dbus auto-launch. If necessary, this behavior can be restored by setting, before launching chrome, DBUS_SESSION_BUS_ADDRESS="autolaunch:" . This workaround will be necessary until libdbus and gspawn are fixed to be multi-threading friendly [2,3] and that fix rolls into the various distributions. The change is introduced in the main embedder rather than in the google-chrome wrapper, as several binaries can be affected by this, for instance: - browser tests (http://crbug.com/693668) - chrome --headless - webdriver/selenium which seem to directly invoke "chrome" see https://github.com/SeleniumHQ/docker-selenium/issues/87 [1] https://dbus.freedesktop.org/doc/dbus-launch.1.html [2] https://bugs.freedesktop.org/show_bug.cgi?id=100843 [3] https://bugs.chromium.org/p/chromedriver/issues/detail?id=1699 BUG= 715658 , 695643 , 713947 TEST=strace -ff -o trace chrome; grep dbus-launch trace* Review-Url: https://codereview.chromium.org/2861163002 Cr-Commit-Position: refs/heads/master@{#469987} [modify] https://crrev.com/8511820ec8280caacbd4f81f3ecd13b6c61681b0/services/service_manager/embedder/main.cc
,
May 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1e78cb7863da28bb3411286cdbcc4fb4510ce173 commit 1e78cb7863da28bb3411286cdbcc4fb4510ce173 Author: iclelland <iclelland@chromium.org> Date: Mon May 08 18:41:31 2017 Revert of Linux: Disable DBus auto-launch (patchset #1 id:1 of https://codereview.chromium.org/2861163002/ ) Reason for revert: Speculative revert -- the TSan bots have been reporting a data race when setting Envvars (in this case, appending to the python path to start a websocket server). The race appeared immediately after this patch landed, so it may be legitimate. Reverting this to see if it clears the failures up; if so, we'll probably just have to serialize the calls to setenv. Filed crbug.com/719633 for this as well. Original issue's description: > Linux: Disable DBus auto-launch > > This is a workaround (ETA ~ 2-3 years) for libdbus not being multi-threading > friendly and causing random hangs when running chrome outside of Linux > desktop environments. > > Background: > ----------- > Typically, Linux desktop environments set the DBUS_SESSION_BUS_ADDRESS > environment variable. This variable allows the dbus client library to > directly connect to the existing bus, which is started by the desktop > environment or systemd. > When this variable is missing, the dbus client library will fallback > to auto-launch mode [1], which causes 4 nested fork() + exec() calls. > Doing this has two problems: (i) slows down startup; (ii) can hang > the browser if the fork() happens while another thread is in a malloc() > (Chrome's tcmalloc has no at-fork handlers). > This situation (no env variable) is very common in test scenarios > (browsertests, chromedriver, etc). > > Change introduced by this CL: > ----------------------------- > This CL sets the bus address env variable to "disabled:" if not set. > This effectively shuts down the dbus auto-launch. If necessary, this > behavior can be restored by setting, before launching chrome, > DBUS_SESSION_BUS_ADDRESS="autolaunch:" . > This workaround will be necessary until libdbus and gspawn are fixed > to be multi-threading friendly [2,3] and that fix rolls into the > various distributions. > The change is introduced in the main embedder rather than in the > google-chrome wrapper, as several binaries can be affected by this, > for instance: > - browser tests (http://crbug.com/693668) > - chrome --headless > - webdriver/selenium which seem to directly invoke "chrome" > see https://github.com/SeleniumHQ/docker-selenium/issues/87 > > [1] https://dbus.freedesktop.org/doc/dbus-launch.1.html > [2] https://bugs.freedesktop.org/show_bug.cgi?id=100843 > [3] https://bugs.chromium.org/p/chromedriver/issues/detail?id=1699 > > BUG= 715658 , 695643 , 713947 > TEST=strace -ff -o trace chrome; grep dbus-launch trace* > > Review-Url: https://codereview.chromium.org/2861163002 > Cr-Commit-Position: refs/heads/master@{#469987} > Committed: https://chromium.googlesource.com/chromium/src/+/8511820ec8280caacbd4f81f3ecd13b6c61681b0 TBR=satorux@google.com,thestig@chromium.org,jam@chromium.org,satorux@chromium.org,primiano@chromium.org # Skipping CQ checks because original CL landed less than 1 days ago. NOPRESUBMIT=true NOTREECHECKS=true NOTRY=true BUG= 715658 , 695643 , 713947 Review-Url: https://codereview.chromium.org/2869843003 Cr-Commit-Position: refs/heads/master@{#470059} [modify] https://crrev.com/1e78cb7863da28bb3411286cdbcc4fb4510ce173/services/service_manager/embedder/main.cc
,
May 9 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/2fc330d0b93d4bfd7bd04b9fdd3102e529901f91 commit 2fc330d0b93d4bfd7bd04b9fdd3102e529901f91 Author: primiano <primiano@chromium.org> Date: Tue May 09 13:07:12 2017 Reland of Linux: Disable DBus auto-launch (patchset #1 id:1 of https://codereview.chromium.org/2869843003/ ) Reason for reland: Adding TSan suppression. The race is independent of this CL, see crbug.com/719633 Original issue's description: > Revert of Linux: Disable DBus auto-launch (patchset #1 id:1 of https://codereview.chromium.org/2861163002/ ) > > Reason for revert: > Speculative revert -- the TSan bots have been reporting a data race when setting Envvars (in this case, appending to the python path to start a websocket server). The race appeared immediately after this patch landed, so it may be legitimate. Reverting this to see if it clears the failures up; if so, we'll probably just have to serialize the calls to setenv. > > Filed crbug.com/719633 for this as well. > > Original issue's description: > > Linux: Disable DBus auto-launch > > > > This is a workaround (ETA ~ 2-3 years) for libdbus not being multi-threading > > friendly and causing random hangs when running chrome outside of Linux > > desktop environments. > > > > Background: > > ----------- > > Typically, Linux desktop environments set the DBUS_SESSION_BUS_ADDRESS > > environment variable. This variable allows the dbus client library to > > directly connect to the existing bus, which is started by the desktop > > environment or systemd. > > When this variable is missing, the dbus client library will fallback > > to auto-launch mode [1], which causes 4 nested fork() + exec() calls. > > Doing this has two problems: (i) slows down startup; (ii) can hang > > the browser if the fork() happens while another thread is in a malloc() > > (Chrome's tcmalloc has no at-fork handlers). > > This situation (no env variable) is very common in test scenarios > > (browsertests, chromedriver, etc). > > > > Change introduced by this CL: > > ----------------------------- > > This CL sets the bus address env variable to "disabled:" if not set. > > This effectively shuts down the dbus auto-launch. If necessary, this > > behavior can be restored by setting, before launching chrome, > > DBUS_SESSION_BUS_ADDRESS="autolaunch:" . > > This workaround will be necessary until libdbus and gspawn are fixed > > to be multi-threading friendly [2,3] and that fix rolls into the > > various distributions. > > The change is introduced in the main embedder rather than in the > > google-chrome wrapper, as several binaries can be affected by this, > > for instance: > > - browser tests (http://crbug.com/693668) > > - chrome --headless > > - webdriver/selenium which seem to directly invoke "chrome" > > see https://github.com/SeleniumHQ/docker-selenium/issues/87 > > > > [1] https://dbus.freedesktop.org/doc/dbus-launch.1.html > > [2] https://bugs.freedesktop.org/show_bug.cgi?id=100843 > > [3] https://bugs.chromium.org/p/chromedriver/issues/detail?id=1699 > > > > BUG= 715658 , 695643 , 713947 > > TEST=strace -ff -o trace chrome; grep dbus-launch trace* > > > > Review-Url: https://codereview.chromium.org/2861163002 > > Cr-Commit-Position: refs/heads/master@{#469987} > > Committed: https://chromium.googlesource.com/chromium/src/+/8511820ec8280caacbd4f81f3ecd13b6c61681b0 > Review-Url: https://codereview.chromium.org/2869843003 > Cr-Commit-Position: refs/heads/master@{#470059} > Committed: https://chromium.googlesource.com/chromium/src/+/1e78cb7863da28bb3411286cdbcc4fb4510ce173 BUG= 715658 , 695643 , 713947 , 719633 TBR=satorux@google.com,thestig@chromium.org,jam@chromium.org Review-Url: https://codereview.chromium.org/2865283002 Cr-Commit-Position: refs/heads/master@{#470301} [modify] https://crrev.com/2fc330d0b93d4bfd7bd04b9fdd3102e529901f91/build/sanitizers/tsan_suppressions.cc [modify] https://crrev.com/2fc330d0b93d4bfd7bd04b9fdd3102e529901f91/services/service_manager/embedder/main.cc
,
May 15 2017
Workaround landed in #12. please reopen if this happens again. |
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by rbyers@chromium.org
, Apr 21 2017