New issue
Advanced search Search tips

Issue 696413 link

Starred by 1 user
Status: Untriaged
Owner: ----
Cc:
machenb...@chromium.org
thomasanderson@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 2
Type: Bug

Blocked on:
issue 691029


Sign in to add a comment

Browser doesn't start under TSAN due to Gtk-related deadlock

Project Member Reported by kbr@chromium.org, Feb 27 2017 
Our group just brought up a Linux TSAN bot to try to catch deadlocks in Chrome's GPU stack. It is crashing the browser upon startup, flagging potential deadlocks in Gtk-related code.

On first glance this report appears wrong. The code will always run single-threaded to the best of my understanding, so any acquisition of locks in a different order should be OK. However, I haven't studied the code in detail.

Could someone knowledgeable in this area (either Chrome's Gtk code or TSAN) tell me whether it's expected that full Chrome fails to start on Linux with TSAN? Looking more deeply, it seems that https://build.chromium.org/p/chromium.memory/builders/Linux%20TSan%20Tests only runs unit tests.

Example failing build:
https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20GPU%20TSAN%20Release/builds/27

Failing shard (from context_lost_tests):
https://chromium-swarm.appspot.com/task?id=3496171338b9b510&refresh=10&show_raw=1

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=6207)
  Cycle in lock order graph: M153 (0x7b0c00023ca0) => M135 (0x7b0c00021b10) => M153

  Mutex M135 acquired here while holding mutex M153 in main thread:
    #0 pthread_mutex_lock ??:? (chrome+0xc166be)
    #1 g_type_class_ref ??:? (libgobject-2.0.so.0+0x2e131)
    #2 GtkUi chrome/browser/ui/libgtkui/gtk_ui.cc:431 (chrome+0x6bf499a)
    #3 BuildGtkUi() chrome/browser/ui/libgtkui/gtk_ui.cc:1081 (chrome+0x6bf93a3)
    #4 PreEarlyInitialization chrome/browser/ui/views/chrome_browser_main_extra_parts_views_linux.cc:70 (chrome+0x683230c)
    #5 PreEarlyInitialization chrome/browser/chrome_browser_main.cc:903 (chrome+0x374fc7d)
    #6 ChromeBrowserMainPartsPosix::PreEarlyInitialization() chrome/browser/chrome_browser_main_posix.cc:257 (chrome+0x36e218c)
    #7 EarlyInitialization content/browser/browser_main_loop.cc:621 (chrome+0x1495ff5)
    #8 Initialize content/browser/browser_main_runner.cc:106 (chrome+0x14a01d6)
    #9 BrowserMain content/browser/browser_main.cc:42 (chrome+0x1494efd)
    #10 RunNamedProcessTypeMain content/app/content_main_runner.cc:491 (chrome+0x32bb404)
    #11 Run content/app/content_main_runner.cc:836 (chrome+0x32bc0e7)
    #12 ContentMain content/app/content_main.cc:20 (chrome+0x32b9dee)
    #13 ChromeMain chrome/app/chrome_main.cc:113 (chrome+0xc5ca9b)
    #14 main chrome/app/chrome_exe_main_aura.cc:17 (chrome+0xc5c9ee)

  Mutex M153 previously acquired by the same thread here:
    #0 pthread_mutex_lock ??:? (chrome+0xc166be)
    #1 g_io_extension_get_type ??:? (libgio-2.0.so.0+0x5e302)
    #2 GtkUi chrome/browser/ui/libgtkui/gtk_ui.cc:431 (chrome+0x6bf499a)
    #3 BuildGtkUi() chrome/browser/ui/libgtkui/gtk_ui.cc:1081 (chrome+0x6bf93a3)
    #4 PreEarlyInitialization chrome/browser/ui/views/chrome_browser_main_extra_parts_views_linux.cc:70 (chrome+0x683230c)
    #5 PreEarlyInitialization chrome/browser/chrome_browser_main.cc:903 (chrome+0x374fc7d)
    #6 ChromeBrowserMainPartsPosix::PreEarlyInitialization() chrome/browser/chrome_browser_main_posix.cc:257 (chrome+0x36e218c)
    #7 EarlyInitialization content/browser/browser_main_loop.cc:621 (chrome+0x1495ff5)
    #8 Initialize content/browser/browser_main_runner.cc:106 (chrome+0x14a01d6)
    #9 BrowserMain content/browser/browser_main.cc:42 (chrome+0x1494efd)
    #10 RunNamedProcessTypeMain content/app/content_main_runner.cc:491 (chrome+0x32bb404)
    #11 Run content/app/content_main_runner.cc:836 (chrome+0x32bc0e7)
    #12 ContentMain content/app/content_main.cc:20 (chrome+0x32b9dee)
    #13 ChromeMain chrome/app/chrome_main.cc:113 (chrome+0xc5ca9b)
    #14 main chrome/app/chrome_exe_main_aura.cc:17 (chrome+0xc5c9ee)

  Mutex M153 acquired here while holding mutex M135 in main thread:
    #0 pthread_mutex_lock ??:? (chrome+0xc166be)
    #1 g_io_extension_get_type ??:? (libgio-2.0.so.0+0x5e302)
    #2 GetSystemColor chrome/browser/ui/libgtkui/native_theme_gtk2.cc:? (chrome+0x6c08be3)
    #3 LoadGtkValues chrome/browser/ui/libgtkui/gtk_ui.cc:842 (chrome+0x6bf57c3)
    #4 Initialize chrome/browser/ui/libgtkui/gtk_ui.cc:461 (chrome+0x6bf4e49)
    #5 ChromeBrowserMainExtraPartsViewsLinux::ToolkitInitialized() chrome/browser/ui/views/chrome_browser_main_extra_parts_views_linux.cc:77 (chrome+0x6832502)
    #6 ToolkitInitialized chrome/browser/chrome_browser_main.cc:915 (chrome+0x374fffd)
    #7 ChromeBrowserMainPartsLinux::ToolkitInitialized() chrome/browser/chrome_browser_main_linux.cc:44 (chrome+0x354dffe)
    #8 InitializeToolkit content/browser/browser_main_loop.cc:1667 (chrome+0x149decc)
    #9 Initialize content/browser/browser_main_runner.cc:109 (chrome+0x14a01e6)
    #10 BrowserMain content/browser/browser_main.cc:42 (chrome+0x1494efd)
    #11 RunNamedProcessTypeMain content/app/content_main_runner.cc:491 (chrome+0x32bb404)
    #12 Run content/app/content_main_runner.cc:836 (chrome+0x32bc0e7)
    #13 ContentMain content/app/content_main.cc:20 (chrome+0x32b9dee)
    #14 ChromeMain chrome/app/chrome_main.cc:113 (chrome+0xc5ca9b)
    #15 main chrome/app/chrome_exe_main_aura.cc:17 (chrome+0xc5c9ee)

  Mutex M135 previously acquired by the same thread here:
    #0 pthread_mutex_lock ??:? (chrome+0xc166be)
    #1 g_type_class_ref ??:? (libgobject-2.0.so.0+0x2e131)
    #2 GetSystemColor chrome/browser/ui/libgtkui/native_theme_gtk2.cc:? (chrome+0x6c08be3)
    #3 LoadGtkValues chrome/browser/ui/libgtkui/gtk_ui.cc:842 (chrome+0x6bf57c3)
    #4 Initialize chrome/browser/ui/libgtkui/gtk_ui.cc:461 (chrome+0x6bf4e49)
    #5 ChromeBrowserMainExtraPartsViewsLinux::ToolkitInitialized() chrome/browser/ui/views/chrome_browser_main_extra_parts_views_linux.cc:77 (chrome+0x6832502)
    #6 ToolkitInitialized chrome/browser/chrome_browser_main.cc:915 (chrome+0x374fffd)
    #7 ChromeBrowserMainPartsLinux::ToolkitInitialized() chrome/browser/chrome_browser_main_linux.cc:44 (chrome+0x354dffe)
    #8 InitializeToolkit content/browser/browser_main_loop.cc:1667 (chrome+0x149decc)
    #9 Initialize content/browser/browser_main_runner.cc:109 (chrome+0x14a01e6)
    #10 BrowserMain content/browser/browser_main.cc:42 (chrome+0x1494efd)
    #11 RunNamedProcessTypeMain content/app/content_main_runner.cc:491 (chrome+0x32bb404)
    #12 Run content/app/content_main_runner.cc:836 (chrome+0x32bc0e7)
    #13 ContentMain content/app/content_main.cc:20 (chrome+0x32b9dee)
    #14 ChromeMain chrome/app/chrome_main.cc:113 (chrome+0xc5ca9b)
    #15 main chrome/app/chrome_exe_main_aura.cc:17 (chrome+0xc5c9ee)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:? in __interceptor_pthread_mutex_lock

Comment 1 by thomasanderson@chromium.org, Feb 27 2017 

To the best of my knowledge, everything in libgtkui is done on the main thread.

Also, it looks like the locks are managed by Gtk, so I'm not sure there's anything we can do about this.  However, there's a chance this deadlock will go away when using use_gtk3=true.

Comment 2 by e...@chromium.org, Feb 27 2017 

Just going off the traces, I suspect that this won't be fixed by migrating to gtk3. These mutexes are held in libgobject and libgio which are below the gtk layer.

I get the same impression that the chrome side of this code is always run single threaded, but I wouldn't count out things like the G-Object class system (the lock in g_type_class_ref()) being accessed from multiple threads. Same with GIO.

Comment 3 by kbr@chromium.org, Feb 28 2017 

Thanks erg@.

I have a feeling that these are really just warnings and aren't the real reason the TSAN-instrumented browser fails to start. I'll try to do a TSAN build on Linux soon and check its behavior locally rather than on the bots.

Comment 4 by e...@chromium.org, Mar 9 2018

Cc: -e...@chromium.org
Un-cc-ing me from all bugs on my final day.

Comment 5 by kbr@chromium.org, Jul 13

Blocking: -691029

Comment 6 by kbr@chromium.org, Jul 13

Blockedon: 691029

Comment 7 by pnangunoori@chromium.org, Nov 21

Labels: Hotlist-DesktopUIToolingRequired Hotlist-DesktopUIChecked
*** UI Mass Triage***
►

Sign in to add a comment