New issue
Advanced search Search tips

Issue 807487 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Jan 2018
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 1
Type: Bug-Regression



Sign in to add a comment

Chrome ToT crashes on launch

Project Member Reported by thomasanderson@chromium.org, Jan 31 2018

Issue description

Version: ToT ed41e2ad7346240d2125d234c99f8bafd30fdcb4

gn args:
is_component_build = true
is_debug = false
use_goma = true

Chrome crashes with this stack trace:
#0  0x0000000000000000 in  ()
#1  0x00007ffff3cc9580 in sqlite3Malloc () at /home/tom/dev/chromium_1/src/out/Release/./libchromium_sqlite3.so
#2  0x00007ffff3ce33f5 in strAccumFinishRealloc () at /home/tom/dev/chromium_1/src/out/Release/./libchromium_sqlite3.so
#3  0x00007ffff3ccbf44 in sqlite3_mprintf () at /home/tom/dev/chromium_1/src/out/Release/./libchromium_sqlite3.so
#4  0x00007fffd05f187d in sdb_BuildFileName (version=9, type=0x7fffd0603745 "cert", prefix=0x216295b30110 "", directory=0x216295b9cda4 "/home/tom/.pki/nssdb") at sdb.c:350
#5  0x00007fffd05f187d in s_open (directory=directory@entry=0x216295b9cda4 "/home/tom/.pki/nssdb", certPrefix=certPrefix@entry=0x216295b30110 "", keyPrefix=keyPrefix@entry=0x216295b30100 "", cert_version=cert_version@entry=9, key_version=key_version@entry=4, flags=flags@entry=4, certdb=0x7fffd180ffa0, keydb=0x7fffd180ff98, newInit=0x7fffd180ff88) at sdb.c:2008
#6  0x00007fffd05f51c6 in sftk_DBInit (configdir=<optimized out>, certPrefix=0x216295b30110 "", keyPrefix=0x216295b30100 "", updatedir=<optimized out>, updCertPrefix=0x0, updKeyPrefix=<optimized out>, updateID=0x216295b86970 "", readOnly=0, noCertDB=0, noKeyDB=0, forceOpen=0, isFIPS=0, certDB=0x7fffd1810058, keyDB=0x7fffd1810060) at sftkdb.c:2593
#7  0x00007fffd05de844 in SFTK_SlotReInit (slot=slot@entry=0x21629582a3c0, configdir=configdir@entry=0x216295b9cda0 "sql:/home/tom/.pki/nssdb", updatedir=updatedir@entry=0x216295b86960 "", updateID=updateID@entry=0x216295b86970 "", params=params@entry=0x216295821258, moduleIndex=moduleIndex@entry=0) at pkcs11.c:2484
#8  0x00007fffd05dec93 in SFTK_SlotInit (configdir=0x216295b9cda0 "sql:/home/tom/.pki/nssdb", updatedir=0x216295b86960 "", updateID=0x216295b86970 "", params=0x216295821258, moduleIndex=moduleIndex@entry=0) at pkcs11.c:2600
#9  0x00007fffd05df82e in nsc_CommonInitialize (pReserved=pReserved@entry=0x7fffd1810220, isFIPS=isFIPS@entry=0) at pkcs11.c:3052
#10 0x00007fffd05dfa58 in NSC_Initialize (pReserved=0x7fffd1810220) at pkcs11.c:3115
#11 0x00007fffeeaee1ae in secmod_ModuleInit (mod=mod@entry=0x216295b98c20, reload=reload@entry=0x7fffd1810390, alreadyLoaded=alreadyLoaded@entry=0x7fffd18102a4) at pk11load.c:245
#12 0x00007fffeeaee7f0 in secmod_LoadPKCS11Module (mod=mod@entry=0x216295b98c20, oldModule=oldModule@entry=0x7fffd1810390) at pk11load.c:511
#13 0x00007fffeeafba5c in SECMOD_LoadModule (modulespec=modulespec@entry=0x216295b97e00 "library= name=\"NSS Internal PKCS #11 Module\" NSS=\"Flags=internal,critical trustOrder=75 cipherOrder=100 slotParams=(1={slotFlags=[RSA,DSA,DH,RC2,RC4,DES,RANDOM,SHA1,MD5,MD2,SSL,TLS,AES,Camellia,SEED,S"..., parent=parent@entry=0x216295b34020, recurse=recurse@entry=1) at pk11pars.c:1672
#14 0x00007fffeeafbb8f in SECMOD_LoadModule (modulespec=modulespec@entry=0x21629573ba80 "name=\"NSS Internal Module\" parameters=\"configdir='sql:/home/tom/.pki/nssdb' certPrefix='' keyPrefix='' secmod='secmod.db' flags=optimizeSpace updatedir='' updateCertPrefix='' updateKeyPrefix='' update"..., parent=parent@entry=0x0, recurse=recurse@entry=1) at pk11pars.c:1707
#15 0x00007fffeeac7ab3 in nss_InitModules (isContextInit=0, optimizeSpace=<optimized out>, forceOpen=<optimized out>, noModDB=<optimized out>, noCertDB=<optimized out>, readOnly=<optimized out>, pwRequired=<optimized out>, configStrings=<optimized out>, configName=<optimized out>, updateName=<optimized out>, updateID=<optimized out>, updKeyPrefix=<optimized out>, updCertPrefix=<optimized out>, updateDir=<optimized out>, secmodName=<optimized out>, keyPrefix=0x7fffeebb062d "", certPrefix=0x7fffeebb062d "", configdir=<optimized out>) at nssinit.c:464
#16 0x00007fffeeac7ab3 in nss_Init (configdir=<optimized out>, certPrefix=certPrefix@entry=0x7fffeebb062d "", keyPrefix=keyPrefix@entry=0x7fffeebb062d "", secmodName=secmodName@entry=0x7fffeebb023f "secmod.db", updateDir=updateDir@entry=0x7fffeebb062d "", updCertPrefix=updCertPrefix@entry=0x7fffeebb062d "", updKeyPrefix=<optimized out>, updateID=<optimized out>, updateName=<optimized out>, initContextPtr=<optimized out>, initParams=<optimized out>, readOnly=<optimized out>, noCertDB=<optimized out>, noModDB=<optimized out>, forceOpen=<optimized out>, noRootInit=<optimized out>, optimizeSpace=<optimized out>, noSingleThreadedModules=<optimized out>, allowAlreadyInitializedModules=<optimized out>, dontFinalizeModules=<optimized out>)
    at nssinit.c:689
#17 0x00007fffeeac8099 in NSS_InitReadWrite (configdir=<optimized out>) at nssinit.c:832
#18 0x00007ffff6719478 in base::LazyInstance<crypto::(anonymous namespace)::NSSInitSingleton, base::internal::LeakyLazyInstanceTraits<crypto::(anonymous namespace)::NSSInitSingleton> >::Get() () at /home/tom/dev/chromium_1/src/out/Release/./libcrcrypto.so
#19 0x00007ffff6a52178 in net::CertDatabase::GetInstance() () at /home/tom/dev/chromium_1/src/out/Release/./libnet.so
#20 0x00007ffff6c6fc3b in net::ClientSocketFactory::GetDefaultFactory() () at /home/tom/dev/chromium_1/src/out/Release/./libnet.so
#21 0x00007ffff6b2ef69 in net::DnsClient::CreateClient(net::NetLog*) () at /home/tom/dev/chromium_1/src/out/Release/./libnet.so
#22 0x0000555556798758 in chrome_browser_net::DnsProbeService::SetSystemClientToCurrentConfig() ()
---Type <return> to continue, or q <return> to quit---
#23 0x00005555567986b7 in chrome_browser_net::DnsProbeService::DnsProbeService() ()
#24 0x0000555556760df0 in IOThread::Init() ()
#25 0x00007ffff5735030 in content::BrowserProcessSubThread::Init() () at /home/tom/dev/chromium_1/src/out/Release/./libcontent.so
#26 0x00007ffff7b3ae9a in base::Thread::ThreadMain() () at /home/tom/dev/chromium_1/src/out/Release/./libbase.so
#27 0x00007ffff7b3457d in base::(anonymous namespace)::ThreadFunc(void*) () at /home/tom/dev/chromium_1/src/out/Release/./libbase.so
#28 0x00007ffff7bbd7fc in start_thread (arg=0x7fffd1811700) at pthread_create.c:465
#29 0x00007fffed5a1b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

 
Labels: -Type-Bug -Pri-3 OS-Linux Pri-1 Type-Bug-Regression
Status: Fixed (was: Assigned)
Offending CL has been reverted

Comment 3 by zcb...@gmail.com, Feb 6 2018

Hi can you give me a link to the related CL? I'm facing the exactly same crash when porting Chromium to a new platform, this would be great help to me.
re c#3
This is the CL that fixes the issue (but didn't land):
https://chromium-review.googlesource.com/c/chromium/src/+/894885
This was the revert:
https://chromium-review.googlesource.com/c/chromium/src/+/894707
Project Member

Comment 5 by bugdroid1@chromium.org, Feb 6 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3d8ec48766b2075fb6f242d13bbc181e850df1fc

commit 3d8ec48766b2075fb6f242d13bbc181e850df1fc
Author: Victor Costan <pwnall@chromium.org>
Date: Tue Feb 06 23:56:15 2018

sqlite: Prefix SQLite API methods with chrome_.

In component builds, SQLite's API methods are exported from the
chromium_sqlite component, which means they are visible to the dynamic
library loader. This opens up the following possibilities:

1) A system library calls into our SQLite instead of calling into the
   system's SQLite library which it was built against. The patches in
   our SQLite version lead to different behavior from the system's
   SQLite, which can cause subtle failures. This happens if the dynamic
   library loader resolves the system library's symbol imports with our
   SQLite's exported symbols.
2) A system library loads the system SQLite, and we end up calling into
   it, instead of calling into our version of SQLite. This happens if
   the dynamic library loader resolves our symbol imports with the
   system's SQLite library.

Both possibilities above lead to the possibility that the component
build will behave differently from the release build, in subtle and
potentially non-deterministic ways. This is not a purely academic
concern.  https://crbug.com/807487  happened because we use NSS on Linux,
and NSS invokes SQLite via a complex plugin system. On non-component builds,
NSS (a system library) loads and uses the system version of SQLite. On
component builds, NSS ends up using our SQLite.

This CL fixes the problem by adding a chrome_ prefix to all the symbols
exported from SQLite3. In C++ libraries, namespaces can make prefixing
easy. Unfortunately, SQLite is a C library, so the prefixing is fairly
heavy-handed. A high-level overview of the approach follows:

* An extract_sqlite_api Python script reads SQLite's header, extracts
  the names of all exported symbols, and writes a header file consisting
  of renaming preprocessor macros, e.g.
      #define sqlite3_init chrome_sqlite3_init
  David Benjamin <davidben@chromium.org> designed the approach and wrote
  the original version of the script.
* The script that we use to generate SQLite's amalgamation now also
  invokes the extract_sqlite_api script described above, and saves the
  output to amalgamation/rename_exports.h.
* The SQLite component exposes an sqlite3.h header that must be used by
  all SQLite3 users in Chromium. This header now #includes
  rename_exports.h (containing the renaming preprocessor macros) before
  #including amalgamation/sqlite3.h.
* sqlite3.c (the main output of the amalgamation process) does not
  #include "sqlite3.h". However, in order to facilitate autoconf builds,
  it does #include a "config.h", if a certain preprocessor define
  exists. We abuse that define to have sqlite.c always load config.h,
  and have config.h load our rename_exports.h.

This CL also adds a PRESUBMIT.py that runs unit tests for the
extract_sqlite_api Python script, which ensures that the script will not
break accidentally. Both the unit tests and the PRESUBIMT script are
inspired from //tools/vim.

Bug:  807093 ,  807487 
Change-Id: If3868ba119ffd4ccbb06d1a6fcd4cc2ecd9ef2ae
Reviewed-on: https://chromium-review.googlesource.com/898549
Reviewed-by: Chris Mumford <cmumford@chromium.org>
Commit-Queue: Victor Costan <pwnall@chromium.org>
Cr-Commit-Position: refs/heads/master@{#534843}
[modify] https://crrev.com/3d8ec48766b2075fb6f242d13bbc181e850df1fc/third_party/sqlite/BUILD.gn
[add] https://crrev.com/3d8ec48766b2075fb6f242d13bbc181e850df1fc/third_party/sqlite/PRESUBMIT.py
[modify] https://crrev.com/3d8ec48766b2075fb6f242d13bbc181e850df1fc/third_party/sqlite/README.chromium
[delete] https://crrev.com/5157f928d980f7913d23144c597438d4c4f65699/third_party/sqlite/amalgamation/README
[add] https://crrev.com/3d8ec48766b2075fb6f242d13bbc181e850df1fc/third_party/sqlite/amalgamation/README.md
[modify] https://crrev.com/3d8ec48766b2075fb6f242d13bbc181e850df1fc/third_party/sqlite/amalgamation/config.h
[add] https://crrev.com/3d8ec48766b2075fb6f242d13bbc181e850df1fc/third_party/sqlite/amalgamation/rename_exports.h
[rename] https://crrev.com/3d8ec48766b2075fb6f242d13bbc181e850df1fc/third_party/sqlite/amalgamation/shell/shell.c
[delete] https://crrev.com/5157f928d980f7913d23144c597438d4c4f65699/third_party/sqlite/google_generate_amalgamation.sh
[add] https://crrev.com/3d8ec48766b2075fb6f242d13bbc181e850df1fc/third_party/sqlite/scripts/extract_sqlite_api.py
[add] https://crrev.com/3d8ec48766b2075fb6f242d13bbc181e850df1fc/third_party/sqlite/scripts/extract_sqlite_api_unittest.py
[add] https://crrev.com/3d8ec48766b2075fb6f242d13bbc181e850df1fc/third_party/sqlite/scripts/generate_amalgamation.sh
[modify] https://crrev.com/3d8ec48766b2075fb6f242d13bbc181e850df1fc/third_party/sqlite/sqlite3.h

Project Member

Comment 6 by bugdroid1@chromium.org, Feb 7 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/86ca43119893fc14e1820ac1e2a7e2415cd25643

commit 86ca43119893fc14e1820ac1e2a7e2415cd25643
Author: Victor Costan <pwnall@chromium.org>
Date: Wed Feb 07 21:34:10 2018

sqlite: Remove USE_SYSTEM_SQLITE option.

We currently use the system-supplied SQLite library on iOS.

This is not the end of the world, because we require iOS 10.0+, which
means SQLite 3.14+, which has the features that Chrome uses. iOS is also
special in that we don't ship Blink, so we don't expose SQLite via
WebSQL -- we control all the other SQL queries we send to SQLite, so
SQLite security bugs aren't easily exploitable.

At the same time, USE_SYSTEM_SQLITE means Chrome for iOS uses an
entirely different SQLite build that most developers don't run. This
makes it more likely that we'll introduce bugs or crashes, and makes it
more difficult to investigate existing bugs/crashes.

In theory, USE_SYSTEM_SQLITE seems appealing on Android and Linux. In
practice, the idea isn't workable on either platform.

* On Android, not shipping our own SQLite would cut down APK size by a few
  hundred KB. However, Android doesn't have a great upgrade story before
  Lollipop (5.0), and we still support KitKat (4.4). SQLite is exposed
  via WebSQL on Android, so we must be in control of SQLite upgrades.

* Linux distributions would like to ship Chromium builds linked against
  their SQLite copies. However,  https://crbug.com/807487  suggests that
  USE_SYSTEM_SQLITE on Linux would result in unacceptable stability. The
  root cause of the bug was that NSS can load SQLite on Linux, resulting
  in SQLite getting initialized outside our control. Even if we could
  fix that bug (via a dependency from //crypto to //sql), we'd still
  have to chase all the libraries that might potentially initialize
  SQLite. Today we believe that NSS is the only library in that set, but
  we simply don't have the bandwidth to justify investigating the bug
  reports that will come if we miss something.

In conclusion, there is no platform where USE_SYSTEM_SQLITE is desirable
and viable. So, this CL removes USE_SYSTEM_SQLITE from the codebase.
This will cause the iOS build to increase by a few hundred KB. This
price buys us one less difference between iOS and every other platform,
and increased OS stability in the long run, as iOS bugs / crashes will
be easier to diagnose.

Bug:  22208 ,  299684 , 372584,  807093 ,  807487 
Cq-Include-Trybots: master.tryserver.chromium.mac:ios-device
Change-Id: I1bd64bd93527624e197bd082c725a952c8b3c430
Reviewed-on: https://chromium-review.googlesource.com/898223
Commit-Queue: Victor Costan <pwnall@chromium.org>
Reviewed-by: Chris Mumford <cmumford@chromium.org>
Reviewed-by: Sylvain Defresne <sdefresne@chromium.org>
Cr-Commit-Position: refs/heads/master@{#535148}
[modify] https://crrev.com/86ca43119893fc14e1820ac1e2a7e2415cd25643/sql/connection_unittest.cc
[modify] https://crrev.com/86ca43119893fc14e1820ac1e2a7e2415cd25643/sql/recovery.cc
[modify] https://crrev.com/86ca43119893fc14e1820ac1e2a7e2415cd25643/sql/sqlite_features_unittest.cc
[modify] https://crrev.com/86ca43119893fc14e1820ac1e2a7e2415cd25643/sql/vfs_wrapper.cc
[modify] https://crrev.com/86ca43119893fc14e1820ac1e2a7e2415cd25643/third_party/sqlite/BUILD.gn
[modify] https://crrev.com/86ca43119893fc14e1820ac1e2a7e2415cd25643/third_party/sqlite/sqlite3.h
[modify] https://crrev.com/86ca43119893fc14e1820ac1e2a7e2415cd25643/tools/metrics/histograms/enums.xml

Sign in to add a comment