New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 652359 link

Starred by 2 users

Issue metadata

Status: Verified
Owner:
ex-Googler
Closed: Mar 2017
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 3
Type: Bug

Blocked on:
issue 654577
issue 533682

Blocking:
issue 332604
issue 489784



Sign in to add a comment

Find alternative to Meta for the sql::Connection mmap code to store status.

Project Member Reported by sh...@chromium.org, Oct 3 2016

Issue description

sql::Connection does some evaluation before enabling mmap on a database, because there are various broken results which can occur with corrupt systems (see issue 489784).  It stores status info in the Meta table so that it only have to evaluate once.  Some databases have no Meta table, so they disable mmap.

These sub-systems know who they are, they could set an alternative place to store the data.
 

Comment 1 by sh...@chromium.org, Oct 3 2016

OK, and my brain immediately suggested using a view.  Like one of:

CREATE VIEW MmapStatus (key, value) AS SELECT 'mmap_status', -1;
CREATE VIEW MmapStatus (value) AS SELECT -1;

Obviously this would be opt-in, and would take precedence over using Meta.

Comment 2 by sh...@chromium.org, Oct 10 2016

Status: Started (was: Assigned)
Status: components/sync/syncable/directory_backing_store.cc is blocking me.  The related test code uses ad-hoc corruption injection, but the way it is done doesn't manifest the same for mmap versus POSIX I/O, so my change causes spurious test failures (the production code is right, but the test is wrong).

I'm grinding through solutions for that.

Comment 3 by sh...@chromium.org, Oct 10 2016

Blockedon: 654577
Grr.  Chromium compiles with SQLITE_ENABLE_MEMORY_MANAGEMENT, which should enable sqlite3_release_memory() to release all of the pcache buffers, which should work well enough as a stand-in for sqlite3_db_release_memory(), problem solved.  BUT, we patch SQLite to add SQLITE_SEPARATE_CACHE_POOLS which defeats sqlite3_release_memory() because it takes precedence over SQLITE_ENABLE_MEMORY_MANAGEMENT.

I need to ponder.  sqlite3_release_memory() was basically a way to dump the page caches without having to wire things through between the test and the originating code.  Maybe things just need to be wired through.

Comment 4 by sh...@chromium.org, Oct 14 2016

Blockedon: 533682

Comment 5 by sh...@chromium.org, Oct 14 2016

Blocking: 332604

Comment 6 by dskiba@chromium.org, Oct 24 2016

Labels: Performance-Memory Performance-Memory-Q4
shess@, would you be able to fix this in Q4?

Comment 7 by sh...@chromium.org, Oct 28 2016

https://codereview.chromium.org/2397753005 is the prospective CL for this.  The problem in #2 has been solved for enough time that I think it will stick, so moving forward.
Project Member

Comment 8 by bugdroid1@chromium.org, Nov 7 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a62504d228eedeed490044b79403a9de9eb8a4cb

commit a62504d228eedeed490044b79403a9de9eb8a4cb
Author: shess <shess@chromium.org>
Date: Mon Nov 07 19:26:12 2016

[sql] Allow storing mmap status in a VIEW instead of meta table.

Some databases do not use the sql::MetaTable to handle versioning.
The mmap verification pass uses the [meta] table to store progress for
larger databases, and it uses detection of the [meta] table to
distinguish new/empty databases from existing databases.  For
non-[meta] databases, store the status in an MmapStatus view.

BUG= 652359 

Review-Url: https://codereview.chromium.org/2397753005
Cr-Commit-Position: refs/heads/master@{#430343}

[modify] https://crrev.com/a62504d228eedeed490044b79403a9de9eb8a4cb/chrome/browser/extensions/activity_log/activity_database.cc
[modify] https://crrev.com/a62504d228eedeed490044b79403a9de9eb8a4cb/chrome/browser/predictors/predictor_database.cc
[modify] https://crrev.com/a62504d228eedeed490044b79403a9de9eb8a4cb/components/sync/syncable/directory_backing_store.cc
[modify] https://crrev.com/a62504d228eedeed490044b79403a9de9eb8a4cb/content/browser/dom_storage/dom_storage_database.cc
[modify] https://crrev.com/a62504d228eedeed490044b79403a9de9eb8a4cb/sql/connection.cc
[modify] https://crrev.com/a62504d228eedeed490044b79403a9de9eb8a4cb/sql/connection.h
[modify] https://crrev.com/a62504d228eedeed490044b79403a9de9eb8a4cb/sql/connection_unittest.cc

Project Member

Comment 9 by bugdroid1@chromium.org, Nov 10 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e28e2e598746a19c53447f8d54c1927aba6e5b59

commit e28e2e598746a19c53447f8d54c1927aba6e5b59
Author: shess <shess@chromium.org>
Date: Thu Nov 10 02:12:34 2016

[sql] Histogram for mmap status VIEW.

https://codereview.chromium.org/2397753005/ landed a change to how
sql::Connection handles mmap tracking.  I neglected to add the
histograms.xml entry to mirror an enum change.

BUG= 652359 

Review-Url: https://codereview.chromium.org/2483783006
Cr-Commit-Position: refs/heads/master@{#431143}

[modify] https://crrev.com/e28e2e598746a19c53447f8d54c1927aba6e5b59/tools/metrics/histograms/histograms.xml

Should this be marked fixed?

Comment 11 by sh...@chromium.org, Mar 17 2017

Status: Verified (was: Started)
AFAICT, this is working fine.  Lots of new successes registered in histograms, a few failures, the failures at first glance appear in line with the failures for the meta-table storage, so I think they just indicate cases where the database is somewhat broken, rather than any new issues.

I did have a moment where I worried that EVENT_MMAP_META_FAILURE_UPDATE and EVENT_MMAP_STATUS_FAILURE_UPDATE were untested and thus perhaps just wrong, but, unfortunately, after verifying that they were untested and testing them, the code seems right.  So I'll land tests for that shortly, but I think the bug can be closed.
Project Member

Comment 12 by bugdroid1@chromium.org, Mar 20 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a7b07acd99ebfd40b319dfd49103081a8c538ec4

commit a7b07acd99ebfd40b319dfd49103081a8c538ec4
Author: shess <shess@chromium.org>
Date: Sun Mar 19 23:59:38 2017

[sql] Tests for updates to mmap state.

Histograms indicate a bit more EVENT_MMAP_STATUS_FAILURE_UPDATE and
EVENT_MMAP_META_FAILURE_UPDATE than I was hoping for, and I realized
that that code was untested.  Add some tests for that case.
Unfortunately for the histograms, the untested code appears to work.

BUG= 652359 

Review-Url: https://codereview.chromium.org/2756843003
Cr-Commit-Position: refs/heads/master@{#458007}

[modify] https://crrev.com/a7b07acd99ebfd40b319dfd49103081a8c538ec4/sql/connection_unittest.cc

Sign in to add a comment