New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 725565 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Sep 20
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 3
Type: Feature


Participants' hotlists:
dmurph-shortlist-features

Show other hotlists

Other hotlists containing this issue:
Hotlist-2
Hotlist-3
Hotlist-1
IDB-Scaling


Sign in to add a comment

Start index tombstones sweeper stats crawling

Project Member Reported by dmu...@chromium.org, May 23 2017

Issue description

ee https://docs.google.com/document/d/18H0k_QKo0a9b9fDndhw3MXm6Db27lmXzFAOjDPFA1T4/edit#heading=h.19lnn0712d8r
for a description of the problem. Basically, we create a lot of IndexedDB Index Key tombstones (not leveldb tombstones) when we operate on object stores that have Indexes.


The bug tracks the CLs for adding the stats collecting / tombstone crawling feature, and should serve as the Finch bug.

Why do we want to use finch? We don't need to gather stats from the entire population, as it is a little risky and also incurs disk activity. We are fine with a small sample. Once we have gathered stats, we gradually turn on deletion.

For both stats and deletion, we want to carefully monitor all errors, as this sweeper does a lot of disk analysis and reports very robust error metrics. We want to be extra careful to not break anything, even though there are tests.

 
Labels: Type-Feature
Project Member

Comment 2 by bugdroid1@chromium.org, Aug 17 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/66da98b692341dd44357b586a52bb5fc265c0011

commit 66da98b692341dd44357b586a52bb5fc265c0011
Author: Daniel Murphy <dmurph@chromium.org>
Date: Thu Aug 17 21:21:06 2017

[IndexedDB] Index tombstone crawler for stats and tombstone deletion.

Bug:  725565 ,  711439 ,  723129 
Change-Id: I29d64e968b62aed0e7cdea4ac5721d0441958b75
Reviewed-on: https://chromium-review.googlesource.com/572700
Commit-Queue: Daniel Murphy <dmurph@chromium.org>
Reviewed-by: Camille Lamy <clamy@chromium.org>
Reviewed-by: Victor Costan <pwnall@chromium.org>
Reviewed-by: Chris Mumford <cmumford@chromium.org>
Cr-Commit-Position: refs/heads/master@{#495322}
[modify] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/BUILD.gn
[modify] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/indexed_db_backing_store.cc
[modify] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/indexed_db_backing_store.h
[modify] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/indexed_db_factory_impl.cc
[modify] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/indexed_db_factory_impl.h
[modify] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/indexed_db_leveldb_coding.cc
[modify] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/indexed_db_leveldb_coding.h
[modify] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/indexed_db_leveldb_coding_unittest.cc
[add] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/indexed_db_pre_close_task_queue.cc
[add] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/indexed_db_pre_close_task_queue.h
[add] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/indexed_db_pre_close_task_queue_unittest.cc
[add] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/indexed_db_tombstone_sweeper.cc
[add] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/indexed_db_tombstone_sweeper.h
[add] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/indexed_db_tombstone_sweeper_unittest.cc
[modify] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/leveldb/leveldb_database.h
[add] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/leveldb/mock_level_db.cc
[add] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/browser/indexed_db/leveldb/mock_level_db.h
[modify] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/common/indexed_db/indexed_db_metadata.cc
[modify] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/common/indexed_db/indexed_db_metadata.h
[modify] https://crrev.com/66da98b692341dd44357b586a52bb5fc265c0011/content/test/BUILD.gn

Comment 3 by dmu...@chromium.org, Sep 30 2017

Summary: Start index tombstones sweeper stats crawling (was: Create finch trial for inspecting idb index tomstones)
Project Member

Comment 4 by bugdroid1@chromium.org, Oct 4 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/532ef169eb6fac02407fddf270e107753c06a29a

commit 532ef169eb6fac02407fddf270e107753c06a29a
Author: Daniel Murphy <dmurph@chromium.org>
Date: Wed Oct 04 20:12:43 2017

[IndexedDB] Pulling out leveldb operations & reporting into own files.

This reduces the size of the extra large backing store class, and leads
towards a smaller class in the future as other classes can now call the
appropriate leveldb operations.

R: cmumford@chromium.org, pwnall@chromium.org
Bug:  725565 
Change-Id: I29e7c72970b47801a51db4548f9e0a1b3ae26d4b
Reviewed-on: https://chromium-review.googlesource.com/693382
Reviewed-by: Victor Costan <pwnall@chromium.org>
Reviewed-by: Nick Carter <nick@chromium.org>
Reviewed-by: Chris Mumford <cmumford@chromium.org>
Commit-Queue: Daniel Murphy <dmurph@chromium.org>
Cr-Commit-Position: refs/heads/master@{#506495}
[modify] https://crrev.com/532ef169eb6fac02407fddf270e107753c06a29a/content/browser/BUILD.gn
[modify] https://crrev.com/532ef169eb6fac02407fddf270e107753c06a29a/content/browser/indexed_db/indexed_db_backing_store.cc
[modify] https://crrev.com/532ef169eb6fac02407fddf270e107753c06a29a/content/browser/indexed_db/indexed_db_leveldb_coding.cc
[modify] https://crrev.com/532ef169eb6fac02407fddf270e107753c06a29a/content/browser/indexed_db/indexed_db_leveldb_coding.h
[add] https://crrev.com/532ef169eb6fac02407fddf270e107753c06a29a/content/browser/indexed_db/indexed_db_leveldb_operations.cc
[add] https://crrev.com/532ef169eb6fac02407fddf270e107753c06a29a/content/browser/indexed_db/indexed_db_leveldb_operations.h
[add] https://crrev.com/532ef169eb6fac02407fddf270e107753c06a29a/content/browser/indexed_db/indexed_db_reporting.cc
[add] https://crrev.com/532ef169eb6fac02407fddf270e107753c06a29a/content/browser/indexed_db/indexed_db_reporting.h

Project Member

Comment 5 by bugdroid1@chromium.org, Oct 12 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d36cedfd5f4705ea41cda5f79e854bf679118c0b

commit d36cedfd5f4705ea41cda5f79e854bf679118c0b
Author: Daniel Murphy <dmurph@chromium.org>
Date: Thu Oct 12 00:04:54 2017

[IndexedDB] Moving metadata reading and writing to dedicated classes

Prerequisite for more cleanup, adding more tests, and adding metadata
for sweeper. Only non-move code changes were to use std::move in
IndexedDBDatabase for metadata classes instead of copying.

Bug:  725565 
Change-Id: I693d3c90fab27cd25f29517a15b4877ab071513c
Reviewed-on: https://chromium-review.googlesource.com/703636
Reviewed-by: Nick Carter <nick@chromium.org>
Reviewed-by: Victor Costan <pwnall@chromium.org>
Reviewed-by: Joshua Bell <jsbell@chromium.org>
Commit-Queue: Daniel Murphy <dmurph@chromium.org>
Cr-Commit-Position: refs/heads/master@{#508183}
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/BUILD.gn
[add] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/fake_indexed_db_metadata_coding.cc
[add] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/fake_indexed_db_metadata_coding.h
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_backing_store.cc
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_backing_store.h
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_backing_store_unittest.cc
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_class_factory.cc
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_class_factory.h
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_database.cc
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_database.h
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_database_unittest.cc
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_factory_impl.cc
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_fake_backing_store.cc
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_fake_backing_store.h
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_leveldb_coding.cc
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_leveldb_coding.h
[add] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_metadata_coding.cc
[add] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_metadata_coding.h
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/indexed_db_transaction_unittest.cc
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/mock_browsertest_indexed_db_class_factory.cc
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/browser/indexed_db/mock_browsertest_indexed_db_class_factory.h
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/common/indexed_db/indexed_db_key_path.cc
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/common/indexed_db/indexed_db_key_path.h
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/common/indexed_db/indexed_db_metadata.cc
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/common/indexed_db/indexed_db_metadata.h
[modify] https://crrev.com/d36cedfd5f4705ea41cda5f79e854bf679118c0b/content/test/BUILD.gn

Comment 6 by dmu...@chromium.org, Oct 12 2017

Description: Show this description

Comment 7 by dmu...@chromium.org, Oct 12 2017

Cc: rkaplow@chromium.org
CCing rkaplow@chromium.org as finch ambassador.

Still waiting on https://chromium-review.googlesource.com/c/chromium/src/+/711298 before enabling this feature, just getting finch stuff set up.
Project Member

Comment 8 by bugdroid1@chromium.org, Oct 25 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f7f1e9b586a74cc941a3e2f9bbda2a2f82aa0a40

commit f7f1e9b586a74cc941a3e2f9bbda2a2f82aa0a40
Author: Daniel Murphy <dmurph@chromium.org>
Date: Wed Oct 25 03:24:13 2017

[IndexedDB] Throttle sweeper globally and by origin

Bug:  725565 
Change-Id: Icb5bb1dd7114d1468ad55947a51ce97219d5c3e5
Reviewed-on: https://chromium-review.googlesource.com/711298
Commit-Queue: Daniel Murphy <dmurph@chromium.org>
Reviewed-by: Victor Costan <pwnall@chromium.org>
Cr-Commit-Position: refs/heads/master@{#511349}
[modify] https://crrev.com/f7f1e9b586a74cc941a3e2f9bbda2a2f82aa0a40/content/browser/indexed_db/indexed_db_backing_store_unittest.cc
[modify] https://crrev.com/f7f1e9b586a74cc941a3e2f9bbda2a2f82aa0a40/content/browser/indexed_db/indexed_db_context_impl.cc
[modify] https://crrev.com/f7f1e9b586a74cc941a3e2f9bbda2a2f82aa0a40/content/browser/indexed_db/indexed_db_factory_impl.cc
[modify] https://crrev.com/f7f1e9b586a74cc941a3e2f9bbda2a2f82aa0a40/content/browser/indexed_db/indexed_db_factory_impl.h
[modify] https://crrev.com/f7f1e9b586a74cc941a3e2f9bbda2a2f82aa0a40/content/browser/indexed_db/indexed_db_factory_unittest.cc
[modify] https://crrev.com/f7f1e9b586a74cc941a3e2f9bbda2a2f82aa0a40/content/browser/indexed_db/indexed_db_leveldb_coding.cc
[modify] https://crrev.com/f7f1e9b586a74cc941a3e2f9bbda2a2f82aa0a40/content/browser/indexed_db/indexed_db_leveldb_coding.h
[modify] https://crrev.com/f7f1e9b586a74cc941a3e2f9bbda2a2f82aa0a40/content/browser/indexed_db/indexed_db_leveldb_operations.cc
[modify] https://crrev.com/f7f1e9b586a74cc941a3e2f9bbda2a2f82aa0a40/content/browser/indexed_db/indexed_db_leveldb_operations.h
[modify] https://crrev.com/f7f1e9b586a74cc941a3e2f9bbda2a2f82aa0a40/content/browser/indexed_db/indexed_db_tombstone_sweeper.cc
[modify] https://crrev.com/f7f1e9b586a74cc941a3e2f9bbda2a2f82aa0a40/content/browser/indexed_db/leveldb_coding_scheme.md

Cc: -rkaplow@chromium.org jwd@chromium.org
moving to jwd instead of rkaplow because rkaplow is OOO.
Status: Fixed (was: Assigned)
Done?

Tentatively resolving

Sign in to add a comment