Since I/O for in-memory databases is fast; eliminate or minimize database caching to reduce RAM consumption
Up for review: https://chromium-review.googlesource.com/c/chromium/src/+/647329
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/13f84c7c23dc76bb5b2c040331d1eb5240a62da5 commit 13f84c7c23dc76bb5b2c040331d1eb5240a62da5 Author: Chris Mumford <cmumford@chromium.org> Date: Fri Sep 22 21:46:10 2017 leveldb: Disable block cache for in-memory db's. I/O for in-memory env's is already fast so reduce memory use by not using a cache. TBR=kinuko@chromium.org,megjablon@chromium.org,nyquist@chromium.org,zea@chromium.org,mvanouwerkerk@chromium.org,nhiroki@chromium.org,phajdan.jr@chromium.org TBRing reviewers: kinuko@chromium.org: chrome/browser/extensions/api/sync_file_system/* chrome/browser/sync_file_system/* content/browser/service_worker/service_worker_database.cc megjablon@chromium.org: components/data_reduction_proxy/core/browser/data_store_impl.cc nyquist@chromium.org: components/leveldb_proto/* zea@chromium.org: components/sync/model_impl/model_type_store_backend.cc mvanouwerkerk@chromium.org: content/browser/notifications/* nhiroki@chromium.org: storage/browser/fileapi/quota/quota_backend_impl_unittest.cc phajdan.jr@chromium.org: PRESUBMIT.py Bug: 761398 Change-Id: I58884dc0bf98fec46d12227c96ac77107a17f034 Reviewed-on: https://chromium-review.googlesource.com/647329 Reviewed-by: Chris Mumford <cmumford@chromium.org> Reviewed-by: Nicolas Zea <zea@chromium.org> Reviewed-by: Victor Costan <pwnall@chromium.org> Commit-Queue: Chris Mumford <cmumford@chromium.org> Cr-Commit-Position: refs/heads/master@{#503866} [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/PRESUBMIT.py [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/extensions/api/sync_file_system/sync_file_system_browsertest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/drive_backend/conflict_resolver_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/drive_backend/drive_backend_sync_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/drive_backend/leveldb_wrapper_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/drive_backend/list_changes_task_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/drive_backend/local_to_remote_syncer_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/drive_backend/metadata_database.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/drive_backend/metadata_database_index_on_disk_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/drive_backend/metadata_database_index_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/drive_backend/metadata_database_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/drive_backend/register_app_task_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/drive_backend/remote_to_local_syncer_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/drive_backend/sync_engine_initializer_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/drive_backend/sync_worker_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/local/local_file_change_tracker.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/local/local_file_change_tracker_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/local/local_file_sync_context_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/local/local_file_sync_service_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/local/syncable_file_operation_runner_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/local/syncable_file_system_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/chrome/browser/sync_file_system/sync_file_system_service_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/components/data_reduction_proxy/core/browser/data_store_impl.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/components/leveldb/leveldb_service_impl.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/components/leveldb_proto/leveldb_database.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/components/sync/model_impl/model_type_store_backend.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/content/browser/indexed_db/indexed_db_tombstone_sweeper_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/content/browser/indexed_db/leveldb/leveldb_database.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/content/browser/notifications/notification_database.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/content/browser/service_worker/service_worker_database.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/storage/browser/fileapi/quota/quota_backend_impl_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/third_party/leveldatabase/README.chromium [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/third_party/leveldatabase/chromium_logger.h [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/third_party/leveldatabase/env_chromium.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/third_party/leveldatabase/env_chromium.h [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/third_party/leveldatabase/env_chromium_unittest.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/third_party/leveldatabase/leveldb_chrome.cc [modify] https://crrev.com/13f84c7c23dc76bb5b2c040331d1eb5240a62da5/third_party/leveldatabase/leveldb_chrome.h
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f20b5542ddd73b666dc36bf4c45beae6fec9e047 commit f20b5542ddd73b666dc36bf4c45beae6fec9e047 Author: Hayato Ito <hayato@chromium.org> Date: Mon Sep 25 10:36:02 2017 Revert "leveldb: Disable block cache for in-memory db's." This reverts commit 13f84c7c23dc76bb5b2c040331d1eb5240a62da5. Reason for revert: This CL is suspicous for continuous Linux ChromiumOS MSan Tests failed. Linux ChromiumOS MSan Tests [32 out of the last 32 builds have failed] - https://uberchromegw.corp.google.com/i/chromium.memory/builders/Linux%20ChromiumOS%20MSan%20Tests?numbuilds=200 - https://uberchromegw.corp.google.com/i/chromium.memory/builders/Linux%20ChromiumOS%20MSan%20Tests/builds/3130 - https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.memory%2FLinux_ChromiumOS_MSan_Tests%2F3130%2F%2B%2Frecipes%2Fsteps%2Fbrowser_tests%2F0%2Flogs%2FSyncFileSystemTest.AuthorizationTest%2F0 [6731:6731:0922/170816.409501:INFO:CONSOLE(0)] "[SUCCESS] (no test)", source: chrome-extension://fplbbjkdcjiaieedkmggclfkfamgmeka/_generated_background_page.html (0) ==6731==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x12f47b6e in key third_party/leveldatabase/src/util/cache.cc:58:9 #1 0x12f47b6e in Unref third_party/leveldatabase/src/util/cache.cc:234:0 #2 0x12f47b6e in Release third_party/leveldatabase/src/util/cache.cc:266:0 #3 0x12f47b6e in leveldb::(anonymous namespace)::ShardedLRUCache::Release(leveldb::Cache::Handle*) third_party/leveldatabase/src/util/cache.cc:372:0 #4 0x12f2d187 in leveldb::Iterator::~Iterator() third_party/leveldatabase/src/table/iterator.cc:16:5 #5 0x12f53539 in ~Iter third_party/leveldatabase/src/table/block.cc:76:14 #6 0x12f53539 in leveldb::Block::Iter::~Iter() third_party/leveldatabase/src/table/block.cc:76:0 #7 0x12f3d93c in ~IteratorWrapper third_party/leveldatabase/src/table/iterator_wrapper.h:23:24 #8 0x12f3d93c in ~TwoLevelIterator third_party/leveldatabase/src/table/two_level_iterator.cc:89:0 #9 0x12f3d93c in leveldb::(anonymous namespace)::TwoLevelIterator::~TwoLevelIterator() third_party/leveldatabase/src/table/two_level_iterator.cc:88:0 #10 0x12f2ec9c in ~IteratorWrapper third_party/leveldatabase/src/table/iterator_wrapper.h:23:24 #11 0x12f2ec9c in ~MergingIterator third_party/leveldatabase/src/table/merger.cc:28:0 #12 0x12f2ec9c in leveldb::(anonymous namespace)::MergingIterator::~MergingIterator() third_party/leveldatabase/src/table/merger.cc:27:0 #13 0x12ecf072 in ~DBIter third_party/leveldatabase/src/db/db_iter.cc:63:5 #14 0x12ecf072 in leveldb::(anonymous namespace)::DBIter::~DBIter() third_party/leveldatabase/src/db/db_iter.cc:62:0 #15 0x1054d42a in operator() buildtools/third_party/libc++/trunk/include/memory:2272:5 Original change's description: > leveldb: Disable block cache for in-memory db's. > > I/O for in-memory env's is already fast so reduce memory use by not using > a cache. > > TBR=kinuko@chromium.org,megjablon@chromium.org,nyquist@chromium.org,zea@chromium.org,mvanouwerkerk@chromium.org,nhiroki@chromium.org,phajdan.jr@chromium.org > > TBRing reviewers: > kinuko@chromium.org: > chrome/browser/extensions/api/sync_file_system/* > chrome/browser/sync_file_system/* > content/browser/service_worker/service_worker_database.cc > megjablon@chromium.org: > components/data_reduction_proxy/core/browser/data_store_impl.cc > nyquist@chromium.org: > components/leveldb_proto/* > zea@chromium.org: > components/sync/model_impl/model_type_store_backend.cc > mvanouwerkerk@chromium.org: > content/browser/notifications/* > nhiroki@chromium.org: > storage/browser/fileapi/quota/quota_backend_impl_unittest.cc > phajdan.jr@chromium.org: > PRESUBMIT.py > > Bug: 761398 > Change-Id: I58884dc0bf98fec46d12227c96ac77107a17f034 > Reviewed-on: https://chromium-review.googlesource.com/647329 > Reviewed-by: Chris Mumford <cmumford@chromium.org> > Reviewed-by: Nicolas Zea <zea@chromium.org> > Reviewed-by: Victor Costan <pwnall@chromium.org> > Commit-Queue: Chris Mumford <cmumford@chromium.org> > Cr-Commit-Position: refs/heads/master@{#503866} TBR=kinuko@chromium.org,zea@chromium.org,nyquist@chromium.org,phajdan.jr@chromium.org,cmumford@chromium.org,nhiroki@chromium.org,mvanouwerkerk@chromium.org,megjablon@chromium.org,pwnall@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. Bug: 761398 Change-Id: I45fa5795dd7fedd6ee826f42c723f111701d3a3b Reviewed-on: https://chromium-review.googlesource.com/680736 Reviewed-by: Hayato Ito <hayato@chromium.org> Reviewed-by: Kinuko Yasuda <kinuko@chromium.org> Commit-Queue: Marc Treib <treib@chromium.org> Cr-Commit-Position: refs/heads/master@{#504022} [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/PRESUBMIT.py [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/extensions/api/sync_file_system/sync_file_system_browsertest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/drive_backend/conflict_resolver_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/drive_backend/drive_backend_sync_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/drive_backend/leveldb_wrapper_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/drive_backend/list_changes_task_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/drive_backend/local_to_remote_syncer_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/drive_backend/metadata_database.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/drive_backend/metadata_database_index_on_disk_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/drive_backend/metadata_database_index_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/drive_backend/metadata_database_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/drive_backend/register_app_task_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/drive_backend/remote_to_local_syncer_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/drive_backend/sync_engine_initializer_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/drive_backend/sync_worker_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/local/local_file_change_tracker.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/local/local_file_change_tracker_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/local/local_file_sync_context_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/local/local_file_sync_service_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/local/syncable_file_operation_runner_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/local/syncable_file_system_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/chrome/browser/sync_file_system/sync_file_system_service_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/components/data_reduction_proxy/core/browser/data_store_impl.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/components/leveldb/leveldb_service_impl.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/components/leveldb_proto/leveldb_database.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/components/sync/model_impl/model_type_store_backend.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/content/browser/indexed_db/indexed_db_tombstone_sweeper_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/content/browser/indexed_db/leveldb/leveldb_database.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/content/browser/notifications/notification_database.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/content/browser/service_worker/service_worker_database.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/storage/browser/fileapi/quota/quota_backend_impl_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/third_party/leveldatabase/README.chromium [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/third_party/leveldatabase/chromium_logger.h [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/third_party/leveldatabase/env_chromium.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/third_party/leveldatabase/env_chromium.h [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/third_party/leveldatabase/env_chromium_unittest.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/third_party/leveldatabase/leveldb_chrome.cc [modify] https://crrev.com/f20b5542ddd73b666dc36bf4c45beae6fec9e047/third_party/leveldatabase/leveldb_chrome.h
Have an internal leveldb fix for empty caches up for review.
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1dc497c0362050f75684809f4c8f0b074f6fb421 commit 1dc497c0362050f75684809f4c8f0b074f6fb421 Author: Chris Mumford <cmumford@chromium.org> Date: Mon Oct 09 22:03:22 2017 leveldb: Disable read caches for in-memory databases. In-memory databases are already fast and a read cache will probably have little or no benefit. Disabling the cache to conserve RAM. This change is a subset of http://crrev.com/c/647329 (reverted) without the changes related to the creation of leveldb_chrome::NewMemEnv(). That was reverted due to a bug in leveldb which was fixed in https://github.com/google/leveldb/commit/1c75e88055e06. Bug: 761398 Change-Id: I64dbf53dbf74ec251afd84aec9b9d751508054ec Reviewed-on: https://chromium-review.googlesource.com/704034 Reviewed-by: Victor Costan <pwnall@chromium.org> Commit-Queue: Chris Mumford <cmumford@chromium.org> Cr-Commit-Position: refs/heads/master@{#507503} [modify] https://crrev.com/1dc497c0362050f75684809f4c8f0b074f6fb421/third_party/leveldatabase/README.chromium [modify] https://crrev.com/1dc497c0362050f75684809f4c8f0b074f6fb421/third_party/leveldatabase/env_chromium.cc [modify] https://crrev.com/1dc497c0362050f75684809f4c8f0b074f6fb421/third_party/leveldatabase/env_chromium.h [modify] https://crrev.com/1dc497c0362050f75684809f4c8f0b074f6fb421/third_party/leveldatabase/env_chromium_unittest.cc [modify] https://crrev.com/1dc497c0362050f75684809f4c8f0b074f6fb421/third_party/leveldatabase/leveldb_chrome.cc [modify] https://crrev.com/1dc497c0362050f75684809f4c8f0b074f6fb421/third_party/leveldatabase/leveldb_chrome.h
Comment 1 by cmumford@chromium.org
, Sep 1 2017