New issue
Advanced search Search tips

Issue 760362 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 1
Type: Bug



Sign in to add a comment

Fix LevelDB's handling of Status::NotFound returned from Env methods

Project Member Reported by pwnall@chromium.org, Aug 29 2017

Issue description

https://crrev.com/c/641789 reverted the LevelDB Env implementation changes in https://crrev.com/c/563652 because it caused https://crbug.com/746155. However, this also brings back  https://crbug.com/738961 , which the reverted changes were aiming to fix.

This issue tracks fixing LevelDB's handling of Status::NotFound coming from Env methods, and re-landing the LevelDB Env implementation changes in https://crrev.com/c/563652.
 
Below is a trace of all the methods involved in opening a LevelDB database in https://crbug.com/746155. Going through everything will be non-trivial.


LazyLevelDb::EnsureDbIsOpen
-> leveldb_env::OpenDB
-> DBTracker::OpenDatabase
-> leveldb::DB::Open
  
leveldb::DB::Open calls:
 1. DBImpl::DBImpl (copies options into new object)
 2. leveldb::port::Mutex::Lock (not very relevant)
 3. DBImpl::Recover (status returned if not OK)
 4. VersionSet::NewFileNumber (uint64_t, no status)
 5. leveldb::LogFileName with the number above (std::string, no status)
 6. Env::NewWritableFile with db_name + "/" + number above + ".log" (status returned if not OK)
    - implemented by to ChromiumEnv::NewWritableFile or MojoEnv::NewWritableFile
 7. VersionEdit::SetLogNumber with the number above (void, no status)
 8. leveldb::log::Writer::Writer with db_name + "/" + number above + ".log" (constructor, no status)
 9. MemTable::MemTable (constructor, no status)
10. MemTable::MemTable::Ref (void, no status)
11. (if save_manifest) VersionEdit::SetPrevLogNumber with 0 (void, no status)
12. (if save_manifest) VersionEdit::SetLogNumber (void, no status)
13. (if save_manifest) VersionSet::LogAndApply (status returned if not OK)
14. DBImpl::DeleteObsoleteFiles (void, no status)
15. DBImpl::MaybeScheduleCompaction (void, no status)
    - will schedule a compaction on the background thread

DBImpl::Recover calls:
 1. Env::CreateDir (status ignored)
 2. Env::LockFile with db_name + "/LOCK" (status returned if not OK)
 3. Env::FileExists (boolean status)
    - if it returns false and !create_if_missing, returns Status::InvalidArgument
 4. DBImpl::NewDB (status returned if not OK)
 5. VersionSet::Recover (status returned if not OK)
 6. VersionSet::LogNumber (uint64_t, no status)
 7. VersionSet::PrevLogNumber (uint64_t, no status)
 8. Env::GetChildren with db_name (status returned if not OK)
 9. VersionSet::AddLiveFiles (void, no status)
10. (loop) leveldb::ParseFileName with the files above
11. (loop) DBImpl::RecoverLogFile with numbers from the log files above (status returned if not OK)
12. VersionSet::MarkFileNumberUsed (void, no status)
13. VersionSet::LastSequence (uint64_t, no status)
14. VersionSet::SetLastSequence (void, no status)

DBImpl::NewDB calls:
1. VersionSet::VersionSet (constructor, no status)
2. VersionSet setters, no return value
3. Env::NewWritableFile with db_name + "/MANIFEST-000001" (status returned if not OK)
4. leveldb::log::Writer::Writer with the WritableFile above
5. VersionEdit::RecordTo, no return value
6. leveldb::log::Writer::AddRecord
   - string operations
   - leveldb::log::Writer::EmitPhysicalRecord
     - more string operations
     - leveldb::WritableFile::Append (status returned if not OK)
     - leveldb::WritableFile::Flush (status returned if not OK)
7. leveldb::SetCurrentFile
   - string operations to obtain "MANIFEST-000001\n"
   - leveldb::WriteStringToFileSync with db_name + "/000001.dbtmp" (status returned if not OK)
   - Env::RenameFile with db_name + "/000001.dbtmp" and db_name + "/CURRENT" (status returned if not OK)
   - (if RenameFile fails) Env::DeleteFile with db_name + "/000001.dbtmp" (status returned if not OK)
8. (if a call above fails) leveldb::DeleteFile with db_name + "/MANIFEST-000001" (status ignored)

VersionSet::Recover calls:
 1. leveldb::ReadFileToString with db_name + "/CURRENT" (status returned if not OK)
 2. Env::NewSequentialFile with db_name + "/" + the file name in "CURRENT" (NotFound converted to Corruption)
 3. VersionSet::Builder::Builder (constructor, no status)
 4. leveldb::log::Reader::Reader with the SequentialFile above
 5. (loop) leveldb::log::Reader::ReadRecord returns true or false
 6. (loop) VersionEdit::DecodeFrom (status returned if not OK)
    - may return Corruption or OK
 7. (loop) VersionSet::Builder::Apply (no status)
 8. VersionSet::MarkFileNumberUsed (no status)
 9. leveldb::Version::Version() (constructor, no status)
10. VersionSet::Builder::SaveTo with the Version above (no status)
11. VersionSet::Finalize with the Version above (no status)
12. VersionSet::AppendVersion with the Version above (no status)
13. VersionSet::ReuseManifest (returns bool, no status)
    - called with dscname = db_name + "/" + the file name in "CURRENT"
    - called with dscbase = the file name in "CURRENT"
    - leveldb::ParseFileName with the file name in "CURRENT"
    - Env::GetFileSize with db_name + "/" + the file name in "CURRENT"
    - Env::NewAppendableFile with db_name + "/" + the file name in "CURRENT" (non-OK status returns false)
    - returns bool, used to set instance fields

DBImpl::RecoverLogFile calls:
 1. leveldb::LogFileName with db_name and the number given to RecoverLogFile (std::string, no status)
 2. Env::NewSequentialFile with db_name + "/" + log_number + ".log" (maybe status returned if not OK)
 3. (if status not OK) DBImpl::MaybeIgnoreError
    - coerces status to OK if options_.paranoid_checks is false
 4. leveldb::log::Reader::Reader with the file above and a custom Reporter (status returned if not OK)
 5. WriteBatch::WriteBatch (constructor, no status)
 6. (loop) leveldb::log::Reader::ReadRecord (bool, no status)
 7. (loop) WriteBatchInternal::SetContents (void, no status)
 8. (once in loop) MemTable::MemTable (constructor, no status)
 9. (loop) WriteBatchInternal::InsertInto (status returned if not OK)
    - (anonymous namespace)::MemTableInserter::MemTableInserter (constructor, no status)
    - WriteBatchInternal::Sequence (SequenceNumber no status)
    - WriteBatch::Iterate (status returned if not OK)
      - may return Corruption or OK
10. (loop) DBImpl::MaybeIgnoreError
    - coerces status to OK if options_.paranoid_checks is false
11. (loop) WriteBatchInternal::Sequence (SequenceNumber, no status)
12. (loop) WriteBatchInternal::Count (int, no status)
13. (loop) MemTable::ApproximateMemoryUsage (size_t, no status)
14. (loop, if MemTable full) DBImpl::WriteLevel0Table (status returned if not OK)
15. (loop, if MemTable full) MemTable::Unref (void, no status)
16. (if reuse_logs) Env::GetFileSize with db_name + "/" + log_number + ".log" (status returned if not OK)
17. (if reuse_logs) Env::NewAppendableFile with db_name + "/" + log_number + ".log" (status returned if not OK)
18. (if reuse_logs) leveldb::log::Writer::Writer with the AppendableFile above (constructor, no status)
19. (if reuse_logs) MemTable::MemTable (constructor, no status)
20. (if reuse_logs) MemTable::Ref (void, no status)
21. (complex if) DBImpl::WriteLevel0Table (status returned if not OK)
22. (complex if) MemTable::Unref (void, no status)

DBImpl::WriteLevel0Table calls:
1. Env::NowMicros (uint64_t, no status)
2. VersionSet::NewFileNumber (uint64_t, no status)
3. MemTable::NewIterator (Iterator, no status)
4. leveldb::BuildTable (status returned if not OK)
5. Version::PickLevelForMemTableOutput (int, no status)
6. VersionEdit::AddFile (void, no status)
7. Env::NowMicros (uint64_t, no status)
8. DBImpl::CompactionStats::Add (void, no status)

leveldb::BuildTable calls:
 1. Iterator::SeekToFirst (void, no status)
 2. leveldb::TableFileName (std::string, no status)
 3. Env::NewWritableFile with db_name + "/" + meta->number + ".ldb" (status returned if not OK)
 4. TableBuilder::TableBuilder with WritableFile above (constructor, no status)
 5. InternalKey::DecodeFrom (void, no status)
 6. (loop) Iterator::Valid (bool, no status)
 7. (loop) Iterator::key (Slice, no status)
 8. (loop) InternalKey::DecodeFrom (void, no status)
 9. (loop) Iterator::Next (void, no status)
10. TableBuilder::Add (void, status indirectly returned if not OK via TableBuilder::Rep)
    - Comparator::Compare (int, no status)
    - Comparator::FindShortestSeparator (void, no status)
    - BlockHandle::EncodeTo on TableBuilder::Rep's pending_handle (void, no status)
    - BlockBuilder::Add on index_block with key = last_key set from above, value = encoded handle from above (void, no status)
    - FilterBlockBuilder::AddKey with key (void, no status)
    - BlockBuilder::Add on data_block with key, value (void, no status)
    - TableBuilder::Flush (void, status indirectly returned if not OK via TableBuilder::Rep)
11. TableBuilder::Finish (status returned if not OK)
12. TableBuilder::FileSize (uint64_t, no status)
13. WriteStatus::Sync (status returned if not OK)
14. WriteStatus::Close (status returned if not OK)
15. TableCache::NewIterator with meta->number (Iterator, status implied in iterator)
16. Iterator::Status on the Iterator above (status returned if not OK)
17. Iterator::Status on the iterator argument (status returned if not OK)
18. (if error or empty file) Env::DeleteFile with db_name + "/" + meta->number + ".ldb" (status ignored)

TableBuilder::Flush calls (void, status indirectly returned if not OK via TableBuilder::Rep):
1. TableBuilder::WriteBlock (void, status indirectly returned if not OK via TableBuilder::Rep)
2. WritableFile::Flush (status indirectly returned if not OK via TableBuilder::Rep)
3. FilterBlockBuilder::StartBlock (void, no status)
   - (loop) FilterBlockBuilder::GenerateFilter (void, no status)
      - FilterPolicy::CreateFilter (void, no status)

TableBuilder::Finish calls  (void, status indirectly returned if not OK via TableBuilder::Rep):
 1. BlockHandle::BlockHandle
 2. TableBuilder::Flush
    - TableBuilder::WriteBlock (void, status indirectly returned if not OK via TableBuilder::Rep)
    - WritableFile::Flush (status indirectly returned if not OK via TableBuilder::Rep)
    - FilterBlockBuilder::StartBlock (void, no status)
      - (loop) FilterBlockBuilder::GenerateFilter (void, no status)
        - FilterPolicy::CreateFilter (void, no status)
 3. TableBuilder::WriteRawBlock with filter block (void, status indirectly returned if not OK via TableBuilder::Rep)
 4. BlockBuilder::BlockBuilder (constructor, no status)
 5. FilterPolicy::Name (const char*, no status)
 6. BlockHandle::EncodeTo on filter_block_handle (void, no status)
 7. BlockBuilder::Add on metablock with key = "filter." + name from above, value = encoded handle from above  (void, no status)
 8. TableBuilder::WriteBlock with metaindex block (void, status indirectly returned if not OK via TableBuilder::Rep)
 9. Comparator::FindShortSuccessor into TableBuilder::Rep's last_key (void, no status)
10. BlockHandle::EncodeTo on TableBuilder::Rep's pending_handle (void, no status)
11. BlockBuilder::Add with key = last_key set from above, value = encoded handle from above (void, no status)
12. TableBuilder::WriteBlock with index block (void, status indirectly returned if not OK via TableBuilder::Rep)
13. leveldb::Footer::Footer (constructor, no status)
14. leveldb::Footer::set_metaindex_handle (void, no status)
15. leveldb::Footer::set_index_handle (void, no status)
16. Footer::EncodeTo (void, no status)
17. WritableFile::Append with encoded footer from above (status indirectly returned if not OK via TableBuilder::Rep)

TableBuilder::WriteBlock calls (void, status indirectly returned if not OK via TableBuilder::Rep):
1. BlockBuilder::Finish (Slice, no status)
2. port::Snappy_Compress (bool, no status)
3. TableBuilder::WriteRawBlock (void, status indirectly returned if not OK via TableBuilder::Rep)

TableBuilder::WriteRawBlock calls (void, status indirectly returned if not OK via TableBuilder::Rep):
1. BlockHandle::set_offset (void, no status)
2. BlockHandle::set_size (void, no status)
3. WritableFile::Append (status set in TableBuilder::Rep, eventually status returned if not OK)
4. leveldb::crc32c::Value (uint32_t, no status)
5. leveldb::crc32c::Mask (uint32_t, no status)
6. leveldb::EncodeFixed32 (void, no status)
7. WritableFile::Append (status set in TableBuilder::Rep, eventually status returned if not OK)

TableCache::NewIterator calls:
1. TableCache::FindTable (status indirectly returned if not OK, via leveldb::Iterator::status)
   - leveldb::EncodeFixed64 with file_number
   - leveldb::Cache::Lookup (leveldb::Cache::Handle*, no status)
   - leveldb::TableFileName with db_name, file_number (std::string, no status)
   - Env::NewRandomAccessFile with db_name + "/" + file_number + ".ldb"
   - (on failure) leveldb::SSTTableFileName with db_name, file_number (std::string, no status)
   - (on failure) Env::NewRandomAccessFile with db_name + "/" + file_number + ".sst" (status returned if not OK)
   - leveldb::Table::Open with RandomAccessFile from above (status returned if not OK)
     - returns Corruption if file is too small
     - RandomAccessFile::Read on file with position and buffer for footer (status returned if not OK)
     - leveldb::Footer::Footer (constructor, no status)
     - leveldb::Footer::DecodeFrom with footer buffer (status returned if not OK)
       - can return Corruption
       - BlockHandle::DecodeFrom on metaindex_handle (status returned if not OK)
         - returns Corruption or OK
       - BlockHandle::DecodeFrom on index_handle (status returned if not OK)
         - returns Corruption or OK
     - leveldb::ReadBlock (status returned if not OK)
     - leveldb::Block::Block (constructor, no status)
     - leveldb::Table::Rep::Rep (constructor, no status)
     - leveldb::Table::Table with rep above (constructor, no status)
     - leveldb::Table::ReadMeta (void, no status)
       - leveldb::ReadBlock with metaindex_handle (non-OK status results in early return, but is not propagated)
       - leveldb::Block::Block (constructor, no status)
       - leveldb::Block::NewIterator with leveldb::BitwiseComparator (constructor, no status)
       - FilterPolicy::Name (const char*, no status)
       - leveldb::Iterator::Seek with "filter." + name above (void, status returned indirectly but ignored)
       - leveldb::Iterator::Valid (bool, no status)
       - leveldb::Table::ReadFilter (void, no status)
   - leveldb::Cache::Insert (leveldb::Cache::Handle*, no status)
2. leveldb::Cache::Value on the handle from FindTable (void*, no status)
3. leveldb::Table::NewIterator (leveldb::Iterator*, status indirectly returned if not OK, via leveldb::Iterator::status)
   - Block::NewIterator on leveldb::Table::Rep::index_block (leveldb::Iterator*, status indirectly returned if not OK, via leveldb::Iterator::status)
     - returns Corruption if the block is too small
   - leveldb::Table::BlockReader (leveldb::Iterator*, status indirectly returned if not OK, via leveldb::Iterator::status)
     - leveldb::EncodeFixed64 on leveldb::Table::Rep::cache_id (void, no status)
     - leveldb::EncodeFixed64 on BlockHandle::offset() (void, no status)
     - BlockCache::Lookup with key composed from the 2 fragments above (leveldb::Cache::Handle*, no status)
     - leveldb::ReadBlock (status returned if not OK)
     - leveldb::Block::Block (constructor, no status)
     - (if the block contents is cacheable) leveldb::Cache::Insert (leveldb::Cache::Handle*, no status)
     - leveldb::Block::NewIterator on the leveldb::Block created above (leveldb::Iterator*, status indirectly returned if not OK, via leveldb::Iterator::status)
       - indirectly returns Corruption if the block contents is too small
     - leveldb::Iterator::RegisterCleanup (void, no status)
   - leveldb::NewTwoLevelIterator (leveldb::Iterator*, no status)

leveldb::ReadBlock calls:
1. RandomAccessFile::Read with position and buffer for whole block (status returned if not OK)
2. returns Corruption if the read didn't get the expected block size
3. crc32c::Unmask (uint32_t, no status)
4. crc32c::Value (uint32_t, no status)
5. returns Corruption if CRC32C doesn't match expected value
6. (if compression) port::Snappy_GetUncompressedLength (bool, no status)
7. (if compression) port::Snappy_Uncompress (bool, no status)


VersionSet::LogAndApply calls:
 1. (skipped) VersionEdit::SetLogNumber (void, no status)
 2. (skipped) VersionEdit::SetPrevLogNumber (void, no status)
 3. VersionEdit::SetNextFile (void, no status)
 4. VersionEdit::SetLastSequence 

Comment 2 by jsb...@chromium.org, Jan 18 2018

Components: -Blink>Storage Internals>Storage

Sign in to add a comment