New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 773380 link

Starred by 1 user

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Feature



Sign in to add a comment

Add a simpler storage backend than LevelDB

Project Member Reported by nyquist@chromium.org, Oct 10 2017

Issue description

OS: All

What steps will reproduce the problem?
(1) Start using //components/leveldb_proto

What is the expected result?
It works fine.

What happens instead?
It works fine, but there is some overhead related to storing data in LevelDB, particularly for things that only write out once or read once, etc.

Notes:
It should be possible to provide the same public interface as //components/leveldb_proto/proto_database.h but instead of being backed by LevelDB, it should be backed by something simple.

A)
One idea would be for a database with the proto message Foo have something like:
###
message FooDB {
  map<string, Foo> foos;
}
###

and serialize that to disk as a single file. (Note, if we would need backwards compatibility, we could use:

###
message FooEntry {
  // Required.
  optional string key;
  // Required.
  optional Foo foo;
}

message FooDB {
  repeated FooEntry foos;
}
###

However, that is already what will happen on the wire with the new format in proto3, so I think we don't have to deal with that.

Reading would require reading all fields, and writing would require writing the whole file.

B)
For databases with known few items, another solution could be to have a single file per unique key.
 

Comment 1 by jming@chromium.org, Oct 19 2017

Cc: shaktisahu@chromium.org
Can we also, as part of this work, evaluate the current overhead of leveldb and make sure it's worth the optimization?  I like this idea, I just want to make sure we know what the impact will be.
Components: Internals>Storage

Comment 4 by jsb...@chromium.org, Jan 20 2018

Status: Available (was: Untriaged)

Comment 5 by jsb...@chromium.org, Feb 10 2018

Cc: pwnall@chromium.org
Crazy related idea that I have rumbling around in my head now:

* build a simple key/value store abstraction on top of *SQLite*
* as above but with protobuf values
* build in a migration tool from existing leveldb-backed stores

I'm frustrated enough with leveldb's bloat and fragility that I'd like to see memory/perf comparisons vs. sqlite when used simple k/v store. SQLite might seem like overkill for scenarios like this, but per pwnall@ it's a scenario they've actually tuned for...



Comment 6 by pwnall@chromium.org, Feb 10 2018

Sorry for being unclear!

I meant that SQLite is actually tuned for being an embedded database. They're careful about their memory allocations, and guard against corruption with reasonable consistence.

SQLite is not tuned for the key/value use case. However, there is an "SQLite 4", which is tuned for that case -- https://sqlite.org/src4/doc/trunk/www/design.wiki

Sign in to add a comment