Add a simpler storage backend than LevelDB |
||||
Issue description
OS: All
What steps will reproduce the problem?
(1) Start using //components/leveldb_proto
What is the expected result?
It works fine.
What happens instead?
It works fine, but there is some overhead related to storing data in LevelDB, particularly for things that only write out once or read once, etc.
Notes:
It should be possible to provide the same public interface as //components/leveldb_proto/proto_database.h but instead of being backed by LevelDB, it should be backed by something simple.
A)
One idea would be for a database with the proto message Foo have something like:
###
message FooDB {
map<string, Foo> foos;
}
###
and serialize that to disk as a single file. (Note, if we would need backwards compatibility, we could use:
###
message FooEntry {
// Required.
optional string key;
// Required.
optional Foo foo;
}
message FooDB {
repeated FooEntry foos;
}
###
However, that is already what will happen on the wire with the new format in proto3, so I think we don't have to deal with that.
Reading would require reading all fields, and writing would require writing the whole file.
B)
For databases with known few items, another solution could be to have a single file per unique key.
,
Oct 19 2017
Can we also, as part of this work, evaluate the current overhead of leveldb and make sure it's worth the optimization? I like this idea, I just want to make sure we know what the impact will be.
,
Nov 15 2017
,
Jan 20 2018
,
Feb 10 2018
Crazy related idea that I have rumbling around in my head now: * build a simple key/value store abstraction on top of *SQLite* * as above but with protobuf values * build in a migration tool from existing leveldb-backed stores I'm frustrated enough with leveldb's bloat and fragility that I'd like to see memory/perf comparisons vs. sqlite when used simple k/v store. SQLite might seem like overkill for scenarios like this, but per pwnall@ it's a scenario they've actually tuned for...
,
Feb 10 2018
Sorry for being unclear! I meant that SQLite is actually tuned for being an embedded database. They're careful about their memory allocations, and guard against corruption with reasonable consistence. SQLite is not tuned for the key/value use case. However, there is an "SQLite 4", which is tuned for that case -- https://sqlite.org/src4/doc/trunk/www/design.wiki |
||||
►
Sign in to add a comment |
||||
Comment 1 by jming@chromium.org
, Oct 19 2017