New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 654819 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Closed: Mar 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux , Windows , Mac
Pri: 3
Type: Bug



Sign in to add a comment

Investigate storing Rice encoded hashes on disk

Project Member Reported by vakh@chromium.org, Oct 11 2016

Issue description

V4Store currently stores RAW hashes on disk so that the load from disk on startup is fast. My non-scientific tests showed that Rice decoding takes a total of ~8 seconds for Malware and SocEng lists on Linux. It is unacceptable to block resource loads for that long.

This is doable if we can find a way to speed up Rice-decoding hash prefixes.
 

Comment 1 by vakh@chromium.org, Mar 7 2017

Cc: nparker@chromium.org sh...@chromium.org
Status: WontFix (was: Assigned)
The current size of the store files is just over 6MB. Given current disk sizes, reducing it to half (or less) doesn't really move the needle.
Also, given that we store the updates in memory in RAW form, using the same amount of space on-disk is a non-issue.

+nparker@, shess@: Please feel free to re-open if you disagree.

Comment 2 by sh...@chromium.org, Mar 7 2017

I agree with this, but something to keep in mind is update I/O costs, especially for low-end devices which may have inadequate flash controllers.  Even there, though, I think the better wins are in strategies which use log files to collect incremental changes with periodic compaction, which can plausibly cut I/O by an amortized order of magnitude.

Sign in to add a comment