New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 665382 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Oct 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 1
Type: Bug

Blocked on:
issue 701518

Blocking:
issue 665384
issue 754419



Sign in to add a comment

Update sqlite3 revision

Project Member Reported by mmoroz@chromium.org, Nov 15 2016

Issue description

Current ToT revision has a fix of a high severity security vulnerability (https://www.sqlite.org/src/info/c5dbc599b910c02a) found by our OSS-Fuzz project (https://clusterfuzz-external.appspot.com/v2/testcase-detail/5770842466156544).

Also upstream contains updated fuzz target. We should use it instead of our existing fuzz target (I'll file a separate bug for this).
 

Comment 1 by mmoroz@chromium.org, Nov 15 2016

Blocking: 665384

Comment 2 by mmoroz@chromium.org, Nov 15 2016

Description: Show this description

Comment 3 by sh...@chromium.org, Nov 15 2016

I don't think the indicated SQLite CL actually fixes this issue.  The variable moved from the constant section to the recursion section is already in the recursion section for our SQLite.  Probably later development attempted to move it up and now that CL is moving it back.

Digging deeper, though.

Comment 4 by sh...@chromium.org, Nov 15 2016

I am having no luck at all reproducing this on Linux or OSX.  Building sql_unittests with gn flags:
is_asan = true
enable_nacl = false  # Necessary until NaCl GN build is more complete.
is_debug = false  # Release build.
use_goma = true

then running this test:
TEST_F(SQLConnectionTest, ASAN) {
  sql::test::ScopedErrorExpecter expecter;
  expecter.ExpectError(SQLITE_ERROR);
  ASSERT_FALSE(db().Execute("CREATE  table   i(O\rUNIQUE\rUNIQUE"));
  ASSERT_TRUE(expecter.SawExpectedErrors());
}

works just fine.

I've spent a couple hours trying to figure out how to repro on a Linux box, but AFAICT basically all of the clusterfuzz and oss-fuzz "How to repro locally" documentation describes some parallel universe of code and output.  So I'm not going to continue trying to track this down.

Comment 5 by mmoroz@chromium.org, Nov 16 2016

Cc: tanin@chromium.org
Have you tried to follow this instruction: https://github.com/google/oss-fuzz/blob/master/docs/reproducing.md ? Doesn't it work?

Sadly to hear about your experience. OSS-Fuzz is in active development stage, so there can be some mistakes or missing points, though reproducing doc should be ok AFAIK.

Would you mind sharing what exactly is incorrect or is "from parallel universe"? We would appreciate your feedback.

Comment 6 by mmoroz@chromium.org, Nov 21 2016

Yet another one high severity bug has been fixed upstream: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=199

Patch link: https://www.sqlite.org/src/info/0a98c8d76ac86412


Comment 7 by aarya@google.com, Nov 21 2016

Labels: -Type-Bug Type-Bug-Security

Comment 8 by sh...@chromium.org, Nov 21 2016

WRT #6, I can't see https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=199 either as shess@chromium.org nor as shess@google.com .

I'm going to try to walk through things, again, and see if I can't figure out where they go awry.  I think the "reproducing.md" document asked for information that wasn't present in the clusterfuzz output, and which I couldn't access in the oss-fuzz bug-tracker, so I assumed that I was supposed to follow the cluster-fuzz repro steps.  But that seems to be for a different edition of clusterfuzz than is used on Chromium project, because it seemed to refer to things which didn't match up with anything in the report.

Comment 9 by sh...@chromium.org, Nov 21 2016

OK, so when I follow the reproducing.md instructions, and run:

docker run --rm -v /tmp/fuzz-3-sqlite3_ossfuzz:/testcase -t ossfuzz/sqlite3 reproduce ossfuzz

I get:
+ ossfuzz /testcase
INFO: Seed: 191371414
INFO: Loaded 0 modules (0 guards): 
ossfuzz: Running 1 inputs 1 time(s) each.
Running: /testcase
Executed /testcase in 1 ms

I couldn't tell you if that reproduced the problem, or not.  I'd guess not, because it didn't print the ASAN backtrace?  Given that it's in some docker container, I'm not sure how to verify that it's even building with ASAN in there.  There's a bin/compile file which references $SANITIZER_FLAGS, though I can't figure out who sets that.  But the compile output contains things like "-fsanitize=address -fsanitize-coverage=edge,indirect-calls,8bit-counters" in CFLAGS, so somebody must be setting it, so the fact that I'm not getting any output maybe means it's flaky?  Or there's something different about my build environment?


Comment 10 by sh...@chromium.org, Nov 21 2016

In the testcase link from OP, I see:
ASAN_OPTIONS = redzone=16:external_symbolizer_path=/bin/llvm-symbolizer:handle_sigill=1:strict_string_check=1:strict_memcmp=1:detect_container_overflow=1:coverage=0:allocator_may_return_null=1:use_sigaltstack=1:allocator_release_to_os=1:detect_stack_use_after_return=1:alloc_dealloc_mismatch=0:detect_leaks=0:print_scariness=1:strip_path_prefix=/workspace/:max_uar_stack_size_log=16:handle_abort=1:check_malloc_usable_size=0:quarantine_size_mb=10:detect_odr_violation=0:symbolize=1:handle_segv=1:fast_unwind_on_fatal=0

But in the reproduce output I only see:
ASAN_OPTIONS=detect_leaks=0

Looks like that's coming from build.sh, maybe I can edit it (or maybe I'll just try to figure out how to pull the sources out of the docker container thing).
shess@

That docker command runs a test with the head version of sqlite3. And the issue was already fixed there.

If you want to run test with your local version of sqlite3, you can mount your sources over container's: -v $home/src/sqlite3:/src/sqlite3.

Note: if your sqlite3 sources are older than couple weeks that wouldn't work either because they don't have the ossfuzz.c target. If this is the case, then taking ossfuzz.c out of sqlite3 head and compiling with your local version should work.


Comment 12 by sh...@chromium.org, Nov 21 2016

Well, I figured out how to spin up a shell in there and patched the longer ASAN_OPTIONS into build.sh, and the build appeared to take that, and ... still no repro.  I dunno, that's all I can think of to poke at from that end.

Comment 13 by sh...@chromium.org, Nov 21 2016

aizatsky@ um ... I guess I would suggest that if you're going to automatically log bugs into the Chromium bug tracker, then they should be easy to reproduce against the Chromium version of SQLite, so that I can verify the bug fixed in Chromium.  Having to comprehend both Chromium and some arbitrary group of third-party systems puts a barrier in the way of getting these things addressed.

Is this something where I can create a separate ossfuzz.c target that builds against Chromium's third_party/sqlite/amalgamation/sqlite3.c, and then you can target the fuzzing infrastructure at that?  The current system builds _an_ amalgamation and fuzzes against that, but AFAICT it doesn't contain the same SQLITE_* flags as Chromium uses when building the sqlite3.c it uses.
shess@, you can add ossfuzz.c the same way as have another sqlite3 fuzzer (https://cs.chromium.org/chromium/src/third_party/sqlite/BUILD.gn?type=cs&q=sqlite3_prepare_v2_fuzzer&sq=package:chromium&l=270). It should be easy to build following Chromium instructions: https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/reproducing.md#Reproducing-LibFuzzer-ASan-bugs (go/libfuzzer-chrome)

The only possible problem I see is path to headers included in ossfuzz.c, since in Chromium we tend to use #include "third_party/......."

Comment 15 by sh...@chromium.org, Nov 21 2016

Given a Chromium src dir and the original ossfuzz sqlite3 dir at /tmp/sqlite3 at $CHROMIUMSRC, then the following seems to work

docker run --rm -v /tmp/fuzz-3-sqlite3_ossfuzz:/testcase -v$CHROMIUMSRC/third_party/sqlite/src:/src/sqlite3 -v/tmp/sqlite3/test/ossfuzz.c:/src/sqlite3/test/ossfuzz.c -t ossfuzz/sqlite3 reproduce ossfuzz

It fails compile in shell.c, which verifies it's copying the right stuff into the container.  So I patched the Chromium change out (it should be irrelevant).  After that, it all runs successfully, which is to say - still no repro.

Note my comment #3.  The SQLite CL which the OP claims fixes the problem actually addresses a change which I believe landed after our version of SQLite.  I haven't dug down to figure out the history between our SQLite and SQLite's head, but it seems plausible to me that the vulnerability in question never existed in Chromium's SQLite.

[The ref in comment #6 is plausible in our SQLite, but since I can't access the ossfuzz report, I have no basis for comment.]
I've CC'ed shess@ on https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=199.

By the way, do I understand correctly that updating sqlite3 in Chromium is not an easy thing and, to get it done, someone should prove a crash or another significant problem?


Comment 17 by sh...@chromium.org, Nov 21 2016

OK, on https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=199 , I get no-repro at SQLite head, repro when I comment out the lines changed by drh, then no-repro when I roll back to version-3.10.2 and layer in ossfuzz.c .  I got no-repro against Chromium's SQLite, also, so I think it's not an issue for Chromium at this time.

Comment 18 by sh...@chromium.org, Nov 21 2016

WRT updating SQLite, basically I am the only person who spends any time on SQLite, and most Chromium SQLite clients are honestly in maintenance mode with poor test coverage (it might take awhile for anyone to notice errors with any subtlety), so I prefer to update about once a year, rather than attempting some sort of rolling integration system.  Actually doing the code update is not hard, but there are a couple local patches we have which are somewhat annoying to validate.

Notably the WebSQL integration was done by monkey-patching SQLite, and while I have made the actual change cleaner to apply over time, the assumptions baked into the implementation mean that we frequently get bitten by valid SQLite implementation changes.  WebSQL has been deprecated for a long time, so there's not much support for rewriting the implementation to be cleaner - yet every time I bring up deleting the feature, I get pushback.  Since WebSQL is the only place where non-Chromium SQLite code can be injected, maybe the time is ripe to take another pass at that.  [Most fuzzer reports seem to be in parsing-related stuff for very odd statements, and thus should not be relevant to Chromium browser-side SQLite usage.]

Also, to be clear, are you asserting that you have more confidence in the security of rolling SQLite integration than in picking a decent release and sticking with it for awhile?  At least from the fuzzer level?  I ask because while I do trust the SQLite team (they are on average more solid than Chromium developers, IMHO), bugs do get in there, and per the point in my first paragraph, I'm the only resource working on this.  I'd prefer not to get on the treadmill of having to track all SQLite issues in case they impact Chromium.

Comment 19 by sh...@chromium.org, Nov 21 2016

Still regardless of how Chromium approaches SQLite integration, I don't think logging SQLite problems against Chromium is worthwhile unless they repro with the SQLite Chromium is actually using.  That just wastes time in wild goose chases.
Thanks a lot shess@ for detailed explanation, it makes sense. I haven't thought that it's so complicated (i.e. we have some patches on top of the upstream, significantly different build configurations, etc.). I followed the common security sense like "If library A has security bugs and we are using it, we should update it".

I've also added ossfuzz.c target and verified that both issues are not reproducible inside Chromium.

Your concern about usage of sqlite in Chromium differs from its usage inside fuzzer is totally valid. This is one of the reasons why we are encouraging developers to write/update fuzz targets - they have better knowledge how target is being used and how it should be tested.

Since we are not going to update sqlite3 for now, let me add ossfuzz.c fuzzer to the Chromium repo. Then, after an update, we will change sources of that fuzzer_test GN target, but will be running on some existing corpus, which is a good thing.
Project Member

Comment 21 by sheriffbot@chromium.org, Nov 22 2016

Labels: M-54
Project Member

Comment 22 by sheriffbot@chromium.org, Nov 22 2016

Status: Assigned (was: Available)
Cc: -sh...@chromium.org michaeln@chromium.org
Owner: sh...@chromium.org
to reflect reality
Project Member

Comment 25 by sheriffbot@chromium.org, Dec 2 2016

Labels: -M-54 M-55
Project Member

Comment 26 by sheriffbot@chromium.org, Dec 6 2016

shess: Uh oh! This issue still open and hasn't been updated in the last 14 days. This is a serious vulnerability, and we want to ensure that there's progress. Could you please leave an update with the current status and any potential blockers?

If you're not the right owner for this issue, could you please remove yourself as soon as possible or help us find the right one?

If the issue is fixed or you can't reproduce it, please close the bug. If you've started working on a fix, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 27 by sheriffbot@chromium.org, Dec 21 2016

shess: Uh oh! This issue still open and hasn't been updated in the last 29 days. This is a serious vulnerability, and we want to ensure that there's progress. Could you please leave an update with the current status and any potential blockers?

If you're not the right owner for this issue, could you please remove yourself as soon as possible or help us find the right one?

If the issue is fixed or you can't reproduce it, please close the bug. If you've started working on a fix, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 28 by sheriffbot@chromium.org, Jan 14 2017

Labels: Deadline-Exceeded
We commit ourselves to a 60 day deadline for fixing for high severity vulnerabilities, and have exceeded it here. If you're unable to look into this soon, could you please find another owner or remove yourself so that this gets back into the security triage queue?

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 29 by sheriffbot@chromium.org, Jan 26 2017

Labels: -M-55 M-56
Labels: -Type-Bug-Security -Restrict-View-SecurityTeam -Security_Impact-Stable -Security_Severity-High Type-Bug
I don't think we need to keep this in the security queue. The bugs in question have been public for a long time and there's no clear indication they affect Chrome's use of them.

Comment 31 by sh...@chromium.org, Mar 27 2017

Blockedon: 701518
I'm rolling a new SQLite at  issue 701518 , which I think would obsolete this bug.
Awesome! Thank you Scott!

Comment 33 by sh...@chromium.org, Jun 21 2017

Labels: ShessReview

Comment 34 by ecacho@google.com, Jun 27 2017

Components: -UI>Localization UI>Browser
Labels: Needs-TestConfirmation
It doesn't look like this is a localization issue. Can't confirm that it's still relevant to the current English UI
Labels: TE-NeedsTriageHelp
Unable to triage this issue from TE-End, adding "TE-NeedsTriageHelp" label for further triage.
Blocking: 754419
Cc: -michaeln@chromium.org -aizatsky@chromium.org pwnall@chromium.org
Owner: michaeln@chromium.org
Since shess@ is not at Google anymore, assigning this to sqlite owner michaeln@ (https://cs.chromium.org/chromium/src/third_party/sqlite/OWNERS?sq=package:chromium)
For more information, please see https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/reference.md.

The link referenced in the description is no longer valid.

(bulk edit)

Comment 39 by costan@google.com, Oct 24 2017

Owner: pwnall@chromium.org
Status: Fixed (was: Assigned)
I have upgraded SQLite since that bug happened. We're on 3.20.1 now.

Comment 40 by costan@google.com, Oct 24 2017

Cc: michaeln@chromium.org

Sign in to add a comment