New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 629003 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Closed: Aug 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug-Security



Sign in to add a comment

Crash in QPCValueToTimeDelta

Project Member Reported by ClusterFuzz, Jul 18 2016

Issue description

Detailed report: https://cluster-fuzz.appspot.com/testcase?key=4582141380001792

Fuzzer: ifratric-browserfuzzer-v3
Job Type: windows_syzyasan_chrome
Platform Id: windows

Crash Type: UNKNOWN
Crash Address: 0x24cdf8c7
Crash State:
  QPCValueToTimeDelta
  QPCNow
  base::TimeTicks::Now
  
Recommended Security Severity: Medium

Regressed: https://cluster-fuzz.appspot.com/revisions?job=windows_syzyasan_chrome&range=405656:405727

Unminimized Testcase: https://cluster-fuzz.appspot.com/download/AMIfv968JJS4oap8bU07zp8jwBslaBtm2eJz0JjuAsKkj92iwYXdIKeNP9m2DyNOaMEDf9mVZNKIMBXb_NH0uNqCi12k2jrn5MBjOASqOLr1HhKQbTvzpelUruzKPwVJSF7S8Fgj3JyfGw_wIYT4TdUmoQNpAZL0CWfSR_M-Ggm3W_k5Gh0TrAk?testcase_id=4582141380001792


Filer: mmoroz

See https://dev.chromium.org/Home/chromium-security/bugs/reproducing-clusterfuzz-bugs for more information.
 

Comment 1 by mmoroz@chromium.org, Jul 18 2016

Cc: charliea@chromium.org w...@chromium.org
Labels: Pri-2
Owner: m...@chromium.org
mui@, could you please take a look? Looks similar to bug 601442.

Comment 2 by mmoroz@chromium.org, Jul 18 2016

Labels: M-53
Project Member

Comment 3 by sheriffbot@chromium.org, Jul 18 2016

Labels: ReleaseBlock-Beta
This issue is a security regression. If you are not able to fix this quickly, please revert the change that introduced it.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 4 by sheriffbot@chromium.org, Jul 18 2016

Labels: -Pri-2 Pri-1
Project Member

Comment 5 by sheriffbot@chromium.org, Jul 18 2016

Status: Assigned (was: Available)

Comment 6 by ta...@google.com, Jul 18 2016

Components: Blink>JavaScript

Comment 7 by ta...@google.com, Jul 18 2016

Ping miu@

Comment 8 by gov...@chromium.org, Jul 19 2016

M53 beta launch is next week.Your bug is labelled as Beta ReleaseBlock, pls make sure to land the fix before 6:00 PM PST, Friday (07/22/16). Thank you.

Comment 9 by m...@chromium.org, Jul 20 2016

Will look into this tomorrow. But, I suspect it's a WontFix since the code hasn't been changed in a long time, and the test case and stack trace don't indicate the fuzzing values can "reach" the function call in any way.
Project Member

Comment 10 by sheriffbot@chromium.org, Jul 21 2016

Labels: -Security_Impact-Head Security_Impact-Beta
Project Member

Comment 11 by sheriffbot@chromium.org, Jul 21 2016

Labels: -ReleaseBlock-Beta ReleaseBlock-Stable
Labels: -Security_Impact-Beta -ReleaseBlock-Stable Security_Impact-Head ReleaseBlock-Beta
I'm afraid sheriffbot's label changes were a hiccup - this is still a blocker for Friday's M53 (unless it is found to be WontFix, of course!)

Comment 13 by m...@chromium.org, Jul 21 2016

This looks almost exactly like bug 601442.


Comment 14 by m...@chromium.org, Jul 21 2016

Cc: awhalley@chromium.org gov...@chromium.org tanin@chromium.org
BTW--I'm probably not the person who should look at this. I'm trying to set up a Windows system now to do some bisecting w/in the rev range the fuzzer mentioned, and it's taking forever just to "install corp updates" and "gclient sync" and foo. Due to scheduling conflicts, it's unlikely I'll be able too isolate the problem change until EOD Friday.

Also, I'm 100% sure I made no code changes w/in the rev range the fuzzer pointed to. Maybe it'd be best for someone else (such as stability sheriff) to run the bisects and triage this issue?
Cc: falken@chromium.org asvitk...@chromium.org
+current and future stability sheriffs
Cc: -falken@chromium.org -asvitk...@chromium.org
Labels: Stability-Sheriff-Desktop
Cc: falken@chromium.org
Cc: asvitk...@chromium.org
Ah, Restrict-View-SecurityTeam means Stability-Sheriff-Desktop won't be effective, I see.
Cc: brucedaw...@chromium.org
+brucedawson: You may be interested in this crash.
Project Member

Comment 20 by sheriffbot@chromium.org, Jul 22 2016

Labels: -Security_Impact-Head Security_Impact-Beta
Project Member

Comment 21 by sheriffbot@chromium.org, Jul 22 2016

Labels: -ReleaseBlock-Beta ReleaseBlock-Stable
I wasn't able to do stability sheriff duties today.

I tried examining the minidump in the report using the documentation at https://code.google.com/p/syzygy/wiki/SyzyASanBug, but "error_info->alloc_stack" doesn't exist. That may be where someone else can start off.
Labels: -Security_Impact-Beta -ReleaseBlock-Stable Security_Impact-Head ReleaseBlock-Beta
awhalley@, do we need this for next week M53 Dev on Tuesday?

Comment 25 by m...@chromium.org, Jul 22 2016

What's strange here is that the fuzzer is providing a rev range for the regression. I am interpreting this [perhaps incorrectly] as some change in that rev range that may be causing non-problematic code to crash. For example, maybe new/changed code is stomping on memory in a bad way.

Also, yesterday, I was able to confirm the issue is not reproducible on a Linux build. This is definitely a Windows-only issue.

Comment 26 by m...@chromium.org, Jul 22 2016

Labels: -ReleaseBlock-Beta
Owner: brucedaw...@chromium.org
I can't repro this at all, using the fuzzer test HTML, from running bisect_builds.py (32-bit on Windows) running on my local Win7 machine.

Looking at this further, the fuzz testing is a renderer (not browser) crash. So, this doesn't seem release-blocking to me. The other labels (w.r.t. this being a security issue) seem wrong too (more sheriffbot@ automation gone wrong?).

At this point I'm not sure what else I can do here. As this is Windows-specific with a call into Windows platform libraries being the crash point, I'm going to ask Bruce to take a look.
I tried taking a look at the crash dump but I couldn't get the binaries or symbols to load. The instructions at:
    https://code.google.com/p/syzygy/wiki/SyzyASanBug
just seem to say "Use the normal Chrome and Microsoft symbol servers", which I already had set up, but when I load the minidump from:

https://cluster-fuzz.appspot.com/testcase?key=4582141380001792

it failed to match the binary or symbols, whereas on normal crash dumps this works perfectly.

The 'crashing' instruction is:
    mov         dword ptr [ecx],eax.
The values of ECX and ESP are:
    ECX = 24CDF8C4
    ESP = 24CDF898
Therefore ECX is clearly a valid stack location (as in, within range) and it's not clear to me what asan is saying the problem is. Without a binary from the symbol server I can't even examine the parent functions to understand where ECX came from.

The error_type: UNKNOWN_BAD_ACCESS message is not very helpful.

It's hard to imagine how this could be a security bug - there is no user input involved, right? And the code involved has not changed recently, and is mostly straight through with no logic or branches or edge cases or pointer math, so it's not clear to me what could be wrong.

Anybody else have any luck getting the symbols to work? With those I could presumably understand this better, but more information on what asan is claiming the problem is would be helpful.

Comment 28 by nick@chromium.org, Jul 22 2016

Cc: stanisc@chromium.org nick@chromium.org
miu: Crashes like this are assumed to be exploitable, hence the release block treatment.

I couldn't repro a crash on win64 non-asan canary using all the flags in the report (except for --disable-gl-drawing-for-tests).

The call to TimeTicks::Now in the crash is less than a month old. +stanisc, who added it in https://chromium.googlesource.com/chromium/src//+/56ee6be040caa993583170bd5edc472e7b969c9e, might have some insight?

Comment 29 by nick@chromium.org, Jul 22 2016

Labels: -Stability-Sheriff-Desktop
I'm removing from the stability sheriff queue; routing clusterfuzz bugs is (I think) handled by the bugs-- team. The stability sheriff is focused on issues that are crashing in the wild.
TimeTicks::Now should be safe to call from anywhere, anytime. The bug, whatever it is, is almost certainly elsewhere.

Any explanation of what the bug is actually saying (what is wrong with the write? It's to mapped stack memory, so what is asan saying is wrong with the address?) or help on getting symbol server working for the crash might help diagnose.

There have apparently been reports in the past which blamed QPCNow for asan bugs and always the cause is elsewhere.

So, presumably a real bug but for some reason asan is not reporting the location of the bug.

Cc: m...@chromium.org
Project Member

Comment 32 by sheriffbot@chromium.org, Jul 23 2016

Labels: -Security_Impact-Head Security_Impact-Beta
Labels: -M-53
Project Member

Comment 34 by sheriffbot@chromium.org, Jul 24 2016

Labels: M-52
Project Member

Comment 35 by sheriffbot@chromium.org, Jul 24 2016

Labels: ReleaseBlock-Stable
Project Member

Comment 36 by sheriffbot@chromium.org, Jul 25 2016

Labels: M-52
Labels: -M-52 -ReleaseBlock-Stable M-53
Project Member

Comment 38 by sheriffbot@chromium.org, Jul 29 2016

Labels: ReleaseBlock-Stable
Status: WontFix (was: Assigned)
Okay, progress update:

1) While the syzyasan documentation says that binaries/symbols should download automatically if you have the Chrome and Microsoft symbol servers enabled this does *not* work. This is a significant blocker because it makes the crash dump useless to anybody who is not persistent or does not have specialized knowledge. I managed to find the binaries and symbols by using "lmv m chrome_child" in windbg to find that the crash dump came from version 405727. I then downloaded https://www.googleapis.com/download/storage/v1/b/chromium-browser-syzyasan/o/win32-release%2Fasan-win32-release-405727.zip?generation=1468587677480000&alt=media and extracted the contents of the syzygy directory, added the extraction path to the symbol path, and reloaded symbols. Messy, but it did eventually work.

Either the documentation should be changed to describe this process, or the symbols should actually be uploaded to the symbol server. I would prefer the latter.

2) Now what? The testcase data on cluster-fuzz.appspot.com includes 29 lines of "Bad access information", but most of this is just noise. The free_tid, free_stack_size, free_stack, and milliseconds_since_free lines are all irrelevant because this is not a use-after-free bug. I think the only relevant lines are these four:

  access_size: 4
  error_type: UNKNOWN_BAD_ACCESS
  location: 0x24cdf8c7 Void
  access_mode: ASAN_WRITE_ACCESS

Of these, the access_size and the fact that it is a write are both visible in the crash dump. The location is 3 greater than the address shown in the crash dump, and I don't understand that. The one crucial piece of information that I desperately want is "what is *wrong* with this write", and the metadata is silent on this - it just says UNKNOWN_BAD_ACCESS. So ultimately the 29 lines of information about the crash seems to add zero information - am I missing something?

The write is to a stack location - presumably to a temporary created by the compiler to store the 64-bit result of QPCValueToTimeDelta. This should be completely static and consistent, so the bug should not be there. So the bug must be elsewhere - perhaps a stray write from another thread? - but without any indication of what are 

I filed  crbug.com/524748  for these issues and more last year, but it appears that none of the issues reported there have been addressed.

I'm closing this as WontFix because I cannot repro the bug. I'm also tempted to mark it as blocked on  crbug.com/524748 , but that bug is closed so that would be a futile gesture.

Cc: siggi@chromium.org chrisha@chromium.org sebmarchand@chromium.org
+cc a bunch of syzygy folks re: syzygy symbols issue in comment 39.
 crbug.com/525098  is the bug for the symbols not going on symbol server.

A short-term fix would be to update the documentation so it doesn't say that the symbols are on symbol server, and instead says where to get them more directly.
Labels: -ReleaseBlock-Stable
Project Member

Comment 43 by sheriffbot@chromium.org, Nov 9 2016

Labels: -Restrict-View-SecurityTeam allpublic
This bug has been closed for more than 14 weeks. Removing security view restrictions.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Sign in to add a comment