New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 629481 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Dec 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 2
Type: Bug

Blocked on:
issue 644672



Sign in to add a comment

Hang in media_vpx_video_decoder_fuzzer

Project Member Reported by ClusterFuzz, Jul 19 2016

Issue description

Components: Tools>Test>FindIt>NoResult Blink>Media>Video
Labels: -Pri-1 Needs-triage Te-Logged Pri-2
Labels: Needs-BlinkMediaTriage
From findit tool:

Author: delcypher
Project: chromium-libfuzzer
Changelist: https://chromium.googlesource.com/chromium/llvm-project/llvm/lib/Fuzzer.git/+/a4501630c591f0992c4010796b0f3edc658dc8d0
Time: Tue Jun 07 23:32:50 2016
The CL last changed line 245 of file FuzzerLoop.cpp, which is stack frame 1.

Author: James Zern
Project: chromium-libvpx
Changelist: https://chromium.googlesource.com/webm/libvpx.git/+/e656f44c241293134445496c3c005a0e543eb022
Time: Fri Jun 20 04:14:51 2014
The CL last changed line 1784 of file vp9_decodeframe.c, which is stack frame 3.

Author: Dmitry Kovalev
Project: chromium-libvpx
Changelist: https://chromium.googlesource.com/webm/libvpx.git/+/88a10abe7c1a6835c0b3f5be561d083d56615170
Time: Thu Mar 20 22:01:37 2014
The CL last changed line 2237 of file vp9_decodeframe.c, which is stack frame 4.

Author: James Zern
Project: chromium-libvpx
Changelist: https://chromium.googlesource.com/webm/libvpx.git/+/6f7ab01451c3b1401a7a4823947d73f86aebe893
Time: Wed Dec 03 00:51:27 2014
The CL last changed line 345 of file vp9_decoder.c, which is stack frame 5.

Author: hkuang
Project: chromium-libvpx
Changelist: https://chromium.googlesource.com/webm/libvpx.git/+/be6aeadaf49de28f13355d5dd7f64b4074dcca7c
Time: Tue Jan 27 20:26:28 2015
The CL last changed line 291 of file vp9_dx_iface.c, which is stack frame 6.

Author: James Zern
Project: chromium-libvpx
Changelist: https://chromium.googlesource.com/webm/libvpx.git/+/820201caa803e78c461e05128b87f4d7b24247a7
Time: Wed Oct 16 14:10:27 2013
The CL last changed line 133 of file vpx_thread.c, which is stack frame 7.

Author: hkuang
Project: chromium-libvpx
Changelist: https://chromium.googlesource.com/webm/libvpx.git/+/be6aeadaf49de28f13355d5dd7f64b4074dcca7c
Time: Tue Jan 27 20:26:28 2015
The CL last changed line 461 of file vp9_dx_iface.c, which is stack frame 8.

Suspected Project: chromium-libvpx

 I am unable to find delcypher@, so keeping the bug as it is.
Owner: mmoroz@chromium.org
Status: Assigned (was: Available)
mmoroz@, as per  issue 644672  assigning this issue to you. could you please take a look and please help us to find correct owner if it is not related your changes.

Thank you.
Labels: -Needs-triage

Comment 6 by mmoroz@google.com, Sep 28 2016

Blockedon: 644672
Cc: jrumm...@chromium.org kcc@chromium.org dalecur...@chromium.org aizatsky@chromium.org
Let's and see if it goes away after resolving of  bug 644672 .
Components: -Tools>Test>FindIt>NoResult
Components: -Blink>Media>Video Internals>Media>Video
Labels: -Needs-BlinkMediaTriage
Cc: fgalligan@chromium.org vigneshv@chromium.org
I am able to reproduce the time out but only with a release build (so unable to get to the bottom it).

Is there any place where i can find the 'gn' command line used by the fuzzer. Here's the one that i'm using:
gn gen out/libfuzzer '--args=is_debug=false use_libfuzzer=true is_asan=true enable_nacl=false proprietary_codecs=true ffmpeg_branding="ChromeOS" use_goma=true'

If i flip the is_debug to true, it does not reproduce.
Might be fixed by https://codereview.chromium.org/2453013005 if initialize is failing or something else is racing the RunUntilIdle().
Still no luck even after patching in https://codereview.chromium.org/2453013005.

Here's the failure log: https://paste.googleplex.com/6589916246966272
Hard to say, does vpx_codec_decode() ever return?
> does vpx_codec_decode() ever return?

It looks like the failure happens in the ConfigureDecoder() step itself. It somehow hits a check and fails. So vpx_codec_decode is never called.
ConfigureDecoder() is failing to return?
Cc: mmoroz@chromium.org
Owner: vigneshv@chromium.org
The original bug is that vpx_codec_decode never returns which causes the fuzzer to timeout. I am able to reproduce that behavior in the release build.

But in my debug build, it hits a DCHECK in ConfigureDecoder(). As i type this i realize how lame my question has been. I'm able to reproduce if i get rid of the DCHECK.

I'll try to get to the bottom of this.
I think the issue is that the code doesn't verify if the configuration is valid (or if Initialize() succeeds). The DCHECK is hit as the code assumes it won't be called with a bad configuration. In the release configuration the DCHECK is skipped, so it continues on and bad things happen. mmoroz@ has been working on a CL to check the configuration (https://codereview.chromium.org/2324843004/).
Does it make sense to land https://codereview.chromium.org/2324843004/ firstly?

Then we will have more or less working debug build. Then we will see if the issue is still reproducible. Since the fuzzer is not correct in terms of initialization stuff at the moment, I think it makes sense to use valid combinations of pixel_format, codec, etc.
Re #18: The underlying libvpx hang is still a valid bug. So the work on your CL (https://codereview.chromium.org/2324843004/) should not affect this bug.
Owner: jzern@chromium.org
James,

This one seems beyond my scope and understanding of libvpx threads.

FWIW, i'm able to reproduce this (even without ASAN) in vpxdec. I've attached the WebM version of the fuzzer's input in this bug. I'm able to reproduce it using the following commandline: "./vpxdec --threads=2 --output=/tmp/decoded.y4m ~/files/fuzzer/629481/629481.webm". (if it doesn't reproduce right away, try running it in a loop ~50 times and it almost always reproduces at least once).

Here's what i've found out so far: At some point in decoding, the check in vp9_decodeframe.c:669 [1] fails and it enters vpx_internal_error. Here's where it gets interesting, there are two paths:
1) When the decode fails gracefully (i.e.) does not hang, error_info->setjmp is set to 0 (which itself seems weird because it seems like this should never happen).
2) When the decode hangs, error_info->setjmp is set to 1, but the call to longjmp hangs (when i tried to step into it in gdb).

Either way, some path in libvpx seems wrong and its being triggered by this input file.

Please take a look, thanks!
Attaching the file for real - still no 'forgot attachment' notification in monorail i see ;)
629481.webm
52.5 KB View Download
Due to code reorganization the original fix for cases like this was lost, meaning a thread would jump back to main's target. I'll have a fix for this soon.
Project Member

Comment 24 by bugdroid1@chromium.org, Nov 2 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/webm/libvpx/+/1961a92a940ce73bb4f522c3e87b1fbdea216112

commit 1961a92a940ce73bb4f522c3e87b1fbdea216112
Author: James Zern <jzern@google.com>
Date: Wed Nov 02 01:45:50 2016

vp9,tile_worker_hook: correctly set jmp target

vp9_init_macroblockd() resets the error_info to cm's global copy; this
needs to be set to the thread-level target to avoid jumping to the
incorrect stack, resulting in hang or crash.
broken since:
1f4a6c8 vp9/tile_worker_hook: add multiple tile decoding
includes v1.5.0, v1.6.0

BUG= 629481 

Change-Id: Icbf1696b25ba8c479e845fbf227b3c3ca73542f5

[modify] https://crrev.com/1961a92a940ce73bb4f522c3e87b1fbdea216112/test/test-data.mk
[modify] https://crrev.com/1961a92a940ce73bb4f522c3e87b1fbdea216112/vp9/decoder/vp9_decodeframe.c
[modify] https://crrev.com/1961a92a940ce73bb4f522c3e87b1fbdea216112/test/invalid_file_test.cc
[modify] https://crrev.com/1961a92a940ce73bb4f522c3e87b1fbdea216112/test/test-data.sha1

Project Member

Comment 25 by bugdroid1@chromium.org, Nov 3 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9334c34dbc59ff79a2c24c6867c9d22802212843

commit 9334c34dbc59ff79a2c24c6867c9d22802212843
Author: jzern <jzern@chromium.org>
Date: Thu Nov 03 21:49:39 2016

Roll src/third_party/libvpx/source/libvpx/ 9a032fa26..1961a92a9 (33 commits).

https://chromium.googlesource.com/webm/libvpx.git/+log/9a032fa2628e..1961a92a940c

$ git log 9a032fa26..1961a92a9 --date=short --no-merges --format='%ad %ae %s'
2016-11-01 jzern vp9,tile_worker_hook: correctly set jmp target
2016-10-25 linfengz Add high bitdepth intra prediction optimization speed test
2016-10-21 linfengz Refine 8-bit intra prediction NEON optimization (mode tm)
2016-10-31 jzern idct32x32_1_add_neon: clear a couple conv warnings
2016-10-31 jzern build/make/Android.mk: s/armv8/arm64/
2016-10-31 jzern build/make/Android.mk: fix armeabi-v7a build
2016-10-31 jzern use .S suffix rather than .s for NEON asm
2016-10-31 marpan vp9-svc: Fix some stats in vp9_spatial_svc_encoder.
2016-10-31 pbos Add vp9_spatial_svc_encoder to .gitignore.
2016-10-18 jzern idct,NEON: add a tran_low_t->s16 load adapter
2016-10-21 linfengz Refine 8-bit intra prediction NEON optimization (mode h and v)
2016-10-30 marpan vp9-svc: Add checks to layer bitrates in vp9_spatial_svc_encoder.
2016-10-28 jimbankoski vpxdec.c : don't double count corrupted frames
2016-10-29 jzern tiny_ssim: fix visual studio build
2016-10-28 pbos Add temporal-layer support to tiny_ssim.
2016-10-27 marpan vp9: Updates to SVC sample encoder.
2016-10-27 pbos Add tools/tiny_ssim for generating SSIM/PSNR.
2016-10-12 paulwilkins Change to KF boost calculation.
2016-10-21 linfengz Refine 8-bit intra prediction NEON optimization (mode d45 and d135)
2016-10-26 debargha Force recode if framesize exceeds max allowed size
(...)

+ add an include path for NEON asm allowing it to include generated sources
  which match the source tree layout, e.g., vpx_dsp/arm/...

R=johannkoenig@google.com
BUG= 629481 

Review-Url: https://codereview.chromium.org/2471423002
Cr-Commit-Position: refs/heads/master@{#429704}

[modify] https://crrev.com/9334c34dbc59ff79a2c24c6867c9d22802212843/DEPS
[modify] https://crrev.com/9334c34dbc59ff79a2c24c6867c9d22802212843/third_party/libvpx/BUILD.gn
[modify] https://crrev.com/9334c34dbc59ff79a2c24c6867c9d22802212843/third_party/libvpx/README.chromium
[modify] https://crrev.com/9334c34dbc59ff79a2c24c6867c9d22802212843/third_party/libvpx/libvpx_srcs.gni
[modify] https://crrev.com/9334c34dbc59ff79a2c24c6867c9d22802212843/third_party/libvpx/source/config/linux/arm-neon-cpu-detect/vpx_dsp_rtcd.h
[modify] https://crrev.com/9334c34dbc59ff79a2c24c6867c9d22802212843/third_party/libvpx/source/config/linux/arm-neon/vpx_dsp_rtcd.h
[modify] https://crrev.com/9334c34dbc59ff79a2c24c6867c9d22802212843/third_party/libvpx/source/config/linux/arm64/vpx_dsp_rtcd.h
[modify] https://crrev.com/9334c34dbc59ff79a2c24c6867c9d22802212843/third_party/libvpx/source/config/vpx_version.h

Comment 26 by jzern@chromium.org, Nov 17 2016

This hasn't caused any problem in dev. Branch is tomorrow for a new beta and this has been present since late 2015, should we skip the merge to the current beta?
Project Member

Comment 27 by sheriffbot@chromium.org, Nov 22 2016

Labels: -Restrict-View-EditIssue
Removing EditIssue view restrictions from ClusterFuzz filed bugs. If you believe that this issue should still be restricted, please reapply the label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 28 by jzern@chromium.org, Dec 12 2016

Status: Fixed (was: Assigned)

Comment 29 by quanto@google.com, Jan 31 2017

this has been assigned CVE-2017-0474

Sign in to add a comment