New issue
Advanced search Search tips

Issue 770690 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Oct 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

WebRTC change broke multiple fuzzing builds

Project Member Reported by mmoroz@chromium.org, Oct 2 2017

Issue description

https://build.chromium.org/p/chromium.fyi/builders/Afl%20Upload%20Linux%20ASan/builds/5035

https://build.chromium.org/p/chromium.fyi/builders/Libfuzzer%20Upload%20Linux%20ASan%20Debug/builds/4854

https://build.chromium.org/p/chromium.fyi/builders/Libfuzzer%20Upload%20Linux%20UBSan/builds/5565

https://build.chromium.org/p/chromium.fyi/builders/Libfuzzer%20Upload%20Linux%20ASan/builds/6037

[577/7394] LINK ./transport_feedback_packet_loss_tracker_fuzzer
FAILED: transport_feedback_packet_loss_tracker_fuzzer 
python "../../build/toolchain/gcc_link_wrapper.py" --output="./transport_feedback_packet_loss_tracker_fuzzer" -- ../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -Werror -Wl,-O1 -Wl,--gc-sections -nostdlib++ --sysroot=../../build/linux/debian_jessie_amd64-sysroot -L../../build/linux/debian_jessie_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_jessie_amd64-sysroot/lib/x86_64-linux-gnu -L../../build/linux/debian_jessie_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_jessie_amd64-sysroot/usr/lib/x86_64-linux-gnu -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -o "./transport_feedback_packet_loss_tracker_fuzzer" -Wl,--start-group @"./transport_feedback_packet_loss_tracker_fuzzer.rsp" ./libc++.so -Wl,--end-group   -ldl -lpthread -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lXext -lX11 -lXcomposite -lXrender -lm 
../../third_party/webrtc/modules/audio_device/audio_device_buffer.cc:80: error: undefined reference to 'rtc::TaskQueue::~TaskQueue()'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[578/7394] CXX obj/third_party/WebKit/Source/platform/scheduler/scheduler/renderer_scheduler.o
[579/7394] LINK ./ulpfec_generator_fuzzer
[580/7394] LINK ./neteq_rtp_fuzzer
[581/7394] LINK ./vp9_qp_parser_fuzzer
[582/7394] CXX obj/third_party/WebKit/Source/platform/scheduler/scheduler/main_thread_task_queue.o
[583/7394] LINK ./turn_unwrap_fuzzer
FAILED: turn_unwrap_fuzzer 
python "../../build/toolchain/gcc_link_wrapper.py" --output="./turn_unwrap_fuzzer" -- ../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -Werror -Wl,-O1 -Wl,--gc-sections -nostdlib++ --sysroot=../../build/linux/debian_jessie_amd64-sysroot -L../../build/linux/debian_jessie_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_jessie_amd64-sysroot/lib/x86_64-linux-gnu -L../../build/linux/debian_jessie_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_jessie_amd64-sysroot/usr/lib/x86_64-linux-gnu -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -o "./turn_unwrap_fuzzer" -Wl,--start-group @"./turn_unwrap_fuzzer.rsp" ./libc++.so -Wl,--end-group   -ldl -lpthread -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lXext -lX11 -lXcomposite -lXrender -lm -lz 
../../third_party/webrtc/rtc_base/task_queue.h:191: error: undefined reference to 'rtc::TaskQueue::PostTask(std::__1::unique_ptr<rtc::QueuedTask, std::__1::default_delete<rtc::QueuedTask> >)'
../../third_party/webrtc/modules/audio_processing/aec_dump/aec_dump_impl.cc:94: error: undefined reference to 'rtc::TaskQueue::PostTask(std::__1::unique_ptr<rtc::QueuedTask, std::__1::default_delete<rtc::QueuedTask> >)'
../../third_party/webrtc/modules/audio_processing/aec_dump/aec_dump_impl.cc:121: error: undefined reference to 'rtc::TaskQueue::PostTask(std::__1::unique_ptr<rtc::QueuedTask, std::__1::default_delete<rtc::QueuedTask> >)'
../../third_party/webrtc/modules/audio_processing/aec_dump/aec_dump_impl.cc:135: error: undefined reference to 'rtc::TaskQueue::PostTask(std::__1::unique_ptr<rtc::QueuedTask, std::__1::default_delete<rtc::QueuedTask> >)'
../../third_party/webrtc/modules/audio_device/audio_device_buffer.cc:80: error: undefined reference to 'rtc::TaskQueue::~TaskQueue()'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[584/7394] CXX obj/third_party/webrtc_overrides/task_queue_impl/task_queue.o
[585/7394] CXX obj/third_party/WebKit/Source/platform/scheduler/scheduler/webthread_base.o
[586/7394] CXX obj/third_party/WebKit/Source/platform/scheduler/scheduler/web_frame_scheduler_impl.o
[587/7394] CXX obj/ui/views/mus/mus/desktop_window_tree_host_mus.o
[588/7394] CXX obj/services/viz/public/interfaces/interfaces/compositor_frame.mojom.o
[589/7394] CXX obj/third_party/WebKit/Source/platform/scheduler/scheduler/web_view_scheduler_impl.o
[590/7394] CXX obj/ui/accessibility/accessibility/aura_window_properties.o
[591/7394] CXX obj/ui/base/base/simple_menu_model.o
[592/7394] CXX obj/ui/aura/aura/drag_drop_delegate.o
[593/7394] LINK ./sdp_parser_fuzzer
FAILED: sdp_parser_fuzzer 
python "../../build/toolchain/gcc_link_wrapper.py" --output="./sdp_parser_fuzzer" -- ../../third_party/llvm-build/Release+Asserts/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -fuse-ld=gold -B../../third_party/binutils/Linux_x64/Release/bin -Wl,--threads -Wl,--thread-count=4 -m64 -Werror -Wl,-O1 -Wl,--gc-sections -nostdlib++ --sysroot=../../build/linux/debian_jessie_amd64-sysroot -L../../build/linux/debian_jessie_amd64-sysroot/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_jessie_amd64-sysroot/lib/x86_64-linux-gnu -L../../build/linux/debian_jessie_amd64-sysroot/usr/lib/x86_64-linux-gnu -Wl,-rpath-link=../../build/linux/debian_jessie_amd64-sysroot/usr/lib/x86_64-linux-gnu -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -Wl,-rpath-link=. -Wl,--disable-new-dtags -Wl,-rpath=\$ORIGIN/. -Wl,-rpath-link=. -Wl,--export-dynamic -Wl,-u_sanitizer_options_link_helper -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=leak -fsanitize-coverage=trace-pc-guard -o "./sdp_parser_fuzzer" -Wl,--start-group @"./sdp_parser_fuzzer.rsp" ./libc++.so -Wl,--end-group   -ldl -lpthread -lrt -lgmodule-2.0 -lgobject-2.0 -lgthread-2.0 -lglib-2.0 -lXext -lX11 -lXcomposite -lXrender -lm -lz 
../../third_party/webrtc/modules/audio_device/audio_device_buffer.cc:80: error: undefined reference to 'rtc::TaskQueue::~TaskQueue()'
../../third_party/webrtc/rtc_base/task_queue.h:191: error: undefined reference to 'rtc::TaskQueue::PostTask(std::__1::unique_ptr<rtc::QueuedTask, std::__1::default_delete<rtc::QueuedTask> >)'
../../third_party/webrtc/modules/audio_processing/aec_dump/aec_dump_impl.cc:94: error: undefined reference to 'rtc::TaskQueue::PostTask(std::__1::unique_ptr<rtc::QueuedTask, std::__1::default_delete<rtc::QueuedTask> >)'
../../third_party/webrtc/modules/audio_processing/aec_dump/aec_dump_impl.cc:121: error: undefined reference to 'rtc::TaskQueue::PostTask(std::__1::unique_ptr<rtc::QueuedTask, std::__1::default_delete<rtc::QueuedTask> >)'
../../third_party/webrtc/modules/audio_processing/aec_dump/aec_dump_impl.cc:135: error: undefined reference to 'rtc::TaskQueue::PostTask(std::__1::unique_ptr<rtc::QueuedTask, std::__1::default_delete<rtc::QueuedTask> >)'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[594/7394] CXX obj/ui/aura/aura/event_client.o
 
Summary: WebRTC change broke fuzzing builds (was: WebRTC change broke fuzzing build)
I suspect this change: https://chromium-review.googlesource.com/c/chromium/src/+/681814

Will create a revert.
Summary: WebRTC change broke multiple fuzzing builds (was: WebRTC change broke fuzzing builds)

Comment 3 by nisse@chromium.org, Oct 3 2017

I would expect the revert to cause some problems with webrtc cl https://webrtc-review.googlesource.com/c/src/+/3940. You may have to revert that one too, if the reverted cl can't be relanded soon.

Comment 4 by nisse@chromium.org, Oct 3 2017

Cc: nisse@chromium.org
Dependent webrtc cl being reverted by hbos@.
Owner: mmoroz@chromium.org
Status: Assigned (was: Untriaged)
Assigning to mmoroz@ since he seems to be driving this. 
mmoroz@: Feel free to reassign.
Owner: nisse@chromium.org
Assigning to nisse@ since I've reverted their change.

Comment 7 by nisse@chromium.org, Oct 4 2017

I'm afraid I know nothing about the chrome fuzzers. Can you tell me how to repro the link failures locally? I take it these are webrtc fuzzers, but built as part of the chromium build, and using chromium's top-level .gn file for settings?
The buildbot is using the following GN args:

enable_nacl = false
ffmpeg_branding = "ChromeOS"
goma_dir = "/b/build/slave/cache/goma_client"
is_asan = true
is_debug = false
optimize_for_fuzzing = true
pdf_enable_xfa = true
proprietary_codecs = true
use_libfuzzer = true


(copied from https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.fyi%2FLibfuzzer_Upload_Linux_ASan%2F6037%2F%2B%2Frecipes%2Fsteps%2Fgenerate_build_files%2F0%2Fstdout)

nd then builds all fuzz targets:

ninja -C out/Release -j 50 transport_feedback_packet_loss_tracker_fuzzer turn_unwrap_fuzzer sdp_parser_fuzzer

(taken from https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.fyi%2FLibfuzzer_Upload_Linux_ASan%2F6037%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout)


We also have documentation that describes all build configurations (https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/reference.md), but I think those are not important for this particular issue.
We've done our best to setup a libfuzzer trybot in WebRTC [1] but apparently it cannot catch all cases (there'll always be a few differences since it's using a WebRTC checkout and not a Chromium one).

How come there are still no libfuzzer trybots in Chromium? If there was, we could run them and prevent breakages like this.

[1] https://build.chromium.org/p/tryserver.webrtc/builders/linux_libfuzzer_rel
Here's a cl under review: https://webrtc-review.googlesource.com/c/src/+/6840  

It solves the problem, when I build locally according to above instructions.
We are rolling a newer version of libFuzzer: https://chromium-review.googlesource.com/c/chromium/src/+/705134

It shouldn't affect anything though, just FYI
Project Member

Comment 12 by bugdroid1@chromium.org, Oct 10 2017

The following revision refers to this bug:
  https://webrtc.googlesource.com/src.git/+/6e8785045fb93e4bd3f953654f83da29a8c31b99

commit 6e8785045fb93e4bd3f953654f83da29a8c31b99
Author: Niels Möller <nisse@webrtc.org>
Date: Tue Oct 10 11:45:48 2017

Unconditionally link fuzzers with rtc_task_queue_impl.

Bug:  webrtc:8166 ,  chromium:770690 
Change-Id: I9480720c99308f8a2a3dcf407a07d762249f5a9c
Reviewed-on: https://webrtc-review.googlesource.com/6840
Reviewed-by: Henrik Kjellander <kjellander@webrtc.org>
Reviewed-by: Mirko Bonadei <mbonadei@webrtc.org>
Commit-Queue: Niels Moller <nisse@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#20219}
[modify] https://crrev.com/6e8785045fb93e4bd3f953654f83da29a8c31b99/test/fuzzers/BUILD.gn

Comment 13 by nisse@chromium.org, Oct 20 2017

Status: Fixed (was: Assigned)
Fuzzer link dependencies fixed in webrtc, and the reverted change now successfully relanded.

Sign in to add a comment