Automatically applying components based on crash stacktrace and information from OWNERS files.

If this is incorrect, please apply the Test-Predator-Wrong-Components label.

Automatically adding ccs based on suspected regression changelists:

WTF: Remove kQuestionMarksForUnencodables as its final real user is gone. by bsittler@chromium.org - https://chromium.googlesource.com/chromium/src/+/6ec8ee4de81c1a053c4cc82b03123969f6567a68

[RootLayerScrolls] Fix ScrollRectToVisible for iframes by szager@chromium.org - https://chromium.googlesource.com/chromium/src/+/08b0819307cc7d1a973e7d938ff156527519c791

If this is incorrect, please apply the Test-Predator-Wrong-CLs label.

Gentle Ping :

bsittler@ / szager@ Could you please look into this issue.

Pretty sure https://chromium.googlesource.com/chromium/src/+/6ec8ee4de81c1a053c4cc82b03123969f6567a68 is not related as its impacts are only visible in form-based file upload handlers (e.g. form post-intercepting service workers or server-side form form handlers.)

My change is a no-op unless the --root-layer-scrolls flag is turned on (which is not the case in the clusterfuzz report).

Unable to provide possible suspect using Predator, CL and Code Search.
Could someone please look into the issue.

Thank You...

I am attempting to reproduce this crash in a clean client. I'll follow up when I know more.

Please pardon my ignorance, but how is this supposed to work? When I run
$ /google/data/ro/teams/clusterfuzz-tools/releases/clusterfuzz reproduce 5810351965995008
... it croaks after a while with this at the end of the output:

E: Unable to locate package libgtk-3-0-dbg
yes: standard output: Broken pipe
The following command failed:  sudo apt-get install --reinstall libasound2:i386 libcap2:i386 libdconf1:i386 libelf-dev:i386 libfontconfig1:i386 libgconf-2-4:i386 libgl1-mesa-glx
:i386 libglib2.0-0:i386 libgpm2:i386 libgtk2.0-0:i386 libgtk-3-0:i386 libncurses5:i386 libnss3:i386 libpango1.0-0:i386 libpci3:i386 libssl1.0.2:i386 libssl-dev:i386 libtinfo-dev:i386 libudev1:i386 libxcomposite1:i386 libxcursor1:i386 libxdamage1:i386 libxi6:i386 libxrandr2:i386 libxss1:i386 libxtst6:i386 linux-libc-dev:i386 ant apache2-bin autoconf binutils-aarch64-linux-gnu binutils-arm-linux-gnueabihf binutils-mips64el-linux-gnuabi64 binutils-mipsel-linux-gnu bison cdbs cmake curl devscripts dpkg-dev elfutils fakeroot flex fonts-ipafont fonts-thai-tlwg g++ g++-6-multilib g++-arm-linux-gnueabihf gawk git-core git-svn g++-mingw-w64-i686 gperf intltool lib32gcc1 lib32ncurses5-dev lib32stdc++6 lib32z1-dev libapache2-mod-php7.1 libasound2 libasound2-dev libatk1.0-0 libav-tools libbluetooth-dev libbrlapi0.6 libbrlapi-dev libbz2-1.0 libbz2-dev libc6 libc6-dbg libc6-dev-armhf-cross libc6-i386 libcairo2 libcairo2-dbg libcairo2-dev libcap2 libcap-dev libcups2 libcups2-dev libcurl4-gnutls-dev libdconf1 libdconf-dev libdrm-dev libelf-dev libexpat1 libffi6 libffi6-dbg libffi-dev libfontconfig1 libfontconfig1-dbg libfreetype6 libgbm-dev libgconf2-dev libgl1-mesa-dev libgles2-mesa-dev libglib2.0-0 libglib2.0-0-dbg libglib2.0-dev libglu1-mesa-dev libgnome-keyring0 libgnome-keyring-dev libgtk2.0-0 libgtk2.0-0-dbg libgtk2.0-dev libgtk-3-0 libgtk-3-0-dbg libgtk-3-dev libjpeg-dev libkrb5-dev libnspr4 libnspr4-dbg libnspr4-dev libnss3 libnss3-dbg libnss3-dev libpam0g libpam0g-dev libpango1.0-0 libpci3 libpci-dev libpcre3 libpcre3-dbg libpixman-1-0 libpixman-1-0-dbg libpng16-16 libpulse0 libpulse-dev libsctp-dev libspeechd2 libspeechd-dev libsqlite3-0 libsqlite3-0-dbg libsqlite3-dev libssl-dev libstdc++6 libtinfo-dev libtool libudev1 libudev-dev libwww-perl li
bx11-6 libx11-xcb1 libxau6 libxau6-dbg libxcb1 libxcb1-dbg libxcomposite1 libxcomposite1-dbg libxcursor1 libxcursor1-dbg libxdamage1 libxdamage1-dbg libxdmcp6 libxdmcp6-dbg libxext6 libxext6-dbg libxfixes3 libxi6 libxinerama1 libxinerama1-dbg libxkbcommon-dev libxrandr2 libxrender1 libxslt1-dev libxss-dev libxt-dev libxtst6 libxtst-dev linux-libc-dev-armhf-cross locales mesa-common-dev openbox patch perl php7.1-cgi pkg-config python python-cherrypy3 python-crypto python-dev python-numpy python-opencv python-openssl python-psutil python-yaml realpath rpm ruby subversion texinfo ttf-mscorefonts-installer wdiff x11-utils xcompmgr xsltproc xutils-dev xvfb zip zlib1g zlib1g-dbg

It produces the following output:
Reading package lists...
Building dependency tree...
Reading state information...
Package libcairo2-dbg is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  libcairo2-dbgsym:i386 libcairo-script-interpreter2-dbgsym:i386
  libcairo-gobject2-dbgsym:i386 libcairo2-dbgsym
  libcairo-script-interpreter2-dbgsym libcairo-gobject2-dbgsym

E: Package 'libcairo2-dbg' has no installation candidate
E: Unable to locate package libgtk-3-0-dbg
yes: standard output: Broken pipe

You will have to install the above packages yourself.

(There is no libgtk-3-0-dbg on the Google workstation I am using, though there is a libgtk-3-0-dbgsym which is already installed.)

I'd email clusterfuzz-dev@google.com to ask about the libgtk dependency.

+ mmoroz@

Interesting! bsittler@, what OS are you using? On my goobuntu desktop I see the following:

$ aptitude search libgtk-3-0-dbg
i   libgtk-3-0-dbg                                                         - GTK+ libraries and debugging symbols                                            
p   libgtk-3-0-dbg:i386                                                    - GTK+ libraries and debugging symbols       

On my workstation (running a variant of Debian Testing) I see:

$ xvfb-run aptitude search libgtk-3-0-dbg
i   libgtk-3-0-dbgsym                                                                - debug symbols for libgtk-3-0                                                              
p   libgtk-3-0-dbgsym:i386                                                           - debug symbols for libgtk-3-0                                                              

See solution in https://bugs.chromium.org/p/chromium/issues/detail?id=790784#c1

I can't reproduce this with a tsan build:

../../out/tsan/content_shell --no-sandbox LayoutTests/787996.html

The test runs and doesn't crash.

I'm reluctant to run the 'clusterfuzz reproduce' command because I expect it to build the world from scratch and take all day each time I run it, which makes debugging impractical.

Does it still do that? Any tips for reproducing from a local content_shell build on Ubuntu?

You need to set the following TSAN_OPTIONS (adjust your out dir path)

TSAN_OPTIONS = external_symbolizer_path=/mnt/scratch0/clusterfuzz/slave-bot/builds/chromium-browser-tsan_linux-release_e1f0dbf462d779d5196ef318a224a998657a9c66/revisions/tsan-linux-release-518630/llvm-symbolizer:flush_memory_ms=2000:stack_trace_format=DEFAULT:report_thread_leaks=0:history_size=3:use_sigaltstack=1:handle_sigfpe=1:handle_sigbus=1:handle_abort=1:suppressions=/mnt/scratch0/clusterfuzz/scripts/suppressions/tsan_suppressions.txt:atexit_sleep_ms=200:handle_segv=1:handle_sigill=1:report_signal_unsafe=0:symbolize_inline_frames=false:symbolize=1:print_summary=1

the one that make a different is handle_abort=1

I still can't recreate this. Output attached from:

TSAN_OPTIONS="external_symbolizer_path=../../third_party/llvm-build/Release+Asserts/bin/llvm-symbolizer:flush_memory_ms=2000:stack_trace_format=DEFAULT:report_thread_leaks=0:history_size=3:use_sigaltstack=1:handle_sigfpe=1:handle_sigbus=1:handle_abort=1:suppressions=../../v8/tools/sanitizers/tsan_suppressions.txt:atexit_sleep_ms=200:handle_segv=1:handle_sigill=1:report_signal_unsafe=0:symbolize_inline_frames=false:symbolize=1:print_summary=1" ../../out/tsan/chrome --no-sandbox LayoutTests/787996.html

Deleted: Tsan-Output.txt 65.9 KB

Tsan-Output.txt 65.9 KB View Download

Above is at Cr-Commit-Position: refs/heads/master@{#520629}

I also cannot reproduce this. On my workstation running a non-Ubuntu Debian derivative I get:

New crash type: Data race WRITE 8
New crash state:
  tzset_internal
  ??
  SysTimeFromTimeStruct

Original crash type: Abrt
Original crash state:
  blink::LayoutBlockFlow::AppendFloatsToLastLine
  blink::LayoutBlockFlow::LayoutRunsAndFloatsInRange
  blink::LayoutBlockFlow::LayoutRunsAndFloats

The stacktrace doesn't match the original stacktrace.
Try again (3 times). Press Ctrl+C to stop trying to reproduce.

DifferentStacktraceError: The original crash cannot be reproduced after trying 3 times.
But it seems we get a different stacktrace. Could you check if the stacktrace is good enough?

Here are things you can try:
- Run outside XVFB (e.g. you will be able to see the launched program on screen.) with `--disable-xvfb`, which is especially useful for Chrome.
- Run with the downloaded build by adding `--build download`.
- Run `build/install-build-deps.sh` to ensure all dependencies are installed.
- Run with more number of trials by adding `-i 10`, which is especially good for gesture-related testcases.
- Use gdb to debug by adding `--enable-debug`.

Detailed log of this run can be found in: /usr/local/google/home/bsittler/.clusterfuzz/logs/output.log

"Detailed log of this run can be found in: /usr/local/google/home/bsittler/.clusterfuzz/logs/output.log"

Could you post the log here? I don't have access to google shares.

Unfortunately that was run on a corporate workstation and the log contains information I can't share. Are there any specific sections or patterns you would look for?

I'm more interested in the command you ran to reproduce, including the gn
args you used to build, and the command-line flags and binary you used to
reproduce the test result.

I ran
$ clusterfuzz reproduce --current 5810351965995008

args.gn:
enable_nacl = false
goma_dir = "/usr/local/google/home/bsittler/goma"
is_component_build = false
is_debug = false
is_tsan = true
strip_absolute_paths_from_debug_symbols = true
use_goma = true

ClusterFuzz has detected this issue as fixed in range 524387:524395.

Detailed report: https://clusterfuzz.com/testcase?key=5810351965995008

Fuzzer: inferno_layout_test_unmodified
Job Type: linux_tsan_chrome_mp
Platform Id: linux

Crash Type: Abrt
Crash Address: 0x03e900002cc0
Crash State:
  blink::LayoutBlockFlow::AppendFloatsToLastLine
  blink::LayoutBlockFlow::LayoutRunsAndFloatsInRange
  blink::LayoutBlockFlow::LayoutRunsAndFloats
  
Sanitizer: thread (TSAN)

Regressed: https://clusterfuzz.com/revisions?job=linux_tsan_chrome_mp&range=518240:518474
Fixed: https://clusterfuzz.com/revisions?job=linux_tsan_chrome_mp&range=524387:524395

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=5810351965995008

See https://github.com/google/clusterfuzz-tools for more information.

If you suspect that the result above is incorrect, try re-doing that job on the test case report page.

ClusterFuzz testcase 5810351965995008 is verified as fixed, so closing issue as verified.

If this is incorrect, please add ClusterFuzz-Wrong label and re-open the issue.

 Issue 797662  has been merged into this issue.

Is this still happening Rob?