New issue
Advanced search Search tips

Issue 798546 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Feb 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 3
Type: Bug



Sign in to add a comment

ToTWinThinLTO64 failing base_unittests ScopedHandleTest.MultiProcess with fatal exception EXCEPTION_ACCESS_VIOLATION

Project Member Reported by inglorion@chromium.org, Jan 2 2018

Issue description

Logs: https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.clang%2FToTWinThinLTO64%2F404%2F%2B%2Frecipes%2Fsteps%2Fbase_unittests%2F0%2Fstdout

Excerpt:
Retrying 1 test (retry #1)
[ RUN      ] ScopedHandleTest.MultiProcess
[3588:5916:0102/103633.954:39965334:WARNING:test_suite.cc(237)] Test launcher output path C:\Users\CHROME~2\AppData\Local\Temp\4036_26355\test_results.xml exists. Not adding test launcher result printer.
Received fatal exception EXCEPTION_ACCESS_VIOLATION
Backtrace:
	RtlAnsiStringToUnicodeString [0x000000007789EEF1+289]
	HeapFree [0x0000000077651A0A+10]
	(No symbol) [0x000007FEFB3FB358]
../../base/win/scoped_handle_unittest.cc(123): error: Expected equality of these values:
  0
  rv
    Which is: -1073741819
[  FAILED  ] ScopedHandleTest.MultiProcess (56 ms)
 
Confirmed locally that this happens with use_thin_lto = true and not with use_thin_lto = false, with otherwise identical configurations.
The problem appears to happen in the call to run_test_function() in scoped_handle_unittest.cc.

The failure is also flaky (1 run without the crash, 2 crashes with different stack traces):

C:\src\chromium\src>out\clang_thinlto\base_unittests --gtest_filter=ScopedHandleTest.MultiProcess
IMPORTANT DEBUGGING NOTE: batches of tests are run inside their
own process. For debugging a test inside a debugger, use the
--gtest_filter=<your_test_name> flag along with
--single-process-tests.
Using sharding settings from environment. This is shard 0/1
Using 1 parallel jobs.
Note: Google Test filter = ScopedHandleTest.MultiProcess
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from ScopedHandleTest
[ RUN      ] ScopedHandleTest.MultiProcess
[11392:7868:0102/145237.425:530140750:WARNING:test_suite.cc(237)] Test launcher output path C:\src\tmp\9572_17759\test_results.xml exists. Not adding test launcher result printer.
Received fatal exception EXCEPTION_ACCESS_VIOLATION
Backtrace:
        RtlFreeHeap [0x00007FFC782ECADB+843]
        (No symbol) [0x00007FFC6486B358]
        (No symbol) [0x000001E836FBA4B0]
        (No symbol) [0x00007FFC00000000]
        (No symbol) [0x000001E836FBA4B0]
[       OK ] ScopedHandleTest.MultiProcess (127 ms)
[----------] 1 test from ScopedHandleTest (127 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (131 ms total)
[  PASSED  ] 1 test.
[1/1] ScopedHandleTest.MultiProcess (127 ms)
SUCCESS: all tests passed.
Tests took 0 seconds.

C:\src\chromium\src>out\clang_thinlto\base_unittests --gtest_filter=ScopedHandleTest.MultiProcess
IMPORTANT DEBUGGING NOTE: batches of tests are run inside their
own process. For debugging a test inside a debugger, use the
--gtest_filter=<your_test_name> flag along with
--single-process-tests.
Using sharding settings from environment. This is shard 0/1
Using 1 parallel jobs.
Note: Google Test filter = ScopedHandleTest.MultiProcess
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from ScopedHandleTest
[ RUN      ] ScopedHandleTest.MultiProcess
[9976:15416:0102/145238.733:530142062:WARNING:test_suite.cc(237)] Test launcher output path C:\src\tmp\25520_1202\test_results.xml exists. Not adding test launcher result printer.
[       OK ] ScopedHandleTest.MultiProcess (246 ms)
[----------] 1 test from ScopedHandleTest (248 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (254 ms total)
[  PASSED  ] 1 test.
[1/1] ScopedHandleTest.MultiProcess (246 ms)
SUCCESS: all tests passed.
Tests took 0 seconds.

C:\src\chromium\src>out\clang_thinlto\base_unittests --gtest_filter=ScopedHandleTest.MultiProcess
IMPORTANT DEBUGGING NOTE: batches of tests are run inside their
own process. For debugging a test inside a debugger, use the
--gtest_filter=<your_test_name> flag along with
--single-process-tests.
Using sharding settings from environment. This is shard 0/1
Using 1 parallel jobs.
Note: Google Test filter = ScopedHandleTest.MultiProcess
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from ScopedHandleTest
[ RUN      ] ScopedHandleTest.MultiProcess
[9184:1940:0102/145513.107:530296437:WARNING:test_suite.cc(237)] Test launcher output path C:\src\tmp\9340_15679\test_results.xml exists. Not adding test launcher result printer.
Received fatal exception EXCEPTION_ACCESS_VIOLATION
Backtrace:
        RtlFreeHeap [0x00007FFC782ECADB+843]
        (No symbol) [0x00007FFC6486B358]
        (No symbol) [0x00000223E9FDA390]
        (No symbol) [0x00007FFC00000000]
        (No symbol) [0x00000223E9FDA390]
        RtlAcquireSRWLockExclusive [0x00007FFC782E5858+184]
        (No symbol) [0x0000000000000230]
        (No symbol) [0x0000000000000040]
        LdrGetDllPath [0x00007FFC782E0000+80]
[       OK ] ScopedHandleTest.MultiProcess (114 ms)
[----------] 1 test from ScopedHandleTest (115 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (122 ms total)
[  PASSED  ] 1 test.
[1/1] ScopedHandleTest.MultiProcess (114 ms)
SUCCESS: all tests passed.
Tests took 0 seconds.
Commenting out the body of InternalRunThreadTest in scoped_handle_test_dll.cc (leaving only the return true) leads to a difference in behavior between the LTO and non-LTO builds: In the non-LTO build, the test passes, and in the LTO build, it fails.
Components: Tests
Labels: Needs-Milestone Triaged-ET TE-NeedsTriageHelp
The issue seems to be out of TE-scope as it is related to unittests. Hence, adding TE-NeedsTriageHelp for further investigation from dev team.

Thanks...!!

Comment 5 by r...@chromium.org, Jan 3 2018

Labels: -TE-NeedsTriageHelp -Needs-Milestone
Status: Available (was: Unconfirmed)
Thanks, I think we already have the right people CC'd on this, so it doesn't need further help to find an owner.
Cc: p...@chromium.org
Owner: inglorion@chromium.org
The culprit here appears to be ActiveVerifier, which is defined in an anonymous namespace, but the dll we use obtains a pointer to an instance from the main module and then calls virtual methods on it. When compiled with ThinLTO and -fwhole-program-vtables, I suspect we optimize based on the assumption that ActiveVerifier isn't used outside the current module. That would normally be a reasonable assumption for a class defined in an anonymous namespace, but in this case is broken by getting a pointer to an object in a different module.
Project Member

Comment 7 by bugdroid1@chromium.org, Feb 13 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f364fc10b1e89b514e79f6b04ad01f2fc71cfbb5

commit f364fc10b1e89b514e79f6b04ad01f2fc71cfbb5
Author: inglorion <inglorion@chromium.org>
Date: Tue Feb 13 20:39:33 2018

Rename ActiveVerifier to ScopedHandleVerifier and add lto_visibility_public

ActiveVerifier instances are passed across module boundaries. This
means that whole program analysis cannot see all uses and not all
optimizations that we can apply to classes only used in a single
module can be correctly applied. This change moves ActiveVerifier
into its own header file, renames it to ScopedHandleVerifier, and
annotates it with clang::lto_visibility_public to inform the compiler
that the class is used outside the current module. This avoids a crash
we were seeing in base_unittests. This change also removes
ActiveVerifier from the CFI blacklist.

R=brucedawson, pcc, robliao, thakis

Bug:  798546 
Change-Id: I8f6ff5303ff5f339aedbfb72488875d5d54d3a9a
Reviewed-on: https://chromium-review.googlesource.com/889644
Reviewed-by: Peter Collingbourne <pcc@chromium.org>
Reviewed-by: Daniel Cheng <dcheng@chromium.org>
Reviewed-by: Robert Liao <robliao@chromium.org>
Commit-Queue: Bob Haarman <inglorion@chromium.org>
Cr-Commit-Position: refs/heads/master@{#536458}
[modify] https://crrev.com/f364fc10b1e89b514e79f6b04ad01f2fc71cfbb5/base/BUILD.gn
[modify] https://crrev.com/f364fc10b1e89b514e79f6b04ad01f2fc71cfbb5/base/win/scoped_handle.cc
[modify] https://crrev.com/f364fc10b1e89b514e79f6b04ad01f2fc71cfbb5/base/win/scoped_handle.h
[modify] https://crrev.com/f364fc10b1e89b514e79f6b04ad01f2fc71cfbb5/base/win/scoped_handle_test_dll.cc
[add] https://crrev.com/f364fc10b1e89b514e79f6b04ad01f2fc71cfbb5/base/win/scoped_handle_verifier.cc
[add] https://crrev.com/f364fc10b1e89b514e79f6b04ad01f2fc71cfbb5/base/win/scoped_handle_verifier.h
[modify] https://crrev.com/f364fc10b1e89b514e79f6b04ad01f2fc71cfbb5/tools/cfi/blacklist.txt

Status: Verified (was: Available)
We got green builds after the fix landed.

Sign in to add a comment