New issue
Advanced search Search tips

Issue 792188 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Dec 2017
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 3
Type: Bug

Blocking:
issue 797267
issue 787920



Sign in to add a comment

chromium.clang/ToTLinux net_unittests fail flakily in net::ReadCRL()

Project Member Reported by h...@chromium.org, Dec 5 2017

Issue description

There are lots of both red and green builds, suggesting this isn't a Clang regression but something being flaky.

From https://ci.chromium.org/buildbot/chromium.clang/ToTLinux/1000

[ RUN      ] CertVerifyProcInternalTest.CRLSetDuringPathBuilding/CertVerifyProcBuiltin
[15857:15857:1205/120912.109406:5806353595:ERROR:cert_verify_proc_builtin.cc(513)] CertVerifyProcBuiltin for 127.0.0.1 failed:
----- Certificate i=3 (CN=D Root CA - Multi-root) -----
ERROR: Certificate is revoked
Received signal 11 SEGV_MAPERR 000000000000
#0 0x7fe6edd4c49c base::debug::StackTrace::StackTrace()
#1 0x7fe6edd4c001 base::debug::(anonymous namespace)::StackDumpSignalHandler()
#2 0x7fe6eda33330 <unknown>
#3 0x7fe6ec3a7521 <unknown>
#4 0x7fe6ecf2ed0f std::__1::basic_string<>::assign()
#5 0x7fe6ed787bf1 net::ReadCRL()
#6 0x7fe6ed78786b net::CRLSetStorage::Parse()
#7 0x00000073b329 net::CertVerifyProcInternalTest_CRLSetDuringPathBuilding_Test::TestBody()
#8 0x000002009ce6 testing::Test::Run()
#9 0x00000200a780 testing::TestInfo::Run()
#10 0x00000200ac67 testing::TestCase::Run()
#11 0x0000020112b7 testing::internal::UnitTestImpl::RunAllTests()
#12 0x000002010f37 testing::UnitTest::Run()
#13 0x00000208faa2 base::TestSuite::Run()
#14 0x00000209284e base::(anonymous namespace)::LaunchUnitTestsInternal()
#15 0x00000209270b base::LaunchUnitTests()
#16 0x000001c1d81b main
#17 0x7fe6ec273f45 __libc_start_main
#18 0x0000005d402a _start
  r8: 00007ffe5dcd5110  r9: 00002e2c0f87dc6d r10: 0000000000000003 r11: 00007fe6ec3dd510
 r12: 00007ffe5dcd53a0 r13: 0000000000000020 r14: 00002e2c1008a000 r15: ffffd1d010c03cf5
  di: 0000000000000020  si: 00002e2c0f87dc8d  bp: 00007ffe5dcd5130  bx: 0000000000000000
  dx: 00007fe6ec3a751c  ax: 0000000000000000  cx: 0000000000000000  sp: 00007ffe5dcd50e8
  ip: 00007fe6ec3a7521 efl: 0000000000010202 cgf: 0000000000000033 erf: 0000000000000006
 trp: 000000000000000e msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(1). Core file will not be generated.



From https://ci.chromium.org/buildbot/chromium.clang/ToTLinux/997

[ RUN      ] CRLSetTest.AddCRLDelta
Received signal 11 SEGV_MAPERR 000000000000
#0 0x7f03e182c49c base::debug::StackTrace::StackTrace()
#1 0x7f03e182c001 base::debug::(anonymous namespace)::StackDumpSignalHandler()
#2 0x7f03e1513330 <unknown>
#3 0x7f03dfe87521 <unknown>
#4 0x7f03e0a0ed0f std::__1::basic_string<>::assign()
#5 0x7f03e1267bf1 net::ReadCRL()
#6 0x7f03e126786b net::CRLSetStorage::Parse()
#7 0x00000074fffd net::CRLSetTest_AddCRLDelta_Test::TestBody()
#8 0x000002009a36 testing::Test::Run()
#9 0x00000200a4d0 testing::TestInfo::Run()
#10 0x00000200a9b7 testing::TestCase::Run()
#11 0x000002011007 testing::internal::UnitTestImpl::RunAllTests()
#12 0x000002010c87 testing::UnitTest::Run()
#13 0x00000208f7f2 base::TestSuite::Run()
#14 0x00000209259e base::(anonymous namespace)::LaunchUnitTestsInternal()
#15 0x00000209245b base::LaunchUnitTests()
#16 0x000001c1d56b main
#17 0x7f03dfd53f45 __libc_start_main
#18 0x0000005d402a _start
  r8: 00007ffe153020d0  r9: 00000000003802c2 r10: 0000000000000003 r11: 00007f03dfebd510
 r12: 00007ffe153022c8 r13: 0000000000000020 r14: 00000ab5eab4d000 r15: fffff549ed542f85
  di: 0000000000000020  si: 00000000003802e2  bp: 00007ffe153020f0  bx: 0000000000000000
  dx: 00007f03dfe8751c  ax: 0000000000000000  cx: 0000000000000000  sp: 00007ffe153020a8
  ip: 00007f03dfe87521 efl: 0000000000010202 cgf: 0000000000000033 erf: 0000000000000006
 trp: 000000000000000e msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(1). Core file will not be generated.



From https://ci.chromium.org/buildbot/chromium.clang/ToTLinux/995

[ RUN      ] CertVerifyProcInternalTest.CRLSetDuringPathBuilding/CertVerifyProcBuiltin
[2252:2252:1204/204742.701699:21828611899:ERROR:cert_verify_proc_builtin.cc(513)] CertVerifyProcBuiltin for 127.0.0.1 failed:
----- Certificate i=3 (CN=D Root CA - Multi-root) -----
ERROR: Certificate is revoked
Received signal 11 SEGV_MAPERR 000000000000
#0 0x7fed52bc849c base::debug::StackTrace::StackTrace()
#1 0x7fed52bc8001 base::debug::(anonymous namespace)::StackDumpSignalHandler()
#2 0x7fed528af330 <unknown>
#3 0x7fed51223521 <unknown>
#4 0x7fed51daad0f std::__1::basic_string<>::assign()
#5 0x7fed52603bf1 net::ReadCRL()
#6 0x7fed5260386b net::CRLSetStorage::Parse()
#7 0x00000073b329 net::CertVerifyProcInternalTest_CRLSetDuringPathBuilding_Test::TestBody()
#8 0x00000200bc86 testing::Test::Run()
#9 0x00000200c720 testing::TestInfo::Run()
#10 0x00000200cc07 testing::TestCase::Run()
#11 0x000002013257 testing::internal::UnitTestImpl::RunAllTests()
#12 0x000002012ed7 testing::UnitTest::Run()
#13 0x000002091ab2 base::TestSuite::Run()
#14 0x00000209485e base::(anonymous namespace)::LaunchUnitTestsInternal()
#15 0x00000209471b base::LaunchUnitTests()
#16 0x000001c1fadb main
#17 0x7fed510eff45 __libc_start_main
#18 0x0000005d402a _start
  r8: 00007ffd06a6a970  r9: 000008428df4fc6d r10: 0000000000000003 r11: 00007fed51259510
 r12: 00007ffd06a6ac00 r13: 0000000000000020 r14: 000008428e765000 r15: fffff7be8ee335b7
  di: 0000000000000020  si: 000008428df4fc8d  bp: 00007ffd06a6a990  bx: 0000000000000000
  dx: 00007fed5122351c  ax: 0000000000000000  cx: 0000000000000000  sp: 00007ffd06a6a948
  ip: 00007fed51223521 efl: 0000000000010202 cgf: 0000000000000033 erf: 0000000000000006
 trp: 000000000000000e msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(1). Core file will not be generated.



https://ci.chromium.org/buildbot/chromium.clang/ToTLinux/993

[ RUN      ] CertVerifyProcInternalTest.CRLSetLeafSerial/CertVerifyProcNSS
Received signal 11 SEGV_MAPERR 000000000000
#0 0x7f961bcd149c base::debug::StackTrace::StackTrace()
#1 0x7f961bcd1001 base::debug::(anonymous namespace)::StackDumpSignalHandler()
#2 0x7f961b9b8330 <unknown>
#3 0x7f961a32c521 <unknown>
#4 0x7f961aeb3d0f std::__1::basic_string<>::assign()
#5 0x7f961b70cbf1 net::ReadCRL()
#6 0x7f961b70c86b net::CRLSetStorage::Parse()
#7 0x000000739987 net::CertVerifyProcInternalTest_CRLSetLeafSerial_Test::TestBody()
#8 0x000002009656 testing::Test::Run()
#9 0x00000200a0f0 testing::TestInfo::Run()
#10 0x00000200a5d7 testing::TestCase::Run()
#11 0x000002010c27 testing::internal::UnitTestImpl::RunAllTests()
#12 0x0000020108a7 testing::UnitTest::Run()
#13 0x00000208f482 base::TestSuite::Run()
#14 0x00000209222e base::(anonymous namespace)::LaunchUnitTestsInternal()
#15 0x0000020920eb base::LaunchUnitTests()
#16 0x000001c1fadb main
#17 0x7f961a1f8f45 __libc_start_main
#18 0x0000005d402a _start
  r8: 00007ffd44359f00  r9: 00000ba05c5ef2ed r10: 0000000000000003 r11: 00007f961a362510
 r12: 00007ffd4435a168 r13: 0000000000000020 r14: 00000ba05ce04000 r15: fffff45c5faf6d33
  di: 0000000000000020  si: 00000ba05c5ef30d  bp: 00007ffd44359f20  bx: 0000000000000000
  dx: 00007f961a32c51c  ax: 0000000000000000  cx: 0000000000000000  sp: 00007ffd44359ed8
  ip: 00007f961a32c521 efl: 0000000000010202 cgf: 0000000000000033 erf: 0000000000000006
 trp: 000000000000000e msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(1). Core file will not be generated.
 

Comment 1 by h...@chromium.org, Dec 5 2017

Blocking: 787920
I can't reproduce with pinned Clang, but I can repro with ToT, which suggests something changed in Clang. I'll try to bisect it.

gn args:
llvm_force_head_revision=true clang_use_chrome_plugins=false clang_base_path="/work/llvm/build.goma" is_debug=false

$ gn clean out/release && ninja -C out/release net_unittests && out/release/net_unittests --gtest_filter="*CRL*" --single-process-tests --gtest_repeat=10

......
[ RUN      ] CertVerifyProcInternalTest.CRLSetDuringPathBuilding/CertVerifyProcNSS
[104667:104667:1205/141256.381477:19897699002986:ERROR:cert_verify_proc_nss.cc(922)] CERT_PKIXVerifyCert for 127.0.0.1 failed err=-8180
[       OK ] CertVerifyProcInternalTest.CRLSetDuringPathBuilding/CertVerifyProcNSS (13 ms)
[ RUN      ] CertVerifyProcInternalTest.CRLSetDuringPathBuilding/CertVerifyProcBuiltin
[104667:104667:1205/141256.392565:19897699014073:ERROR:cert_verify_proc_builtin.cc(513)] CertVerifyProcBuiltin for 127.0.0.1 failed:
----- Certificate i=3 (CN=E Root CA - Multi-root) -----
ERROR: Certificate is revoked


Received signal 11 <unknown> 000000000000
#0 0x000002c73b6c base::debug::StackTrace::StackTrace()
#1 0x000002c736e1 base::debug::(anonymous namespace)::StackDumpSignalHandler()
#2 0x7f3026fc9330 <unknown>
#3 0x7f3025206ba4 <unknown>
#4 0x0000022b24d7 std::__1::basic_string<>::assign()
#5 0x0000029b42b6 net::X509Certificate::GetDEREncoded()
#6 0x0000029faa11 net::(anonymous namespace)::ParseCertificateFromOSHandle()
#7 0x0000029f9dfe net::(anonymous namespace)::CertVerifyProcBuiltin::VerifyInternal()
#8 0x0000029f8b34 net::CertVerifyProc::Verify()
#9 0x0000009e3cc7 net::CertVerifyProcInternalTest_CRLSetDuringPathBuilding_Test::TestBody()
#10 0x00000231a436 testing::Test::Run()
#11 0x00000231aed0 testing::TestInfo::Run()
#12 0x00000231b3b7 testing::TestCase::Run()
#13 0x0000023219e7 testing::internal::UnitTestImpl::RunAllTests()
#14 0x000002321667 testing::UnitTest::Run()
#15 0x000002f307d2 base::TestSuite::Run()
#16 0x000002f33c6e base::(anonymous namespace)::LaunchUnitTestsInternal()
#17 0x000002f33b2b base::LaunchUnitTests()
#18 0x000001ebff3b main
#19 0x7f302518df45 __libc_start_main
#20 0x00000087d02a _start
  r8: 0000000000000000  r9: 0e6121640e6c1845 r10: 0000000000000013 r11: 00007fff1b90e5e1
 r12: 0000000000000000 r13: e6121640e6c18450 r14: 00007fff1b90e790 r15: 0000000000000016
  di: 0000000000000000  si: 303c5fd11a4bab2a  bp: 00007fff1b90e730  bx: e6121640e6c18460
  dx: e6121640e6c18450  ax: e6121640e6c18450  cx: 0000000000000000  sp: 00007fff1b90e6e8
  ip: 00007f3025206ba4 efl: 0000000000010246 cgf: 0000000000000033 erf: 0000000000000000
 trp: 000000000000000d msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(1). Core file will not be generated.

Comment 2 by h...@chromium.org, Dec 5 2017

Status: Started (was: Assigned)
Bisection points to
r319482 - [memcpyopt] Teach memcpyopt to optimize across basic blocks
(+r319483 which adds a missing file)

Comment 3 by h...@chromium.org, Dec 5 2017

There's a bug about this upstream: https://bugs.llvm.org/show_bug.cgi?id=35519
And a patch in review: https://reviews.llvm.org/D40802
I wonder if that fixes our problem too..

Comment 4 by h...@chromium.org, Dec 6 2017

No, D40802 doesn't fix our problem, so we might be having a different failure mode.

Comment 5 by h...@chromium.org, Dec 6 2017

Pretty sure it's (at least) ./obj/net/net/crl_set_storage.o that gets miscompiled.

Attaching the preprocessed source. This is the build command:

clang++ -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -funwind-tables -fPIC -pthread -Xclang -mllvm -Xclang -instcombine-lower-dbg-declare=0 -no-canonical-prefixes -m64 -march=x86-64 -O2 -fno-ident -fdata-sections -ffunction-sections -fno-omit-frame-pointer -g0 -fvisibility=hidden -std=gnu++14 -fno-exceptions -fno-rtti -nostdinc++ -fvisibility-inlines-hidden -c /tmp/crl_set_storage.ii -o /tmp/crl_set_storage.o
crl_set_storage.ii
2.5 MB Download

Comment 6 by h...@chromium.org, Dec 6 2017

Diffing before and after the memcpyopt change:

@@ -396,7 +396,7 @@
 
 _ZNSt3__16vectorINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENS4_IS6_EEE9push_backEOS6_.exit.thread: ; preds = %if.end6
   %28 = bitcast %"class.std::__1::basic_string"* %26 to i8*
-  call void @llvm.memcpy.p0i8.p0i8.i64(i8* %28, i8* nonnull %15, i64 24, i32 8, i1 false) #13
+  call void @llvm.memset.p0i8.i64(i8* %28, i8 0, i64 24, i32 8, i1 false)
   call void @llvm.memset.p0i8.i64(i8* nonnull %15, i8 0, i64 24, i32 8, i1 false) #13
   %29 = load %"class.std::__1::basic_string"*, %"class.std::__1::basic_string"** %__end_.i.i.i, align 8
   %incdec.ptr.i = getelementptr inbounds %"class.std::__1::basic_string", %"class.std::__1::basic_string"* %29, i64 1
@@ -813,26 +813,24 @@
   br i1 %cmp.i147, label %_ZNSt3__16vectorINS_4pairINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENS0_IS7_NS5_IS7_EEEEEENS5_ISA_EEE9push_backEOSA_.exit.thread, label %_ZNSt3__16vectorINS_4pairINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENS0_IS7_NS5_IS7_EEEEEENS5_ISA_EEE9push_backEOSA_.exit
 
 _ZNSt3__16vectorINS_4pairINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENS0_IS7_NS5_IS7_EEEEEENS5_ISA_EEE9push_backEOSA_.exit.thread: ; preds = %for.body
-  %43 = bitcast %"struct.std::__1::pair"* %41 to i8*
-  call void @llvm.memcpy.p0i8.p0i8.i64(i8* %43, i8* nonnull %31, i64 24, i32 8, i1 false) #13
   call void @llvm.memset.p0i8.i64(i8* nonnull %31, i8 0, i64 24, i32 8, i1 false) #13
   %second.i.i.i.i.i = getelementptr inbounds %"struct.std::__1::pair", %"struct.std::__1::pair"* %41, i64 0, i32 1


I haven't looked enough at the code to figure out what's going on, but these seem to be the two changes: replacing the memcpy with memset, and further down, a memcpy disappears completely.

Comment 7 by h...@chromium.org, Dec 6 2017

_ZNSt3__16vectorINS_4pairINS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEENS0_IS7_NS5_IS7_EEEEEENS5_ISA_EEE9push_backEOSA_.exit.thread: ; preds = %for.body
-  %43 = bitcast %"struct.std::__1::pair"* %41 to i8*
-  call void @llvm.memcpy.p0i8.p0i8.i64(i8* %43, i8* nonnull %31, i64 24, i32 8, i1 false) #13


We were copying from %31 which points to an alloca, initialized in for.body as:

call void @llvm.memset.p0i8.i64(i8* nonnull %31, i8 0, i64 24, i32 8, i1 false)

I think maybe this is similar to PR35519, where memcpyopt thinks %31 is uninitialized and drops the copy, failing to realize it was initialized by the memset?

Comment 8 by h...@chromium.org, Dec 6 2017

Revert is in LLVM r319873 while this gets figured out on https://bugs.llvm.org/show_bug.cgi?id=35519

Comment 9 by h...@chromium.org, Dec 12 2017

Status: Fixed (was: Started)
This is currently fixed since the revert is in.

Comment 10 by a...@chromium.org, Dec 22 2017

FYI this ended up relanding in a Clang roll and caused bug 797267.

Comment 11 by kbr@chromium.org, Dec 22 2017

Blocking: 797267

Sign in to add a comment