gl_tests don't work under asan, tsan, or msan |
|||||||||
Issue descriptionhttps://ci.chromium.org/buildbot/tryserver.chromium.linux/linux_chromium_msan_rel_ng/982 https://ci.chromium.org/buildbot/tryserver.chromium.linux/linux_chromium_tsan_rel_ng/283782 One example failure (via the swarming link); but there are many: [ RUN ] OpenGLES3FunctionTest.GetFragDataLocationInvalid [26902:26902:0406/053054.788318:16822540442:FATAL:gl_factory_x11.cc(122)] Check failed: false. #0 0x00000058c8bc (/b/s/w/ir/out/Release/gl_tests+0x58c8bb) #1 0x000000b7895a (/b/s/w/ir/out/Release/gl_tests+0xb78959) #2 0x000000b76b7f (/b/s/w/ir/out/Release/gl_tests+0xb76b7e) #3 0x000000b74cbc (/b/s/w/ir/out/Release/gl_tests+0xb74cbb) #4 0x000000d8733d (/b/s/w/ir/out/Release/gl_tests+0xd8733c) #5 0x000000d867b6 (/b/s/w/ir/out/Release/gl_tests+0xd867b5) #6 0x00000065dff9 (/b/s/w/ir/out/Release/gl_tests+0x65dff8) #7 0x00000065d783 (/b/s/w/ir/out/Release/gl_tests+0x65d782) #8 0x0000005e4356 (/b/s/w/ir/out/Release/gl_tests+0x5e4355) #9 0x000000b3a667 (/b/s/w/ir/out/Release/gl_tests+0xb3a666) #10 0x000000b3b67e (/b/s/w/ir/out/Release/gl_tests+0xb3b67d) #11 0x000000b3bec7 (/b/s/w/ir/out/Release/gl_tests+0xb3bec6) #12 0x000000b4c497 (/b/s/w/ir/out/Release/gl_tests+0xb4c496) #13 0x000000b4bd9c (/b/s/w/ir/out/Release/gl_tests+0xb4bd9b) #14 0x000000c0ecc7 (/b/s/w/ir/out/Release/gl_tests+0xc0ecc6) #15 0x000000684a02 (/b/s/w/ir/out/Release/gl_tests+0x684a01) #16 0x000000684b05 (/b/s/w/ir/out/Release/gl_tests+0x684b04) #17 0x000000c12341 (/b/s/w/ir/out/Release/gl_tests+0xc12340) #18 0x000000c12a44 (/b/s/w/ir/out/Release/gl_tests+0xc12a43) #19 0x000000684902 (/b/s/w/ir/out/Release/gl_tests+0x684901) #20 0x7f11cec76f45 (/lib/x86_64-linux-gnu/libc-2.19.so+0x21f44) #21 0x00000054c02a (/b/s/w/ir/out/Release/gl_tests+0x54c029)
,
Apr 10 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a8fc38f72151224ad8cc4582cda864a7721560bc commit a8fc38f72151224ad8cc4582cda864a7721560bc Author: Nico Weber <thakis@chromium.org> Date: Tue Apr 10 02:20:11 2018 Remove msan, tsan test lists and just use asan test lists on all memory bots. For example, angle_unittests wasn't running on the MSan bot, which wasn't intentional. If a test really doesn't work under e.g. TSan, it can be added to the exceptions file instead. For another example, https://chromium-review.googlesource.com/#/c/981787/ added cronet tests, but again not to msan. Disable a few tests that don't pass on tsan or msan. Bug: 830659 , 830648 ,830653, 830819 ,830944, 830945 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Change-Id: Icac81720f32c24497942b9c056c6d6999a866ec6 Reviewed-on: https://chromium-review.googlesource.com/987477 Reviewed-by: Dan Erat <derat@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#549382} [modify] https://crrev.com/a8fc38f72151224ad8cc4582cda864a7721560bc/chromeos/dbus/cras_audio_client_unittest.cc [modify] https://crrev.com/a8fc38f72151224ad8cc4582cda864a7721560bc/testing/buildbot/chromium.memory.json [modify] https://crrev.com/a8fc38f72151224ad8cc4582cda864a7721560bc/testing/buildbot/test_suite_exceptions.pyl [modify] https://crrev.com/a8fc38f72151224ad8cc4582cda864a7721560bc/testing/buildbot/test_suites.pyl [modify] https://crrev.com/a8fc38f72151224ad8cc4582cda864a7721560bc/testing/buildbot/waterfalls.pyl [modify] https://crrev.com/a8fc38f72151224ad8cc4582cda864a7721560bc/third_party/blink/renderer/platform/text/text_run_test.cc [modify] https://crrev.com/a8fc38f72151224ad8cc4582cda864a7721560bc/ui/gl/gl_context_glx_unittest.cc [modify] https://crrev.com/a8fc38f72151224ad8cc4582cda864a7721560bc/ui/gl/gl_image_shared_memory_unittest.cc
,
Apr 10 2018
,
Apr 11 2018
asan too: https://ci.chromium.org/buildbot/tryserver.chromium.linux/linux_chromium_asan_rel_ng/573930
,
Apr 12 2018
,
Apr 12 2018
CFI too. Dupe has a symbolized stack: [ RUN ] GLApplyScreenSpaceAntialiasingCHROMIUMTest.TextureStatePreserved [31744:31744:0411/214112.790867:1485531039:FATAL:gl_factory_x11.cc(122)] Check failed: false. #0 0x000000b685cc base::debug::StackTrace::StackTrace() #1 0x000000b66676 logging::LogMessage::~LogMessage() #2 0x000000d0755b gl::init::CreateOffscreenGLSurfaceWithFormat() #3 0x000000d06c7d gl::init::CreateOffscreenGLSurface() #4 0x0000007fd1e0 gpu::GLManager::InitializeWithWorkaroundsImpl() #5 0x0000007fccb8 gpu::GLManager::Initialize() #6 0x0000007b4a93 gpu::GLApplyScreenSpaceAntialiasingCHROMIUMTest::SetUp() #7 0x000000b3e9ed testing::Test::Run() #8 0x000000b3f2fd testing::TestInfo::Run() #9 0x000000b3fa72 testing::TestCase::Run() #10 0x000000b46773 testing::internal::UnitTestImpl::RunAllTests() #11 0x000000b463fe testing::UnitTest::Run() #12 0x000000bdab39 base::TestSuite::Run() #13 0x0000008174b4 (anonymous namespace)::RunHelper() #14 0x000000817631 _ZN4base8internal7InvokerINS0_9BindStateIPFiPNS_9TestSuiteEEJNS0_17UnretainedWrapperIS3_EEEEEFivEE7RunImplIRKS6_RKNSt3__15tupleIJS8_EEEJLm0EEEEiOT_OT0_NSF_16integer_sequenceImJXspT1_EEEE #15 0x000000b52f95 _ZNO4base12OnceCallbackIFivEE3RunEv #16 0x000000bdd880 base::(anonymous namespace)::LaunchUnitTestsInternal() #17 0x000000bddb6d base::LaunchUnitTestsSerially() #18 0x0000008173db main #19 0x7f7311c24f45 __libc_start_main #20 0x00000078302a _start
,
Apr 13 2018
,
Apr 13 2018
At the start of the tests there is the following log message: [31888:31888:0330/084254.849080:2918057683:ERROR:gl_surface_glx.cc(418)] XOpenDisplay failed. [31888:31888:0330/084254.849644:2918058176:ERROR:gl_initializer_x11.cc(157)] GLSurfaceGLX::InitializeOneOff failed. GLX initialization failed because there is no display. Are the tests running on GPU trybots?
,
Apr 13 2018
Probably not. Do you know where that needs to be configured?
,
Apr 13 2018
Aha. There are several places in src/testing/buildbot/test_suite_exceptions.pyl where other tests (like gl_unittests) are specified to run on physical hardware. Let me add these exceptions for gl_tests.
,
Apr 20 2018
,
Apr 20 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/345ea06924f373f6e20ac042a55f758acfbc401f commit 345ea06924f373f6e20ac042a55f758acfbc401f Author: Kenneth Russell <kbr@chromium.org> Date: Fri Apr 20 03:35:51 2018 Run ASAN/MSAN/TSAN gl_tests on physical hardware. We should add a better way to define mixins like these, but for the time being, add exceptions to trigger these jobs on physical hardware. Bug: 830659 , 835072 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_chromium_asan_rel_ng;master.tryserver.chromium.linux:linux_chromium_cfi_rel_ng Change-Id: I30c377d28e6c2ea1202d0d64f86de9665c0266ad Reviewed-on: https://chromium-review.googlesource.com/1013127 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Nico Weber <thakis@chromium.org> Cr-Commit-Position: refs/heads/master@{#552253} [modify] https://crrev.com/345ea06924f373f6e20ac042a55f758acfbc401f/gpu/command_buffer/tests/gl_oes_egl_image_unittest.cc [modify] https://crrev.com/345ea06924f373f6e20ac042a55f758acfbc401f/testing/buildbot/chromium.memory.json [modify] https://crrev.com/345ea06924f373f6e20ac042a55f758acfbc401f/testing/buildbot/test_suite_exceptions.pyl
,
Apr 20 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/615a8d0f9b333be87ff27a884aec29880bef281a commit 615a8d0f9b333be87ff27a884aec29880bef281a Author: Morten Stenshorne <mstensho@chromium.org> Date: Fri Apr 20 12:19:21 2018 Revert "Run ASAN/MSAN/TSAN gl_tests on physical hardware." This reverts commit 345ea06924f373f6e20ac042a55f758acfbc401f. Reason for revert: Probably what turned Linux MSan Tests red. crbug.com/835261 Original change's description: > Run ASAN/MSAN/TSAN gl_tests on physical hardware. > > We should add a better way to define mixins like these, but for the > time being, add exceptions to trigger these jobs on physical hardware. > > Bug: 830659 , 835072 > Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_chromium_asan_rel_ng;master.tryserver.chromium.linux:linux_chromium_cfi_rel_ng > Change-Id: I30c377d28e6c2ea1202d0d64f86de9665c0266ad > Reviewed-on: https://chromium-review.googlesource.com/1013127 > Commit-Queue: Kenneth Russell <kbr@chromium.org> > Reviewed-by: Nico Weber <thakis@chromium.org> > Cr-Commit-Position: refs/heads/master@{#552253} TBR=thakis@chromium.org,kbr@chromium.org Change-Id: I2c4a5fdfeaf2b0948e4d256b151915d988204124 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 830659 , 835072 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_chromium_asan_rel_ng;master.tryserver.chromium.linux:linux_chromium_cfi_rel_ng Reviewed-on: https://chromium-review.googlesource.com/1021571 Reviewed-by: Morten Stenshorne <mstensho@chromium.org> Commit-Queue: Morten Stenshorne <mstensho@chromium.org> Cr-Commit-Position: refs/heads/master@{#552307} [modify] https://crrev.com/615a8d0f9b333be87ff27a884aec29880bef281a/gpu/command_buffer/tests/gl_oes_egl_image_unittest.cc [modify] https://crrev.com/615a8d0f9b333be87ff27a884aec29880bef281a/testing/buildbot/chromium.memory.json [modify] https://crrev.com/615a8d0f9b333be87ff27a884aec29880bef281a/testing/buildbot/test_suite_exceptions.pyl
,
Apr 21 2018
This was the failure: https://ci.chromium.org/buildbot/chromium.memory/Linux%20MSan%20Tests/9291 Going to try to initialize those variables: https://chromium.googlesource.com/chromium/src/+/00226e5a08d7c0d45e972668f5e9cb5b8911836d/ui/gl/gl_surface_glx.cc#422 I feel we've seen this problem before where we can't see what the un-instrumented driver does.
,
Apr 21 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a5f991706198e9c6cc36c66163e96ddd6ecda92b commit a5f991706198e9c6cc36c66163e96ddd6ecda92b Author: Kenneth Russell <kbr@chromium.org> Date: Sat Apr 21 03:03:59 2018 Initialize variables passed by pointer to glXQueryVersion. Attempt to make the MSAN bot happy. Tbr: thakis@chromium.org Bug: 830659 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel Change-Id: I33920bc08cab060d396fb91ce222a7bc169fdcac Reviewed-on: https://chromium-review.googlesource.com/1023276 Reviewed-by: Kenneth Russell <kbr@chromium.org> Reviewed-by: Nico Weber <thakis@chromium.org> Commit-Queue: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#552565} [modify] https://crrev.com/a5f991706198e9c6cc36c66163e96ddd6ecda92b/ui/gl/gl_surface_glx.cc
,
Apr 21 2018
The gl_tests won't run on MSAN right now. It looks like the OpenGL driver would have to be compiled with MSAN instrumentation in order for that to work: https://ci.chromium.org/buildbot/tryserver.chromium.linux/linux_chromium_msan_rel_ng/1011 In the reland, https://chromium-review.googlesource.com/1023333 , disabling these tests on MSAN again and just enabling them on ASAN, CFI, and TSAN bots.
,
Apr 28 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/1a515be834afd445b40b0d6bcd64568b08e32262 commit 1a515be834afd445b40b0d6bcd64568b08e32262 Author: Kenneth Russell <kbr@chromium.org> Date: Sat Apr 28 03:54:45 2018 Reland: Run ASAN/TSAN gl_tests on physical hardware. We should add a better way to define mixins like these, but for the time being, add exceptions to trigger these jobs on physical hardware. It's not feasible to run these tests on MSAN right now. Tbr: thakis@chromium.org Bug: 830659 , 835072 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_chromium_asan_rel_ng;master.tryserver.chromium.linux:linux_chromium_msan_rel_ng;master.tryserver.chromium.linux:linux_chromium_tsan_rel_ng;master.tryserver.chromium.linux:linux_chromium_cfi_rel_ng Change-Id: Ifbc4d86262dfd203dd8b57670aac989e433b1f46 Reviewed-on: https://chromium-review.googlesource.com/1023333 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#554625} [modify] https://crrev.com/1a515be834afd445b40b0d6bcd64568b08e32262/gpu/command_buffer/tests/gl_oes_egl_image_unittest.cc [modify] https://crrev.com/1a515be834afd445b40b0d6bcd64568b08e32262/testing/buildbot/chromium.memory.json [modify] https://crrev.com/1a515be834afd445b40b0d6bcd64568b08e32262/testing/buildbot/test_suite_exceptions.pyl
,
May 1 2018
Looks like that CL is sticking. Closing as Fixed.
,
May 2 2018
Nice! There are still 3 references to this bug: https://cs.chromium.org/search/?q=830659&sq=package:chromium&type=cs I suppose the two msan bot lines should have a comment saying # Can't run on MSan because gl_tests uses the hardware driver, which # isn't instrumented. above them instead, and the third line can just be deleted?
,
May 3 2018
Thanks for pointing this out. Cleaning these up in https://chromium-review.googlesource.com/1041565 .
,
May 3 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/55bbec5e9e8fc165cc189417d56fccfe6bb042d9 commit 55bbec5e9e8fc165cc189417d56fccfe6bb042d9 Author: Kenneth Russell <kbr@chromium.org> Date: Thu May 03 18:44:55 2018 Removed leftover references to crbug.com/830659 . Thanks to thakis for pointing these out. Bug: 830659 Change-Id: I45f285d29c16c666292a7479248c97222b6ff39f Tbr: thakis@chromium.org Reviewed-on: https://chromium-review.googlesource.com/1041565 Reviewed-by: Kenneth Russell <kbr@chromium.org> Reviewed-by: Nico Weber <thakis@chromium.org> Cr-Commit-Position: refs/heads/master@{#555826} [modify] https://crrev.com/55bbec5e9e8fc165cc189417d56fccfe6bb042d9/testing/buildbot/test_suite_exceptions.pyl |
|||||||||
►
Sign in to add a comment |
|||||||||
Comment 1 Deleted