Add Linux TSAN bot to chromium.gpu.fyi waterfall |
|||||||||||||||||
Issue descriptionPer discussion with the V8 team, there is a desire to add a TSAN bot running GPU tests. So that the V8 team doesn't end up chasing down lots of bugs that aren't in V8, the GPU team should add one such bot to the chromium.gpu.fyi waterfall first.
,
Feb 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/3797588513ef7108c73e868da142d9fae9cdf43f commit 3797588513ef7108c73e868da142d9fae9cdf43f Author: Kenneth Russell <kbr@chromium.org> Date: Fri Feb 24 21:09:46 2017 Deploy Linux GPU TSAN bot. BUG= 691029 Change-Id: I89d275cb3cde02d1545f993b69d7f8a18ca57320 TBR=dpranke@chromium.org Reviewed-on: https://chromium-review.googlesource.com/446861 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> [add] https://crrev.com/3797588513ef7108c73e868da142d9fae9cdf43f/scripts/slave/recipes/chromium.expected/full_chromium_gpu_fyi_Linux_GPU_TSAN_Release.json [modify] https://crrev.com/3797588513ef7108c73e868da142d9fae9cdf43f/scripts/slave/recipe_modules/chromium_tests/chromium_gpu_fyi.py [modify] https://crrev.com/3797588513ef7108c73e868da142d9fae9cdf43f/masters/master.chromium.gpu.fyi/builders.pyl
,
Feb 24 2017
,
Feb 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/bb480a041f8788d627f86322cce1fedf07f4706e commit bb480a041f8788d627f86322cce1fedf07f4706e Author: kbr <kbr@chromium.org> Date: Fri Feb 24 22:38:52 2017 Deploy Linux GPU TSAN bot. BUG= 691029 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel TBR=dpranke@chromium.org,zmo@chromium.org Review-Url: https://codereview.chromium.org/2713303002 Cr-Commit-Position: refs/heads/master@{#452963} [modify] https://crrev.com/bb480a041f8788d627f86322cce1fedf07f4706e/content/test/gpu/generate_buildbot_json.py [modify] https://crrev.com/bb480a041f8788d627f86322cce1fedf07f4706e/testing/buildbot/chromium.gpu.fyi.json [modify] https://crrev.com/bb480a041f8788d627f86322cce1fedf07f4706e/tools/mb/mb_config.pyl
,
Feb 24 2017
Scheduling chromium.gpu.fyi waterfall restart for about 7 PM Pacific today to deploy the new bot.
,
Feb 24 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager/+/8ff282a554a763fecdb90bf967023f6b477da278 commit 8ff282a554a763fecdb90bf967023f6b477da278 Author: Kenneth Russell <kbr@chromium.org> Date: Fri Feb 24 23:04:18 2017
,
Feb 25 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/6ea0149f43ecab479dfc4d31354908a413110ae3 commit 6ea0149f43ecab479dfc4d31354908a413110ae3 Author: Kenneth Russell <kbr@chromium.org> Date: Sat Feb 25 08:32:04 2017 Fix scheduler for Linux GPU TSAN bot. BUG= 691029 Change-Id: Id96e8338c8d64aad290785eccce33fbd575f07c7 TBR=dpranke@chromium.org Reviewed-on: https://chromium-review.googlesource.com/446902 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> [modify] https://crrev.com/6ea0149f43ecab479dfc4d31354908a413110ae3/masters/master.chromium.gpu.fyi/builders.pyl
,
Feb 25 2017
The following revision refers to this bug: https://chrome-internal.googlesource.com/infradata/master-manager/+/1c7c7ec5f85f897c797fac63d91cb0902dd7020f commit 1c7c7ec5f85f897c797fac63d91cb0902dd7020f Author: Kenneth Russell <kbr@chromium.org> Date: Sat Feb 25 09:17:05 2017
,
Feb 27 2017
,
Feb 27 2017
,
Feb 27 2017
The browser doesn't start correctly under TSAN, and all of the Telemetry-based tests are failing on this bot. Filed a couple of bugs. Let's see whether these are known issues and/or whether it's feasible to get the full browser to start under TSAN.
,
Feb 28 2017
danakj@ as current pixel wrangler: FYI, this newly added bot on the FYI waterfall is heavily red.
We could add a new filter to src/content/test/gpu/generate_buildbot_json.py , like a new field to tester_config like 'instrumentation_types': ['asan', 'msan', 'tsan'], and then (a) mark "Linux GPU TSAN Release" as a "tsan" bot, and add:
'disabled_tester_configs': [
{
'instrumentation_types': ['tsan']
}
]
to all the tests that are failing.
,
Mar 3 2017
I think I've done what you said in #12 but presubmit fails. I am noting that content/test/gpu/run_unittests.py fails before my patch too though. So probably not me and I'll upload bypassing hooks.
,
Mar 3 2017
Was it that a gclient sync was needed in order to get third_party/webgl/src in sync with webgl[2]_conformance_expectations.py? content/test/gpu/run_unittests.py passes here (on macOS, synced earlier this afternoon).
,
Mar 3 2017
You're right, gclient sync fixes things.
,
Mar 3 2017
Well, maybe not. The output is a bit confusing but I get 2 failures it looks like, let me make a bug.
,
Mar 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0cf54c3293b5ed38cbfeedcbf8b8108d9bd25f6f commit 0cf54c3293b5ed38cbfeedcbf8b8108d9bd25f6f Author: danakj <danakj@chromium.org> Date: Sat Mar 04 00:00:00 2017 Skip failing test suites on the GPU FYI TSAN bot. This adds an 'instrumentation_type' field to the TSAN bot and checks for 'disabled_instrumentation_types' field for each test suite. If both are present and the disabled type is the bot's type, then the test suite is skipped. R=kbr@chromium.org BUG= 691029 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Review-Url: https://codereview.chromium.org/2727643011 Cr-Commit-Position: refs/heads/master@{#454718} [modify] https://crrev.com/0cf54c3293b5ed38cbfeedcbf8b8108d9bd25f6f/content/test/gpu/generate_buildbot_json.py [modify] https://crrev.com/0cf54c3293b5ed38cbfeedcbf8b8108d9bd25f6f/testing/buildbot/chromium.gpu.fyi.json
,
Mar 6 2017
Confirmed that the TSAN bot is green now. But the following suites are disabled: tab_capture_end2end_tests context_lost depth_capture gpu_process_launch_tests hardware_accelerated_feature maps_pixel_test pixel_test screenshot_sync trace_test webgl_conformance webgl_conformance_angle_tests webgl2_conformance_tests webgl2_conformance_angle_tests
,
Mar 7 2017
Thank you danakj@ for getting this bot to a reasonable state of green! There are two legitimate flaky failures of the angle_end2end_tests and angle_perftests suites on this bot which look like real lock inversion problems. I'm not sure whether this is a bug in ANGLE's DisplayGLX or just something in the GLX implementation which needs to be suppressed. Could someone from the ANGLE team please take a look at this? Log excerpt from https://chromium-swarm.appspot.com/task?id=34bee746f22b1010&refresh=10&show_raw=1 : ----- ERR: initialize(428): ANGLE Display::initialize error 0: Could not create GL context. ================== WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=25818) Cycle in lock order graph: M47 (0x7fc4e2b95a00) => M50 (0x7fc4e2b95a88) => M47 Mutex M50 acquired here while holding mutex M47 in main thread: #0 pthread_mutex_lock ??:? (angle_perftests+0x46b79e) #1 __glXLookupDisplay ??:? (libGLX.so.0+0x9a8c) #2 initialize third_party/angle/src/libANGLE/renderer/gl/glx/DisplayGLX.cpp:288 (angle_perftests+0x8199fc) #3 initialize third_party/angle/src/libANGLE/Display.cpp:424 (angle_perftests+0x77c5c2) #4 Initialize third_party/angle/src/libGLESv2/entry_points_egl.cpp:88 (angle_perftests+0x5d8e3d) #5 eglInitialize third_party/angle/src/libEGL/libEGL.cpp:87 (angle_perftests+0x5d8b79) #6 initializeGL third_party/angle/util/EGLWindow.cpp:206 (angle_perftests+0x5d4892) #7 IsPlatformAvailable third_party/angle/src/tests/test_utils/angle_test_instantiate.cpp:84 (angle_perftests+0x4f9c9b) #8 FilterTestParams<(anonymous namespace)::BufferSubDataParams> third_party/angle/src/tests/test_utils/angle_test_instantiate.h:33 (angle_perftests+0x4bd7b0) #9 gtest_BufferSubDataBenchmark_EvalGenerator_ third_party/angle/src/tests/perf_tests/BufferSubData.cpp:386 (angle_perftests+0x4bd7b0) #10 RegisterTests testing/gtest/include/gtest/internal/gtest-param-util.h:554 (angle_perftests+0x4c3f15) #11 RegisterTests testing/gtest/include/gtest/internal/gtest-param-util.h:714 (angle_perftests+0x5c4c4c) #12 RegisterParameterizedTests testing/gtest/src/gtest.cc:2620 (angle_perftests+0x5c4c4c) #13 PostFlagParsingInit testing/gtest/src/gtest.cc:4452 (angle_perftests+0x5c4c4c) #14 InitGoogleTestImpl<char> testing/gtest/src/gtest.cc:5354 (angle_perftests+0x5c7277) #15 testing::InitGoogleTest(int*, char**) testing/gtest/src/gtest.cc:5372 (angle_perftests+0x5c70c0) #16 InitializeFromCommandLine base/test/test_suite.cc:165 (angle_perftests+0x592df2) #17 TestSuite base/test/test_suite.cc:141 (angle_perftests+0x592df2) #18 main gpu/angle_perftests_main.cc:23 (angle_perftests+0x4fa849) Mutex M47 previously acquired by the same thread here: #0 pthread_mutex_lock ??:? (angle_perftests+0x46b79e) #1 glXGetCurrentReadDrawable ??:? (libGLX.so.0+0x753e) #2 initialize third_party/angle/src/libANGLE/renderer/gl/glx/DisplayGLX.cpp:288 (angle_perftests+0x8199fc) #3 initialize third_party/angle/src/libANGLE/Display.cpp:424 (angle_perftests+0x77c5c2) #4 Initialize third_party/angle/src/libGLESv2/entry_points_egl.cpp:88 (angle_perftests+0x5d8e3d) #5 eglInitialize third_party/angle/src/libEGL/libEGL.cpp:87 (angle_perftests+0x5d8b79) #6 initializeGL third_party/angle/util/EGLWindow.cpp:206 (angle_perftests+0x5d4892) #7 IsPlatformAvailable third_party/angle/src/tests/test_utils/angle_test_instantiate.cpp:84 (angle_perftests+0x4f9c9b) #8 FilterTestParams<(anonymous namespace)::BufferSubDataParams> third_party/angle/src/tests/test_utils/angle_test_instantiate.h:33 (angle_perftests+0x4bd7b0) #9 gtest_BufferSubDataBenchmark_EvalGenerator_ third_party/angle/src/tests/perf_tests/BufferSubData.cpp:386 (angle_perftests+0x4bd7b0) #10 RegisterTests testing/gtest/include/gtest/internal/gtest-param-util.h:554 (angle_perftests+0x4c3f15) #11 RegisterTests testing/gtest/include/gtest/internal/gtest-param-util.h:714 (angle_perftests+0x5c4c4c) #12 RegisterParameterizedTests testing/gtest/src/gtest.cc:2620 (angle_perftests+0x5c4c4c) #13 PostFlagParsingInit testing/gtest/src/gtest.cc:4452 (angle_perftests+0x5c4c4c) #14 InitGoogleTestImpl<char> testing/gtest/src/gtest.cc:5354 (angle_perftests+0x5c7277) #15 testing::InitGoogleTest(int*, char**) testing/gtest/src/gtest.cc:5372 (angle_perftests+0x5c70c0) #16 InitializeFromCommandLine base/test/test_suite.cc:165 (angle_perftests+0x592df2) #17 TestSuite base/test/test_suite.cc:141 (angle_perftests+0x592df2) #18 main gpu/angle_perftests_main.cc:23 (angle_perftests+0x4fa849) Mutex M47 acquired here while holding mutex M50 in main thread: #0 pthread_mutex_lock ??:? (angle_perftests+0x46b79e) #1 __glXDisplayClosed ??:? (libGLX.so.0+0x6f8a) #2 IsPlatformAvailable third_party/angle/src/tests/test_utils/angle_test_instantiate.cpp:90 (angle_perftests+0x4f9cc7) #3 FilterTestParams<(anonymous namespace)::BufferSubDataParams> third_party/angle/src/tests/test_utils/angle_test_instantiate.h:33 (angle_perftests+0x4bd7b0) #4 gtest_BufferSubDataBenchmark_EvalGenerator_ third_party/angle/src/tests/perf_tests/BufferSubData.cpp:386 (angle_perftests+0x4bd7b0) #5 RegisterTests testing/gtest/include/gtest/internal/gtest-param-util.h:554 (angle_perftests+0x4c3f15) #6 RegisterTests testing/gtest/include/gtest/internal/gtest-param-util.h:714 (angle_perftests+0x5c4c4c) #7 RegisterParameterizedTests testing/gtest/src/gtest.cc:2620 (angle_perftests+0x5c4c4c) #8 PostFlagParsingInit testing/gtest/src/gtest.cc:4452 (angle_perftests+0x5c4c4c) #9 InitGoogleTestImpl<char> testing/gtest/src/gtest.cc:5354 (angle_perftests+0x5c7277) #10 testing::InitGoogleTest(int*, char**) testing/gtest/src/gtest.cc:5372 (angle_perftests+0x5c70c0) #11 InitializeFromCommandLine base/test/test_suite.cc:165 (angle_perftests+0x592df2) #12 TestSuite base/test/test_suite.cc:141 (angle_perftests+0x592df2) #13 main gpu/angle_perftests_main.cc:23 (angle_perftests+0x4fa849) Mutex M50 previously acquired by the same thread here: #0 pthread_mutex_lock ??:? (angle_perftests+0x46b79e) #1 __glXAddVendorDrawableMapping ??:? (libGLX.so.0+0xa68a) #2 IsPlatformAvailable third_party/angle/src/tests/test_utils/angle_test_instantiate.cpp:90 (angle_perftests+0x4f9cc7) #3 FilterTestParams<(anonymous namespace)::BufferSubDataParams> third_party/angle/src/tests/test_utils/angle_test_instantiate.h:33 (angle_perftests+0x4bd7b0) #4 gtest_BufferSubDataBenchmark_EvalGenerator_ third_party/angle/src/tests/perf_tests/BufferSubData.cpp:386 (angle_perftests+0x4bd7b0) #5 RegisterTests testing/gtest/include/gtest/internal/gtest-param-util.h:554 (angle_perftests+0x4c3f15) #6 RegisterTests testing/gtest/include/gtest/internal/gtest-param-util.h:714 (angle_perftests+0x5c4c4c) #7 RegisterParameterizedTests testing/gtest/src/gtest.cc:2620 (angle_perftests+0x5c4c4c) #8 PostFlagParsingInit testing/gtest/src/gtest.cc:4452 (angle_perftests+0x5c4c4c) #9 InitGoogleTestImpl<char> testing/gtest/src/gtest.cc:5354 (angle_perftests+0x5c7277) #10 testing::InitGoogleTest(int*, char**) testing/gtest/src/gtest.cc:5372 (angle_perftests+0x5c70c0) #11 InitializeFromCommandLine base/test/test_suite.cc:165 (angle_perftests+0x592df2) #12 TestSuite base/test/test_suite.cc:141 (angle_perftests+0x592df2) #13 main gpu/angle_perftests_main.cc:23 (angle_perftests+0x4fa849) SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:? in __interceptor_pthread_mutex_lock
,
Mar 16 2017
webgl_conformance_gl_passthrough_tests are also failing Do they need to be disabled as well? [4941:4941:0315/163241.228717:ERROR:gles2_cmd_decoder_passthrough_doers.cc(3045)] Not implemented reached in error::Error gpu::gles2::GLES2DecoderPassthroughImpl::DoTraceBeginCHROMIUM(const char *, const char *) Looks suspicious to me. As to angle_perftests problem from comment #19, this is definitely in libGLX.so and not in ANGLE. I think angle_perftests don't run on Linux, so they can be disabled for now?
,
Mar 16 2017
Yes, the intent was to disable all the Telemetry-based tests, so webgl_conformance_gl_passthrough_tests should be disabled too. angle_perftests seems to be run on both the Win and Linux NVIDIA bots. I agree that the TSAN report looks like a bug inside the GLX implementation, but I think those should be only warnings, and shouldn't be preventing the test suite from running. It's fine with me to disable it, but given that other ANGLE based tests like angle_deqp_gles2_gl_tests are running OK on this bot, shouldn't we figure out the essential difference and get angle_perftests working? The essential error seems to be: ERR: initialize(428): ANGLE Display::initialize error 0: Could not create GL context. Why is this working OK for the other test suites built with TSAN?
,
Mar 16 2017
Oh, I've missed them running on Linux Release NVIDIA. Maybe they don't run in Debug? The difference between the bots seems to be [15546:15551:0315/171909.815570:22488072931:ERROR:kill_posix.cc(84)] Unable to terminate process group 15742: No such process messages.
,
Mar 16 2017
Ah, interesting, good point. The same "Display::initialize error 0" error shows up on the runs on https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20Release%20(NVIDIA), too. It also looks like angle_perftests are a no-op on that configuration. Sounds fine to skip angle_perftests on the TSAN bot to get it green then. We can deal with the GLX error later.
,
Mar 16 2017
The webgl_conformance_gl_passthrough_tests seem to be failing due to a TSAN failure from a renderer process. Looks like this is the only version of the WebGL tests currently being run on this bot. Should we suppress the failure or remove the tests?
,
Mar 16 2017
TSAN looks unreliable. angle_perftests suddenly started passing in https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20GPU%20TSAN%20Release/builds/441 I'll disable webgl_conformance_gl_passthrough_tests on it.
,
Mar 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/5484f6789849305b5a86787dd9710c0f682e77e0 commit 5484f6789849305b5a86787dd9710c0f682e77e0 Author: ynovikov <ynovikov@chromium.org> Date: Thu Mar 16 18:21:28 2017 Skip all webgl_conformance suites on the GPU FYI TSAN bot. Previous CL missed a few. BUG= 691029 TBR=kbr@chromium.org CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Review-Url: https://codereview.chromium.org/2748363003 Cr-Commit-Position: refs/heads/master@{#457496} [modify] https://crrev.com/5484f6789849305b5a86787dd9710c0f682e77e0/content/test/gpu/generate_buildbot_json.py [modify] https://crrev.com/5484f6789849305b5a86787dd9710c0f682e77e0/testing/buildbot/chromium.gpu.fyi.json
,
Mar 16 2017
I took care of webgl_conformance_gl_passthrough_tests, but don't know what to do with angle_perftests and angle_end2end_tests. I couldn't find an option in TSAN to not report problems in functions which are not compiled by it, i.e. libGLX.
,
Mar 16 2017
Can't the TSAN suppression mechanism be used to match portions of the stack trace? I looked into this but didn't have time to try it locally.
,
Mar 16 2017
Indeed, looks like there is a way to suppress mutex errors: https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions So, how do we set TSAN_OPTIONS at runtime?
,
Mar 16 2017
Create a file like src/content/test/gpu/gpu_tests/tsan_suppressions.txt . Looks like V8 is one of the few projects with a file like this checked in, at https://cs.chromium.org/chromium/src/v8/tools/sanitizers/tsan_suppressions.txt . First get it working locally; try modifying src/content/test/gpu/run_gpu_integration_test.py to see if it can modify its own environment to set TSAN_OPTIONS. If that doesn't work, try modifying telemetry's browser_test_runner to take a new --tsan-options command line flag and set that in the environment for the sub-process. If that doesn't work, let's ask for help. Once it's working locally then we need to make sure the path name is scoped properly so it works when run via isolate.
,
Mar 16 2017
I don't think angle_perftests and angle_end2end_tests use run_gpu_integration_test.py or browser_test_runner. angle_perftests seems to use run_gtest_perf_test.py. No idea how angle_end2end_tests are run.
,
Mar 17 2017
Ah yes, right, sorry. angle_end2end_tests are a GTest-based test. The recipes invoke them via the LocalGTestTest and SwarmingGTestTest classes in https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/steps.py . Looking through there, I don't see any obvious functionality for setting arbitrary environment variables, or suppressions. So this might involve some recipe changes. I'd suggest you file a Build Infrastructure bug (type=Feature), block this on it, describe the need to set TSAN_OPTIONS for GTest type tests, and CC: phajdan.jr, martiniss, and sergiyb. Ask whether this functionality already exists -- or look through this code as well as src/base/test/launcher/ .
,
Mar 17 2017
,
Apr 6 2017
Maybe using tsan_suppressions.cc might work instead.
,
Apr 7 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b3d846d907794a018dca0793f45320064dc882f8 commit b3d846d907794a018dca0793f45320064dc882f8 Author: jbauman <jbauman@chromium.org> Date: Fri Apr 07 19:28:17 2017 Add TSAN suppression for deadlocks in libGLX.so This software isn't under our control, so suppress deadlock warnings we can't fix. BUG= 691029 Review-Url: https://codereview.chromium.org/2807523002 Cr-Commit-Position: refs/heads/master@{#462960} [modify] https://crrev.com/b3d846d907794a018dca0793f45320064dc882f8/build/sanitizers/tsan_suppressions.cc
,
Jun 26 2017
,
Jun 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a2c0592a18bd438cedc03892bdaaf54132f18898 commit a2c0592a18bd438cedc03892bdaaf54132f18898 Author: Kenneth Russell <kbr@chromium.org> Date: Mon Jun 26 18:14:57 2017 Try running context_lost tests on Linux TSAN GPU bot again. Suppressions were added a while back; let's see if they've helped. BUG= 691029 TBR=zmo@chromium.org NOTRY=true Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Change-Id: I9aa633145871c3e279b2967e8942831562336cea Reviewed-on: https://chromium-review.googlesource.com/548958 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#482324} [modify] https://crrev.com/a2c0592a18bd438cedc03892bdaaf54132f18898/content/test/gpu/generate_buildbot_json.py [modify] https://crrev.com/a2c0592a18bd438cedc03892bdaaf54132f18898/testing/buildbot/chromium.gpu.fyi.json
,
Jun 26 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/692e54ad8c94b21dbc6c7e262de15f140bf68bc5 commit 692e54ad8c94b21dbc6c7e262de15f140bf68bc5 Author: Kenneth Russell <kbr@chromium.org> Date: Mon Jun 26 22:32:36 2017 Revert "Try running context_lost tests on Linux TSAN GPU bot again." This reverts commit a2c0592a18bd438cedc03892bdaaf54132f18898. Reason for revert: browser still doesn't start under TSAN; see https://luci-milo.appspot.com/buildbot/chromium.gpu.fyi/Linux%20GPU%20TSAN%20Release/4419 . Original change's description: > Try running context_lost tests on Linux TSAN GPU bot again. > > Suppressions were added a while back; let's see if they've helped. > > BUG= 691029 > TBR=zmo@chromium.org > NOTRY=true > > Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel > Change-Id: I9aa633145871c3e279b2967e8942831562336cea > Reviewed-on: https://chromium-review.googlesource.com/548958 > Commit-Queue: Kenneth Russell <kbr@chromium.org> > Reviewed-by: Kenneth Russell <kbr@chromium.org> > Cr-Commit-Position: refs/heads/master@{#482324} TBR=zmo@chromium.org,kbr@chromium.org Change-Id: I655d27f702a005f5f111f3c351b06af3ce30cef0 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 691029 Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Reviewed-on: https://chromium-review.googlesource.com/549069 Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#482439} [modify] https://crrev.com/692e54ad8c94b21dbc6c7e262de15f140bf68bc5/content/test/gpu/generate_buildbot_json.py [modify] https://crrev.com/692e54ad8c94b21dbc6c7e262de15f140bf68bc5/testing/buildbot/chromium.gpu.fyi.json
,
Jun 26 2017
The browser still doesn't start successfully when built under TSAN. If someone from the TSAN team wants to pick this up and get the browser working in this configuration, I'll be happy to turn on all the tests.
,
Jun 26 2017
Unfortunately the sanitizer team doesn't have a lot of bandwidth to work on Chromium right now. It's basically just me at this point, and I'm focused on CFI. Copying a few people so that they are aware, though.
,
Jun 26 2017
,
Jun 27 2017
How does the problem manifest itself? I skimmed through the bug, but can't find this information.
,
Jun 27 2017
The browser doesn't start. See this failing build: https://luci-milo.appspot.com/buildbot/chromium.gpu.fyi/Linux%20GPU%20TSAN%20Release/4419 and the stdout from the failing step: https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.gpu.fyi%2FLinux_GPU_TSAN_Release%2F4419%2F%2B%2Frecipes%2Fsteps%2Fcontext_lost_tests_on_NVIDIA_GPU_on_Linux_on_Ubuntu%2F0%2Fstdout This would be reproducible by building the browser with TSAN turned on and running: ./content/test/gpu/run_gpu_integration_test.py context_lost --browser=release while cd'd into src/ in a Chromium checkout.
,
Jun 28 2017
It's unclear from the log what exactly goes wrong. There are these errors, but it continues after all of them: [21930:21965:0626/122040.334370:ERROR:browser_gpu_channel_host_factory.cc(103)] Failed to launch GPU process. [1:1:0626/122045.984365:ERROR:memory_mapped_file.cc(52)] Couldn't open /b/s/w/ir/out/Release/chrome_200_percent.pak [1:1:0626/122045.984501:ERROR:data_pack.cc(164)] Failed to mmap datapack [22025:22025:0626/122046.123554:ERROR:memory_mapped_file.cc(52)] Couldn't open /b/s/w/ir/out/Release/chrome_200_percent.pak [22025:22025:0626/122046.123668:ERROR:data_pack.cc(164)] Failed to mmap datapack [22025:22082:0626/122046.508152:ERROR:devtools_http_handler.cc(804)] [22025:22058:0626/122050.625456:ERROR:service_manager_context.cc(175)] Attempting to run unsupported native service: /b/s/w/ir/out/Release/content_renderer.service [22025:22058:0626/122050.626515:ERROR:service_manager_context.cc(175)] Attempting to run unsupported native service: /b/s/w/ir/out/Release/content_renderer.service And there this guy: (WARNING) 2017-06-26 12:21:00,993 gpu_integration_test.StartBrowser:86 Browser start failed (attempt 3 of 3) but it is confusing, because browser obviously started (assuming the previous output is from browser). Maybe it just needs to wait longer? Also, what happens when browser is started manually? used to work fine long time ago
,
Jul 24 2017
,
Sep 11
Forgot to actually close this. The bot is now located here: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20FYI%20GPU%20TSAN%20Release?limit=200 It's not running any browser-based tests yet unfortunately. |
|||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||
Comment 1 by kbr@chromium.org
, Feb 10 2017