I took one builder from each successive group, the exception log looks the same in all of them.

The bot tries to run WebContentsVideoCaptureDeviceBrowserTest.ErrorsOutIfWebContentsHasGoneBeforeDeviceStart. It fails with "use-of-uninitialized-value". Then the same stack trace is repeated a lot of times. The test time-outs. Then the test is retried. During one of such stack traces, the bot fails with TIMED_OUT.

Relevant bits:

======= the test with the first MemorySanitizer stack trace ======
[ RUN      ] WebContentsVideoCaptureDeviceBrowserTest.ErrorsOutIfWebContentsHasGoneBeforeDeviceStart
[28754:28771:0124/210004.488775:ERROR:network_service_impl.cc(43)] Not implemented reached in std::unique_ptr<net::NetworkChangeNotifier> content::(anonymous namespace)::CreateNetworkChangeNotifierIfNeeded()
DevTools listening on ws://127.0.0.1:34160/devtools/browser/5a51379f-fc81-4bdc-b24f-0c3160fd346f
[28754:28754:0124/210004.548265:ERROR:proxy_service.cc(1507)] ProxyConfigService for ChromeOS should be created in profile_io_data.cc::CreateProxyConfigService and this should be used only for examples.
ATTENTION: default value of option force_s3tc_enable overridden by environment.
Uninitialized bytes in __interceptor_strlen at offset 20 inside [0x71f000000e00, 3280)
==28775==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x7f644a2f94c8 in _init ??:0:0
    #1 0x7f644a2fe599 in glXMakeCurrent ??:?
    #2 0x7f644a2fe599 in ?? ??:0
    #3 0x7f644a2fa7f0 in glXQueryVersion ??:0:0
    #4 0xa53b943 in gl::GLSurfaceGLX::InitializeOneOff() ui/gl/gl_surface_glx.cc:425:8
    #5 0x3535d4d in ui::GLOzoneGLX::InitializeGLOneOffPlatform() ui/ozone/platform/x11/gl_ozone_glx.cc:27:8
    #6 0xdcfdd67 in gl::init::InitializeGLOneOffPlatform() ui/gl/init/gl_initializer_ozone.cc:18:26
    #7 0xdcf9cb5 in gl::init::InitializeGLOneOffImplementation(gl::GLImplementation, bool, bool, bool, bool) ui/gl/init/gl_factory.cc:88:43
    #8 0xdcf9356 in gl::init::(anonymous namespace)::InitializeGLOneOffHelper(bool) ui/gl/init/gl_factory.cc:65:10
    #9 0xdcf97ec in gl::init::InitializeGLNoExtensionsOneOff() ui/gl/init/gl_factory.cc:79:10
    #10 0x103152c4 in gpu::GpuInit::InitializeAndStartSandbox(base::CommandLine*, gpu::GpuPreferences const&, gpu::GPUInfo const*, gpu::GpuFeatureInfo const*) gpu/ipc/service/gpu_init.cc:222:22
    #11 0x17601854 in content::GpuMain(content::MainFunctionParams const&) content/gpu/gpu_main.cc:299:39
    #12 0x75af050 in content::RunNamedProcessTypeMain(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, content::MainFunctionParams const&, content::ContentMainDelegate*) content/app/content_main_runner.cc:421:14
    #13 0x75b1d95 in content::ContentMainRunnerImpl::Run() content/app/content_main_runner.cc:712:12
    #14 0x10b2bbcb in service_manager::Main(service_manager::MainParams const&) services/service_manager/embedder/main.cc:456:29
    #15 0x41f9334 in content::ContentMain(content::ContentMainParams const&) content/app/content_main.cc:19:10
    #16 0x9afd66e in content::LaunchTests(content::TestLauncherDelegate*, unsigned long, int, char**) content/public/test/test_launcher.cc:621:12
    #17 0x9a74b41 in main content/test/content_test_launcher.cc:138:10
    #18 0x7f644d7a8f44 in __libc_start_main /build/eglibc-ripdx6/eglibc-2.19/csu/libc-start.c:287:0
    #19 0x5cbaa9 in _start ??:0:0
  Uninitialized value was created by a heap allocation
    #0 0x5f169d in __interceptor_malloc /b/build/slave/linux_upload_clang/build/src/third_party/llvm/compiler-rt/lib/msan/msan_interceptors.cc:939:3
    #1 0x7f644a2feae3 in glXMakeCurrent ??:?
    #2 0x7f644a2feae3 in ?? ??:0
SUMMARY: MemorySanitizer: use-of-uninitialized-value (/usr/lib/x86_64-linux-gnu/mesa/libGL.so.1+0x194c8)
Exiting
ATTENTION: default value of option force_s3tc_enable overridden by environment.
Uninitialized bytes in __interceptor_strlen at offset 20 inside [0x71f000000e00, 3280)
==28807==WARNING: MemorySanitizer: use-of-uninitialized-value
==================================================================

=========== last stack trace with bot failure message ============
ATTENTION: default value of option force_s3tc_enable overridden by environment.
Uninitialized bytes in __interceptor_strlen at offset 20 inside [0x71f000000e00, 3280)
==2233==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x7f97983f94c8 in _init ??:0:0
    #1 0x7f97983fe599 in glXMakeCurrent ??:?
    #2 0x7f97983fe599 in ?? ??:0
    #3 0x7f97983fa7f0 in glXQueryVersion ??:0:0
    #4 0xa53b943 in gl::GLSurfaceGLX::InitializeOneOff() ui/gl/gl_surface_glx.cc:425:8
    #5 0x3535d4d in ui::GLOzoneGLX::InitializeGLOneOffPlatform() ui/ozone/platform/x11/gl_ozone_glx.cc:27:8
    #6 0xdcfdd67 in gl::init::InitializeGLOneOffPlatform() ui/gl/init/gl_initializer_ozone.cc:18:26
    #7 0xdcf9cb5 in gl::init::InitializeGLOneOffImplementation(gl::GLImplementation, bool, bool, bool, bool) ui/gl/init/gl_factory.cc:88:43
    #8 0xdcf9356 in gl::init::(anonymous namespace)::InitializeGLOneOffHelper(bool) ui/gl/init/gl_factory.cc:65:10
    #9 0xdcf97ec in gl::init::InitializeGLNoExtensionsOneOff() ui/gl/init/gl_factory.cc:79:10
    #10 0x103152c4 in gpu::GpuInit::InitializeAndStartSandbox(base::CommandLine*, gpu::GpuPreferences const&, gpu::GPUInfo const*, gpu::GpuFeatureInfo const*) gpu/ipc/service/gpu_init.cc:222:22
    #11 0x17601854 in content::GpuMain(content::MainFunctionParams const&) content/gpu/gpu_main.cc:299:39
    #12 0x75af050 in content::RunNamedProcessTypeMain(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, content::MainFunctionParams const&, content::ContentMainDelegate*) content/app/content_main_runner.cc:421:14
    #13 0x75b1d95 in content::ContentMainRunnerImpl::Run() content/app/content_main_runner.cc:712:12
    #14 0x10b2bbcb in service_manager::Main(service_manager::MainParams const&) services/service_manager/embedder/main.cc:456:29
    #15 0x41f9334 in content::ContentMain(content::ContentMainParams const&) content/app/content_main.cc:19:10
    #16 0x9afd66e in content::LaunchTests(content::TestLauncherDelegate*, unsigned long, int, char**) content/public/test/test_launcher.cc:621:12
    #17 0x9a74b41 in main content/test/content_test_launcher.cc:138:10
    #18 0x7f979b8f6f44 in __libc_start_main /build/eglibc-ripdx6/eglibc-2.19/csu/libc-start.c:287:0
    #19 0x5cbaa9 in _start ??:0:0
[0124/212109.923018:FATAL:test_launcher.cc(492)] Check failed: ReadFileToString(output_filename, &output_file_contents). 
#0 0x000000607dd1 (/b/swarming/w/ir/out/Release/content_browsertests (deleted)+0x607dd0)
#1 0x00000acd7b7f (/b/swarming/w/ir/out/Release/content_browsertests (deleted)+0xacd7b7e)
#2 0x00000ad28d2e (/b/swarming/w/ir/out/Release/content_browsertests (deleted)+0xad28d2d)
#3 0x000009b902b6 (/b/swarming/w/ir/out/Release/content_browsertests (deleted)+0x9b902b5)
#4 0x000009ba2801 (/b/swarming/w/ir/out/Release/content_browsertests (deleted)+0x9ba2800)
#5 0x00000ae9a3ca (/b/swarming/w/ir/out/Release/content_browsertests (deleted)+0xae9a3c9)
#6 0x00000ae969e5 (/b/swarming/w/ir/out/Release/content_browsertests (deleted)+0xae969e4)
#7 0x00000aea6b81 (/b/swarming/w/ir/out/Release/content_browsertests (deleted)+0xaea6b80)
#8 0x00000ae8f4e6 (/b/swarming/w/ir/out/Release/content_browsertests (deleted)+0xae8f4e5)
#9 0x7f025da54184 <unknown>
#10 0x7f025883403d <unknown>
+---------------------------------------------------------------------------+
| End of shard 0                                                            |
|  Pending: 0.4s  Duration: 3600.0s  Bot: swarm986-c4  Exit: -15  TIMED_OUT |
+---------------------------------------------------------------------------+
Total duration: 3600.0s
WARNING:root:collect_cmd had non-zero return code: 241
Test runtime exceeded allocated time
some shards did not complete: 0
step returned non-zero exit code: 241
==================================================================

The file with that test has been touched recently in https://chromium-review.googlesource.com/889934, which is a reland of https://chromium-review.googlesource.com/884996, which was reverted in https://chromium-review.googlesource.com/890380.

So, the group 5415 - 5416 does not seem to be caused by the CLs from #2.
However, the initial CL (currently reverted) is present in 5444 [1] (i.e. beginning of the second group 5444 - 5453).

https://uberchromegw.corp.google.com/i/chromium.memory/builders/Linux%20ChromiumOS%20MSan%20Tests/builds/5444

The second group ended because of the mentioned revert (the revert is in 5454 and it fails for other reasons, but without an exception).

https://uberchromegw.corp.google.com/i/chromium.memory/builders/Linux%20ChromiumOS%20MSan%20Tests/builds/5454

And the last group started when the reland landed (5482).

https://uberchromegw.corp.google.com/i/chromium.memory/builders/Linux%20ChromiumOS%20MSan%20Tests/builds/5482

This seems like a strong evidence, I am reverting the reland https://chromium-review.googlesource.com/#/c/889934/.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad

commit 13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad
Author: vitaliii <vitaliii@chromium.org>
Date: Mon Jan 29 11:37:48 2018

Revert "Re-land "FrameSinkVideoCapturer: Fix refresh logic for damaged sources.""

This reverts commit d26dcac0a6390fb039dee8eaec0906411164c7a9.

Reason for revert: Seems to cause an exception (time out) on Linux ChromiumOS MSan Tests (see  crbug.com/806715 ). The original CL (the one relanded) caused this exception too.

Original change's description:
> Re-land "FrameSinkVideoCapturer: Fix refresh logic for damaged sources."
> 
> This reverts commit 52c31b740b1d124a4a9fa521e65ee0c0dc20b724, to re-land
> the original CL (https://chromium-review.googlesource.com/884996). The
> viz_content_browsertests were failing because of a bug in the posix
> shmem impl (in base/memory/shared_memory_helper.cc) when the pool is
> being run in the VIZ process. For now, the tests have been added to the
> viz_content_browsertests filter file; and once the root-cause issue has
> been resolved, they can be re-enabled.
> 
> TBR=xjz@chromium.org,kylechar@chromium.org
> 
> Bug:  806635 ,  785072 ,  754872 
> Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel
> Change-Id: I0f419ce2aee170a0fbd966ff8e10bb99c3b7067f
> Reviewed-on: https://chromium-review.googlesource.com/889934
> Commit-Queue: Yuri Wiitala <miu@chromium.org>
> Reviewed-by: Yuri Wiitala <miu@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#532322}

TBR=miu@chromium.org,kylechar@chromium.org,xjz@chromium.org

Change-Id: Ia3715ed63494598badbe85e2f418b792c952dadb
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  806635 ,  785072 ,  754872 ,  806715 
Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel
Reviewed-on: https://chromium-review.googlesource.com/890445
Reviewed-by: vitaliii <vitaliii@chromium.org>
Commit-Queue: vitaliii <vitaliii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#532395}
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/components/viz/service/frame_sinks/compositor_frame_sink_support.cc
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/components/viz/service/frame_sinks/compositor_frame_sink_support.h
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/components/viz/service/frame_sinks/video_capture/capturable_frame_sink.h
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/components/viz/service/frame_sinks/video_capture/frame_sink_video_capturer_impl.cc
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/components/viz/service/frame_sinks/video_capture/frame_sink_video_capturer_impl.h
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/components/viz/service/frame_sinks/video_capture/frame_sink_video_capturer_impl_unittest.cc
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/content/browser/media/capture/aura_window_capture_machine.cc
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/content/browser/media/capture/web_contents_video_capture_device_browsertest.cc
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/media/capture/content/android/screen_capture_machine_android.cc
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/media/capture/content/screen_capture_device_core.cc
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/media/capture/content/screen_capture_device_core.h
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/media/capture/content/thread_safe_capture_oracle.cc
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/media/capture/content/thread_safe_capture_oracle.h
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/media/capture/content/video_capture_oracle.cc
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/media/capture/content/video_capture_oracle.h
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/media/capture/content/video_capture_oracle_unittest.cc
[modify] https://crrev.com/13c588bc9a1e2855f1f6cba4664a1f3502c4b0ad/testing/buildbot/filters/viz.content_browsertests.filter

And only now the revert has reached a build:

https://uberchromegw.corp.google.com/i/chromium.memory/builders/Linux%20ChromiumOS%20MSan%20Tests/builds/5488

+kbr, piman

Seems that the tests failed on MSAN because of something internal to the OpenGL library on CrOS. I'm not sure who to contact for upstream resolution of this.

For now, I'll just disable the tests that triggered the MSAN failure so they don't run on CrOS+MSAN.

+marcheu re: possible CrOS OpenGL issue per #8

This is a hybrid ChromeOS-Chrome on linux build. Chrome OS doesn't use GLX, so the Chrome OS drivers aren't used here.

I guess that's an issue inside of the driver that's running on these bots?

Yes looks like it...

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/da1deebdab4e3538b0c78432a00413ac075ce556

commit da1deebdab4e3538b0c78432a00413ac075ce556
Author: Yuri Wiitala <miu@chromium.org>
Date: Tue Jan 30 00:25:45 2018

Re-enable WebContentsVideoCaptureDeviceBrowserTests, sans CrOS+MSAN.

TBR=xjz@chromium.org

Bug:  754872 ,  806715 
Change-Id: Iea4cdc3fd17315ad688b48312a0185c114cc8908
Reviewed-on: https://chromium-review.googlesource.com/892053
Reviewed-by: Yuri Wiitala <miu@chromium.org>
Commit-Queue: Yuri Wiitala <miu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#532691}
[modify] https://crrev.com/da1deebdab4e3538b0c78432a00413ac075ce556/content/browser/media/capture/web_contents_video_capture_device_browsertest.cc

In the sites page (https://www.chromium.org/developers/testing/memorysanitizer), there is explicit discussion about not enabling HW GL, but it does say there that OSMesa is an option. And, from the log output, this MSAN error is reporting on OSMesa: /usr/lib/x86_64-linux-gnu/mesa/libGL.so

I downloaded the latest source for mesa and took a look at the code. There are, in fact, a few places where memory might be getting used uninitialized in the relevant functions. I'll see about sending them a bug report for those...

So, at this point, how do we proceed? Should we blacklist this stack?

I'd suggest filing an upstream bug against Mesa and suppressing the MSAN failure. This seems unlikely to cause failures or other issues in Chrome. (If you can confirm a fix and upload a patch to Mesa that'd be ideal since they're unlikely to reproduce your report.)

Also, Chrome OS doesn't use GLX any more, so this is an issue specific to desktop Linux, even though it's the Chromium OS flavored build.

This path (/usr/lib/x86_64-linux-gnu/mesa/libGL.so) points to the library not being part of prebuilt_instrumented_libraries.

This is probably a false positive.

Are you passing --use-gl=osmesa? osmesa doesn't refer to /usr/lib/x86_64-linux-gnu/mesa/libGL.so, but rather the libosmesa.so included with the chromium build.

Oh! I didn't realize we had an instrumented build available. I'll update the test code to use the instrumented mesa for MSAN bulids.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c8c7d153f4c1010f6795ec5e84679857f40ea4ce

commit c8c7d153f4c1010f6795ec5e84679857f40ea4ce
Author: Yuri Wiitala <miu@chromium.org>
Date: Tue Feb 06 19:23:19 2018

Use software GL for WCVideoCaptureDeviceBrowserTests on MSAN.

Fixes the test SetUp() to use the instrumented software GL impl when
the WebContentsVideoCaptureDevice browser tests are MSAN-built. This
prevents false testing failures caused by memory bugs in the platform-
specific libraries.

Bug:  806715 
Change-Id: Idec6245674b36d0c8b9cca0488e0601e02d932c8
Reviewed-on: https://chromium-review.googlesource.com/903584
Reviewed-by: Xiangjun Zhang <xjz@chromium.org>
Commit-Queue: Yuri Wiitala <miu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#534745}
[modify] https://crrev.com/c8c7d153f4c1010f6795ec5e84679857f40ea4ce/content/browser/media/capture/web_contents_video_capture_device_browsertest.cc

The bots have been green on content_browsertests since the change landed. Looks like the issue is completely resolved.