This is the first build I can find where the flake started: https://uberchromegw.corp.google.com/i/chromium.chromiumos/builders/Linux%20ChromiumOS%20Tests%20%281%29/builds/29495 which corresponds to 432287-432297. The crash is in ui::ws::GpuCompositorFrameSink, which last changed two days ago, so I doubt that is it. Most likely something higher up in cc changed the ordering, or perhaps we're not unregistering correctly.

https://codereview.chromium.org/2502663003 is the blame list and impacts cc. I don't see anything else, but +fsamuel/+sadrul know this code better and can hopefully help isolate what may have caused the flake.

asan gives some useful trace:

=================================================================
==20069==ERROR: AddressSanitizer: heap-use-after-free on address 0x61a000000c88 at pc 0x7f31d88dd8e5 bp 0x7ffc37a83670 sp 0x7ffc37a83668
READ of size 8 at 0x61a000000c88 thread T0
    #0 0x7f31d88dd8e4 in Send ./out/cros/../../gpu/ipc/client/gpu_channel_host.cc:112:17
    #1 0x7f31d88de431 in InternalFlush ./out/cros/../../gpu/ipc/client/gpu_channel_host.cc:181:3
    #2 0x7f31d88ddbe3 in OrderingBarrier ./out/cros/../../gpu/ipc/client/gpu_channel_host.cc:158:7
    #3 0x7f31d88cb16d in Flush ./out/cros/../../gpu/ipc/client/command_buffer_proxy_impl.cc:265:41
    #4 0x7f31d8556a28 in Flush ./out/cros/../../gpu/command_buffer/client/cmd_buffer_helper.cc:180:22
    #5 0x7f31cfe8adae in FlushHelper ./out/cros/../../gpu/command_buffer/client/gles2_implementation.cc:1389:33
    #6 0x7f31cfe820af in SetAggressivelyFreeResources ./out/cros/../../gpu/command_buffer/client/gles2_implementation.cc:434:5
    #7 0x7f31d7aaf4c5 in ClientBecameNotVisible ./out/cros/../../cc/output/context_cache_controller.cc:85:23
    #8 0x7f31d7ad4da8 in ~GLRenderer ./out/cros/../../cc/output/gl_renderer.cc:422:23
    #9 0x7f31d7ad763d in ?? ./out/cros/../../cc/output/gl_renderer.cc:409:27
    #10 0x7f31d7430bed in operator() ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:63:2
    #11 0x7f31d7430bed in reset ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:245:0
    #12 0x7f31d7430bed in ~unique_ptr ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:169:0
    #13 0x7f31d7430bed in ~Display ./out/cros/../../cc/surfaces/display.cc:72:0
    #14 0x7f31d7430fdd in ?? ./out/cros/../../cc/surfaces/display.cc:56:21
    #15 0xbff4d30 in operator() ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:63:2
    #16 0xbff4d30 in reset ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:245:0
    #17 0xbff4d30 in ~unique_ptr ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:169:0
    #18 0xbff4d30 in ~GpuCompositorFrameSink ./out/cros/../../services/ui/ws/gpu_compositor_frame_sink.cc:59:0
    #19 0xbff5004 in ~GpuCompositorFrameSink ./out/cros/../../services/ui/ws/gpu_compositor_frame_sink.cc:51:51
    #20 0xbff5004 in ?? ./out/cros/../../services/ui/ws/gpu_compositor_frame_sink.cc:51:0
    #21 0xbfdec47 in Close ./out/cros/../../mojo/public/cpp/bindings/strong_binding.h:75:18
    #22 0xbfdec47 in OnConnectionError ./out/cros/../../mojo/public/cpp/bindings/strong_binding.h:106:0
    #23 0x7f31df46800f in Run ./out/cros/../../base/callback.h:64:12
    #24 0x7f31df46800f in NotifyError ./out/cros/../../mojo/public/cpp/bindings/lib/interface_endpoint_client.cc:298:0
    #25 0x7f31df47e13b in ProcessNotifyErrorTask ./out/cros/../../mojo/public/cpp/bindings/lib/multiplex_router.cc:741:13
    #26 0x7f31df47839b in ProcessTasks ./out/cros/../../mojo/public/cpp/bindings/lib/multiplex_router.cc:658:15
    #27 0x7f31df474308 in OnPipeConnectionError ./out/cros/../../mojo/public/cpp/bindings/lib/multiplex_router.cc:628:3
    #28 0x7f31df454ed9 in Run ./out/cros/../../base/callback.h:64:12
    #29 0x7f31df454ed9 in HandleError ./out/cros/../../mojo/public/cpp/bindings/lib/connector.cc:321:0
    #30 0x7f31df4564ad in OnHandleReadyInternal ./out/cros/../../mojo/public/cpp/bindings/lib/connector.cc:202:5
    #31 0x7f31e6cef7a6 in Run ./out/cros/../../base/callback.h:64:12
    #32 0x7f31e6cef7a6 in OnHandleReady ./out/cros/../../mojo/public/cpp/system/watcher.cc:122:0
    #33 0x7f31e6cefa07 in WillDestroyCurrentMessageLoop ./out/cros/../../mojo/public/cpp/system/watcher.cc:32:17
    #34 0x7f31e691db67 in ~MessageLoop ./out/cros/../../base/message_loop/message_loop.cc:128:14
    #35 0x5caa6ee in StartChildApp ./out/cros/../../chrome/test/base/mash_browser_tests_main.cc:137:1
    #36 0x5cabe24 in Invoke<mojo::InterfaceRequest<service_manager::mojom::Service> > ./out/cros/../../base/bind_internal.h:164:12
    #37 0x5cabe24 in MakeItSo<void (*const &)(mojo::InterfaceRequest<service_manager::mojom::Service>), mojo::InterfaceRequest<service_manager::mojom::Service> > ./out/cros/../../base/bind_internal.h:285:0
    #38 0x5cabe24 in RunImpl<void (*const &)(mojo::InterfaceRequest<service_manager::mojom::Service>), const std::tuple<> &> ./out/cros/../../base/bind_internal.h:361:0
    #39 0x5cabe24 in Run ./out/cros/../../base/bind_internal.h:339:0
    #40 0x5ca7576 in Run ./out/cros/../../base/callback.h:64:12
    #41 0x5ca7576 in ChildProcessMainWithCallback ./out/cros/../../services/service_manager/runner/host/child_process_base.cc:125:0
    #42 0x5ca9ef3 in RunMashBrowserTests ./out/cros/../../chrome/test/base/mash_browser_tests_main.cc:153:5
    #43 0x5ca9bfb in main ./out/cros/../../chrome/test/base/browser_tests_main_chromeos.cc:14:7
    #44 0x7f31bdf68f44 in __libc_start_main /build/eglibc-oGUzwX/eglibc-2.19/csu/libc-start.c:287:0

0x61a000000c88 is located 8 bytes inside of 1200-byte region [0x61a000000c80,0x61a000001130)
freed by thread T0 here:
    #0 0xac729b in operator delete(void*) ??:?
    #1 0xbf6cb9e in operator() ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:63:2
    #2 0xbf6cb9e in reset ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:245:0
    #3 0xbf6cb9e in ~unique_ptr ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:169:0
    #4 0xbf6cb9e in ~WindowServer ./out/cros/../../services/ui/ws/window_server.cc:80:0
    #5 0xbf6d55d in ?? ./out/cros/../../services/ui/ws/window_server.cc:65:31
    #6 0x5b21ccd in operator() ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:63:2
    #7 0x5b21ccd in reset ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:245:0
    #8 0x5b21ccd in ~Service ./out/cros/../../services/ui/service.cc:96:0
    #9 0x5b221bd in ?? ./out/cros/../../services/ui/service.cc:92:21
    #10 0x8b243ee in operator() ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:63:2
    #11 0x8b243ee in reset ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:245:0
    #12 0x8b243ee in ~unique_ptr ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:169:0
    #13 0x8b243ee in ~ServiceContext ./out/cros/../../services/service_manager/public/cpp/lib/service_context.cc:45:0
    #14 0x8b2453d in ?? ./out/cros/../../services/service_manager/public/cpp/lib/service_context.cc:45:35
    #15 0x5a5c7f5 in operator() ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:63:2
    #16 0x5a5c7f5 in reset ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:245:0
    #17 0x5a5c7f5 in ~unique_ptr ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:169:0
    #18 0x5a5c7f5 in ~MashPackagedService ./out/cros/../../mash/package/mash_packaged_service.cc:30:0
    #19 0x5a5c86d in ?? ./out/cros/../../mash/package/mash_packaged_service.cc:30:45
    #20 0x8b243ee in operator() ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:63:2
    #21 0x8b243ee in reset ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:245:0
    #22 0x8b243ee in ~unique_ptr ./out/cros/../../build/linux/ubuntu_precise_amd64-sysroot/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/bits/unique_ptr.h:169:0
    #23 0x8b243ee in ~ServiceContext ./out/cros/../../services/service_manager/public/cpp/lib/service_context.cc:45:0
    #24 0x5caa6dd in StartChildApp ./out/cros/../../chrome/test/base/mash_browser_tests_main.cc:137:1


Looks like WindowServer has been destroyed, but GpuCompositorFrameSink is still alive (because its lifetime is not tied to the WindowServer), and indirectly through some layers, trying to access GpuServiceProxy (which has been already destroyed with the WindowServer).

 Issue 666632  has been merged into this issue.

This flake is plaguing the waterfall. Please investigate ASAP and revert/fix.

Issue 666983 has been merged into this issue.

I have a fix in code review: https://codereview.chromium.org/2481263002/

@fsamuel - As part of this, can you implement a sane timeout for the mash_browser_tests step in the recipe? 

This will help with two things:

1-Less clogging of the CQ w/ builds stuck on a hung step.
2-Proper reporting of failure.

I have a short term fix in the CQ: https://codereview.chromium.org/2521763002/ 

#8: How do I set the timeout for mash_browser_tests?

You can pass a timeout as an argument to api.step(), e.g.

api.step('timeout', ['sleep', '20'], timeout=1)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1426008b49521475a9b937fd65901abc22e3a349

commit 1426008b49521475a9b937fd65901abc22e3a349
Author: fsamuel <fsamuel@chromium.org>
Date: Mon Nov 21 22:50:20 2016

Give ServerWindowCompositorFrameSinkManager ownership of GpuCompositorFrameSink

GpuCompositorFrameSink was outliving WindowServer and GpuServiceProxy but
was reaching into those objects at tear down. This was causing segfaults.

This CL gives ServerWindowCompositorFrameSinkManager ownership of
GpuCompositorFrameSink thereby eliminating the lifetime issue.

BUG= 666481 

Review-Url: https://codereview.chromium.org/2521763002
Cr-Commit-Position: refs/heads/master@{#433679}

[modify] https://crrev.com/1426008b49521475a9b937fd65901abc22e3a349/services/ui/ws/gpu_compositor_frame_sink.cc
[modify] https://crrev.com/1426008b49521475a9b937fd65901abc22e3a349/services/ui/ws/gpu_compositor_frame_sink.h
[modify] https://crrev.com/1426008b49521475a9b937fd65901abc22e3a349/services/ui/ws/server_window_compositor_frame_sink_manager.cc
[modify] https://crrev.com/1426008b49521475a9b937fd65901abc22e3a349/services/ui/ws/server_window_compositor_frame_sink_manager.h

What does api.step('timeout', ['sleep', '20'], timeout=1) mean?
We currently have a single test in mash_browsertests, but that's going to change. At some point we'll be running all browser_tests through it, at which point 20 seconds won't be realistic for all.

Sorry, that's not a great example. I meant to illustrate that the step api takes a keyword argument "timeout" that can be used to configure the timeout on that step. If you're not familiar with the recipe, though, this is probably not very helpful. 

Who is familiar with this recipe?

I think these steps were taking upwards of an hour, so we can set the timeout at something between 20 seconds and an hour. The goal is to find a point where we won't be killing builds that shouldn't be killed, but we also won't be taking up CQ time on a step that should realistically never take more than X minutes.

Dirk likely knows about the recipe used for mash_browser_tests. I suspect it's the same as browser_tests.

I believe that if you add `"hard_timeout": 60` to 

https://cs.chromium.org/chromium/src/testing/buildbot/chromium.chromiumos.json?rcl=0&l=74

you'll get a timeout of 60 seconds on the swarming job.

There are two entries for mash_browser_tests in that file, so update both of them.

(katthomas@ - the steps are configured indirectly via the generate_* methods in 

https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/steps.py

from the entries in the //testing/buildbot/*.json files in the chromium checkout.

60 seems a bit low. What is the default, and what do we typically use for browser_tests?

I think we don't normally specify timeouts for steps, and so I think browser_tests normally gets the default, which is an hour (?).

However, from looking at the filter file, mash_browser_tests is running all of one test  and usually takes about 10 seconds, which suggests that 60 is more than enough :). 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/38d113d305a42f7087ebefec448860f163ecdb16

commit 38d113d305a42f7087ebefec448860f163ecdb16
Author: sky <sky@chromium.org>
Date: Tue Nov 22 03:56:27 2016

Makes mash_browser_tests timeout after 1 minute.

BUG= 666481 
TEST=none
R=dpranke@chromium.org

Review-Url: https://codereview.chromium.org/2520183002
Cr-Commit-Position: refs/heads/master@{#433783}

[modify] https://crrev.com/38d113d305a42f7087ebefec448860f163ecdb16/testing/buildbot/chromium.chromiumos.json

Looks fixed. Thanks!

 Issue 663453  has been merged into this issue.

Hmmm... Sheriff-o-Matic says that mash_browser_tests are consistently failing on chromium.chromiumos/Linux ChromiumOS Tests (1).  Looking at https://uberchromegw.corp.google.com/i/chromium.chromiumos/builders/Linux%20ChromiumOS%20Tests%20%281%29/builds/29845/steps/steps/logs/stdio I see the following output at the bottom:

[1/1] BrowserTest.Title (9973 ms)
SUCCESS: all tests passed.
[1123/100234:ERROR:native_widget_mus.cc(872)] Not implemented reached in virtual void views::NativeWidgetMus::ViewRemoved(views::View *)
[1123/100234:ERROR:wm_shell_mus.cc(405)] Not implemented reached in virtual void ash::mus::WmShellMus::RemoveDisplayObserver(ash::WmDisplayObserver *)
[1123/100234:ERROR:surface_manager.cc(124)] No reference from SurfaceId(FrameSinkId(0, 0), LocalFrameId(0, (B6A4A0EAC7C01DB45EDF32E1A78A95B)lu)) to SurfaceId(FrameSinkId(2, 1), LocalFrameId(1, (C33D84FD43523075F7F304B208F461D)lu))

command timed out: 2400 seconds without output, attempting to kill
process killed by signal 9


This seems to be the same issue, right?

Reactivating per #c22.  If I misunderstood something and this is not really the same issue, could you please open a separate bug to track the purpleness of https://uberchromegw.corp.google.com/i/chromium.chromiumos/builders/Linux%20ChromiumOS%20Tests%20%281%29?numbuilds=200 ?

This should be fixed now.

https://codereview.chromium.org/2453313004/ has been repeatedly failing the commit queue for some weeks now on mash_browser_tests. None of the chromium-mojo people raised any concerns about the CL itself, and mash_browser_tests seems to pass on my local machine (https://groups.google.com/a/chromium.org/d/msg/chromium-mojo/fOPYqYMfJGE/TSJehQ3aAwAJ). Any hints on how to further debug this?

A recent buildbot run is at https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_chromeos_ozone_rel_ng/builds/279976 but I don't know how to read these tea leaves very well.

I'm kind of at a loss. I thought this was fixed now. Anyone have any thoughts?

I looked at the failure logs, and it looks like the crash is not the reason for the failure. For example, the test does actually fail:

Value of: browser()->GetWindowTitleForCurrentTab( true )
  Actual: Chromium - about:blank
Expected: LocaleWindowCaptionFromPageTitle(test_title)
Which is: Chromium - Title Of Awesomeness
../../chrome/browser/ui/browser_browsertest.cc:464: Failure
Value of: tab_title
  Actual: title2.html
Expected: test_title
Which is: Title Of Awesomeness

Looking at the logs a bit more, the following line looks most relevant to the CL:

[14284:14339:1125/164952:ERROR:render_process_host_impl.cc(2999)] Terminating render process for bad Mojo message: Received bad user message: InterfaceProviderSpec "service_manager:connector" prevented service: content_renderer from binding interface: chrome::mojom::FieldTrialRecorder exposed by: content_browser


Perhaps the manifest addition is not being picked up by mash_browser_tests?

Removing sheriff-chromium, as this seems to be being worked on already

I think these might new occurrences of this bug?

https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/342119
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/342042

Except in this case we're seeing the crash in webkit_tests.

Here is another place where a step-defined timeout would be handy, continuing the discussion from #18. 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f244dba45916187f9e025c43bf4d3c49983282d3

commit f244dba45916187f9e025c43bf4d3c49983282d3
Author: kylechar <kylechar@chromium.org>
Date: Tue Dec 06 21:11:29 2016

Check client_ isn't null in GpuCompositorframeSink.

All other methods that call something on client_ check it's not null
first. There is a crash during shutdown for mash_browser_tests that is
caused by this.

BUG= 666481 

Review-Url: https://codereview.chromium.org/2549393003
Cr-Commit-Position: refs/heads/master@{#436722}

[modify] https://crrev.com/f244dba45916187f9e025c43bf4d3c49983282d3/services/ui/surfaces/gpu_compositor_frame_sink.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/15ee57778085a2abef4a6548cd79c28daa16e3c3

commit 15ee57778085a2abef4a6548cd79c28daa16e3c3
Author: fsamuel <fsamuel@chromium.org>
Date: Tue Dec 06 21:27:44 2016

cc: Unregister FrameSink hierarchy on CompositorFrameSinkSupport destruction

A CompositorFrameSinkSupport may be destroyed during test tear down without
Unregistering all BeginFrame children. This causes a DCHECK to fail in
SurfaceManager. With this change, CompositorFrameSinkSupport keeps a set
of all child FrameSinkIds, and Unregisters the FrameSinks at destruction if
any remain.

BUG= 666481 
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_trusty_blink_rel

Review-Url: https://codereview.chromium.org/2557853002
Cr-Commit-Position: refs/heads/master@{#436735}

[modify] https://crrev.com/15ee57778085a2abef4a6548cd79c28daa16e3c3/cc/surfaces/compositor_frame_sink_support.cc
[modify] https://crrev.com/15ee57778085a2abef4a6548cd79c28daa16e3c3/cc/surfaces/compositor_frame_sink_support.h

This should be fixed for realz now. Marking as FIXED again.