Capture the failure log:

[ RUN      ] NavigationControllerBrowserTest.UtilizationOfSpareRenderProcessHost
[15626:15726:0921/142548.552011:ERROR:network_service.cc(88)] Not implemented reached in std::unique_ptr<net::NetworkChangeNotifier> network::(anonymous namespace)::CreateNetworkChangeNotifierIfNeeded()
[15626:15626:0921/142548.578960:ERROR:gpu_interface_provider.cc(87)] Not implemented reached in virtual void content::GpuInterfaceProvider::RegisterOzoneGpuInterfaces(service_manager::BinderRegistry *)
DevTools listening on ws://127.0.0.1:46547/devtools/browser/c1cd0acc-3032-48c1-8846-8b6528c815e7
[15626:15626:0921/142548.579790:ERROR:proxy_resolution_service.cc(1555)] ProxyConfigService for ChromeOS should be created in profile_io_data.cc::CreateProxyConfigService and this should be used only for examples.
[15626:15626:0921/142548.587620:ERROR:render_widget_host_view_aura.cc(1278)] Not implemented reached in virtual base::i18n::TextDirection content::RenderWidgetHostViewAura::GetTextDirection() const
[1:8:0921/142548.634413:ERROR:layer_tree_host_impl.cc(3098)] Forcing zero-copy tile initialization as worker context is missing
[15626:15626:0921/142548.637429:ERROR:layer_tree_host_impl.cc(3098)] Forcing zero-copy tile initialization as worker context is missing
[1:8:0921/142548.647190:ERROR:layer_tree_host_impl.cc(3098)] Forcing zero-copy tile initialization as worker context is missing
[15626:15626:0921/142548.669664:ERROR:render_widget_host_view_aura.cc(1278)] Not implemented reached in virtual base::i18n::TextDirection content::RenderWidgetHostViewAura::GetTextDirection() const
[15626:15626:0921/142548.688227:ERROR:render_widget_host_view_aura.cc(1278)] Not implemented reached in virtual base::i18n::TextDirection content::RenderWidgetHostViewAura::GetTextDirection() const
[15749:15749:0921/142548.698800:WARNING:ipc_message_attachment_set.cc(49)] MessageAttachmentSet destroyed with unconsumed attachments: 0/1
../../content/browser/frame_host/navigation_controller_impl_browsertest.cc:8019: Failure
Expected equality of these values:
  prev_spare
    Which is: 0x7963c8f3a00
  curr_host
    Which is: 0x7963c936000
Stack trace:
#0 0x0000013ec18c testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop()
#1 0x0000013ebb59 testing::internal::AssertHelper::operator=()
#2 0x0000007407e6 content::NavigationControllerBrowserTest_UtilizationOfSpareRenderProcessHost_Test::RunTestOnMainThread()
#3 0x00000255dc5b content::BrowserTestBase::ProxyRunTestOnMainThreadLoop()
#4 0x0000025c9048 content::ShellBrowserMainParts::PreMainMessageLoopRun()
#5 0x000001fa37e1 content::BrowserMainLoop::PreMainMessageLoopRun()
#6 0x00000231bd47 content::StartupTaskRunner::RunAllTasksNow()
#7 0x000001fa215d content::BrowserMainLoop::CreateStartupTasks()
#8 0x000001fa5c83 content::BrowserMainRunnerImpl::Initialize()
#9 0x0000025b08d9 ShellBrowserMain()
#10 0x0000025a6f92 content::ShellMainDelegate::RunProcess()
#11 0x000001ed4bbd content::ContentMainRunnerImpl::Run()
#12 0x000003bf3840 service_manager::Main()
#13 0x0000016347e4 content::ContentMain()
#14 0x00000255d891 content::BrowserTestBase::SetUp()
../../content/browser/frame_host/navigation_controller_impl_browsertest.cc:8023: Failure
Expected: (prev_spare) != (curr_spare), actual: 0x7963c8f3a00 vs 0x7963c8f3a00
Stack trace:
#0 0x0000013ec18c testing::internal::UnitTestImpl::CurrentOsStackTraceExceptTop()
#1 0x0000013ebb59 testing::internal::AssertHelper::operator=()
#2 0x000000740a05 content::NavigationControllerBrowserTest_UtilizationOfSpareRenderProcessHost_Test::RunTestOnMainThread()
#3 0x00000255dc5b content::BrowserTestBase::ProxyRunTestOnMainThreadLoop()
#4 0x0000025c9048 content::ShellBrowserMainParts::PreMainMessageLoopRun()
#5 0x000001fa37e1 content::BrowserMainLoop::PreMainMessageLoopRun()
#6 0x00000231bd47 content::StartupTaskRunner::RunAllTasksNow()
#7 0x000001fa215d content::BrowserMainLoop::CreateStartupTasks()
#8 0x000001fa5c83 content::BrowserMainRunnerImpl::Initialize()
#9 0x0000025b08d9 ShellBrowserMain()
#10 0x0000025a6f92 content::ShellMainDelegate::RunProcess()
#11 0x000001ed4bbd content::ContentMainRunnerImpl::Run()
#12 0x000003bf3840 service_manager::Main()
#13 0x0000016347e4 content::ContentMain()
#14 0x00000255d891 content::BrowserTestBase::SetUp()
[15749:15749:0921/142548.829129:ERROR:gles2_cmd_decoder.cc(17815)] [.BrowserCompositor-0x7963ca91500]GL ERROR :GL_INVALID_OPERATION : glCreateAndConsumeTextureCHROMIUM: invalid mailbox name
[15749:15749:0921/142548.829196:ERROR:gles2_cmd_decoder.cc(10222)] [.BrowserCompositor-0x7963ca91500]RENDER WARNING: texture bound to texture unit 0 is not renderable. It maybe non-power-of-2 and have incompatible texture filtering.
[  FAILED  ] NavigationControllerBrowserTest.UtilizationOfSpareRenderProcessHost, where TypeParam =  and GetParam() =  (410 ms)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f78979a7db3f89fee97266eb8c3dfa0654fdfcf2

commit f78979a7db3f89fee97266eb8c3dfa0654fdfcf2
Author: Evan Stade <estade@chromium.org>
Date: Wed Sep 26 17:57:34 2018

Disable a test on single process mash content browser tests

Disabling due to flake:
  NavigationControllerBrowserTest.UtilizationOfSpareRenderProcessHost

TBR=xiyuan@chromium.org

Bug:  888156 
Change-Id: Ice1001691d141c2ffb2cec2399dffe8855acdf3a
Reviewed-on: https://chromium-review.googlesource.com/1246145
Reviewed-by: Evan Stade <estade@chromium.org>
Reviewed-by: Xiyuan Xia <xiyuan@chromium.org>
Commit-Queue: Evan Stade <estade@chromium.org>
Cr-Commit-Position: refs/heads/master@{#594381}
[modify] https://crrev.com/f78979a7db3f89fee97266eb8c3dfa0654fdfcf2/testing/buildbot/filters/chromeos.single_process_mash.content_browsertests.filter

I made this test flaky again it looks like by not making the renderer compositor visible for swapped out widgets. Maybe this is extremely timing sensitive?

(with CL https://chromium-review.googlesource.com/c/chromium/src/+/1290140)

+lukasza who might be able to help given that this involves the spare RPH.

I think I understand the source of the flakiness - SiteInstanceImpl retains its |process_| until SiteInstanceImpl::RenderProcessHostDestroyed.  If SiteInstanceImpl::GetProcess is called before the old process is destroyed, then the old process will be returned, instead of (as the test expects) associating the new frame with the spare process.

The above source of flakiness can be fixed by the following changes in the test:
+  RenderProcessHostWatcher prev_host_watcher(
+      prev_host, RenderProcessHostWatcher::WATCH_FOR_HOST_DESTRUCTION);
   EXPECT_TRUE(NavigateToURL(shell(), second_url));
+  prev_host_watcher.Wait();

I am not sure if such wait can be added to the generic NavigateToURL (it only makes sense if there are no other frames / service workers / etc. in the process).  I haven't yet looked at the other tests.

I am wondering if NavigateToURL could wait for destruction of the old frame/process.  I think this is something we wanted to do to avoid flakiness of tests that print the frame tree (e.g. see a recent issue 889002).  I wonder if NavigateToURL could wait until WebContentsObserver::RenderFrameDeleted (if the navigation is cross-process).

+arthursonzogni@ who AFAIK was thinking of tweaking WCO::RenderFrameDeleted notifications in presence of unload handlers.  I think in the case of the tests, we want to be notified when the frame and process *really* go away (not just when they go out of the frame tree).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/078178a731236fe79c2ae2466e88565e80ad32c4

commit 078178a731236fe79c2ae2466e88565e80ad32c4
Author: Lukasz Anforowicz <lukasza@chromium.org>
Date: Thu Oct 25 14:46:56 2018

Deflake Nav...ControllerBrowserTest.UtilizationOfSpareRenderProcessHost.

This CL tries to deflake
NavigationControllerBrowserTest.UtilizationOfSpareRenderProcessHost
by making it wait for the old process to go away, before trying another
navigation (that the test expects to grab a fresh spare, rather than
reusing the old process that before this CL might flakily be available).

Bug:  888156 
Change-Id: I7b847a28aabd8827f04c9303dbdab200e47d496c
Reviewed-on: https://chromium-review.googlesource.com/c/1298478
Reviewed-by: Alex Moshchuk <alexmos@chromium.org>
Commit-Queue: Łukasz Anforowicz <lukasza@chromium.org>
Cr-Commit-Position: refs/heads/master@{#602705}
[modify] https://crrev.com/078178a731236fe79c2ae2466e88565e80ad32c4/content/browser/frame_host/navigation_controller_impl_browsertest.cc
[modify] https://crrev.com/078178a731236fe79c2ae2466e88565e80ad32c4/testing/buildbot/filters/chromeos.single_process_mash.content_browsertests.filter

No alerts raised by FindIt, so I think this can be marked as fixed...