It seems that the issue is also affecting many public bots:

https://uberchromegw.corp.google.com/i/chromium.android/builders/Jelly%20Bean%20Tester
https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Low-end%20Tester
https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Phone%20Tester
https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Tablet%20Tester
https://uberchromegw.corp.google.com/i/chromium.android/builders/Marshmallow%20Tablet%20Tester

Wanted to understand exactly which tests are failing. It seems pretty flaky, they are not always the same, and it varies from bot to bot. Maybe the issue is that they are hitting a timeout or something like that?

# Jelly Bean Tester: 5440 -> 5431 #
UUU---C--- BackgroundTracingManagerBrowserTest.CrashWhenSubprocessWithoutArgumentFilter
--C------U TracingControllerTest.NotWhitelistedMetadataStripped

# Lollipop Low-end Tester: 3803 -> 3794 #
U-U---U--- BackgroundTracingManagerBrowserTest.CallPreemptiveTriggerWithDelay
UUUU------ BackgroundTracingManagerBrowserTest.NoWhitelistedArgsStripped
--U------- BackgroundTracingManagerBrowserTest.ReactiveSecondTriggerTermination
--UU------ BackgroundTracingManagerBrowserTest.ReceiveTraceSucceedsOnHigherHistogramSample
UUU-U----- BackgroundTracingManagerBrowserTest.TraceMetadataInTrace
U-UU-UU--- DownloadContentTest.CookiePolicy
--U------- DownloadContentTest.RemoveInterruptedDownload
UUUU-UU--- DownloadContentTest.ResumeRestoredDownload_CorrectHash
-UUUU-U--- DownloadContentTest.ResumeRestoredDownload_EtagMismatch
U-U-UUU--- DownloadContentTest.ResumeRestoredDownload_NoFile
UUUUUUU--- DownloadContentTest.ResumeRestoredDownload_NoHash
U-U-U-U--- DownloadContentTest.ResumeRestoredDownload_ShortFile
-U-U------ DownloadContentTest.ResumeRestoredDownload_WrongHash
U--UUU---- DragDownloadFileTest.DragDownloadFileTest_Complete
UUUUUU---- DragDownloadFileTest.DragDownloadFileTest_NetError
-U------U- MediaCanPlayTypeTest.CodecSupportTest_AAC_ADTS
UU-------- WebContentsImplBrowserTest.DownloadImage_Deny_FileImage

# Lollipop Phone Tester: 6153 -> 6144 #
--U------- BackgroundTracingManagerBrowserTest.CrashWhenSubprocessWithoutArgumentFilter
-U-----U-- BackgroundTracingManagerBrowserTest.TraceMetadataInTrace
-U-U---U-- DownloadContentTest.ResumeRestoredDownload_CorrectHash
--U-----UU DownloadContentTest.ResumeRestoredDownload_EtagMismatch
-U-------- DownloadContentTest.ResumeRestoredDownload_NoFile
-U----U--- DownloadContentTest.ResumeRestoredDownload_WrongHash
-U-U--UU-- DragDownloadFileTest.DragDownloadFileTest_Complete
UU-UU-U--- DragDownloadFileTest.DragDownloadFileTest_NetError
U-U-U---UU UtilityProcessHostImplBrowserTest.LaunchProcess
-UUU---UU- UtilityProcessMojoClientBrowserTest.CallService
U-UU---UU- UtilityProcessMojoClientBrowserTest.ConnectionError

# Lollipop Tablet Tester: 4344 -> 4335 #
U--------- DownloadContentTest.ResumeRestoredDownload_ShortFile
--U-----UU TracingControllerTest.NotWhitelistedMetadataStripped
U--------U UtilityProcessHostImplBrowserTest.LaunchProcess
UU-----U-- UtilityProcessMojoClientBrowserTest.CallService

# Marshmallow Tablet Tester: 5035 -> 5026 #
UU----UU-- BackgroundTracingManagerBrowserTest.NoWhitelistedArgsStripped
--U---UU-- BackgroundTracingManagerBrowserTest.ReceiveTraceFinalContentsOnTrigger
UUU---U-U- BackgroundTracingManagerBrowserTest.TraceMetadataInTrace
UUUUUUUU-- DownloadContentTest.CookiePolicy
-U-----U-- DownloadContentTest.DownloadCancelled
UUU-UU-U-- DownloadContentTest.ResumeRestoredDownload_CorrectHash
UU-----U-- DownloadContentTest.ResumeRestoredDownload_LongFile
UUU-UU-U-- DownloadContentTest.ResumeRestoredDownload_NoFile
--UUU-UUU- DownloadContentTest.ResumeRestoredDownload_NoHash
-UUU-UUUU- DownloadContentTest.ResumeRestoredDownload_ShortFile
UUUUUU-UU- DownloadContentTest.ResumeRestoredDownload_WrongHash
UUUUUU-UU- DragDownloadFileTest.DragDownloadFileTest_Complete
UUUUUUUUU- DragDownloadFileTest.DragDownloadFileTest_NetError
UUUUUUUUU- UtilityProcessHostImplBrowserTest.LaunchProcess
UUUUUUUUU- UtilityProcessMojoClientBrowserTest.CallService
UUUUUUUUU- UtilityProcessMojoClientBrowserTest.ConnectionError

# asan-clang-phone: 773 -> 764 #
UUU------- BackgroundTracingManagerBrowserTest.CallMultipleTriggersOnlyGatherOnce
UU-------- BackgroundTracingManagerBrowserTest.CallPreemptiveTriggerWithDelay
UUU------- BackgroundTracingManagerBrowserTest.CallTriggersMoreThanOnceOnlyGatherOnce
UUU------- BackgroundTracingManagerBrowserTest.CrashWhenSubprocessWithoutArgumentFilter
U-U------- BackgroundTracingManagerBrowserTest.NoWhitelistedArgsStripped
UUU------- BackgroundTracingManagerBrowserTest.ReactiveSecondTriggerMustMatchForTermination
UUU------- BackgroundTracingManagerBrowserTest.ReactiveSecondTriggerTermination
UUU------- BackgroundTracingManagerBrowserTest.ReactiveThirdTriggerTimeout
UUU------- BackgroundTracingManagerBrowserTest.ReactiveTimeoutTermination
UUU------- BackgroundTracingManagerBrowserTest.ReceiveReactiveTraceSucceedsOnHigherHistogramSample
UUU------- BackgroundTracingManagerBrowserTest.ReceiveTraceFinalContentsOnTrigger
UU-------- BackgroundTracingManagerBrowserTest.ReceiveTraceSucceedsOnHigherHistogramSample
UU-------- BackgroundTracingManagerBrowserTest.TraceMetadataInTrace
UUU------- DownloadContentTest.CookiePolicy
UUU------- DownloadContentTest.ResumeRestoredDownload_CorrectHash
UUU------- DownloadContentTest.ResumeRestoredDownload_EtagMismatch
UUU------- DownloadContentTest.ResumeRestoredDownload_LongFile
UUU------- DownloadContentTest.ResumeRestoredDownload_NoFile
UUU------- DownloadContentTest.ResumeRestoredDownload_NoHash
UUU------- DownloadContentTest.ResumeRestoredDownload_ShortFile
UUU------- DownloadContentTest.ResumeRestoredDownload_WrongHash
UUU------- DragDownloadFileTest.DragDownloadFileTest_Complete
UU-------- DragDownloadFileTest.DragDownloadFileTest_NetError
UUU------- TracingControllerTest.NotWhitelistedMetadataStripped

There is a clear "wall" where tests start failing more consistently on "Lollipop Low-end Tester", "Marshmallow Tablet Tester", and "asan-clang-phone", respectively builds:

https://build.chromium.org/p/chromium.android/builders/Lollipop%20Low-end%20Tester/builds/3797
https://build.chromium.org/p/chromium.android/builders/Marshmallow%20Tablet%20Tester/builds/5018
https://uberchromegw.corp.google.com/i/internal.client.clank/builders/asan-clang-phone/builds/773

Also they have an overlapping cl range!
http://test-results.appspot.com/revision_range?start=410519&end=410552

I'm largely taking a stab in the dark, but the following seem somewhat suspicious (they talk about changes to "background" stuff):

8366b12610e369c46f257ea8d7ee6bf7a381d590 Change signature of OnConnect
47cd270d10b49ebdaea567566f8044ae5a89ef9e Change database scheme - add state and start tracking

+ben, +petewil, do you think any of your changes could have caused these tests to fail or take longer than usual to run?

+khushalsagar@ who also made a substantial change to content/browser in the regression range - content: Add ContextProviderFactory to create a render ContextProvider https://chromium.googlesource.com/chromium/src/+/c81e43b82548d619a0b8ebd4dfd2362a8bbf5e2a

petewil@'s change seems unlikely since it was for the offline pages component

Uh oh, I'll try to repro locally to see if its my change.

Thank you Khushal :) Let me know what you find.

ben@ - your patch seems most likely, so will you please look into this asap (there are quite a few bots consistently failing)?

Some of the failures are in UtilityProcessMojoClientBrowserTest, and the other tests failures generally look like they may have to do with connection things, so your change to the signature of OnConnect seems most likely ( 8366b12610e369c46f257ea8d7ee6bf7a381d590 )

I couldn't cleanly revert the OnConnect patch. Will you please try reverting locally to see if the content browsertests pass?  If this doesn't have to do with your change, please set status back to "Untriaged" so either I or the next sheriff can keep digging.

When I tried running some of these locally and they just seem to hang forever.

I also couldn't cleanly revert 81e43b82548d619a0b8ebd4dfd2362a8bbf5e2a (content: Add ContextProviderFactory); I think that's the next one to try if it isn't OnConnect.

I successfully reverted the following two patches; neither had an effect on the tests passing (just spot checked but they seem much less likely anyway):
6b45bc6f0ae42a9bf00ff559b6ae21056e5f4e0c (touched content/browser/ssl/ssl_policy.cc)
b7dd1d295efb200353dacc07e96792df206d891c (touched content/browser/renderer_host/render_widget_host_view_android.cc)

I saw this some during my shift yesterday, too, but didn't get a chance to investigate too much. It appears to be affecting both the main waterfall:

http://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=content_browsertests&builder=chromium.linux%3AAndroid%20Tests

and the CQ:

http://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=content_browsertests%20(with%20patch)&builder=tryserver.chromium.android%3Alinux_android_rel_ng

TracingControllerTest.NotWhitelistedMetadataStripped and ackgroundTracingManagerBrowserTest.CrashWhenSubprocessWithoutArgumentFilter appear to be the biggest culprits.

The TracingControllerTest appears to be hitting a check failure:

I  285.639s Main  pid: 16997, tid: 17031, name: Chrome_ChildIOT  >>> org.chromium.content_browsertests_apk:privileged_process0 <<<
I  285.639s Main  signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
I  285.639s Main      r0 00000000  r1 00004287  r2 00000006  r3 00000000
I  285.639s Main      r4 00000006  r5 0000001e  r6 00004287  r7 0000010c
I  285.639s Main      r8 7c5ef73c  r9 7c5ef2c0  sl 00000000  fp 0000000a
I  285.639s Main      ip 4013f384  sp 7c5ef140  lr 40103fe5  pc 40112f90
I  285.639s Main  
I  285.639s Main  Stack Trace:
I  285.639s Main    RELADDR   FUNCTION                                                                    FILE:LINE
I  285.639s Main    00021f90  tgkill+12                                                                   /system/lib/libc.so
I  285.639s Main    00012fe1  pthread_kill+48                                                             /system/lib/libc.so
I  285.639s Main    000131f5  raise+10                                                                    /system/lib/libc.so
I  285.639s Main    00011f2b  <unknown>                                                                   /system/lib/libc.so
I  285.639s Main    00021844  abort+4                                                                     /system/lib/libc.so
I  285.639s Main    v------>  DebugBreak                                                                  /b/c/b/android/src/base/debug/debugger_posix.cc:219
I  285.639s Main    006a3649  base::debug::BreakDebugger()                                                /b/c/b/android/src/base/debug/debugger_posix.cc:249
I  285.639s Main    006b58d7  logging::LogMessage::~LogMessage()                                          /b/c/b/android/src/base/logging.cc:748
I  285.639s Main    00703953  base::trace_event::TraceLog::FinishFlush(int, bool)                         /b/c/b/android/src/base/trace_event/trace_log.cc:976
I  285.639s Main    v------>  base::Callback<void (), (base::internal::CopyMode)1>::Run() const           /b/c/b/android/src/base/callback.h:389
I  285.639s Main    006a5b9f  base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask const&)  /b/c/b/android/src/base/debug/task_annotator.cc:54
I  285.639s Main    006ba555  base::MessageLoop::RunTask(base::PendingTask const&)                        /b/c/b/android/src/base/message_loop/message_loop.cc:496
I  285.639s Main    006bac13  base::MessageLoop::DeferOrRunPendingTask(base::PendingTask)                 /b/c/b/android/src/base/message_loop/message_loop.cc:505
I  285.639s Main    006bad11  base::MessageLoop::DoWork()                                                 /b/c/b/android/src/base/message_loop/message_loop.cc:629
I  285.639s Main    006bcedd  base::MessagePumpLibevent::Run(base::MessagePump::Delegate*)                /b/c/b/android/src/base/message_loop/message_pump_libevent.cc:217
I  285.639s Main    006bbf33  base::MessageLoop::RunHandler()                                             /b/c/b/android/src/base/message_loop/message_loop.cc:459
I  285.639s Main    006d1669  base::RunLoop::Run()                                                        /b/c/b/android/src/base/run_loop.cc:35
I  285.639s Main    006ed79b  base::Thread::Run(base::RunLoop*)                                           /b/c/b/android/src/base/threading/thread.cc:228
I  285.639s Main    006ee291  base::Thread::ThreadMain()                                                  /b/c/b/android/src/base/threading/thread.cc:301
I  285.639s Main    006ea36f  ThreadFunc                                                                  /b/c/b/android/src/base/threading/platform_thread_posix.cc:71
I  285.640s Main    0000d170  __thread_entry+72                                                           /system/lib/libc.so
I  285.640s Main    0000d308  pthread_create+240                                                          /system/lib/libc.so

(from https://build.chromium.org/p/chromium.linux/builders/Android%20Tests%20%28dbg%29/builds/35661)

i.e., a CHECK failure here: https://codesearch.chromium.org/chromium/src/base/trace_event/trace_log.cc?rcl=0&l=976

Confirmed that the failure in DownloadContentTest is from my change. Working on fixing it.

Fix up for review.
https://codereview.chromium.org/2233023003

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3e1eed893f3fe23256d5c418eecfc72ddd95fb8b

commit 3e1eed893f3fe23256d5c418eecfc72ddd95fb8b
Author: khushalsagar <khushalsagar@google.com>
Date: Thu Aug 11 03:38:30 2016

content: Don't keep requesting the GpuChannelHost during shutdown.

During shutdown, the Gpu Channel initialization returns a nullptr.
Early out in that case rather than keep queueing more requests.

BUG= 636294 

Review-Url: https://codereview.chromium.org/2233023003
Cr-Commit-Position: refs/heads/master@{#411254}

[modify] https://crrev.com/3e1eed893f3fe23256d5c418eecfc72ddd95fb8b/content/browser/gpu/browser_gpu_channel_host_factory.h
[modify] https://crrev.com/3e1eed893f3fe23256d5c418eecfc72ddd95fb8b/content/browser/renderer_host/context_provider_factory_impl_android.cc

Fixed rolled into all of these bots, and tests are no longer failing:
- Jelly Bean Tester
- Lollipop Low-end Tester (just now!)
- Lollipop Phone Tester
- Lollipop Tablet Tester
- Marshmallow Tablet Tester

yay!

Fix also rolled into "asan-clang-phone", but build hasn't finished running the test:
https://uberchromegw.corp.google.com/i/internal.client.clank/builders/asan-clang-phone/builds/778

Will reopen if needed, but seems that this can be regarded as fixed.

+oysteine for the tracing crash in comment #8

BackgroundTracingManagerBrowserTest.CrashWhenSubprocessWithoutArgumentFilter is actually specifically meant to hit that CHECK(). Is the test actually flaky, or is the resulting renderer crash being interpreted as a flake?

Here's a sample recent failure: https://build.chromium.org/p/chromium.android/builders/KitKat%20Tablet%20Tester/builds/5093

It's possible that we just pulled the wrong tombstone out since correlating tombstones to tests isn't an exact science.

It sounds like this is a separate issue from the one Khushal fixed - should we open a separate bug?

On second thought - maybe it's fixed; there haven't been any failures since the patch in #11 landed. I'll keep an eye on this and open a new bug if there are any new flakes.

http://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=content_browsertests&tests=CrashWhenSubprocessWithoutArgumentFilter%20