Suspect https://codereview.chromium.org/2504163005/ is the culprit.

Logs: https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Phone%20Tester/builds/8104

Will take a look. Probably just need to change the tests to make sure they initialize field trial list.

Working on a fix now. Hoping can just land today without having to revert the CL.

(There's a dependent CL already landed and would be bad to have to revert the chain.)

Here's the symbolized stack:

Using toolchain from: /usr/local/google/code/clankium/src/third_party/android_tools/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-
[FATAL:field_trial.cc(676)] Check failed: global_->create_trials_from_command_line_called_. 

Stack Trace:
  RELADDR   FUNCTION                                                                                                                                                                                    FILE:LINE
  0009923b  logging::LogMessage::~LogMessage()                                                                                                                                                          /usr/local/google/code/clankium/src/base/logging.cc:532
  000a69e3  base::FieldTrialList::GetInitiallyActiveFieldTrials(base::CommandLine const&, std::__ndk1::vector<base::FieldTrial::ActiveGroup, std::__ndk1::allocator<base::FieldTrial::ActiveGroup> >*)  /usr/local/google/code/clankium/src/base/metrics/field_trial.cc:676
  0028c4df  chrome_variations::ChildProcessFieldTrialSyncer::InitFieldTrialObserving(base::CommandLine const&)                                                                                          /usr/local/google/code/clankium/src/chrome/common/variations/child_process_field_trial_syncer.cc:41
  0028c481  ChromeContentGpuClient::Initialize(base::FieldTrialList::Observer*)                                                                                                                         /usr/local/google/code/clankium/src/chrome/gpu/chrome_content_gpu_client.cc:72
  004aab2f  content::InProcessGpuThread::Init()                                                                                                                                                         /usr/local/google/code/clankium/src/content/gpu/in_process_gpu_thread.cc:66
  000dd619  base::Thread::ThreadMain()                                                                                                                                                                  /usr/local/google/code/clankium/src/base/threading/thread.cc:322
  000d8065  ThreadFunc                                                                                                                                                                                  /usr/local/google/code/clankium/src/base/threading/platform_thread_posix.cc:71
  00046abf  <unknown>                                                                                                                                                                                   /system/lib/libc.so

Ah, so this happens because Android runs the GPU thread code in the same process as the browser.

So we actually need to *not* run the ChildProcessFieldTrialSyncer code in that case, since they're in the same process.

GPU runs in the same process as browser on low-end Android devices. On high-end devices, they are in separate processes. This test specifically forces the "low-end" mode.

Makes sense - testing a fix now.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cbe6717b654cfd51f6eec2e12b90a91eb5186775

commit cbe6717b654cfd51f6eec2e12b90a91eb5186775
Author: asvitkine <asvitkine@chromium.org>
Date: Wed Nov 23 23:51:22 2016

Fix field trial DCHECK when gpu process is in-process.

The DCHECK was introduced by my CL here:
https://codereview.chromium.org/2504163005/

In fact, it caught the problem that we were running
the field trial syncer code even when the GPU process
was part of the browser process, which is unneeded.
This change fixes that by not running that code in
that case.

BUG=668217

Review-Url: https://codereview.chromium.org/2525973003
Cr-Commit-Position: refs/heads/master@{#434286}

[modify] https://crrev.com/cbe6717b654cfd51f6eec2e12b90a91eb5186775/chrome/gpu/chrome_content_gpu_client.cc

Tests no longer failing with latest run:

https://uberchromegw.corp.google.com/i/chromium.android/builders/Lollipop%20Phone%20Tester/builds/8126

(There's an unrelated org.chromium.blimp.app.auth.RetryingTokenSourceTest_testFailedTokenSource failure however.) 

testDownloadingToast is flaking again:
http://build.chromium.org/p/chromium.android/builders/Lollipop%20Phone%20Tester/builds/8146
http://build.chromium.org/p/chromium.android/builders/Lollipop%20Phone%20Tester/builds/8155

junit.framework.AssertionFailedError: Activity did not regain focus.
	at org.chromium.content.browser.test.util.CriteriaHelper.pollInstrumentationThread(CriteriaHelper.java:74)
	at org.chromium.content.browser.test.util.CriteriaHelper.pollInstrumentationThread(CriteriaHelper.java:89)
	at org.chromium.chrome.test.util.browser.contextmenu.ContextMenuUtils.selectOpenContextMenuItem(ContextMenuUtils.java:165)
	at org.chromium.chrome.test.util.browser.contextmenu.ContextMenuUtils.selectContextMenuItemByJs(ContextMenuUtils.java:118)
	at org.chromium.chrome.test.util.browser.contextmenu.ContextMenuUtils.selectContextMenuItem(ContextMenuUtils.java:99)
	at org.chromium.chrome.browser.hardware_acceleration.ToastHWATest.testDownloadingToast(ToastHWATest.java:79)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
	at org.chromium.chrome.test.ChromeActivityTestCaseBase.runTest(ChromeActivityTestCaseBase.java:726)
	at org.chromium.base.test.BaseTestResult.runParameterized(BaseTestResult.java:161)
	at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:124)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1853)

Oops, s/ianwen/dskiba/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/31793fa1aa3500139091f4b1c26e9302598ad0c8

commit 31793fa1aa3500139091f4b1c26e9302598ad0c8
Author: johnme <johnme@chromium.org>
Date: Fri Nov 25 16:32:25 2016

Mark ToastHWATest#testDownloadingToast flaky

It failed in the following builds:
http://build.chromium.org/p/chromium.android/builders/Lollipop%20Phone%20Tester/builds/8146
http://build.chromium.org/p/chromium.android/builders/Lollipop%20Phone%20Tester/builds/8155

TBR=dskiba@chromium.org
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
BUG=668217

Review-Url: https://codereview.chromium.org/2532743002
Cr-Commit-Position: refs/heads/master@{#434523}

[modify] https://crrev.com/31793fa1aa3500139091f4b1c26e9302598ad0c8/chrome/android/javatests/src/org/chromium/chrome/browser/hardware_acceleration/ToastHWATest.java

I'm also seeing flakes for #testOpenedInBackgroundToast on the Marshmallow Tablet Tester:

https://build.chromium.org/p/chromium.android/builders/Marshmallow%20Tablet%20Tester/builds/7638

C 1426.116s Main  [FAIL] org.chromium.chrome.browser.hardware_acceleration.ToastHWATest#testOpenedInBackgroundToast:
C 1426.116s Main  junit.framework.AssertionFailedError: '2131690266' is not visible
C 1426.116s Main  	at org.chromium.chrome.test.util.browser.contextmenu.ContextMenuUtils.selectOpenContextMenuItem(ContextMenuUtils.java:152)
C 1426.116s Main  	at org.chromium.chrome.test.util.browser.contextmenu.ContextMenuUtils.selectContextMenuItemByJs(ContextMenuUtils.java:116)
C 1426.116s Main  	at org.chromium.chrome.test.util.browser.contextmenu.ContextMenuUtils.selectContextMenuItem(ContextMenuUtils.java:97)
C 1426.116s Main  	at org.chromium.chrome.browser.hardware_acceleration.ToastHWATest.testOpenedInBackgroundToast(ToastHWATest.java:119)
C 1426.117s Main  	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
C 1426.117s Main  	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
C 1426.117s Main  	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
C 1426.117s Main  	at org.chromium.chrome.test.ChromeActivityTestCaseBase.runTest(ChromeActivityTestCaseBase.java:752)
C 1426.117s Main  	at org.chromium.base.test.BaseTestResult.runParameterized(BaseTestResult.java:161)
C 1426.117s Main  	at org.chromium.base.test.BaseTestResult.run(BaseTestResult.java:124)
C 1426.117s Main  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
C 1426.117s Main  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
C 1426.117s Main  	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:555)
C 1426.117s Main  	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1879)

Is it possible to force bot to take a screenshot when this happens?

Disabling several tests (bot-detected, see newly linked bugs for details) in https://crrev.com/c/1179665.

dskiba: Could you please investigate or help to find a recent owner?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2db0839d6a455786e586c14cfcffa02fc96af665

commit 2db0839d6a455786e586c14cfcffa02fc96af665
Author: Friedrich Horschig <fhorschig@chromium.org>
Date: Fri Aug 17 10:06:09 2018

[Flaky][Android] Mark several ToastHWATests flaky

Theses tests were known to be flaky for a while and this is not the
first time they are disabled for unknown reasons. Reopened and linked
bugs with details.

TBR=dskiba@chromium.org, tedchoc@chromium.org

Bug: 668217, 875130,  875129 ,  875092 
Change-Id: Ifee7d29bbdf28349aea3837a6b1a2525484c6e9a
Reviewed-on: https://chromium-review.googlesource.com/1179665
Reviewed-by: Friedrich Horschig <fhorschig@chromium.org>
Commit-Queue: Friedrich Horschig <fhorschig@chromium.org>
Cr-Commit-Position: refs/heads/master@{#584010}
[modify] https://crrev.com/2db0839d6a455786e586c14cfcffa02fc96af665/chrome/android/javatests/src/org/chromium/chrome/browser/hardware_acceleration/ToastHWATest.java

Hi, please find someone else as I'm not working on Clank anymore.

tedchoc: do you know who may be a good POC? The directory hasn't had any changes in almost 9 months and you're in the next owners file up. Thanks!