There doesn't appear to be a single systemic issue here, just a lot of individual ones. :(

ChromeHomeDownloadManagerTest#testShowDownloadManagerLastTrackedActivityNull_TabbedActivityRunning: segfault in the WebContents dtor. Stack visible in the tombstones. +twellington

MinidumpUploaderImplTest: failure in assertion here: https://codesearch.chromium.org/chromium/src/components/minidump_uploader/android/javatests/src/org/chromium/components/minidump_uploader/MinidumpUploaderImplTest.java?rcl=e21758bf780cf8c25927525be22dac8f54132fd8&l=245. Doesn't get picked up as a test failure because the assertion runs on another thread. +gsennton

OmahaBaseTest: the instrumentation reported two test failures, which may have thrown off our parsing logic. Errors are visible in the swarming task's stdout. 1: "java.lang.ClassCastException: android.app.Application cannot be cast to Ti" in BaseChromiumApplication.initCommandLine. Maybe proguard malfeasance? 2: NPE in the InstrumentationResultPrinter, which is a new one to me. +nyquist

DownloadActivityTest.testFilters: https://bugs.chromium.org/p/chromium/issues/detail?id=778255

DevToolsProtocolTest: DCHECK failure in weak_ptr.cc: "WeakPtrs must be invalidated on the same sequenced thread". Visible w/ symbolized trace in the stdout. Not entirely sure who this should go to -- +pfeldman?

ChromeHomeDownloadManagerTest#testShowDownloadManagerLastTrackedActivityNull_TabbedActivityRunning is relatively new. I won't be able to look at it until after Thanksgiving week at the earliest. I'm WF Boulder tomorrow and out half of Friday, so if someone (sheriffing) has time to disable for now until I have time to look at it that would be greatly appreciated.

What I see is random tests failing with "java.util.concurrent.TimeoutException: waitForCallback timed out"

https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/431790
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432090
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432093
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432095
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432098
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432110
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432114
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432119
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432126
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432139
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432150
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432154
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432158
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432160
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432172
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432175
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432198
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng/builds/432224

Does seem like a systemic issue to me.

For some value of "systemic," perhaps. I would argue that, of the builds you selected, the test failures aren't "random."

In the builds you've linked, the following tests failed:

org.chromium.chrome.browser.NavigateTest#testNavigateRedirect
org.chromium.chrome.browser.banners.AppBannerManagerTest#testBannerAppearsThenDoesNotAppearAgainForWeeks
org.chromium.chrome.browser.payments.PaymentRequestCanMakePaymentMetricsTest#testNoQuery_UserAbort
org.chromium.chrome.browser.payments.PaymentRequestCanMakePaymentQueryTest#testPayViaFastBobPay
org.chromium.chrome.browser.payments.PaymentRequestContactDetailsAndFreeShippingTest#testPay
org.chromium.chrome.browser.payments.PaymentRequestContactDetailsTest#testAddContactAndCancelEditorShouldKeepContactSelected
org.chromium.chrome.browser.payments.PaymentRequestEmailAndFreeShippingTest#testPaymentRequestEventsMetric
org.chromium.chrome.browser.payments.PaymentRequestEmailAndPhoneTest#testSuggestionsDeduped
org.chromium.chrome.browser.payments.PaymentRequestIdTest#testPaymentResponse
org.chromium.chrome.browser.payments.PaymentRequestJourneyLoggerTest#testNoContactInfoHistogram
org.chromium.chrome.browser.payments.PaymentRequestJourneyLoggerTest#testNumberOfSelectionAdds_ContactInfo_Completed
org.chromium.chrome.browser.payments.PaymentRequestJourneyLoggerTest#testUserDidNotHaveCompleteSuggestions_ShippingAndPayment_IncompleteShipping
org.chromium.chrome.browser.payments.PaymentRequestMultipleContactDetailsTest#testContactDetailsDedupe_Capitalization
org.chromium.chrome.browser.payments.PaymentRequestNameAndFreeShippingTest#testPay
org.chromium.chrome.browser.payments.PaymentRequestNoShippingTest#testEditAndCloseDialog
org.chromium.chrome.browser.payments.PaymentRequestNoShippingTest#testQuickDismissAndPayShouldNotCrash
org.chromium.chrome.browser.payments.PaymentRequestPaymentAppCanMakePaymentQueryTest#testNoInstrumentsInFastBobPay
org.chromium.chrome.browser.payments.PaymentRequestServiceWorkerPaymentAppTest#testHasMastercardCreditCapabilities
org.chromium.chrome.browser.payments.PaymentRequestTabTest#testDismissOnTabNavigate
org.chromium.chrome.browser.payments.PaymentRequestTabTest#testDismissOnTabSwitch

All of the payments test failures share the same 12 frames at the top of the stack trace:

	at org.chromium.base.test.util.CallbackHelper.waitForCallback(CallbackHelper.java:191)
	at org.chromium.base.test.util.CallbackHelper.waitForCallback(CallbackHelper.java:202)
	at org.chromium.content.browser.test.util.TestCallbackHelperContainer$OnEvaluateJavaScriptResultHelper.waitUntilHasValue(TestCallbackHelperContainer.java:160)
	at org.chromium.content.browser.test.util.JavaScriptUtils.executeJavaScriptAndWaitForResult(JavaScriptUtils.java:56)
	at org.chromium.content.browser.test.util.JavaScriptUtils.executeJavaScriptAndWaitForResult(JavaScriptUtils.java:30)
	at org.chromium.content.browser.test.util.DOMUtils.scrollNodeIntoViewByJs(DOMUtils.java:306)
	at org.chromium.content.browser.test.util.DOMUtils.scrollNodeIntoView(DOMUtils.java:296)
	at org.chromium.content.browser.test.util.DOMUtils.clickNode(DOMUtils.java:234)
	at org.chromium.chrome.browser.payments.PaymentRequestTestCommon.clickNodeAndWait(PaymentRequestTestCommon.java:194)
	at org.chromium.chrome.browser.payments.PaymentRequestTestCommon.openPageAndClickNodeAndWait(PaymentRequestTestCommon.java:164)
	at org.chromium.chrome.browser.payments.PaymentRequestTestCommon.triggerUIAndWait(PaymentRequestTestCommon.java:180)
	at org.chromium.chrome.browser.payments.PaymentRequestTestRule.triggerUIAndWait(PaymentRequestTestRule.java:146)

I've filed https://bugs.chromium.org/p/chromium/issues/detail?id=786137 for that.


On another note: while I do appreciate y'all filing bugs, omnibugs like this are significantly harder to wrangle than more narrow bugs.

Ah, sorry, I've just filtered "waitForCallback" problems out of the rest of the failures, which was a lot of work by itself. Thank you for bucketing those further.

Sorry for filing a meta-bug. Doing tons of triage right now and no time to investigate each deeply. Please do split this into multiple ones.

I'm trying to debug the minidump failure (or rather, I want to double-check that it's failing in that assertion), how do I get the logcat of the device being tested?

#8: use the result_details link in the step. e.g., 

From https://ci.chromium.org/buildbot/tryserver.chromium.android/linux_android_rel_ng/430011

result_details for chrome_public_test_apk: https://storage.cloud.google.com/chromium-result-details/html/chrome_public_test_apk_linux_android_rel_ng_430011_2017_11_14_T23_00_53-UTC

From there, you should see the failed MinidumpUploaderImplTest suite. Drilling into it will show you the list of tests, including the logcat for each. The logcat for that run for MinidumpUploaderImplTest#testFailingThenPassingUpload is https://luci-logdog.appspot.com/v/?s=chromium%2Fandroid%2Fswarming%2Flogcats%2F39d5808612466b11%2F%2B%2Flogcat_logcat_org.chromium.components.minidump_uploader.MinidumpUploaderImplTest.testCancelFailedUploadCausesReschedule_20171114T224852-UTC_0695f25c003be2ec

Thanks John, could we add some more description under the first link rather than just "result_details" ? Something like "result_details (logcats / flakiness links)", there's no way for me to know which link I should click otherwise :)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/f51a223063a7c3f72dfbf19604babbec2876e392

commit f51a223063a7c3f72dfbf19604babbec2876e392
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Wed Dec 06 21:22:35 2017

Change name of result_details link in build page.

Because "result_details" is awfully ambiguous.

Recipe-Nontrivial-Roll: build_limited_scripts_slave
Bug: 785055
Change-Id: I61f590427fb66bc72f8948bc253321f7aab2bff5
Reviewed-on: https://chromium-review.googlesource.com/806997
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipes/cronet.expected/android_cronet_marshmallow_64bit_perf.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipes/cronet.expected/android_cronet_marshmallow_64bit_builder.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipes/chromium.expected/dynamic_instrumentation_test.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipe_modules/chromium_android/api.py
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipes/chromium.expected/dynamic_instrumentation_test_with_timeout_scale.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipe_modules/chromium_android/examples/full.expected/webview_cts_basic.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipes/cronet.expected/android_cronet_tester.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipe_modules/chromium_tests/tests/steps/web_view_cts_test.expected/basic.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_cts_test.expected/basic.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipes/cronet.expected/android_cronet_kitkat_builder.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipe_modules/chromium_android/examples/full.expected/result_details_basic.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipe_modules/cronet/examples/full.expected/mb_test.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipes/chromium.expected/dynamic_instrumentation_nodefault_build.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipes/cronet.expected/android_cronet_armv6_builder.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipes/cronet.expected/cronet_try.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipes/cronet.expected/android_cronet_builder__dbg_.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipe_modules/cronet/examples/full.expected/local_test.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipes/cronet.expected/local_test.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipes/cronet.expected/android_cronet_builder_asan.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipes/chromium.expected/dynamic_instrumentation_nodefault_test.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipe_modules/cronet/examples/full.expected/gn_test.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipe_modules/chromium_android/examples/full.expected/upload_result_details_failures.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/README.recipes.md
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_instrumentation_test.expected/basic.json
[modify] https://crrev.com/f51a223063a7c3f72dfbf19604babbec2876e392/scripts/slave/recipes/cronet.expected/android_cronet_lollipop_builder.json

ToolbarProgressBarTest tests have started flaking aggressively on the trybot. Filed  bug 792724  for that.

(And after the change in #11 + https://chromium-review.googlesource.com/c/chromium/src/+/812207 the result_details link name should be a bit more descriptive.)

Anecdotally, larng seems a bit happier now, so downgrading pri. I'd pull out some numbers to support that claim, but unfortunately the data's not wholly there, so I'm blocking this on my effort to start tracking flakiness-per-bot (bug 783858)

Once we can visualize which bots like to flake, I'll close this out if we deem larng to be under an acceptable level.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2279adad772a0fae6551d7d2bcaba4ec7a16ae02

commit 2279adad772a0fae6551d7d2bcaba4ec7a16ae02
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Wed Dec 13 20:51:44 2017

android: Change name of result_details link for swarming tests.

Bug: 785055
Change-Id: I9441eaa8d0b8a4eec253d4199045f6fc0ece9126
Reviewed-on: https://chromium-review.googlesource.com/812207
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>
Cr-Commit-Position: refs/heads/master@{#523869}
[modify] https://crrev.com/2279adad772a0fae6551d7d2bcaba4ec7a16ae02/build/android/pylib/results/presentation/test_results_presentation.py

linux_android_rel_ng looks to me like it's gotten flakier recently. See the two attached screenshots. Blocked this bug on a few flakes of tests which seem like they're happening all over the place.

Deleted: Screen Shot 2017-12-20 at 4.27.35 PM.png 924 KB

	Screen Shot 2017-12-20 at 4.27.35 PM.png 924 KB View Download

Deleted: Screen Shot 2017-12-20 at 4.27.44 PM.png 860 KB

	Screen Shot 2017-12-20 at 4.27.44 PM.png 860 KB View Download