crbug.com/712607  is another example. 

Another incident: https://codereview.chromium.org/2858933002
with failure for ItemChooserDialogTest 
https://build.chromium.org/p/chromium.android/builders/Lollipop%20Phone%20Tester/builds/11739

Another incident is https://codereview.chromium.org/2861453002

The new BottomSheetContentControllerTest#testSelectContent passed the CQ but fails consistently on the bots.

https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=chrome_public_test_apk&tests=BottomSheetContentControllerTest%23testSelectContent

http://build.chromium.org/p/chromium.android/builders/Lollipop%20Phone%20Tester/builds/11873

The BottomSheetContentControllerTest is likely because it's limited to NMR1

@MinAndroidSdkLevel(Build.VERSION_CODES.LOLLIPOP_MR1)

Hmm, supposedly android_n5x_swarming_rel should have ran that test as well, since its devices is running M build android. And that Test is failing on M bots consistently too. https://build.chromium.org/p/chromium.android/builders/Marshmallow%2064%20bit%20Tester/builds/11985

I added this CL to test if the CQ run would catch it. https://codereview.chromium.org/2879513002

Also, shouldn't we disable the breaking tests on chromium.android bots?

I was just notified a couple of hours ago. The fix/disable is out for review: https://codereview.chromium.org/2875813002/

nice

What I found is that CQ's linux_android_rel bot runs entirely using K devices on N5s, and android_n5x_swarming_rel runs using n5x on marshmallow.

However, android_n5x_swarming_rel doesn't run chrome_public_test_apk currently due to flakiness (it's actually our team's OKR to make chrome_public_test_apk run again on n5x bot)

So any tests that either does not run on K or only fails on L and above can't be caught right now.