What is the bug or feature:
Shard failures result in all tests running on that shard being marked as flaky.
Context or examples:
Saw the email that FindIt is now live. It's a great idea. However, in checking out the results I saw that it reports 2 failures in */EncryptedMediaTest.Playback_Encryption_CENC/*. Following that link shows:
2018-09-17 11:11:45 UTC content_browsertests on Android device Nexus 5 (with patch) MSE_ExternalClearKey/EncryptedMediaTest.Playback_Encryption_CENC/0
2018-09-17 11:11:45 UTC content_browsertests on Android device Nexus 5 (with patch) SRC_ExternalClearKey/EncryptedMediaTest.Playback_Encryption_CENC/0
Both failures occurred on the same run. However, when I look at the logs [1] they report a large number of tests failed. It appears that shard#4 failed, so all the tests scheduled to run on that shard are marked as failing. But looking at the shard result [2], I can't find any line that indicates the test was ever run.
The end of the log does show:
C 720.062s Main ********************************************************************************
C 720.062s Main [==========] 291 tests ran.
C 720.062s Main [ PASSED ] 0 tests.
C 720.062s Main ********************************************************************************
So it does appear that the test was run, but there is no output to provide any data on what failed. Given that all 291 tests on the shard "failed", can we attribute that to a shard failure and not a flaky test?
[1] https://ci.chromium.org/p/chromium/builders/luci.chromium.try/android-kitkat-arm-rel/82304
[2] https://chromium-swarm.appspot.com/task?id=4000009b8cf40010&refresh=10&show_raw=1
Comment 1 by phanindra.mandapaka@chromium.org
, Sep 21