New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 887736 link

Starred by 1 user

Issue metadata

Status: Unconfirmed
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

Findit: Should tests on a shard that fail be counted as flaky?

Project Member Reported by jrumm...@chromium.org, Sep 20

Issue description

What is the bug or feature:
Shard failures result in all tests running on that shard being marked as flaky.

Context or examples:
Saw the email that FindIt is now live. It's a great idea. However, in checking out the results I saw that it reports 2 failures in */EncryptedMediaTest.Playback_Encryption_CENC/*. Following that link shows:

2018-09-17 11:11:45 UTC	content_browsertests on Android device Nexus 5 (with patch)	MSE_ExternalClearKey/EncryptedMediaTest.Playback_Encryption_CENC/0
2018-09-17 11:11:45 UTC	content_browsertests on Android device Nexus 5 (with patch)	SRC_ExternalClearKey/EncryptedMediaTest.Playback_Encryption_CENC/0

Both failures occurred on the same run. However, when I look at the logs [1] they report a large number of tests failed. It appears that shard#4 failed, so all the tests scheduled to run on that shard are marked as failing. But looking at the shard result [2], I can't find any line that indicates the test was ever run.

The end of the log does show:
C  720.062s Main  ********************************************************************************
C  720.062s Main  [==========] 291 tests ran.
C  720.062s Main  [  PASSED  ] 0 tests.
C  720.062s Main  ********************************************************************************

So it does appear that the test was run, but there is no output to provide any data on what failed. Given that all 291 tests on the shard "failed", can we attribute that to a shard failure and not a flaky test?

[1] https://ci.chromium.org/p/chromium/builders/luci.chromium.try/android-kitkat-arm-rel/82304
[2] https://chromium-swarm.appspot.com/task?id=4000009b8cf40010&refresh=10&show_raw=1
 
Labels: Needs-Milestone
Labels: -Needs-Milestone

Sign in to add a comment