So, as far as I can tell, RetrySuccess is supposed to fail too. It's set up to pass after running dummy_Fail 5 times. But control.retry_alwaysflake only runs dummy_Fail 2 times:
RetryFail
RetrySuccess
If I changed the test to run
RetryFail
RetryFail
RetryFail
RetryFail
RetrySuccess
then it should succeed.
The (almost made) decision here is to kill the RETRIES feature.
Note that there are two separate notions of retries, that can both be controlled via control file constants.
- RETRIES (this one): A retry within autoserv. skylab/afe does not know that the test was retried, there aren't separate TKO entries for the retried test. i.e., we're blind that the retry happened.
- JOB_RETRIES: Suite level option -- test is retried (possibly on a different host). Suite has other related options that controls how many tests can be retried across the suite etc.
We will kill RETRIES because:
- No that many things use it: http://shortn/_KTM7dKkHdr
- We're blind when this happens / how often this happens.
- Often when a test fails, we want it to run on a different DUT.
- It is redundant with JOB_RETRIES
- It is non-trivial to make it work with Skylab.
There is a possible performance benefit of RETRIES over JOB_RETRIES, but such retries should be done at an even tighter scope -- the test started a little webserver on the DUT that failed to come up? Well, retry starting the webserver, don't ask autoserv to retry the test for you.
So, I'm going to go ahead and cull this ancient feature.
This is also not specified in that many control files: http://shortn/_9OWKA1F2nm
All tests that use this feature will be bumped to instead use JOB_RETRIES.
https://chromium-review.googlesource.com/c/chromiumos/third_party/autotest/+/1173661
This stack drops the feature by not respecting RETRIES anymore within autoserv, and migrates the smattering of users to JOB_RETRIES.
Cleanup is to remove the test.retries argument from all over the place.
+haddowk to make sure nobody uses this on moblab either.
test_push is failing. It is still expected to run "dummy_Fail.RetrySuccess" as they use the old package like gandof-release/R64-10176.65.0.
__main__.TestPushException: 1 test(s) are not expected to be run:
dummy_Fail.RetrySuccess
Comment 1 by ayatane@chromium.org
, Aug 13