Right now an autotest test passes when it does not raise an exception.
It fails when it raises any exception.
Except sometimes when it raises a special exception like WARN or TEST_NA, which sometimes is not a failure, but that depends on the caller and dashboard and generally is nowhere in agreement.

This is subtle and gets messier. One problem is the use of exceptions, which are then serialized (client tests) and re-interpreted from the logs.

Sometimes the test writer using TEST_NA or WARN wants to punt the decision on PASS/FAIL to the result reader, because there is context missing to the test (which is bad and should be fixed in the test). But generally a writer would have just wanted not to alert sheriffs immediately to a failure, just examine unusual (but passing) situations on a dashboard when there was spare time (non-urgent, minor issues).

Proposal:
0) Don't use exceptions.
1) It is better to force autotest writers to categorize each return path as PASS or FAIL.
2) It should be possible to annotate both PASS and FAIL with a string (not just FAIL as it is now).
3) Maybe it should be possible to annotate each PASS and FAIL with an individual color for dashboards like stainless/. (Or maybe stainless should hash different strings into slightly different shades of green/red.)
4) Any python exception escaping a test should be converted into a very simple failure (encouraging test writers to handle/annotate them, e.g. annotate with "Please fix your test").
5) Each test should also be able to set a test timeout in the control file (I think this is not cleanly handled right now). This timeout should be enforced by the harness, not the test. A test can of course have its own timeouts, but should return a an annotated failure.
6) All other python error/exceptions (like TEST_NA) should be left to autotest harness/scheduling internals, and not be available to tests themselves.

Benefits:
1) Less ambiguity: the test writer decides what is PASS or FAIL. The result reader does not need to interpret if a WARN or a TEST_NA is a PASS or a FAIL.
2) Better mapping when interfacing with external dashboards (like sponge).
3) Better control over text annotation and (suggested) dashboard colors.