+100

Hmhh, I still think it's important to distinguish between disabling because it temporarily failed on platform X and disabling because it cannot run on platform X.

Here is a concrete use case: later we would want to implement the system to try out reenabling telemetry disabled tests & see which one still passing. I imagine for such system, we would do '--force-run-disabled" which allow running tests only on platforms that were the stories were failing but not including platforms which the tests cannot run.

I think we can have --force-run-disabled (-f) taking multilevel (similar to log verbosity flag -v). 

-f: run only temporarily disabled stories
-f -f: run disabled stories regardless of the platform.

+charliea FYI

I agree we should "distinguish between disabling because it temporarily failed on platform X and disabling because it cannot run on platform X."

I've been saying that for quite some time! But the distinction should come from the disabling mechanism:

a) disabled due to temporary failure? -> expectations.config
b) disabled due to "can't possibly work on platform X" -> some method on the story/benchmark (e.g. CanRunOnPlatform)

For your use case maybe it's enough to run a try-job with expectations.config cleared?

#4: I believe your proposal of (a) & (b) is implemented in Telemetry now.

clearing expecations.config is an interesting idea. Though I think it's harder than just passing a flag. What don't you like about making --force-run-disabled flag support different level of enabling tests?

> #4: I believe your proposal of (a) & (b) is implemented in Telemetry now.

Almost. We still disable some WebView stories (e.g. I remember one dealing with omnibox) through expectations.config which should be shared through code.

Also the ways to disable through code are a bit messy today, it needs some cleanup.

I don't have anything specific against -f -f, that's probably a good idea too.

Yep, we have already implemented a) and b) in Telemetry now. If something _can't_ run on a platform (e.g. a test that interacts with the browser Chrome running on webview) then it should be excluded from the SUPPORTED_PLATFORMS list in the benchmark (https://cs.chromium.org/search/?q=SUPPORTED_PLATFORMS%5C+%3D+f:tools/perf&sq=package:chromium&type=cs). If something can run on a platform but just fails, it should be listed in the expectations.config.

Can you give an example of when we would want to run a story that's not listed in SUPPORTED_PLATFORMS? This generally would mean something like running a mobile story on a desktop.

Ohhh, I haven't seen that we can now use expectation conditions in SUPPORTED_PLATFORMS, that is indeed neat! Does it already work for stories as well as benchmarks?

The story I mentioned in #6 is:
https://cs.chromium.org/chromium/src/tools/perf/expectations.config?rcl=5bea452cef8cd89ebf50409b96d59a70fe373e52&l=216

Which I could move to SUPPORTED_PLATFORMS if that is the way to do that sort of things now.

It should, based on https://cs.chromium.org/chromium/src/tools/perf/page_sets/system_health/chrome_stories.py?type=cs&q=f:page_set+SUPPORTED_PLATFORM&sq=package:chromium&l=39

I'd try it and verify that it works, just to be sure though.

I gave this a try on:
https://chromium-review.googlesource.com/c/chromium/src/+/966821

But it doesn't work. After a bit more of digging up it looks like

- benchmark.SUPPORTED_PLATFORMS is a list of test conditions from expectations.py, but
- system_health_story.SUPPORTED_PLATFORMS is a frozen set of strings from [1]

The main point is that the feature I requested in #4 does not seem yet implemented at the *story* level.

Sounds like we need some more cleanup on all of these different things of achieving almost but not quite the same things. :(

[1]: https://cs.chromium.org/chromium/src/tools/perf/page_sets/system_health/platforms.py

Ah, and I forgot to reply to Charlie:

> Can you give an example of when we would want to run a story that's not listed in SUPPORTED_PLATFORMS? This generally would mean something like running a mobile story on a desktop.

Yes, that's what chrishtr originally requested in issue 813168.