I don't think I quite understand what you're asking for here. Are you asking that a sheriff not disable something in the source code, because that'll affect steps that use filter files?

Or something else?

Basically sheriffs (especially the new ones) are not aware of some filter files instead they might disable the test everywhere. e.g. in the following patch the correct thing to do is adding a expectation to filter rather than disabling the entire test. https://chromium-review.googlesource.com/c/chromium/src/+/1295188

If somehow we can manage to have the filter files more visible to sheriffs, they can do the right thing from the very beginning and reduce the side effect of disabling a test.

Got it, thanks.

I think what we ultimately want to do here is have all tests that are disabled by sheriffs to be in filter files, and then we'll wire those up to sheriff-o-matic so you can disable tests by clicking buttons.

We'll probably still also disable tests in source files for things that are never meant to run on a given platform, though.

Dirk, I think you're proposing we have filter files for *all* suites, which we don't have. I know Ojan was a proponent of not disabling tests in code, but instead using something similar to filter files.

IMO, we should just move everything over to filter files that apply to the local dev's machine as well and tell people to stop disabling tests in C++ entirely. Having filter files that only apply to the bots means that local devs end up with a bunch of test flakes and that they rarely understand whether a passing trybot actually ran the test in question or not.

re #c4 - yes, eventually, I think we want to get to the point where every test is configured to have a filter file when run on the bots, so that we can make things work consistently for sheriffs and tooling (as per ojan's comments in #c5).

I owe people a larger design doc / discussion around this, because a lot of people have only heard bits and pieces of it.

@ojan - I don't think we should move away from disabling in c++ entirely. I think that's the right way to handle tests that are *never* meant to run (i.e., NeverFixTests), because managing those things via a filter file is less clear and a pain. I don't think temporarily disabling tests until they are fixed in source code is good, either, it's too hard to build tooling around that, but you don't really need tooling for tracking stuff that's never meant to work.

Added to chrome-client-infra-backlog hotlist for tracking, and marking Available so it's not in our triage queue.

Do I understand correctly that I>C>C here is for migrating to filter files, and I>S>S - for surfacing it in SoM? Maybe it's better to file two separate bugs under an umbrella?

Yes, that sounds right.

(Adding Milestone for SoM triage)