Here are some notes of the discussions on this topic.

Things discussed:
How to decide which flaky tests to disable?
  1. Depends on how we define flakiness.
  2. liaoyuke@ is working on FindIt Flake Detection, and one of the goals is to measure the flakiness of detected flaky tests and rank them accordingly. Once we have the data, we will have a better understanding of the problem space, and can then design a policy.

Can we automatically disable gtests in source code?
  1. Very hard. There are too many corner cases such as Macros and disable only on a specific platform. A  better way to go is to support test expectation files for gtests.

What is the status of supporting test expectation files for gtests?
  1. We're not there yet, there were people who were against this, but they were mostly convinced, and  the more we can show expectation format are useful, the easier for us to convince everyone dpranke@ is working on it, but no ETA.

Automatically disable tests might be risky because if we're not careful, we may eventually disable all tests.

With making test disabling easier & frictionless for sheriffs, we need to make sure we track state of test disabling & report it back to test owners
  1. lindsayw@ already implemented something to track disabled tests. We can sync with them on this.
  2. Letting Sheriffs manually disable flaky tests is too much work because they need to have the checkout and need to compile.
  3. Having a one-click button on Sheriff-o-matic to easily disable flaky tests would be huge win! seanmccullough@ already did a bunch of work to easily disable webkit_layout_tests. It may not be working properly, but his work can be reused.

Conclusion: it is a little bit too early to work on automatically disable flaky tests because it depends on liaoyuke@'s work to rank flaky tests and dpranke@'s work to support test expectation files for gtests. Nothing will happen now, but will revisit this topic in Q4.