If we want to do this, we need a reasonable way to identify high-impact bugs that satisfies these properties:
1. Based on existing information (i.e. bug labels), adding more labels won't fly due to the difficulty to get people to use them and complicating the process.
2. Low false-positive rate, otherwise we'd be losing the benefits of automation
3. Reasonably low false-negative, otherwise the whole exercise is pointless.

Existing signals I'm aware that we may consider:

1. Priority. We could require manual review for all P0 bugs. This wouldn't have kicked in on  issue 722261  though which was high-impact but not critical urgency since the response work took place across several months.

2. Security labels. The only label that makes sense to trigger this is Restrict-View-SecurityEmbargo. Sheriffbot already skips issues marked with that label, so nothing to improve here.

3. Restrict-View-Google. Probably used too frequently and would cause a significant number of false positives.

4. Some heuristics on CC list: Bugs with lots of @google.com CCs are more likely to be sensitive. Maybe we could consider requiring manual review for bugs with more than 5 @google.com CCs?

5. Figuring out from comment history whether the bug affects external entities (partners, upstream projects, etc.)? Hard to do reliably without significant investment. Also highest-impact security bugs tend to omit partner information as a measure of caution.


palmer@, jorgelo@ any other ideas? IMHO, of the above only #4 makes some sense (even though I'm not really convinced this'll work in pratice).

Quick question given signal #2: Are we interested in catching high-impact bugs that may be mis-categorized (not already labeled "Security" related) or are we more interested in bugs that are already "Security" labeled but having some automated process for detecting when those bugs might be high-enough-impact to mandate manual approval (i.e., missing Security_Severity labels)?

(Or perhaps some combination of both?)

To throw a couple ideas on the wall:

Do we have a way to find all bugs that were Restrict-View-Security, then made public, and then re-restricted? I'm not sure if this is feasible with the search functionality though. If we had a set of true-positives to look at, a set of mutual heuristics might be clearer and allow us to tweak how "noisy" the manual approvals are. (I'm imagining something vaguely inspired by the anomaly scoring system of Ho et al https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/ho).

(On a related note: An archive of the issue tracker in something like BigQuery could be very useful for this kind of stuff, but we might have to be careful to not have it be another venue for leaking non-public issues.)