Purpose: Avoid excess noise from Tricium results.
According to the paper about Tricorder from 2014:
We define the “not-useful rate” of an analyzer as:
NOT USEFUL/(NOT USEFUL + PLEASE FIX + APPLY FIX)
Analysis writers are expected to check these numbers
through a dashboard we provide. A rate ≥ 10% puts the analyzer
on probation, and the analysis writer must show progress
toward addressing the issue.
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43322.pdf
In the specific case of Tricium, we don't (yet) have a "please fix" or "apply fix" signal, so the number of "not useful" clicks for a particular category is the main signal we have.
Design decisions:
- Should the filtering be based only on the result "category" string? This may be the simplest and most transparent to describe to analzyer writers.
- How many "not useful" clicks will be sufficient for probation?
- What are the conditions for lifting probation?
Comment 1 by qyears...@chromium.org
, May 15 2018Summary: Automatic filtering/probation of analyzers based on "not useful" feedback (was: Automatic filtering/probation of results based on "not useful" feedback)