File this meta bug to track adding perf testing on ChromeOS CQ to track regressions.
Few examples of large regression that may have been prevented earlier if we reject them through CQ:
https://bugs.chromium.org/p/chromium/issues/detail?id=921550#c1 (86.2% regression in shutdown)
https://bugs.chromium.org/p/chromium/issues/detail?id=920934 (19% regression in booting)
Some discussion points we already have:
+ Once a system is in place, there's a tendency to abuse it by adding a lot of metrics, and tracking small regressions. At the pre-submit or CQ level, we need to run a small number of benchmarks that truly measure user experience (time to login, FPS, scrolling speed etc), with an acceptable threshold value.
+ We need to be good at dealing with noise, or picking a good rejection strategy to avoid falsely reject the CL when regression is just due to noise
+ Adding a fix threshold would be problematic when slow creep tipping the threshold over.
+ Alerting algorithm used in perf dashboard: http://go/perf-dashboard-data
+ We need more elastic fleet capacity before we can tackle this.
Comment 1 by jclinton@chromium.org
, Jan 15