Issue metadata
Sign in to add a comment
|
9.1%-35.4% regression in webrtc_perf_tests at 20764:20764 |
||||||||||||||||||||
Issue descriptionRegression at chromium roll followed by recovery a few chromium rolls later. However, both chromium rolls are virtually empty. Nearby CLs are also irrelevant. Should we stop testing on the device if it doesn't produce any useful data?
,
Dec 4 2017
I looked back in the graphs, and there were ~5 alerts the past 5 months. Out of those, 3 were legit. I think that's an acceptable S/N ratio.
,
Dec 4 2017
Out of those 3, how many were only detected on the nexus 5? Moreover, if there is no cause and no action to be taken then maybe sheriffs have been ignoring alerts instead of filing bugs? Fwiw, I have had at least 3 alerts involving the nexus 5 during the last week, all of them false. Note that it isn't noise in the normal sense of a drawing random values from a distribution. Here there are extremely clear and persistent changes in the metric without any apparent cause. It would be good to at least understand what affects the tests in this way. |
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by 42576172...@developer.gserviceaccount.com
, Dec 1 2017