New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 791122 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Last visit > 30 days ago
Closed: Dec 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression



Sign in to add a comment

9.1%-35.4% regression in webrtc_perf_tests at 20764:20764

Project Member Reported by terelius@chromium.org, Dec 1 2017

Issue description

Regression at chromium roll followed by recovery a few chromium rolls later. However, both chromium rolls are virtually empty. Nearby CLs are also irrelevant.

Should we stop testing on the device if it doesn't produce any useful data?
 
All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=791122

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=c5cda4d40dcd9b1d03bde8f4c7c758653c9e6248d6946379ecf91890812eb6b7


Bot(s) for this bug's original alert(s):

webrtc-android-tests-nexus5-kitkat
Status: WontFix (was: Assigned)
I looked back in the graphs, and there were ~5 alerts the past 5 months. Out of those, 3 were legit. I think that's an acceptable S/N ratio.
Out of those 3, how many were only detected on the nexus 5? Moreover, if there is no cause and no action to be taken then maybe sheriffs have been ignoring alerts instead of filing bugs?

Fwiw, I have had at least 3 alerts involving the nexus 5 during the last week, all of them false.

Note that it isn't noise in the normal sense of a drawing random values from a distribution. Here there are extremely clear and persistent changes in the metric without any apparent cause. It would be good to at least understand what affects the tests in this way.

Sign in to add a comment