New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 707157 link

Starred by 1 user
Status: WontFix
Owner:
█ kjellander@chromium.org
Last visit > 30 days ago
Closed: Apr 2017
Cc:
█ kjellander@chromium.org
█ ivoc@chromium.org
hlundin@chromium.org
peah@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: Bug-Regression



Sign in to add a comment

WebRTC apm_timing perf tests are randomly changing

Project Member Reported by hlundin@chromium.org, Mar 31 2017 
Several of the apm_timing tests, in particular the ones running on mobile platforms, are showing changes that have no correlation at all with code changes. They can be both positive and negative changes.

This seems to be a general problem with these tests, or how they are run on bots, that something other than the code under test changes from time to time. How can we make those tests more robust?

Comment 1 by hlundin@chromium.org, Mar 31 2017 

All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=707157

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgnPbb9gkM


Bot(s) for this bug's original alert(s):

webrtc-android-tests-nexus5-kitkat

Comment 2 by hlundin@chromium.org, Mar 31 2017

Cc: kjellander@chromium.org
 Issue 705882  has been merged into this issue.

Comment 3 by hlundin@chromium.org, Mar 31 2017

Components: Blink>WebRTC>Audio Blink
Labels: OS-Android

Comment 4 by hlundin@chromium.org, Mar 31 2017 

 Issue 705471  has been merged into this issue.

Comment 5 by rtoy@chromium.org, Apr 4 2017

Components: -Blink

Comment 6 by kjellander@chromium.org, Apr 18 2017

Status: WontFix (was: Assigned)
I suspect this is either caused by slightly different device characteristics (since these bots have >1 device connected) and/or temperature changes on the devices due to high load.

I wanted to look at logs to see how many devices were used but 712551 prevented me from doing so. The graphs have recovered to their previous behavior now so at least it was temporary.

In general we might get a little more stable metrics if we limit ourselves to one device per bot, but then in turn we get a higher load on that device and queued builds and/or longer blamelists when the device goes down. I think the current setup is better.
►

Sign in to add a comment