New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 737075 link

Starred by 1 user
Status: WontFix
Owner: ----
Closed: Jun 2018
Cc:
█ kjellander@chromium.org
mflodman@chromium.org
sprang@chromium.org
brandtr@chromium.org
terelius@chromium.org
holmer@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: Bug-Regression



Sign in to add a comment

Used android device affects performance and causes invalid alerts.

Project Member Reported by srte@chromium.org, Jun 27 2017 
The choice of device used by the build bot affects the encode_time in linked graph (and likely others) this causes a lot of invalid alerts and can hide real performance regressions.

For some examples taken from logs at different times, note that although the build number is increasing, the reported value is correlated with the device identifier.

Build 2492: I  558.981s run_tests_on_device(07413249003b5789)  RESULT encode_time: simulcast_vp8_3sl_high = {71.818182, 7.732084} ms
Build 2502: I  578.538s run_tests_on_device(07413249003b5789)  RESULT encode_time: simulcast_vp8_3sl_high = {71.863636, 9.240698} ms

Build 2517: I  586.097s run_tests_on_device(0746ae42003bccf9)  RESULT encode_time: simulcast_vp8_3sl_high = {78.545455, 8.940344} ms
Build 2518: I  605.640s run_tests_on_device(0746ae42003bccf9)  RESULT encode_time: simulcast_vp8_3sl_high = {78.818182, 8.663593} ms

Build 2537: I  631.664s run_tests_on_device(03832b22f0e944a0)  RESULT encode_time: simulcast_vp8_3sl_high = {86.250000, 9.070745} ms
Build 2569: I  649.815s run_tests_on_device(03832b22f0e944a0)  RESULT encode_time: simulcast_vp8_3sl_high = {87.795455, 12.004885} ms

Build 2577: I  609.851s run_tests_on_device(0746ae42003bccf9)  RESULT encode_time: simulcast_vp8_3sl_high = {80.795455, 12.151654} ms
Build 2582: I  616.428s run_tests_on_device(0746ae42003bccf9)  RESULT encode_time: simulcast_vp8_3sl_high = {79.840909, 10.860569} ms

Build 2595: I  667.596s run_tests_on_device(03832b22f0e944a0)  RESULT encode_time: simulcast_vp8_3sl_high = {87.545455, 8.459676} ms
Build 2596: I  684.539s run_tests_on_device(03832b22f0e944a0)  RESULT encode_time: simulcast_vp8_3sl_high = {87.545455, 8.566464} ms

Comment 1 by srte@chromium.org, Jun 27 2017 

All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=737075

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgtqiFvQsM


Bot(s) for this bug's original alert(s):

webrtc-android-tests-nexus5-kitkat

Comment 2 by kjellander@chromium.org, Jun 27 2017

Cc: kjellander@chromium.org
Labels: OS-Android
Owner: ----
Status: Available (was: Assigned)
We're aware of that there are device differences and it is possible to run with a single device per device bot. The downside with that approach is:
1) the device will get very hot running 30-40 minute 100% CPU tests.
2) devices go offline every now and then, in that case we'll loose coverage and end up with large blamelists.

I'm open to discussing cutting down on the devices in favor for single-device tests. It's a call to make for the webrtc-perf-sheriffs@, not me. 

Do you have proof for this variance being not cause by a regression? IT seems one of the changes in the 3-revision span was causing a lot of alerts: https://chromeperf.appspot.com/group_report?rev=18632
Project Member

Comment 3 by sheriffbot@chromium.org, Jun 27 2018

Labels: Hotlist-Recharge-Cold
Status: Untriaged (was: Available)
This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue.

Sorry for the inconvenience if the bug really should have been left as Available.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 4 by tedc...@chromium.org, Jun 27 2018

Components: Blink>WebRTC>Video
Status: WontFix (was: Untriaged)
Based on the lack of feedback in #2, I'm going to close this out.  Otherwise, it should be assigned to someone on WebRTC to investigate.
►

Sign in to add a comment