New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 737095 link

Starred by 1 user
Status: WontFix
Owner:
ilnik@chromium.org
Closed: Jun 2017
Cc:
█ kjellander@chromium.org
█ srte@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression



Sign in to add a comment

simulcast_vp8_3sl_low perf test sensitive to run order

Project Member Reported by srte@chromium.org, Jun 27 2017 
When simulcast_vp8_3sl_low is run after simulcast_vp8_3sl_medium, the  sender_time and  encode_frame_rate seems to go up on some systems. 

Compare the files
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fclient.webrtc.perf%2FAndroid32_Tests__K_Nexus5_%2F2668%2F%2B%2Frecipes%2Fsteps%2Fwebrtc_perf_tests%2F0%2Fstdout
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fclient.webrtc.perf%2FAndroid32_Tests__K_Nexus5_%2F2669%2F%2B%2Frecipes%2Fsteps%2Fwebrtc_perf_tests%2F0%2Fstdout

Search for "sender_time: simulcast_vp8_3sl_low" to see how the value changes depending on whether FullStackTest.VP9SVC_3SL_Medium has been run immediately before.

Comment 1 by srte@chromium.org, Jun 27 2017 

All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=737095

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDgjqnDsgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDg9sm6swkM


Bot(s) for this bug's original alert(s):

webrtc-android-tests-nexus5-kitkat

Comment 2 by ilnik@chromium.org, Jun 27 2017 

I don't see any significant change in sender_time in provided files. 89ms vs 92ms is negligible, considering that deviation is around 20 there (see the second number in brackets, e.g 20.539674 in 'sender_time: simulcast_vp8_3sl_low = {89.640916, 20.539674}')

But alas, some interfering among tests is possible via, e.g., thermal throttling on android devices. These simulcast tests are heavy. But it's very unlikely and I don't see it in provided logs.

Comment 3 by ilnik@chromium.org, Jun 27 2017

Cc: srte@chromium.org
Status: WontFix (was: Assigned)

Comment 4 by ilnik@chromium.org, Jun 27 2017

Status: Assigned (was: WontFix)

Comment 5 by ilnik@chromium.org, Jun 27 2017

Cc: kjellander@chromium.org
Update:
Even if we can't see any significant difference in provided files, we could see them in the graphs.

+kjellander,
I am pretty sure it's thermal throttling or something like that. Because we see the same increase in average encode_time around point 18646: https://chromeperf.appspot.com/report?sid=4c71df5dbd8407a5d1317f65117f457ef1b50e1fe69c8550fa44af07645d2e0e

Can we add some cooldown periods between individual tests on android bots? Or should we add some warm-ups to each perf test on android to always overload android devices? WDYT?

Comment 6 by kjellander@chromium.org, Jun 27 2017

Status: WontFix (was: Assigned)
I'm not sure the thermal throttling explains this behavior. It would rather explain the variation we're seeing between runs.

For this particular device we have 6 unit connected to the Linux machine that have the test sharded among themselves. 2 were offline for this particular run: https://build.chromium.org/p/client.webrtc.perf/builders/Android32%20Tests%20%28K%20Nexus5%29/builds/2709

Either way it seems these metrics have also recovered a bit later, tracked in bug 735433 so we can close this.
►

Sign in to add a comment