Issue metadata
Sign in to add a comment
|
Benchmark regressions due to Android devices getting hot |
||||||||||||||||||||||
Issue descriptionSee the link to graphs below.
,
Apr 24 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8981460718411921840
,
Apr 24 2017
Again, the only interesting V8 change seems to be https://codereview.chromium.org/2827263004. Bisection running.
,
Apr 24 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: android_nexus5X_perf_bisect Benchmark : v8.runtimestats.browsing_mobile_classic Metric : GC:duration_avg/browse_news/browse_news_toi Revision Result N chromium@466004 1933.97 +- 2033.14 21 good chromium@466184 1980.82 +- 1906.33 21 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.news.toi v8.runtimestats.browsing_mobile_classic Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8981460718411921840 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5022202103070720 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 24 2017
Ryan, can you find someone to look into this? Seems to be on the Chrome side.
,
Apr 24 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8981445042050786544
,
Apr 24 2017
#3: Nope, https://codereview.chromium.org/2827263004 is *not* in the regression range "466005:466184" (it rolled in at 466514). The fact that "everything" tanked at the same time here (e.g.: IC misses, GC, bytecode generation, bytecode execution, optimized compilation, Blink calls, V8 C++ operations) makes it pretty unlikely that any one CL would have caused this.
,
Apr 24 2017
The lack of reference build makes it difficult to know for sure. +sullivan, benhenry do we know of anything that changed on the bots around this time that could explain the shift?
,
Apr 24 2017
I checked the buildbot status pages, there do not seem to be any device changes or OS flashes between the builds: https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20%281%29/builds/4783 https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5X%20Perf%20%281%29/builds/4784 I also don't see anything related to the benchmark or catapult in the cl range: https://chromium.googlesource.com/chromium/src/+log/975b115ff82c6d407cd346afb4d4838f5d380eab%5E..a6c3346f39d2e05d591d3719f4756b55916ad313?pretty=fuller +Stephen, Ned: could a recipe change have affected N5X?
,
Apr 24 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: android_nexus5X_perf_bisect Benchmark : v8.runtimestats.browsing_mobile_classic Metric : GC:duration_avg/browse_news/browse_news_toi Revision Result N chromium@466004 2065.39 +- 2634.53 21 good chromium@466184 2168.24 +- 1553.03 21 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.news.toi v8.runtimestats.browsing_mobile_classic Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8981445042050786544 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5022202103070720 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 25 2017
If bisect retry "before" & "after" revisions & can't find the difference, we can be pretty much sure that the change due to some platform/recipe change. Though I can't really tell what are the platform level differences without at least the android trace. However, I am not sure why the bisect fails to find the regression here? Revision Result N chromium@466004 2065.39 +- 2634.53 21 good chromium@466184 2168.24 +- 1553.03 21 bad
,
Apr 25 2017
Hmhh, it's worth noting that the temperature of the devices are clearly different here: Text search "Current battery temperature" in Before regression points: https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5X_Perf__1_%2F4756%2F%2B%2Frecipes%2Fsteps%2Fv8.runtimestats.browsing_mobile_classic%2F0%2Fstdout Temperature: 227 -> 260 https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5X_Perf__1_%2F4767%2F%2B%2Frecipes%2Fsteps%2Fv8.runtimestats.browsing_mobile_classic%2F0%2Fstdout Temperature: 222 -> 265 After regression points: https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5X_Perf__1_%2F4786%2F%2B%2Frecipes%2Fsteps%2Fv8.runtimestats.browsing_mobile_classic%2F0%2Fstdout Temperature: 335 -> 350 https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5X_Perf__1_%2F4785%2F%2B%2Frecipes%2Fsteps%2Fv8.runtimestats.browsing_mobile_classic%2F0%2Fstdout Temperature: 317 -> 350 (these datapoint are from https://chromeperf.appspot.com/report?sid=cb29af7a81f139fcc8b7bdfdb22897b5a2a48e037f38678081d1bba326ed39a7)
,
Apr 25 2017
,
Apr 25 2017
Hmhh, I think we can try to make the device cooler through waiting longer and see if that "fix" the regression. Probably 25 C instead of 35 C as in https://github.com/catapult-project/catapult/blob/master/telemetry/telemetry/internal/story_runner.py#L235? The down side is that probably increase the cycle time of our benchmark run (issue 713345). Though I still it still worth trying out to cool the device down further & see how much does cycle time increase. If it's too much, we can revert the CL & work with lab folks on how to make our devices cool faster (put them in the fridge? :P) If no one oppose, I will make a CL in 2 days from now.
,
Apr 25 2017
,
Apr 25 2017
,
Apr 25 2017
+vhang@ for what device temperature we can expect from the lab (see #14). We don't want to set 'wait until device temperature is 25 C' if the device rack's average temperature is 30C.
,
Apr 25 2017
Issue 714288 has been merged into this issue.
,
Apr 25 2017
,
Apr 25 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8981331559016282176
,
Apr 25 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: android_nexus5X_perf_bisect Benchmark : v8.runtimestats.browsing_mobile_classic Metric : API:duration_avg/browse_news/browse_news_toi Revision Result N chromium@466004 2065.39 +- 2634.53 21 good chromium@466184 2168.24 +- 1553.03 21 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.news.toi v8.runtimestats.browsing_mobile_classic Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8981331559016282176 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5022202103070720 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 25 2017
Sorry for my hasty conclusion about increasing the wait time for device to cool down. I think we need to sync up with labs on this topic to make sure that the lab's room temperature is well controlled, otherwise we may stall the bots waiting for a low temperature that can't be achieved in the lab. I will try to coordinate with the lab folks as the next steps. Assign this to me for now.
,
Apr 25 2017
I believe build73-b1 is a bot that you're concerned about. It has 7 N5X with battOrs. We have fans racked on the phone tray. We can look into high CFM fans to improve air flow. As for the room temperature, it's automatically regulated by the chillers. We can ask for a lower temperature but the current temp we have set has kept all the phones fairly healthy as long as we have a tray fan. Since your phones has battOrs, could it be the cause of the higher temperatures? I'll head into the lab today and see what we can do to improve the cooling of these devices.
,
Apr 25 2017
Thanks Vince for looking into the devices! +Aaron: please see #23, is it possible for battor to make the phone hotter?
,
Apr 26 2017
The slaves running this are as follows? All three are running the Sharded Perf Tests step. Checked the battery temperatures. The devices on build{74,75}-b1 are running much hotter and the devices on these two are in a different tray than those of build73-b1. I'll check the tray fans tomorrow morning.
chrome-bot@build73-b1: Android Nexus5X Perf (1)
Checking 01ffbe95c2bb3f2e...
temperature: 240
Checking 01e40655c29b4f29...
temperature: 197
Checking 01d30cfd7bc7d319...
temperature: 207
Checking 020035a11c7fb806...
temperature: 245
Checking 00d093103eb3ad71...
temperature: 232
Checking 020017d5e678c3d2...
temperature: 210
Checking 01e63b611c979218...
temperature: 222
chrome-bot@build74-b1: Android Nexus5X Perf (2)
Checking 01e2d69f26a8408e...
temperature: 337
Checking 020205bfa0bc13ec...
temperature: 337
Checking 00d433abbb812842...
temperature: 340
Checking 01e15d4fa2659c8d...
temperature: 332
Checking 020b198fc2930d27...
temperature: 350
Checking 01e0fbd77be7b503...
temperature: 302
Checking 020ad055c25b1129...
temperature: 347
chrome-bot@build75-b1: Android Nexus5X Perf (3)
Checking 01e18134a2a5948c...
temperature: 302
Checking 01e14fdcc2bb4526...
temperature: 342
Checking 01e1828ba23dac8c...
temperature: 335
Checking 01e607df1caf8c0a...
temperature: 310
Checking 01e450dc1c47be08...
temperature: 335
Checking 01e4b402cce744d9...
temperature: 350
Checking 020afd5fc2433722...
temperature: 307
,
Apr 26 2017
The tray fan for devices connected to build{74,75}-b1 was off. It's connection to the power supply is very touchy so I just swapped the entire fan assembly out.
It's only been a few min. but temperatures are coming down.
chrome-bot@build74-b1: Android Nexus5X Perf (2)
Checking 020205bfa0bc13ec...
temperature: 225
Checking 00d433abbb812842...
temperature: 290
Checking 01e15d4fa2659c8d...
temperature: 270
Checking 020b198fc2930d27...
temperature: 257
Checking 01e0fbd77be7b503...
temperature: 232
Checking 01e2d69f26a8408e...
temperature: 237
Checking 020ad055c25b1129...
temperature: 255
chrome-bot@build75-b1: Android Nexus5X Perf (3)
Checking 01e18134a2a5948c...
temperature: 245
Checking 01e14fdcc2bb4526...
temperature: 277
Checking 01e450dc1c47be08...
temperature: 292
Checking 01e1828ba23dac8c...
temperature: 280
Checking 01e607df1caf8c0a...
temperature: 257
Checking 020afd5fc2433722...
temperature: 275
Checking 01e4b402cce744d9...
temperature: 280
,
Apr 26 2017
Wow, the graphs are recovered now: https://chromeperf.appspot.com/group_report?bug_id=714497 Thanks for the quick response, Peter! Though I think we should have a better system of detecting issue like this. Filed a new bug to keep track of this: issue 715605
,
Apr 26 2017
Ahh, I forgot to post this earlier this morning. My apologies if I was sending confusing signals.
The tray fan for devices connected to build{74,75}-b1 was off. It's connection to the power supply is very touchy so I just swapped the entire fan assembly out.
,
May 2 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8980694400838651936
,
May 2 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: android_nexus5X_perf_bisect Benchmark : v8.runtimestats.browsing_mobile_classic Metric : API:duration_avg/browse_media/browse_media_imgur Revision Result N chromium@466004 2065.39 +- 2634.53 21 good chromium@466184 2168.24 +- 1553.03 21 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.media.imgur v8.runtimestats.browsing_mobile_classic Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8980694400838651936 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5022202103070720 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you! |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by bmeurer@google.com
, Apr 24 2017