Issue metadata
Sign in to add a comment
|
10.9% regression in power.typical_10_mobile at 1527628978:1527641281 |
||||||||||||||||||||
Issue descriptionSee the link to graphs below.
,
May 31 2018
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8945006103813554784
,
Jun 1 2018
=== BISECT JOB RESULTS === Bisect was unable to run to completion Error: INFRA_FAILURE The bisect was able to narrow the range, you can try running with: good_revision: 53ce5696671da9931a1cb01d602128ef1a95db74 bad_revision : 6cf25da95bee042ec69811e16116fdadb0ea600d If failures persist contact the team (see below) and report the error. Bisect Details Configuration: clankium-low-end-phone-perf-bisect Benchmark : power.typical_10_mobile Metric : application_energy_consumption_mwh/application_energy_consumption_mwh Change : 11.33% | 0.260303736364 -> 0.289806062121 Revision Result N android-chrome@4ab819e0f9 0.260304 +- 0.000649818 6 good android-chrome@4ab819e0f9,chromium@562238 0.263273 +- 0.00290753 6 good android-chrome@4ab819e0f9,chromium@562249 0.288257 +- 0.0044292 6 bad android-chrome@4ab819e0f9,chromium@562259 0.289123 +- 0.00301661 6 bad android-chrome@4ab819e0f9,chromium@562301 0.289411 +- 0.00159778 6 bad android-chrome@471ef73f11 0.289806 +- 0.00229887 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chrome --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests power.typical_10_mobile More information on addressing performance regressions: http://g.co/ChromePerformanceRegressions Debug information about this bisect: https://chromeperf.appspot.com/buildbucket_job_status/8945006103813554784 For feedback, file a bug with component Speed>Bisection
,
Jun 1 2018
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8944914162275919632
,
Jun 1 2018
Trying a bisect with a story filter, hopefully will be shorter+more accurate.
,
Jun 2 2018
=== Auto-CCing suspected CL author gab@chromium.org === Hi gab@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : Gabriel Charette Commit : 7e7dd55cce56f643d5945f5a2dbd42d475c73e1b Date : Mon May 28 13:39:21 2018 Subject: [TaskScheduler] Slower latency heartbeat Bisect Details Configuration: clankium-low-end-phone-perf-bisect Benchmark : power.typical_10_mobile Metric : application_energy_consumption_mwh/http___m.huffpost.com_us_entry_6004486 Change : 12.72% | 0.236305016667 -> 0.266359483333 Revision Result N android-chrome@4ab819e0f9 0.236305 +- 0.00445757 6 good android-chrome@4ab819e0f9,chromium@562238 0.237206 +- 0.00160218 6 good android-chrome@4ab819e0f9,chromium@562244 0.236281 +- 0.00571958 6 good android-chrome@4ab819e0f9,chromium@562247 0.238065 +- 0.00425235 6 good android-chrome@4ab819e0f9,chromium@562248 0.264031 +- 0.00233938 6 bad <-- android-chrome@4ab819e0f9,chromium@562249 0.273173 +- 0.00776998 6 bad android-chrome@4ab819e0f9,chromium@562259 0.267757 +- 0.00832359 6 bad android-chrome@4ab819e0f9,chromium@562301 0.265708 +- 0.00513507 6 bad android-chrome@471ef73f11 0.266359 +- 0.00429064 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chrome --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http...m.huffpost.com.us.entry.6004486 power.typical_10_mobile More information on addressing performance regressions: http://g.co/ChromePerformanceRegressions Debug information about this bisect: https://chromeperf.appspot.com/buildbucket_job_status/8944914162275919632 For feedback, file a bug with component Speed>Bisection
,
Jun 4 2018
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8944617910281270576
,
Jun 5 2018
=== BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : Gabriel Charette Commit : 7e7dd55cce56f643d5945f5a2dbd42d475c73e1b Date : Mon May 28 13:39:21 2018 Subject: [TaskScheduler] Slower latency heartbeat Bisect Details Configuration: clankium-low-end-phone-perf-bisect Benchmark : power.typical_10_mobile Metric : application_energy_consumption_mwh/application_energy_consumption_mwh Change : 8.83% | 0.262402960606 -> 0.285575851515 Revision Result N android-chrome@4ab819e0f9 0.262403 +- 0.00118833 6 good android-chrome@4ab819e0f9,chromium@562238 0.262249 +- 0.00126189 6 good android-chrome@4ab819e0f9,chromium@562244 0.263222 +- 0.00153264 6 good android-chrome@4ab819e0f9,chromium@562247 0.264324 +- 0.00284169 6 good android-chrome@4ab819e0f9,chromium@562248 0.289469 +- 0.00326781 6 bad <-- android-chrome@4ab819e0f9,chromium@562249 0.28869 +- 0.00356045 6 bad android-chrome@4ab819e0f9,chromium@562259 0.288626 +- 0.00441652 6 bad android-chrome@4ab819e0f9,chromium@562301 0.290389 +- 0.00230635 6 bad android-chrome@471ef73f11 0.285576 +- 0.00512216 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chrome --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests power.typical_10_mobile More information on addressing performance regressions: http://g.co/ChromePerformanceRegressions Debug information about this bisect: https://chromeperf.appspot.com/buildbucket_job_status/8944617910281270576 For feedback, file a bug with component Speed>Bisection
,
Jun 7 2018
@benhenry : who's the right contact for "health-plan-clankium-low-end-phone" from the System Health metrics? I can't figure out why this would have regressed @ r562248 but not @ r560052 (the latest CL is merely making the initial CL less aggressive...). Is this using Clang as a compiler? A static initializer for |g_heartbeat_for_testing| is the only thing I can see... but even then... not 10% difference...
,
Jun 7 2018
Adding Charlie and Maria, as I don't know anyone clank specific.
,
Jun 7 2018
Swapping me out for Juan (who owns this benchmark)
,
Jun 7 2018
,
Jun 18 2018
It should be using clang. We only support clang on Android these days. My only guess as to why this caused a regression, but the previous CL seems to coincide with an improvement alert is because something else landed at the same time as the regression to make the "net" improvement, see this graph: https://chromeperf.appspot.com/group_report?sid=3b04a394179538536eb72ebc4c7a33913ff5dcc85145d95666f5aea61823d770
,
Jun 18 2018
+thildebr FYI
,
Jun 19 2018
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8943326880668236720
,
Jun 19 2018
=== BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : Gabriel Charette Commit : 7e7dd55cce56f643d5945f5a2dbd42d475c73e1b Date : Mon May 28 13:39:21 2018 Subject: [TaskScheduler] Slower latency heartbeat Bisect Details Configuration: clankium-low-end-phone-perf-bisect Benchmark : power.typical_10_mobile Metric : application_energy_consumption_mwh/application_energy_consumption_mwh Change : 10.34% | 0.261519163636 -> 0.288566225758 Revision Result N android-chrome@4ab819e0f9 0.261519 +- 0.000629634 6 good android-chrome@4ab819e0f9,chromium@562238 0.261649 +- 0.00108604 6 good android-chrome@4ab819e0f9,chromium@562244 0.2632 +- 0.000886633 6 good android-chrome@4ab819e0f9,chromium@562247 0.262342 +- 0.00123737 6 good android-chrome@4ab819e0f9,chromium@562248 0.288006 +- 0.00475372 6 bad <-- android-chrome@4ab819e0f9,chromium@562249 0.289111 +- 0.00262294 6 bad android-chrome@4ab819e0f9,chromium@562259 0.289593 +- 0.00526017 6 bad android-chrome@4ab819e0f9,chromium@562301 0.292543 +- 0.00314437 6 bad android-chrome@471ef73f11 0.288566 +- 0.00864792 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chrome --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests power.typical_10_mobile More information on addressing performance regressions: http://g.co/ChromePerformanceRegressions Debug information about this bisect: https://chromeperf.appspot.com/buildbucket_job_status/8943326880668236720 For feedback, file a bug with component Speed>Bisection
,
Jul 11
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8941266239912830128
,
Jul 11
The only thing I can see is the call to RandUtil which although it's only invoked 59 minutes after startup may change the loader's logic (not sure why that'd affect much of anything beyond startup performance though... and specifically not power..) There's a matching improvement on the graph in https://chromium.googlesource.com/chromium/src/+log/9de442989747113525ec0b18fd15b93dd9677a5a%5E..e580db8610d8033ada999e5717b9dad7bef78799?pretty=fuller&n=1000 I launched a bisect to see what that pinpoints...
,
Jul 12
=== BISECT JOB RESULTS === Bisect was unable to run to completion Please try rerunning the bisect. If failures persist contact the team (see below) and report the error. Bisect Details Configuration: clankium-low-end-phone-perf-bisect Benchmark : power.typical_10_mobile Metric : application_energy_consumption_mwh/application_energy_consumption_mwh Change : 4.99% | 0.276934519697 -> 0.263118548485 Revision Result N android-chrome@1d65273090 0.276935 +- 0.003409 6 good android-chrome@1d65273090,chromium@565290 0.264611 +- 0.00123219 6 bad android-chrome@1d65273090,chromium@565293 0.262888 +- 0.001518 6 bad android-chrome@1d65273090,chromium@565299 0.264028 +- 0.00164243 6 bad android-chrome@1d65273090,chromium@565312 0.263796 +- 0.00105606 6 bad android-chrome@1d65273090,chromium@565338 0.263999 +- 0.00177075 6 bad android-chrome@1d65273090,chromium@565390 0.264021 +- 0.0021557 6 bad android-chrome@1d65273090,chromium@565493 0.263182 +- 0.00138846 6 bad android-chrome@ffc18f550f 0.263119 +- 0.00146991 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chrome --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests power.typical_10_mobile More information on addressing performance regressions: http://g.co/ChromePerformanceRegressions Debug information about this bisect: https://chromeperf.appspot.com/buildbucket_job_status/8941266239912830128 For feedback, file a bug with component Speed>Bisection
,
Jul 12
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8941174237205828224
,
Jul 12
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8941174227845988768
,
Jul 14
=== Auto-CCing suspected CL author gab@chromium.org === Hi gab@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : Gabriel Charette Commit : 7e7dd55cce56f643d5945f5a2dbd42d475c73e1b Date : Mon May 28 13:39:21 2018 Subject: [TaskScheduler] Slower latency heartbeat Bisect Details Configuration: clankium-low-end-phone-perf-bisect Benchmark : power.typical_10_mobile Metric : application_energy_consumption_mwh/application_energy_consumption_mwh Change : 10.89% | 0.261007684848 -> 0.289436162121 Revision Result N android-chrome@4ab819e0f9 0.261008 +- 0.000857321 6 good android-chrome@4ab819e0f9,chromium@562238 0.260563 +- 0.000705119 6 good android-chrome@4ab819e0f9,chromium@562244 0.261632 +- 0.00109015 6 good android-chrome@4ab819e0f9,chromium@562247 0.261036 +- 0.00095427 6 good android-chrome@4ab819e0f9,chromium@562248 0.288839 +- 0.00586059 6 bad <-- android-chrome@4ab819e0f9,chromium@562249 0.286398 +- 0.00250888 6 bad android-chrome@4ab819e0f9,chromium@562259 0.289301 +- 0.00290086 6 bad android-chrome@4ab819e0f9,chromium@562301 0.289559 +- 0.00362998 6 bad android-chrome@471ef73f11 0.289436 +- 0.00257839 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chrome --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests power.typical_10_mobile More information on addressing performance regressions: http://g.co/ChromePerformanceRegressions Debug information about this bisect: https://chromeperf.appspot.com/buildbucket_job_status/8941174237205828224 For feedback, file a bug with component Speed>Bisection
,
Jul 14
=== BISECT JOB RESULTS === Bisect was unable to run to completion Please try rerunning the bisect. If failures persist contact the team (see below) and report the error. Bisect Details Configuration: clankium-low-end-phone-perf-bisect Benchmark : power.typical_10_mobile Metric : application_energy_consumption_mwh/application_energy_consumption_mwh Change : 5.98% | 0.279413145455 -> 0.262696021212 Revision Result N android-chrome@1d65273090 0.279413 +- 0.00544378 6 good android-chrome@1d65273090,chromium@565338 0.263055 +- 0.000622299 6 bad android-chrome@1d65273090,chromium@565390 0.26327 +- 0.00111397 6 bad android-chrome@1d65273090,chromium@565493 0.262958 +- 0.00115782 6 bad android-chrome@ffc18f550f 0.262696 +- 0.000592699 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chrome --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests power.typical_10_mobile More information on addressing performance regressions: http://g.co/ChromePerformanceRegressions Debug information about this bisect: https://chromeperf.appspot.com/buildbucket_job_status/8941174227845988768 For feedback, file a bug with component Speed>Bisection
,
Jul 18
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/548db20100afee184fcda974d60a06345b05c597 commit 548db20100afee184fcda974d60a06345b05c597 Author: Gabriel Charette <gab@chromium.org> Date: Wed Jul 18 19:16:43 2018 [TaskScheduler] Temporarily disable heartbeat latency metrics on Android to bisect power regression R=fdoray@chromium.org Bug: 848255 Change-Id: Ic47d338fb15e0bf3579ecd9d1acca67099c6d570 Reviewed-on: https://chromium-review.googlesource.com/1137849 Reviewed-by: Robert Liao <robliao@chromium.org> Commit-Queue: Gabriel Charette <gab@chromium.org> Cr-Commit-Position: refs/heads/master@{#576158} [modify] https://crrev.com/548db20100afee184fcda974d60a06345b05c597/base/task_scheduler/service_thread.cc [modify] https://crrev.com/548db20100afee184fcda974d60a06345b05c597/base/task_scheduler/service_thread_unittest.cc
,
Jul 19
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8940568883479032272
,
Jul 20
=== Auto-CCing suspected CL author gab@chromium.org === Hi gab@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : Gabriel Charette Commit : 903fd04e262f060fecf5ed75549076681e63049d Date : Thu Jun 07 16:24:08 2018 Subject: [TaskScheduler] Consider the idle thread on top of the idle stack as active. Bisect Details Configuration: clankium-low-end-phone-perf-bisect Benchmark : power.typical_10_mobile Metric : application_energy_consumption_mwh/application_energy_consumption_mwh Change : 7.28% | 0.281980236364 -> 0.261438289394 Revision Result N android-chrome@1d65273090 0.28198 +- 0.00425623 6 good android-chrome@1d65273090,chromium@565287 0.282939 +- 0.00280111 6 good android-chrome@1d65273090,chromium@565288 0.262591 +- 0.000655372 6 bad <-- android-chrome@1d65273090,chromium@565290 0.262068 +- 0.000964619 6 bad android-chrome@1d65273090,chromium@565293 0.264195 +- 0.00195476 6 bad android-chrome@1d65273090,chromium@565299 0.263917 +- 0.00184747 6 bad android-chrome@1d65273090,chromium@565312 0.262695 +- 0.000409379 6 bad android-chrome@1d65273090,chromium@565338 0.262512 +- 0.00120573 6 bad android-chrome@1d65273090,chromium@565390 0.262156 +- 0.0010626 6 bad android-chrome@1d65273090,chromium@565493 0.261616 +- 0.00158751 6 bad android-chrome@ffc18f550f 0.261438 +- 0.00109021 6 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chrome --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests power.typical_10_mobile More information on addressing performance regressions: http://g.co/ChromePerformanceRegressions Debug information about this bisect: https://chromeperf.appspot.com/buildbucket_job_status/8940568883479032272 For feedback, file a bug with component Speed>Bisection
,
Jul 24
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d21e49e7ff9ffc7cce7c48a1191d5f33c6904f7c commit d21e49e7ff9ffc7cce7c48a1191d5f33c6904f7c Author: Gabriel Charette <gab@chromium.org> Date: Tue Jul 24 16:51:20 2018 Revert "[TaskScheduler] Temporarily disable heartbeat latency metrics on Android to bisect power regression" This reverts commit 548db20100afee184fcda974d60a06345b05c597. Reason for revert: regression had recovered on its own and this CL doesn't seem to help further : https://chromeperf.appspot.com/group_report?sid=3b04a394179538536eb72ebc4c7a33913ff5dcc85145d95666f5aea61823d770 Original change's description: > [TaskScheduler] Temporarily disable heartbeat latency metrics on Android to bisect power regression > > R=fdoray@chromium.org > > Bug: 848255 > Change-Id: Ic47d338fb15e0bf3579ecd9d1acca67099c6d570 > Reviewed-on: https://chromium-review.googlesource.com/1137849 > Reviewed-by: Robert Liao <robliao@chromium.org> > Commit-Queue: Gabriel Charette <gab@chromium.org> > Cr-Commit-Position: refs/heads/master@{#576158} TBR=gab@chromium.org,robliao@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. Bug: 848255 Change-Id: Ie402a1ec2381f296863522bcf9ba7c387648df6d Reviewed-on: https://chromium-review.googlesource.com/1148481 Reviewed-by: Gabriel Charette <gab@chromium.org> Commit-Queue: Gabriel Charette <gab@chromium.org> Cr-Commit-Position: refs/heads/master@{#577581} [modify] https://crrev.com/d21e49e7ff9ffc7cce7c48a1191d5f33c6904f7c/base/task_scheduler/service_thread.cc [modify] https://crrev.com/d21e49e7ff9ffc7cce7c48a1191d5f33c6904f7c/base/task_scheduler/service_thread_unittest.cc
,
Jul 25
,
Jul 26
If the regression has recovered, can we close this out?
,
Jul 26
Yep, closing
,
Jul 26
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9442f946aef55baec873dce19fdbc7f1fbd255ea commit 9442f946aef55baec873dce19fdbc7f1fbd255ea Author: Gabriel Charette <gab@chromium.org> Date: Thu Jul 26 20:26:30 2018 [TaskScheduler] Require constant initialization for g_heartbeat_for_testing Somehow r562248 caused a regression whereby it should have been strictly more performant, the only difference is the potential static initializer (which if incorrectly initialized could result in it being non-zero? and somehow cause the timer to be fast?). Let's try to suppress it. R=fdoray@chromium.org Bug: 848255 Change-Id: I21111ddd4c069b9171badc58fc4a1914bb068641 Reviewed-on: https://chromium-review.googlesource.com/1090794 Reviewed-by: François Doray <fdoray@chromium.org> Cr-Commit-Position: refs/heads/master@{#578420} [modify] https://crrev.com/9442f946aef55baec873dce19fdbc7f1fbd255ea/base/task_scheduler/service_thread.cc
,
Jul 26
Sorry no, I'm not sure it has recovered. While it did recover what it lost through r565288, that CL is expected to have improved power. However, it wasn't directly intended to heal r562248 so I'm not yet convinced that it has recovered (it's just an improvement after a regression as far as I'm concerned at the moment). r578420 is trying to decipher this further but the bots haven't caught up to it yet. Doesn't need to block Beta though.
,
Jul 27
Ok r578420 didn't budge the graph, don't think there's anything left to assess here. Still confused though...
,
Jul 27
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/510d3437426dd68f3d50d5be7c2b34f4f125e5c9 commit 510d3437426dd68f3d50d5be7c2b34f4f125e5c9 Author: Gabriel Charette <gab@chromium.org> Date: Fri Jul 27 20:05:36 2018 Revert "[TaskScheduler] Require constant initialization for g_heartbeat_for_testing" This reverts commit 9442f946aef55baec873dce19fdbc7f1fbd255ea. Reason for revert: didn't budge the graph (https://chromeperf.appspot.com/report?sid=d99ad2294f639cf4e0d0bbfa0e9aee1cac5a0352ef9f35d266f375f52466a608&start_rev=1526716391&end_rev=1532706078) Original change's description: > [TaskScheduler] Require constant initialization for g_heartbeat_for_testing > > Somehow r562248 caused a regression whereby it should have been strictly > more performant, the only difference is the potential static > initializer (which if incorrectly initialized could result in it being > non-zero? and somehow cause the timer to be fast?). > Let's try to suppress it. > > R=fdoray@chromium.org > > Bug: 848255 > Change-Id: I21111ddd4c069b9171badc58fc4a1914bb068641 > Reviewed-on: https://chromium-review.googlesource.com/1090794 > Reviewed-by: François Doray <fdoray@chromium.org> > Cr-Commit-Position: refs/heads/master@{#578420} TBR=gab@chromium.org,fdoray@chromium.org Change-Id: I457751dc2d9d410e049685e22cf3f79cf39d3c11 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 848255 Reviewed-on: https://chromium-review.googlesource.com/1153230 Reviewed-by: Gabriel Charette <gab@chromium.org> Commit-Queue: Gabriel Charette <gab@chromium.org> Cr-Commit-Position: refs/heads/master@{#578762} [modify] https://crrev.com/510d3437426dd68f3d50d5be7c2b34f4f125e5c9/base/task_scheduler/service_thread.cc |
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by 42576172...@developer.gserviceaccount.com
, May 31 2018