Issue metadata
Sign in to add a comment
|
System health stories crashing on low end devices |
||||||||||||||||||||||
Issue descriptionSeveral stories have started failing pretty consistently on: https://luci-milo.appspot.com/buildbot/chromium.perf/Android%20One%20Perf Stories with most failures: config story history 0 common browse:media:imgur 111111111----------- 1 common browse:news:cnn 1111-1111----------- 2 common load:media:imgur 1111-1111----------- 3 common browse:social:tumblr_infinite_scroll 11--1--11--------~~~ 4 common browse:social:pinterest_infinite_scroll 1-11---11--------~~~ 5 memory background:social:facebook 22222??2?-???------- 6 memory background:news:nytimes 2212-??-?-???------- 7 memory load:media:imgur 31-1-?1-1-???------- 8 memory browse:media:imgur 11111?111-???------- Similar failures are seen on downstream clankium/webview low end bots. Build with first failures is: https://luci-milo.appspot.com/buildbot/chromium.perf/Android%20One%20Perf/134 Commit range: http://test-results.appspot.com/revision_range?start=476641&end=476837 Logcat with microdump attached. Will try a bisect.
,
Jun 5 2017
=== Auto-CCing suspected CL author hpayer@chromium.org === Hi hpayer@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Test failure found with culprit Suspected Commit Author : hpayer Commit : 502c6ae6a03979efbd3e006e6a0b8c3369ca2bbc Date : Fri Jun 02 09:40:16 2017 Subject: [heap] Activate memory reducer on external memory activity. Bisect Details Configuration: android_one_perf_bisect Benchmark : system_health.common_mobile Metric : cpuTimeToFirstMeaningfulPaint_avg/browse_media/browse_media_imgur Revision Exit Code N chromium@476640 0 +- N/A 5 good chromium@476653 0 +- N/A 5 good chromium@476656 0 +- N/A 5 good chromium@476656,v8@d1a9603e57 0 +- N/A 5 good chromium@476656,v8@502c6ae6a0 1 +- N/A 5 bad <-- chromium@476656,v8@3b7ff7a5e5 1 +- N/A 5 bad chromium@476657 1 +- N/A 5 bad chromium@476658 1 +- N/A 5 bad chromium@476659 1 +- N/A 5 bad chromium@476665 1 +- N/A 5 bad chromium@476690 1 +- N/A 5 bad chromium@476739 1 +- N/A 5 bad chromium@476837 1 +- N/A 5 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.media.imgur system_health.common_mobile Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8977631932273831488 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=6416378426294272 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Jun 5 2017
hpayer hope you can find a quick fix or revert, since your CL is causing many tests to fail on different bots.
,
Jun 6 2017
,
Jun 6 2017
,
Jun 6 2017
,
Jun 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/v8/v8.git/+/11fc9fab94d48b389cb239d1fa06413682a12eb6 commit 11fc9fab94d48b389cb239d1fa06413682a12eb6 Author: ulan <ulan@chromium.org> Date: Tue Jun 06 17:31:24 2017 [heap] Guard against re-entering GC on external memory change. TBR=hpayer@chromium.org BUG=chromium:729868, chromium:729521 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_rel_ng Review-Url: https://codereview.chromium.org/2929463002 Cr-Commit-Position: refs/heads/master@{#45745} [modify] https://crrev.com/11fc9fab94d48b389cb239d1fa06413682a12eb6/src/api.cc
,
Jun 6 2017
,
Jun 7 2017
The crashing piece should be fixed by #8. The performance regressions need investigation.
,
Jun 7 2017
The following revision refers to this bug: https://chromium.googlesource.com/v8/v8.git/+/cf8f7bdc9d5ee314a10fe043e421ec95604d4eca commit cf8f7bdc9d5ee314a10fe043e421ec95604d4eca Author: hpayer <hpayer@chromium.org> Date: Wed Jun 07 13:21:28 2017 [heap] Increase memory reducer activation limit for external memory changes. BUG= chromium:729521 CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_rel_ng Review-Url: https://codereview.chromium.org/2923563006 Cr-Commit-Position: refs/heads/master@{#45763} [modify] https://crrev.com/cf8f7bdc9d5ee314a10fe043e421ec95604d4eca/include/v8.h
,
Jun 9 2017
,
Jun 12 2017
,
Jun 14 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8976828311836906096
,
Jun 14 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8976828289952992304
,
Jun 14 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: android_one_perf_bisect Benchmark : memory.top_10_mobile Metric : memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size_avg/background/after_http_search_yahoo_com_search__ylt_p_google Revision Result N chromium@476000 5398821 +- 298707 21 good chromium@477000 5459383 +- 258537 21 bad Please refer to the following doc on diagnosing memory regressions: https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests memory.top_10_mobile Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8976828311836906096 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=6386692182769664 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Jun 14 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8976796722568648896
,
Jun 14 2017
=== BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : Evan Stade Commit : 90cf9d727da221f76e5cb7d34aa09c8e711c1e68 Date : Fri Jun 02 00:37:26 2017 Subject: Clean up views::Background. Bisect Details Configuration: android_one_perf_bisect Benchmark : memory.top_10_mobile Metric : memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size_avg/background/after_https_m_facebook_com_rihanna Change : 1.47% | 5472109.71429 -> 5552469.33333 Revision Result N chromium@476000 5472110 +- 306411 21 good chromium@476500 5480903 +- 139348 9 good chromium@476501 5750443 +- 168992 6 bad <-- chromium@476502 5729280 +- 223556 6 bad chromium@476504 5618859 +- 109318 6 bad chromium@476508 5628416 +- 169079 6 bad chromium@476516 5605717 +- 111213 6 bad chromium@476532 5587285 +- 106535 9 bad chromium@476563 5592292 +- 112349 9 bad chromium@476625 5591609 +- 141350 9 bad chromium@476750 5604238 +- 225821 9 bad chromium@477000 5552469 +- 245351 21 bad Please refer to the following doc on diagnosing memory regressions: https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests memory.top_10_mobile Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8976828289952992304 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=6449650296422400 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Jun 15 2017
=== Auto-CCing suspected CL author zpeng@chromium.org === Hi zpeng@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : zpeng Commit : d2f0dd39eaacdd2184c8bd53bb9af5b768a56360 Date : Thu Jun 01 19:01:55 2017 Subject: Update WebApkInfo to include badge icon. Bisect Details Configuration: android_one_perf_bisect Benchmark : memory.top_10_mobile Metric : memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size_avg/background/after_http_search_yahoo_com_search__ylt_p_google Change : 1.05% | 5393261.71429 -> 5449971.80952 Revision Result N chromium@476000 5393262 +- 277794 21 good chromium@476250 5377536 +- 117307 6 good chromium@476375 5414181 +- 209400 14 good chromium@476376 5635925 +- 92754.5 6 bad <-- chromium@476377 5635925 +- 139966 6 bad chromium@476379 5636096 +- 110383 6 bad chromium@476383 5551104 +- 38641.6 6 bad chromium@476391 5604523 +- 140884 6 bad chromium@476406 5548885 +- 184288 6 bad chromium@476437 5656235 +- 108840 6 bad chromium@476500 5481399 +- 258111 14 bad chromium@477000 5449972 +- 232567 21 bad Please refer to the following doc on diagnosing memory regressions: https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests memory.top_10_mobile Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8976796722568648896 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=6386692182769664 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Jun 15 2017
+hpayer is there a separate bug where you are investigating the performance regressions? This just got duped into issue 732315 because #18 and #19 are bisecting into a problematic metric; but the earlier bisect attempts coming from issue 729755 , issue 729631 , and issue 731716 were on good metrics and pointed clearly to your CL.
,
Jun 17 2017
,
Aug 17 2017
ulan, hablich: I just saw this when triaging old bugs. Looks like hpayer may have missed it when OOO. See comment #20, is anyone looking at the regressions?
,
Aug 17 2017
This might be related to https://bugs.chromium.org/p/chromium/issues/detail?id=733655
,
Aug 17 2017
To clarify 23: Regressions 729631 and 731716 are expected as this CL is a trade-off between GC time and memory usage. Increase of memory usage in 729755 is unexpected and might be related to 733655, which is caused by a bug in V8/Oilpan GC interaction.
,
Jan 9 2018
|
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by 42576172...@developer.gserviceaccount.com
, Jun 5 2017