Issue metadata
Sign in to add a comment
|
6.3% regression in system_health.memory_mobile at 428103:428228 |
||||||||||||||||||||||
Issue descriptionSee the link to graphs below.
,
Nov 2 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8997111201051784544
,
Nov 3 2016
===== BISECT JOB RESULTS ===== Status: failed ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@428102 163775283 100598 5 good chromium@428118 164032512 97389.5 4 good chromium@428122 163834266 60863.9 5 good chromium@428126 166042010 201530 5 bad chromium@428134 170803200 6475826 5 bad chromium@428165 176761242 2143544 5 bad chromium@428228 175974810 2529612 5 bad Bisect job ran on: android_nexus6_perf_bisect Bug ID: 661498 Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests system_health.memory_mobile Test Metric: memory:chrome:all_processes:reported_by_os:gpu_memory:proportional_resident_size_avg/load_social/load_social_twitter Relative Change: 7.45% Score: 0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus6_perf_bisect/builds/2715 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8997111201051784544 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5266449243832320 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 4 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8996893710537151920
,
Nov 4 2016
I kicked off a bisect with a narrower revision range. But the test is failing, not sure what we can do here.
,
Nov 4 2016
===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : Showing previews UI for Offline Previews Author : ryansturm Commit description: This changes the functionality of is_offline_preview() in OfflinePageTabHelper to check the provisional information as well. Additionally, this CL addresses other consumers of this information that want access to it in DidFinishNavigation. Specifically, PreviewsPageLoadMetricsObserver will access the is_offline_previews bit as will PreviewsInfoBarHelper. This also prevents showing the offline pages snackbar and replaces it with the previews infobar. This leaves the Offline omnibox and other UI features. BUG=615564, 649148 Committed: https://crrev.com/21364834a65e541369e8895d93695dfaf114056b Review-Url: https://codereview.chromium.org/2362033002 Cr-Original-Commit-Position: refs/heads/master@{#427902} Cr-Commit-Position: refs/heads/master@{#428124} Commit : e28ec590f2f8f5103fb724c587d3abc1a6782cb8 Date : Thu Oct 27 20:29:23 2016 ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@428122 165462357 3850351 12 good chromium@428123 165819051 4551996 12 good chromium@428124 165990400 235928 8 bad <-- chromium@428126 166010880 217887 8 bad Bisect job ran on: android_nexus6_perf_bisect Bug ID: 661498 Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.social.twitter system_health.memory_mobile Test Metric: memory:chrome:all_processes:reported_by_os:gpu_memory:proportional_resident_size_avg/load_social/load_social_twitter Relative Change: 0.25% Score: 0.0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus6_perf_bisect/builds/2727 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8996893710537151920 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=4516542585241600 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 11 2016
Looks like the bisect succeeded but didn't assign the CL owner to the bug (+sullivan as FYI). Asssigning to ryansturm re #6. PTAL.
,
Nov 11 2016
About not assigning the owner, that was due to the low confidence score. However we are removing this score as it's not very useful for people looking at the bisect results.
,
Nov 11 2016
This change was reverted at 428324 and re-landed at 428409 with a fix. Originally, this CL could have caused a memory regression as it was causing an infobar to be shown on nearly every android page (depending on initialized state). What I find interesting is that for the builds after or containing 428324, the memory regression is still present (without the change that is). And when 428409 is first built, the chart does not seem affected at all. Would it make sense to bisect 428229 - 428327 to see if there is another memory issue besides the reverted CL?
,
Nov 11 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8996271091743951040
,
Nov 11 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8996271054328746944
,
Nov 11 2016
I started two bisect jobs. One for 428228 - 428324 (all changes before the revert in that bucket) and 428324-428327 (all changes after the revert in that bucket). Since there might be hidden behavior that offset reverting the CL, I thought this would be the only way to accurately bisect the issue. Hopefully, this yields some result, but otherwise I don't know how to determine what is causing the problem. Like I said, multiple builds with the revert and without the re-land had the bad behavior.
,
Nov 11 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8996268465910075504
,
Nov 11 2016
On more bisect job that will bisect from the revision of the reverted CL until the revision before it was reverted to see if anything between the two changed behavior.
,
Nov 11 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@428324 N/A N/A 0 good chromium@428327 N/A N/A 0 bad Bisect job ran on: android_nexus6_perf_bisect Bug ID: 661498 Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.social.twitter system_health.memory_mobile Test Metric: memory:chrome:all_processes:reported_by_os:gpu_memory:proportional_resident_size_avg/load_social/load_social_twitter Relative Change: None Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus6_perf_bisect/builds/2740 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8996271091743951040 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5214218716971008 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 11 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@428228 N/A N/A 0 good chromium@428324 N/A N/A 0 bad Bisect job ran on: android_nexus6_perf_bisect Bug ID: 661498 Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.social.twitter system_health.memory_mobile Test Metric: memory:chrome:all_processes:reported_by_os:gpu_memory:proportional_resident_size_avg/load_social/load_social_twitter Relative Change: None Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus6_perf_bisect/builds/2741 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8996271054328746944 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5813972916240384 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 11 2016
Looking at the original bisect jobs' results, it's really clear that even if my CL caused a regression from 163MB to 166MB, there is a regression afterwards to 177MB and then back to 174MB when my CL is reverted. I am guessing that the reverted CL had a pretty negative impact of 3MB, but another CL has an impact of 11MB. This is pretty speculative and it could be the case that multiple CLs after mine caused regressions (chromium@428134 is at 170MB, so that could be random deviation or that could be in between bad CL's): Revision Mean Std Dev N Good? chromium@428102 163775283 100598 5 good chromium@428118 164032512 97389.5 4 good chromium@428122 163834266 60863.9 5 good chromium@428126 166042010 201530 5 bad chromium@428134 170803200 6475826 5 bad chromium@428165 176761242 2143544 5 bad chromium@428228 175974810 2529612 5 bad Revision Mean Std Dev N Good? chromium@428122 165462357 3850351 12 good chromium@428123 165819051 4551996 12 good chromium@428124 165990400 235928 8 bad <-- chromium@428126 166010880 217887 8 bad
,
Nov 11 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@428124 N/A N/A 0 good chromium@428323 N/A N/A 0 bad Bisect job ran on: android_nexus6_perf_bisect Bug ID: 661498 Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.social.twitter system_health.memory_mobile Test Metric: memory:chrome:all_processes:reported_by_os:gpu_memory:proportional_resident_size_avg/load_social/load_social_twitter Relative Change: None Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus6_perf_bisect/builds/2742 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8996268465910075504 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5216963872161792 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 11 2016
Hmm. I don't really know to go from here, but I don't think the difference between 428123 and 428124 is significant practically or statistically (165819051 -> 165990400), so I am done looking at this because reverting my CL had no effect anyway.
,
Nov 14 2016
Dashboard trace before regression: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_44-2016-10-27_15-53-56-89536.html And after regression: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_44-2016-10-27_21-12-46-35188.html This looks very similar to bug 663179 . ericrk, since you're owning that one, can you take a look at this one too? Screenshots of what I'm seeing before/after for gl>gpu>textures attached.
,
Nov 17 2016
In the after trace we have memory which has been deleted in the renderer, but is still retained by the GPU process. A quick experiment shows that we just need to flush the GL command buffer to ensure that the GL process sees the delete issued by the renderer, but will need to work on a more robust solution.
,
Nov 17 2016
Issue 663771 has been merged into this issue.
,
Nov 21 2016
Issue 667258 has been merged into this issue.
,
Dec 5 2016
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982651803569832816
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982651764345962208
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982632274354713632
,
Apr 11 2017
=== Auto-CCing suspected CL author csharrison@chromium.org === Hi csharrison@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : csharrison Commit : 90f9f786e88b196d391e6daffb4c9ab7ca7087ab Date : Thu Oct 27 21:33:49 2016 Subject: Add field trial config for CSSExternalScanner experiment Bisect Details Configuration: android_nexus6_perf_bisect Benchmark : system_health.memory_mobile Metric : memory:chrome:all_processes:reported_by_os:gpu_memory:proportional_resident_size_avg/load_social/load_social_twitter Change : 6.28% | 165877077.333 -> 176286378.667 Revision Result N chromium@428124 165877077 +- 262826 6 good chromium@428137 165866155 +- 255576 6 good chromium@428143 165956267 +- 316357 6 good chromium@428145 165999957 +- 343218 6 good chromium@428146 177510400 +- 1024684 6 bad <-- chromium@428149 176025600 +- 5036002 6 bad chromium@428174 176286379 +- 5527314 6 bad chromium@428224 176083627 +- 5196691 6 bad chromium@428323 176286379 +- 5371481 6 bad Please refer to the following doc on diagnosing memory regressions: https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.social.twitter system_health.memory_mobile Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982651764345962208 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5216963872161792 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 11 2017
I see this issue is marked as a duplicate. Should I be concerned about my patch here?
,
Apr 11 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: android_nexus6_perf_bisect Benchmark : system_health.memory_mobile Metric : memory:chrome:all_processes:reported_by_os:gpu_memory:proportional_resident_size_avg/load_social/load_social_twitter Revision Result N chromium@428324 174721512 +- 7918242 21 good chromium@428327 175323624 +- 4352402 21 bad Please refer to the following doc on diagnosing memory regressions: https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.social.twitter system_health.memory_mobile Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982651803569832816 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5214218716971008 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you! |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by primiano@chromium.org
, Nov 2 2016