Issue metadata
Sign in to add a comment
|
13.5% regression in system_health.memory_mobile at 469003:469166 |
||||||||||||||||||||
Issue descriptionSee the link to graphs below.
,
May 5 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8980436734752770064
,
May 5 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8980436717323504240
,
May 5 2017
=== Auto-CCing suspected CL author kbr@chromium.org === Hi kbr@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : kbr Commit : 1d44ab0e3d0a7cd05ce1da956bb87a85a8b88174 Date : Wed May 03 16:54:08 2017 Subject: Fix bug in virtualized GL context state management upon creation. Bisect Details Configuration: android_nexus7_perf_bisect Benchmark : system_health.memory_mobile Metric : memory:chrome:all_processes:reported_by_os:gpu_memory:proportional_resident_size_avg/load_tools/load_tools_dropbox Change : 11.32% | 57559917.7143 -> 64078409.1429 Revision Result N chromium@469002 57559918 +- 683613 14 good chromium@469005 58810953 +- 17640166 14 good chromium@469006 57519218 +- 313540 9 good chromium@469007 64111177 +- 15425746 14 bad <-- chromium@469008 64651719 +- 11565776 9 bad chromium@469013 65215147 +- 8454983 6 bad chromium@469023 65364553 +- 17655924 14 bad chromium@469043 66770716 +- 13136647 9 bad chromium@469084 66566827 +- 59825.9 6 bad chromium@469166 64078409 +- 15356365 14 bad Please refer to the following doc on diagnosing memory regressions: https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.tools.dropbox system_health.memory_mobile Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8980436717323504240 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=6471367857274880 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
May 5 2017
=== BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : kbr Commit : 1d44ab0e3d0a7cd05ce1da956bb87a85a8b88174 Date : Wed May 03 16:54:08 2017 Subject: Fix bug in virtualized GL context state management upon creation. Bisect Details Configuration: android_nexus7_perf_bisect Benchmark : system_health.memory_mobile Metric : memory:chrome:all_processes:reported_by_os:gpu_memory:proportional_resident_size_avg/load_tools/load_tools_dropbox Change : 12.25% | 57535341.7143 -> 64582542.2222 Revision Result N chromium@469002 57535342 +- 656880 14 good chromium@469005 57451861 +- 241537 6 good chromium@469006 57470976 +- 219509 6 good chromium@469007 63838891 +- 9991995 6 bad <-- chromium@469012 64811008 +- 14541218 14 bad chromium@469023 62843758 +- 16725484 14 bad chromium@469044 64680846 +- 11622762 9 bad chromium@469084 62156800 +- 16714872 14 bad chromium@469166 64582542 +- 11444787 9 bad Please refer to the following doc on diagnosing memory regressions: https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.tools.dropbox system_health.memory_mobile Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8980436734752770064 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5313711281012736 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
May 5 2017
Issue 718862 has been merged into this issue.
,
May 5 2017
,
May 5 2017
Issue 718793 has been merged into this issue.
,
May 5 2017
Issue 718849 has been merged into this issue.
,
May 6 2017
Issue 718849 has been merged into this issue.
,
May 16 2017
What is the real impact of this regression? This was a longstanding bug, and the fix is absolutely required. Is the regression really only seen on the Nexus 7? If so I am going to close this as WontFix.
,
May 16 2017
,
May 16 2017
This is a 8Mb regression in GPU memory. I am not sure whether we can just close this as WontFix. Is it possible to implement fix differently or the extra memory is unavoidable? +Primiano & Ben for opinions.
,
May 16 2017
Re #11: This was seen on both Nexus 7 and Nexus 5. primiano, hjd, perezju, can you help kbr in understanding the impact of the regression? As Ned noted, we're regressing 2.6-8MiB on over a dozen pages, on at least the N5 and N7.
,
May 16 2017
gpu_memory:proportional_resident_size_avg is regressed, but I don't see the regression in the trace files. For example for nexus7v2: (before the peak) https://00e9e64bac598ee91387dd62a74236bf7f53df6c1ffd981b62-apidata.googleusercontent.com/download/storage/v1/b/chrome-telemetry-output/o/trace-file-id_24-2017-05-03_12-47-09-96554.html?qk=AD5uMEsEz8Zm_JWMRe72gs-JQdGLsTYNztNpAKbX6IS5KvwLV5p5Ee0WI0Lltaf2HGzwB1rhafK_924YYGqRoVSmX8IrDol_9k1N1Ds3tMU6SIz_GnnpZwuLtb7FEMJN-TQu97tLp4EhtQwDx0Rk2fwvwUqKiC0Xq_dSSpazFsgkHACvJ6Lo6NtW9A8gPCr3P1tZB5BH7DPisDmjE49XrNlyG8WIffNB9FWzXIw_WEm-SCMjygdve342sA4DDmxaTQVr2NFLYIkd3jgcDdknr_VJUxUuomFlWJnsmyDZNqsEMtlV-SwuaaL8cxy1c3uTOCQ_oXRi4Tp3JM_9lGxaBfNpCToWouac6Q9W_duY-Tj78u5lvS9Q9LqNcEttnBTaOtGD9h01VZXItI0DNlTjhdjnJ4b2S9b5DlHkL-1ZRCSsTkFGJvioKr95c0IyjlhPmAf3juTCftpUDLpQ66NrXT8pHIE93pJDjdMQQIkcStuf0xkw5b9wY-PisqlhRjmS3QRt2zjAdPtJiC-IG_qPG7NilO7eVkLvw7TVJNh_vhtT7K9dOVX0PV_hUwz7bkm1pc8VV8-g9XsnVaGhU0MW3uE_IkKVqlz1qPNvpI6uCYYfrQYoSKEelqp2V1nuSnWFvpuMINiGUhMKno2i8sBcdIihVw5pXPhgR8qtssoa55v3TsoOQ7Fck9YHZl5IYMHBlS7OUL2XjOlEVw4keqbRt9O1WHxw_7lQrDY9q88lt6evbBAjYXtQxB_nnTsfXi_Vj7wshROZFURib7K3CIkLbd9eo18fFFuno8DCeVGCyAtEi64Wmp2RauVi7rJXgpZOPcIri6ZY5LKaMUezij2w_ECH10v79wL0Dw (on the peak) https://00e9e64bacfd864c48eb9b0d28777e54d2580c8d21726a0647-apidata.googleusercontent.com/download/storage/v1/b/chrome-telemetry-output/o/trace-file-id_24-2017-05-03_18-34-06-87205.html?qk=AD5uMEvIB8Xg_SXlR7Pxd63EUNrBldoRU1I9PHD4yjmsUddu5utHiQEYBECly8pwcABJ2NR2KG0Feu8ax61yeUwlSlfxkutWoxMOPTOT4S7tukzQWWmO4oH3iZUgxvkqUCRtJ4WSOoxXfcY8YLhSkAqkjv0fWSNkVzmcVaIvyYqjaIBbtqmUAZDB3SPOuc4_u6s5C9tmbQUGsfM39feBt0wfwS7w0Gb_YIcTX7AW1XbcOTDemiNLukAupef5punfhqzaV-itnQg58olgMYHY3gw_2b-FFJmW7CKsXpggB-bHkqxWD7x-Szfwp9UaTekXQqkvVDc-ltwJilJmNd9yKlO1_6ivv1ZATiCuu2vHgPQQoCLCYa3_U62vy9FrLm4eBzVup3CUHLb7emfiGQ0_iY52z3RNQ_QHeoo0WIw7_b0SvwuK4lfItXNaFOP_KWAB8zi_qSuNjAdxZRBBz2169fd81cgh-RMQIoK1mFyuAbIrvzKIh7mBsmD7HQLqHzqlzLqt5K1mtFSJOU1SFROOFfC2lBmj_lUv_eJU9v7NaqkGhgkC3DV1gpRPSXIoFz1eIWsD9dPKPs76V_ZYN4UyRXNsqk1uLcMWbmXSoqZBdPQ8P4YmxCdF4z3mrj3p2FV3AyVAih8LSk0JPRKiOPuG3_fk7GaCkWODWPdicQ9yFoM4L49aL2Rg1R6v8-KZ-J1iTE8hQFuLiZoeZn71B6Q1XBoRVFDwZmxb4F1h5TUVWnQMWnoGN_SLn_Sb0FQY9wCX39WeDs5aP2q7g92xKfLnHVawNJ_l64IGmRzpY4VWTRtdktpArBO5VpaMtDab7fcjNTRdk2ah-Sh8mVvG7RxqQpsm_5OEzxUBFQ Judging from these trace files, there is no regression. In fact some metrics even improved: Total PSS went 111.4 MiB -> 108.5 MiB, Total Private Dirty went 65.7 MiB -> 63.2 MiB
,
May 30 2017
Hi, sorry for the delay getting back to you. Here are a couple of traces (need to copy the link from console.developers.google.com before it expands, otherwise it's not viewable by others). before: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_35-2017-05-03_12-47-24-48150.html after: https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_35-2017-05-03_18-34-19-72482.html Clicking on the memory dump, then "gpu" on the gpu process reveals a "memtrack_pss" column. (Also see screenshot) That is the one increasing from 21,812.0 KiB to 30,772.0 KiB. As sullivan@ mentioned already in #15; the regression is significant (up to 8MiB in several pages) and reproduces on both N5 and N7.
,
Jun 5 2017
Hmm, I've seen similar thing before, and the issue is that Android sometimes doesn't create third buffer of it's triple buffering scheme until necessary. I.e. sometimes opening a specific URL (i.e. avoiding NTP) uses ~8MiB less of graphics memory because Android didn't have to create a third buffer. However, any interaction with a page (scrolling) brings that buffer back.
,
Jun 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/958a49a3e9f2f2fe39e5059f73f660e89de8fba3 commit 958a49a3e9f2f2fe39e5059f73f660e89de8fba3 Author: Kenneth Russell <kbr@chromium.org> Date: Fri Jun 16 23:11:38 2017 Avoid redundant eglMakeCurrent calls. The previous fix for initialization of a new virtual context caused the real context to be released. Avoid this by adding a new entry point to release only the virtual context so it can be quickly made current again. BUG= 718809 , 724071 Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Change-Id: Ieb8c4bfd1db5dfa90168f3a859182efa9c39d0d3 Reviewed-on: https://chromium-review.googlesource.com/534779 Reviewed-by: John Bauman <jbauman@chromium.org> Commit-Queue: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#480211} [modify] https://crrev.com/958a49a3e9f2f2fe39e5059f73f660e89de8fba3/gpu/command_buffer/service/gl_context_virtual.cc [modify] https://crrev.com/958a49a3e9f2f2fe39e5059f73f660e89de8fba3/gpu/command_buffer/service/gl_context_virtual.h [modify] https://crrev.com/958a49a3e9f2f2fe39e5059f73f660e89de8fba3/gpu/ipc/service/gpu_command_buffer_stub.cc [modify] https://crrev.com/958a49a3e9f2f2fe39e5059f73f660e89de8fba3/ui/gl/gl_context.cc [modify] https://crrev.com/958a49a3e9f2f2fe39e5059f73f660e89de8fba3/ui/gl/gl_context.h
,
Jun 21 2017
How do I tell whether this change has affected these graphs? I'm sorry for letting this go for so long, but when I scroll to the right the last data point I see for some of them was generated 12 days ago (June 8).
,
Jun 21 2017
FYI, I looked at system_health.memory_mobile / memory:chrome:all_processes:reported_by_os:proportional_resident_size_avg / load_tools / load_tools_dropbox on the Nexus 5 and 7v2, as well as a couple of the other graphs. https://chromeperf.appspot.com/report?sid=9c387bda39e4330d5cddea8174bccf7ef5677e21ec7e24dc57cf7379ef9fa5b0&start_rev=470442&end_rev=477931
,
Jun 21 2017
,
Jun 21 2017
Hmm. It seems that the system_health.memory_mobile has been deeply unhappy on the N5 bot recently. See https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus5%20Perf?numbuilds=200 Here is a sample error: https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5_Perf%2F163%2F%2B%2Frecipes%2Fsteps%2Fsystem_health.memory_mobile_on_Android%2F0%2Fstdout We are also missing data for N7v2 recently this is stranger as that bot doesn't seem nearly as unhappy (relatively speaking). See: https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus7v2%20Perf
,
Jun 21 2017
,
Jun 21 2017
,
Jun 21 2017
The problem on N5 looks like a device issue; and on the N7v2 there appears to be a rouge story failing and causing a bunch of other stories to be skipped. Sorry for the trouble, I'll try to see if we have some data anywhere to asses the impact of your change in #19.
,
Jun 22 2017
We have fixes on the blocking bugs, and a few new data points on the relevant metrics. Things do seem to have improved, and metrics returned to their values prior to this regression: https://chromeperf.appspot.com/report?sid=03079dce6411e5985cd1cee6ffe89e3c05594929f3361c2b73fe902b1d3a24f3&start_rev=465453&end_rev=481486 We had a black out with no data for >3000 CLs, so it's hard to say if it's all due to r480211 ... but at least it's in the range.
,
Jun 22 2017
If https://chromium-review.googlesource.com/c/534779/ reverts cleanly we could try doing a try job of the revert, I think the command line would be like: ./tools/perf/run_benchmark trybot android-nexus5 system_health.memory_mobile --story-filter='load:tools:dropbox' Then if we see a regression in the tryjob result we can say the CL probably fixed things.
,
Jun 22 2017
Thanks for the tip. Since things have recovered I'm going to call this fixed.
,
Aug 4 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/08cdea0d506e49a0f7a5322e1b664483b6850ab0 commit 08cdea0d506e49a0f7a5322e1b664483b6850ab0 Author: Hector Dearman <hjd@google.com> Date: Fri Aug 04 12:16:03 2017 memory-infra: Remove ProcessMetricsDumpProvider Now that memory-infra has switched to collecting OS metrics separately from the old path we can remove a lot of the old code path. Specifically we can remove: - ProcessMetricsDumpProvider - The content/ stub for ProcessMetricsDumpProvider - The support for OS Metrics in ProcessMemoryDumo - (Including process_memory_maps and process_memory_totals) - The logic for registering of ProcessMetricsDumpProvider - The support in OS metrics for ProcessMetricsDumpProvider - The PrivatePlatformFootprint structtraits We also remove the following memory-infra features: - Peak RSS on recent versions of Android (Little used feature, superseded by general peak detection support in memory-infra). - General peak detection support in Linux (Currently unused, coming back soon) - Some OS stats of little use on MacOS This removes 10 files and ~1000 lines. Bug: 718809 Change-Id: I858ddc91ce342176c25dfda58cb9d3b1ebb39c6f Reviewed-on: https://chromium-review.googlesource.com/586721 Commit-Queue: Hector Dearman <hjd@chromium.org> Reviewed-by: Daniel Cheng <dcheng@chromium.org> Reviewed-by: Siddhartha S <ssid@chromium.org> Reviewed-by: Primiano Tucci <primiano@chromium.org> Reviewed-by: Avi Drissman <avi@chromium.org> Cr-Commit-Position: refs/heads/master@{#491998} [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/base/BUILD.gn [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/base/trace_event/memory_dump_request_args.h [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/base/trace_event/process_memory_dump.cc [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/base/trace_event/process_memory_dump.h [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/base/trace_event/process_memory_dump_unittest.cc [delete] https://crrev.com/48a0ca6d697c2a43d8ff67069f016daa108458fd/base/trace_event/process_memory_maps.cc [delete] https://crrev.com/48a0ca6d697c2a43d8ff67069f016daa108458fd/base/trace_event/process_memory_maps.h [delete] https://crrev.com/48a0ca6d697c2a43d8ff67069f016daa108458fd/base/trace_event/process_memory_totals.cc [delete] https://crrev.com/48a0ca6d697c2a43d8ff67069f016daa108458fd/base/trace_event/process_memory_totals.h [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/components/tracing/BUILD.gn [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/components/tracing/child/child_trace_message_filter.cc [delete] https://crrev.com/48a0ca6d697c2a43d8ff67069f016daa108458fd/components/tracing/common/DEPS [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/components/tracing/common/graphics_memory_dump_provider_android.cc [delete] https://crrev.com/48a0ca6d697c2a43d8ff67069f016daa108458fd/components/tracing/common/process_metrics_memory_dump_provider.cc [delete] https://crrev.com/48a0ca6d697c2a43d8ff67069f016daa108458fd/components/tracing/common/process_metrics_memory_dump_provider.h [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/content/browser/browser_main_loop.cc [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/content/browser/tracing/tracing_controller_impl.cc [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/BUILD.gn [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/memory_instrumentation/coordinator_impl.cc [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/memory_instrumentation/coordinator_impl_unittest.cc [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/public/cpp/BUILD.gn [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/public/cpp/memory_instrumentation/client_process_impl.cc [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/public/cpp/memory_instrumentation/memory_instrumentation.typemap [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/public/cpp/memory_instrumentation/memory_instrumentation_struct_traits.cc [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/public/cpp/memory_instrumentation/memory_instrumentation_struct_traits.h [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/public/cpp/memory_instrumentation/os_metrics.cc [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/public/cpp/memory_instrumentation/os_metrics.h [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/public/cpp/memory_instrumentation/os_metrics_linux.cc [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/public/cpp/memory_instrumentation/os_metrics_mac.cc [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/public/cpp/memory_instrumentation/os_metrics_unittest.cc [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/public/cpp/memory_instrumentation/os_metrics_win.cc [delete] https://crrev.com/48a0ca6d697c2a43d8ff67069f016daa108458fd/services/resource_coordinator/public/cpp/memory_instrumentation/process_metrics_memory_dump_provider.cc [delete] https://crrev.com/48a0ca6d697c2a43d8ff67069f016daa108458fd/services/resource_coordinator/public/cpp/memory_instrumentation/process_metrics_memory_dump_provider.h [delete] https://crrev.com/48a0ca6d697c2a43d8ff67069f016daa108458fd/services/resource_coordinator/public/cpp/memory_instrumentation/process_metrics_memory_dump_provider_unittest.cc [modify] https://crrev.com/08cdea0d506e49a0f7a5322e1b664483b6850ab0/services/resource_coordinator/public/interfaces/memory_instrumentation/memory_instrumentation.mojom |
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by kraynov@chromium.org
, May 5 2017