New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 776025 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression

Blocked on:
issue 777365



Sign in to add a comment

2.7% regression in system_health.memory_mobile at 509317:509376

Project Member Reported by chiniforooshan@chromium.org, Oct 18 2017

Issue description

See the link to graphs below.
 
Project Member

Comment 1 by 42576172...@developer.gserviceaccount.com, Oct 18 2017

All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=776025

(For debugging:) Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?sid=4eddd63f96abc94fcf2b3de6bfdbf4bf07c20a325f1c66d8ca287eb5bf286b8c


Bot(s) for this bug's original alert(s):

android-webview-nexus5X
Project Member

Comment 3 by 42576172...@developer.gserviceaccount.com, Oct 18 2017

Cc: jgruber@chromium.org
Owner: jgruber@chromium.org
Status: Assigned (was: Untriaged)

=== Auto-CCing suspected CL author jgruber@chromium.org ===

Hi jgruber@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : Jakob Gruber
  Commit : f7b09b3bdd96b4bd686b413c7a26e13f08ac4601
  Date   : Mon Oct 16 16:04:12 2017
  Subject: Reland "[snapshot] Ship lazy TFJ builtins"

Bisect Details
  Configuration: android_webview_arm64_aosp_perf_bisect
  Benchmark    : system_health.memory_mobile
  Metric       : memory:webview:all_processes:reported_by_chrome:v8:effective_size_avg/load_news/load_news_reddit
  Change       : 2.74% | 14189536.0 -> 14578852.0

Revision                           Result                   N
chromium@509316                    14189536 +- 130436       6      good
chromium@509324                    14223061 +- 135774       6      good
chromium@509325                    14168981 +- 93012.3      6      good
chromium@509325,v8@972d460f4f      14157077 +- 85612.5      6      good
chromium@509325,v8@f7b09b3bdd      14531252 +- 102875       6      bad       <--
chromium@509325,v8@efa038361d      14523801 +- 63065.1      6      bad
chromium@509325,v8@6627b0aaf7      14540119 +- 84559.0      6      bad
chromium@509326                    14518428 +- 102431       6      bad
chromium@509328                    14518731 +- 82571.3      6      bad
chromium@509331                    14523355 +- 61980.2      6      bad
chromium@509346                    14553945 +- 80461.5      6      bad
chromium@509376                    14578852 +- 105627       6      bad

Please refer to the following doc on diagnosing memory regressions:
  https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.news.reddit system_health.memory_mobile

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8965372509988487152


For feedback, file a bug with component Speed>Bisection
Project Member

Comment 4 by 42576172...@developer.gserviceaccount.com, Oct 19 2017

Cc: hubbe@chromium.org
 Issue 776171  has been merged into this issue.
Project Member

Comment 5 by 42576172...@developer.gserviceaccount.com, Oct 19 2017

 Issue 776169  has been merged into this issue.
Project Member

Comment 6 by 42576172...@developer.gserviceaccount.com, Oct 19 2017

 Issue 776170  has been merged into this issue.
Project Member

Comment 10 by 42576172...@developer.gserviceaccount.com, Oct 19 2017


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : Jakob Gruber
  Commit : f7b09b3bdd96b4bd686b413c7a26e13f08ac4601
  Date   : Mon Oct 16 16:04:12 2017
  Subject: Reland "[snapshot] Ship lazy TFJ builtins"

Bisect Details
  Configuration: winx64_high_dpi_perf_bisect
  Benchmark    : system_health.memory_desktop
  Metric       : memory:chrome:renderer_processes:reported_by_chrome:v8:heap:effective_size_avg/load_games/load_games_lazors
  Change       : 9.40% | 5578752.0 -> 6103040.0

Revision                           Result                 N
chromium@509301                    5578752 +- 0.0         6      good
chromium@509320                    5578752 +- 0.0         6      good
chromium@509325                    5578752 +- 0.0         6      good
chromium@509325,v8@972d460f4f      5491371 +- 478607      6      good
chromium@509325,v8@f7b09b3bdd      6103040 +- 0.0         6      bad       <--
chromium@509325,v8@efa038361d      6103040 +- 0.0         6      bad
chromium@509325,v8@6627b0aaf7      6103040 +- 0.0         6      bad
chromium@509326                    6103040 +- 0.0         6      bad
chromium@509327                    6103040 +- 0.0         6      bad
chromium@509329                    6103040 +- 0.0         6      bad
chromium@509338                    6015659 +- 478607      6      bad
chromium@509375                    6103040 +- 0.0         6      bad

Please refer to the following doc on diagnosing memory regressions:
  https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.games.lazors system_health.memory_desktop

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8965323886958348704


For feedback, file a bug with component Speed>Bisection
Project Member

Comment 11 by 42576172...@developer.gserviceaccount.com, Oct 19 2017

 Issue 776294  has been merged into this issue.
Project Member

Comment 12 by 42576172...@developer.gserviceaccount.com, Oct 19 2017


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : Jakob Gruber
  Commit : f7b09b3bdd96b4bd686b413c7a26e13f08ac4601
  Date   : Mon Oct 16 16:04:12 2017
  Subject: Reland "[snapshot] Ship lazy TFJ builtins"

Bisect Details
  Configuration: winx64intel_perf_bisect
  Benchmark    : media.desktop
  Metric       : memory:chrome:all_processes:reported_by_chrome:v8:effective_size_avg/video.html?src_crowd1080.webm
  Change       : 10.34% | 5070848.0 -> 5595136.0

Revision                           Result                 N
chromium@509259                    5070848 +- 0.0         6      good
chromium@509299                    5070848 +- 0.0         6      good
chromium@509319                    5070848 +- 0.0         6      good
chromium@509324                    5158229 +- 478607      6      good
chromium@509325                    5070848 +- 0.0         6      good
chromium@509325,v8@972d460f4f      5129102 +- 494303      9      good
chromium@509325,v8@f7b09b3bdd      5595136 +- 0.0         6      bad       <--
chromium@509325,v8@efa038361d      5595136 +- 0.0         6      bad
chromium@509325,v8@6627b0aaf7      5595136 +- 0.0         6      bad
chromium@509326                    5595136 +- 0.0         9      bad
chromium@509327                    5682517 +- 478607      6      bad
chromium@509329                    5682517 +- 478607      6      bad
chromium@509338                    5595136 +- 0.0         6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=video.html.src.crowd1080.webm media.desktop

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8965323813939088880


For feedback, file a bug with component Speed>Bisection
Cc: mlippautz@chromium.org
This seems to be an additional allocated page in code-space, see e.g.:

https://chromeperf.appspot.com/report?sid=4b396cecb6b34f6508ba8394c527c633af6fc17c3c0274fe25cb625c2b0a9d4b&start_rev=505526&end_rev=510020

https://chromeperf.appspot.com/report?sid=25e638f06c7e8cb439a684ea1757e4a58a7e996bbb9d354bfccba4be75b146bc&start_rev=505469&end_rev=509980

The reports so far are only from Windows machines. 

mlippautz@, I remember you mentioned this at some point. Is there a difference in how different OS's report these numbers? Are the additional 512K actually used or only requested from the OS and not necessarily mapped?
Windows does not have on demand paging afaik. So any new page you allocate there will immediately result in a +512k delta for committed memory.

This can happen if a CL pushes you across a certain threshold. 

I you can confirm that this can happen and is sort of expected, then this is fine.
Project Member

Comment 15 by 42576172...@developer.gserviceaccount.com, Oct 20 2017


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : Jakob Gruber
  Commit : f7b09b3bdd96b4bd686b413c7a26e13f08ac4601
  Date   : Mon Oct 16 16:04:12 2017
  Subject: Reland "[snapshot] Ship lazy TFJ builtins"

Bisect Details
  Configuration: android_webview_arm64_aosp_perf_bisect
  Benchmark    : system_health.memory_mobile
  Metric       : memory:webview:all_processes:reported_by_chrome:v8:effective_size_avg/load_news/load_news_reddit
  Change       : 2.84% | 14174364.0 -> 14576646.6667

Revision                           Result                   N
chromium@509316                    14174364 +- 137479       6      good
chromium@509324                    14186612 +- 42697.4      6      good
chromium@509325                    14181871 +- 132617       6      good
chromium@509325,v8@972d460f4f      14149037 +- 72157.6      6      good
chromium@509325,v8@f7b09b3bdd      14543889 +- 100859       6      bad       <--
chromium@509325,v8@efa038361d      14527969 +- 76186.0      6      bad
chromium@509325,v8@6627b0aaf7      14560411 +- 58568.6      6      bad
chromium@509326                    14547548 +- 76010.5      6      bad
chromium@509328                    14537981 +- 70261.4      6      bad
chromium@509331                    14537008 +- 69083.3      6      bad
chromium@509346                    14575076 +- 160357       6      bad
chromium@509376                    14576647 +- 125932       6      bad

Please refer to the following doc on diagnosing memory regressions:
  https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=load.news.reddit system_health.memory_mobile

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8965323834609413920


For feedback, file a bug with component Speed>Bisection
Project Member

Comment 17 by 42576172...@developer.gserviceaccount.com, Oct 20 2017


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : Jakob Gruber
  Commit : f7b09b3bdd96b4bd686b413c7a26e13f08ac4601
  Date   : Mon Oct 16 16:04:12 2017
  Subject: Reland "[snapshot] Ship lazy TFJ builtins"

Bisect Details
  Configuration: android_webview_arm64_aosp_perf_bisect
  Benchmark    : memory.top_10_mobile
  Metric       : memory:webview:all_processes:reported_by_chrome:v8:effective_size_avg/background/after_http_www_amazon_com_gp_aw_s_k_nexus
  Change       : 2.49% | 10021480.0 -> 10270588.0

Revision                           Result                   N
chromium@509316                    10021480 +- 259544       9      good
chromium@509324                    9992361 +- 126530        6      good
chromium@509325                    9987477 +- 248329        6      good
chromium@509325,v8@972d460f4f      10002264 +- 97437.4      6      good
chromium@509325,v8@f7b09b3bdd      10241617 +- 127312       6      bad       <--
chromium@509325,v8@efa038361d      10345048 +- 384298       6      bad
chromium@509325,v8@6627b0aaf7      10280985 +- 259967       6      bad
chromium@509326                    10324107 +- 187594       6      bad
chromium@509328                    10242509 +- 160826       6      bad
chromium@509331                    10299856 +- 273851       6      bad
chromium@509346                    10218133 +- 183188       6      bad
chromium@509376                    10270588 +- 291511       6      bad

Please refer to the following doc on diagnosing memory regressions:
  https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-webview --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http.www.amazon.com.gp.aw.s.k.nexus memory.top_10_mobile

More information on addressing performance regressions:
  http://g.co/ChromePerformanceRegressions

Debug information about this bisect:
  https://chromeperf.appspot.com/buildbucket_job_status/8965228267222581616


For feedback, file a bug with component Speed>Bisection
Project Member

Comment 18 by 42576172...@developer.gserviceaccount.com, Oct 20 2017

 Issue 776853  has been merged into this issue.
Project Member

Comment 19 by 42576172...@developer.gserviceaccount.com, Oct 21 2017

 Issue 776854  has been merged into this issue.
Project Member

Comment 20 by 42576172...@developer.gserviceaccount.com, Oct 21 2017

 Issue 776855  has been merged into this issue.
Project Member

Comment 21 by 42576172...@developer.gserviceaccount.com, Oct 21 2017

 Issue 776860  has been merged into this issue.
Status: Started (was: Assigned)
All of these regressions seem to behave similarly:

Code space usage decreased by one page at point ID 502483 (probably due to csa write barriers). At point ID 509350, it increases by one page. This happens only on specific windows bots, all other machines show significant improvements for this range.

See e.g. https://chromeperf.appspot.com/report?sid=3dffc3ac07df5a3003fc9996123acd63cc8e1ffe6eff14d1317c36b6e0bc37ae&start_rev=500809&end_rev=510719

So far, I don't have a local repro on my windows machine.
Project Member

Comment 23 by 42576172...@developer.gserviceaccount.com, Oct 23 2017

 Issue 777288  has been merged into this issue.
Project Member

Comment 24 by 42576172...@developer.gserviceaccount.com, Oct 23 2017

 Issue 777287  has been merged into this issue.
Blockedon: 777365
Project Member

Comment 26 by 42576172...@developer.gserviceaccount.com, Oct 25 2017

 Issue 776851  has been merged into this issue.
Still unable to repro this locally.

On win10, I ran system_health.memory_desktop {yandex,hackernews,lazers}. No change in memory consumption with {hackernews,lazers}.

yandex has a 1M regression in old space, this may be related to GC timing.
Seems related to increased fragmentation. allocated_objects_size_avg decreases by ~300K while effective_size_avg goes up by 512K; so we're actually allocating fewer objects but these use more pages. Maybe related to GC heuristics or timing? 

See https://chromeperf.appspot.com/report?sid=ac813a819bf2ac802aee34e06a1f916e759342eb931cd6838e262c69847440c8&rev=509350

+cc Ulan and Hannes fyi.
Cc: u...@chromium.org hpayer@chromium.org
Interesting that OLD_SPACE shows the same symptoms of increased fragmentation:

https://chromeperf.appspot.com/report?sid=51ede95d81dc196b5d71324ccad64d76de7fc0e71803a15a8885e8bf406c8dca&rev=509350
Status: Fixed (was: Started)
These regressions are fixed by enabling lazy bytecode handlers (see the experiment at point id 515171).

Sign in to add a comment