Issue metadata
Sign in to add a comment
|
2.7%-3% regression in memory.top_10_mobile at 460432:460529 |
||||||||||||||||||||||
Issue descriptionSee the link to graphs below.
,
Mar 30 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8983671136542789504
,
Mar 31 2017
=== Auto-CCing suspected CL author asvitkine@chromium.org === Hi asvitkine@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : asvitkine Commit : d47c7180be6c7531200d034d36c470fb5980f0e4 Date : Wed Mar 29 19:44:58 2017 Subject: Fix perfbotnot feature param setting from trial testing config. Bisect Details Configuration: android_nexus6_perf_bisect Benchmark : memory.top_10_mobile Metric : memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size_avg/foreground/https_www_google_co_uk_hl_en_q_science Change : 1.59% | 18656256.0 -> 18947730.2857 Revision Result N chromium@460431 18656256 +- 263890 6 good chromium@460465 18590379 +- 135158 6 good chromium@460482 18597888 +- 176936 6 good chromium@460491 18660181 +- 200946 6 good chromium@460495 18631461 +- 195588 14 good chromium@460497 18748709 +- 1020417 21 good chromium@460498 19169166 +- 799398 9 bad <-- chromium@460499 18947730 +- 278993 14 bad Please refer to the following doc on diagnosing memory regressions: https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests memory.top_10_mobile Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8983671136542789504 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5769322523262976 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Mar 31 2017
+sullivan Annie, so this is one of the regressions from my CL fixing field_trial_testing_config. So ideally I guess we'd want to figure out what caused this by finding which exact field trial is doing it. I can narrow down the list of things in field trial config by: a) those that apply to Android, b) that use Feature API and c) which use params and access through their features. Once I have this list, is there a good way to narrow down which one is the cause? e.g. can I just run a perf try job or something that will tell me whether the regression goes away - or would we need to actually commit them to see their effect?
,
Mar 31 2017
+nednguyen as well
,
Mar 31 2017
asvitkine@: you can also look at the memory trace before & after regression to see which component regress most. +memory experts for help with this
,
Mar 31 2017
Here are the relevant trials that apply to Android and have both feature and params: [u'--force-fieldtrials=AutofillCreditCardDropdownVariations/LastUsedDateAndExpDate_IncreaseDropdownItemHeight_Experiment/AutofillCreditCardSigninPromo/EnabledThree/AutofillProfileOrderByFrecency/EnabledLimitTo3/BackgroundVideoOptimizations/BackgroundOptimizationEnabled5sOrLess/BrowserScheduler/Default/CSSExternalScanner/Enabled_ScanAndPreload/DataCompressionProxyLoFi/Enabled/DataReductionProxyFREPromo/Enabled_Warmup/DataReductionProxyServerExperiments/Brotli/DisallowFetchForDocWrittenScriptsInMainFrame/DocumentWriteEvaluatorGroup/ExpensiveBackgroundTimerThrottling/Enabled_30sMaxThrottilingDelay/LowPriorityIFrames/Enabled/NTPArticleSuggestions/EnabledSoftFetches/NTPBookmarkSuggestions/Phase2Enabled3wWithDesktop/NTPPersonalizedSectionRanking/Enabled-ClickBasedRanker/NTPRecentForeignTabs/Enabled-ThreeHour/NetProxyPreconnects/Enabled/NetworkQualityEstimator/Enabled/NetworkTimeQueries/NetworkTimeQueriesEnabledBackgroundAndOnDemand/NoStatePrefetchValidation/Enabled/OmniboxBundledExperimentV1/Beta_Android_PhysWeb_QuerySuggest_Experiment/PermissionPromptUIAndroid/ModalGestureEnabled/PersistentHistograms/EnabledOnDiskNoStability3/Precache/EnabledCGRUV20b/ProgressBarAnimationAndroid/SmoothIndeterminateAnimation/ReportCertificateErrors/ShowAndPossiblySend/SafeBrowsingAndroid/Enabled/SafeBrowsingThreatDomDetailsTagAttributes/Enabled/SafeBrowsingUseLocalBlacklist/Use3PAPI/TranslateServerStudy/SmartRendering/UMA_EnableCellularLogUpload/Enabled_wo_sampling/UpdateMenuItem/EnabledWithNewerVersionAvailableSummary/WebBluetoothBlocklist/TestGroup', u'--force-fieldtrial-params=AutofillCreditCardDropdownVariations.LastUsedDateAndExpDate_IncreaseDropdownItemHeight_Experiment:dropdown_item_height/56/show_expiration_date/true,AutofillCreditCardSigninPromo.EnabledThree:impression_limit/3,AutofillProfileOrderByFrecency.EnabledLimitTo3:limit/3,BackgroundVideoOptimizations.BackgroundOptimizationEnabled5sOrLess:max_keyframe_distance_ms/5000,BrowserScheduler.Default:RedirectSequencedWorkerPools/true,CSSExternalScanner.Enabled_ScanAndPreload:cssExternalScannerNoPreload/false/cssExternalScannerPreload/true,DataCompressionProxyLoFi.Enabled:hysteresis_period_seconds/120/rtt_msec/2000,DataReductionProxyFREPromo.Enabled_Warmup:enable_warmup/true,DataReductionProxyServerExperiments.Brotli:exp/allow_brotli,DisallowFetchForDocWrittenScriptsInMainFrame.DocumentWriteEvaluatorGroup:disallowFetchForDocWrittenScriptsInMainFrame/false/disallowFetchForDocWrittenScriptsInMainFrameOnSlowConnections/false,ExpensiveBackgroundTimerThrottling.Enabled_30sMaxThrottilingDelay:cpu_budget/0%2E01/initial_budget/1%2E0/max_budget/3%2E0/max_delay/-1,LowPriorityIFrames.Enabled:lowPriorityIframes/true,NTPArticleSuggestions.EnabledSoftFetches:fetching_interval_hours-wifi-active_ntp_user/8/scheduler_trigger_types/persistent_scheduler_wake_up%2Cntp_opened/send_top_languages/true/soft_fetching_interval_hours-active-active_ntp_user/2/soft_fetching_interval_hours-active-active_suggestions_consumer/1/soft_fetching_interval_hours-active-rare_ntp_user/4,NTPBookmarkSuggestions.Phase2Enabled3wWithDesktop:bookmarks_consider_desktop_visits/true/bookmarks_max_age_in_days/21,NTPPersonalizedSectionRanking.Enabled-ClickBasedRanker:category_ranker/click_based,NTPRecentForeignTabs.Enabled-ThreeHour:max_foreign_tabs_age_in_minutes/180,NetProxyPreconnects.Enabled:restrict_to_one_preconnect_for_proxies/true,NetworkQualityEstimator.Enabled:2G%2EDefaultMedianKbps/81/2G%2EDefaultMedianRTTMsec/1337/3G%2EDefaultMedianKbps/658/3G%2EDefaultMedianRTTMsec/297/4G%2EDefaultMedianKbps/1756/4G%2EDefaultMedianRTTMsec/159/Bluetooth%2EDefaultMedianKbps/449/Bluetooth%2EDefaultMedianRTTMsec/135/Ethernet%2EDefaultMedianKbps/3263/Ethernet%2EDefaultMedianRTTMsec/106/HalfLifeSeconds/60/None%2EDefaultMedianKbps/567/None%2EDefaultMedianRTTMsec/272/Slow2G%2EThresholdMedianHttpRTTMsec/2000/Unknown%2EDefaultMedianKbps/1916/Unknown%2EDefaultMedianRTTMsec/121/WiFi%2EDefaultMedianKbps/2736/WiFi%2EDefaultMedianRTTMsec/168/correlation_logging_probability/0%2E001,NetworkTimeQueries.NetworkTimeQueriesEnabledBackgroundAndOnDemand:FetchBehavior/background-and-on-demand,NoStatePrefetchValidation.Enabled:mode/no_state_prefetch,OmniboxBundledExperimentV1.Beta_Android_PhysWeb_QuerySuggest_Experiment:PhysicalWebAfterTyping/true/PhysicalWebZeroSuggest/true,PermissionPromptUIAndroid.ModalGestureEnabled:require_gesture/true,PersistentHistograms.EnabledOnDiskNoStability3:send_unreported_metrics/no/storage/MappedFile,Precache.EnabledCGRUV20b:config_url/https%3A%2F%2Fwww%2Egstatic%2Ecom%2Fchrome%2Fwifiprefetch%2Fprecache_config_g20,ProgressBarAnimationAndroid.SmoothIndeterminateAnimation:progress-bar-animation/smooth-indeterminate,ReportCertificateErrors.ShowAndPossiblySend:sendingThreshold/1%2E0,SafeBrowsingAndroid.Enabled:enabled/true/types_to_check/0%2C1%2C3%2C6%2C7%2C8%2C9%2C10%2C11%2C13%2C14%2C15,SafeBrowsingThreatDomDetailsTagAttributes.Enabled:tag_attribute_csv/div%2Cid%2Ciframe%2Cid,SafeBrowsingUseLocalBlacklist.Use3PAPI:check_local_blacklist/false,TranslateServerStudy.SmartRendering:server_params/smrd,UMA_EnableCellularLogUpload.Enabled_wo_sampling:Enabled/true/Optimize/true/Sample_Probability/100/Uma_Quota/204800/Uma_Ratio/0%2E05,UpdateMenuItem.EnabledWithNewerVersionAvailableSummary:enable_update_badge/true/enable_update_menu_item/true/show_summary/true,WebBluetoothBlocklist.TestGroup:blocklist_additions/00060000%3Ae%2Cfffd%3Ae%2Ced5f25a4%3Ae', u'--enable-features=AutofillCreditCardLastUsedDateDisplay<AutofillCreditCardDropdownVariations,AutofillCreditCardPopupLayout<AutofillCreditCardDropdownVariations,AutofillCreditCardSigninPromo<AutofillCreditCardSigninPromo,BackgroundVideoTrackOptimization<BackgroundVideoOptimizations,DocumentWriteEvaluator<DisallowFetchForDocWrittenScriptsInMainFrame,ExpensiveBackgroundTimerThrottling<ExpensiveBackgroundTimerThrottling,NTPArticleSuggestions<NTPArticleSuggestions,NTPBookmarkSuggestions<NTPBookmarkSuggestions,ContentSuggestionsCategoryRanker<NTPPersonalizedSectionRanking,NTPForeignSessionsSuggestions<NTPRecentForeignTabs,NetworkTimeServiceQuerying<NetworkTimeQueries,NoStatePrefetch<NoStatePrefetchValidation,ModalPermissionPrompts<PermissionPromptUIAndroid,PersistentHistograms<PersistentHistograms,ThreatDomDetailsTagAttributes<SafeBrowsingThreatDomDetailsTagAttributes', u'--disable-features=DisplayPersistenceToggleInPermissionPrompts<PermissionPromptUIAndroid'] Next, we could narrow which of these actually look up their params by feature, vs trial (the latter we can ignore), but I suspect this won't narrow down the list much, since best practice is to query by feature.
,
Mar 31 2017
Re # 7, yep, you should be able to run perf try jobs with changes to the field trial config to try and binary search the cause.
,
Mar 31 2017
OK. I don't have time to look at this today - so will attempt that next week. Unless someone wants to help out with this? It should be a matter of just removing entries from field_trial_testing_config.json corresponding to the above command-line (i.e. for each fieldtrials entry) and running through perf try jobs.
,
Apr 5 2017
Okay, so I'm now trying to figure out which trial config is the cause. I'm doing this by bisecting through the configs - so here's my perf try bot invocation for a CL that removes the testing configs for the first half of the relevant studies in comment 7: $ tools/perf/run_benchmark try android-nexus5X memory.top_10_mobile WARNING:root:Benchmark memory.top_10_mobile has ShouldDisable() method defined. If your trybot run does not produce any results, it is possible that the benchmark is disabled on the target trybot platform. Running try job.... view progress here https://codereview.chromium.org/2800753002. Repo Name: src Path: /Users/asvitkine/chromium/src Branch: fts_1 Perf Try job sent to rietveld for android platform. WARNING:root:psutil is not installed on the system. Not listing possible leaked processes. To install psutil, see: https://pypi.python.org/pypi/psutil
,
Apr 5 2017
Also sent a CL to disable the second half of trials for completeness: e$ tools/perf/run_benchmark try android-nexus5X memory.top_10_mobile WARNING:root:Benchmark memory.top_10_mobile has ShouldDisable() method defined. If your trybot run does not produce any results, it is possible that the benchmark is disabled on the target trybot platform. Running try job.... view progress here https://codereview.chromium.org/2804723002. Repo Name: src Path: /Users/asvitkine/chromium/src Branch: fts_2 Perf Try job sent to rietveld for android platform. WARNING:root:psutil is not installed on the system. Not listing possible leaked processes. To install psutil, see: https://pypi.python.org/pypi/psutil
,
Apr 6 2017
,
Apr 6 2017
So with the first CL (which disables first half of trials), we see memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size / https_www_google_co_uk_hl_en_q_science go from 15.1 MB TOT to 14.1 MB with patch. So it sounds like it's one of those. Will continue bisecting.
,
Apr 7 2017
Narrowed this down to be one of the following field trial testing configs: NTPArticleSuggestions NTPBookmarkSuggestions NTPPersonalizedSectionRanking NTPRecentForeignTabs Still running more try jobs to narrow down to exact one. But given they're all NTP, updating labels and cc'ing folks.
,
Apr 7 2017
Could it be related to issue 708710 ?
,
Apr 7 2017
,
Apr 7 2017
+tedchoc: re #15
,
Apr 7 2017
Update: It's one of NTPArticleSuggestions or NTPBookmarkSuggestions that's causing the regression. Will prepare one more perf try job to narrow down which one. In the meanwhile, cc'ing authors of those testing configs. (+jkrcal +treib) Summary: field trial config params were not working correctly before for testing configs. When I fixed this, we saw this regression and I narrowed it down to one of those two configs.
,
Apr 8 2017
And my bisecting through entries in field trial config has concluded with https://codereview.chromium.org/2800223002/ which showed that the testing config for NTPArticleSuggestions is the cause of the regression. In the perf try run results, we can see the regression from 11.1MB to 11.7MB in "memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size". Assigning to jkrcal@ based on the above.
,
Apr 8 2017
,
Apr 10 2017
The NTPArticleSuggestions feature is enabled by default anyway. Does that mean it's actually one of the params that's causing the memory regression?
,
Apr 10 2017
Is there a way to run these tests locally? Or is the simplest way what you actually did, creating a CL, uploading it, and running android_perf_bisect_builder and android_nexus5X_perf_bisect bots?
,
Apr 10 2017
To run locally you do, e.g.: $ tools/perf/run_benchmark --browser android-chromium memory.top_10_mobile More details on memory benchmarks in general: https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md
,
Apr 10 2017
Yes, it's one of the params as previously they weren't being applied on perf bots.
,
Apr 12 2017
I have it on my radar. Did not manage to run the perf benchmark locally so far. Will continue tomorrow (was too much occupied by 59 BP so far).
,
Apr 13 2017
I am looking into it. 1) I've started a few variants of the perf bot: https://codereview.chromium.org/2807973005/ (sanity check, just removing the params) https://codereview.chromium.org/2817923002/ (removing params related to the scheduler for article suggestions) https://codereview.chromium.org/2808763011/ (removing params related to the language model) 2) The results table feels weird (link below). The table says (for memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size) that - ToT 11.1 MB - Patch 11.7MB As the patch removes the parameters, it looks as if the default config (obtained by removing params) had worse memory footprint than the experiment with the overridden param (ToT). Aren't the "ToT" and "Patch" columns swapped? https://00e9e64bac2c180a24e942b208c680c2eeb5dbfd5a6325cf1d-apidata.googleusercontent.com/download/storage/v1/b/chromium-telemetry/o/html-results%2Fresults-2017-04-07_17-42-56?qk=AD5uMEtkwACFWwguBbY109_mBI2lj6zaIvOVgkKiJ1rrX93pIRtkYbMRcw4tag6AtUzI-bSOtxF8EDdpzkN9V399yXdZlQCiPKs-ORLkOvYniLHezHkXJuxIyAn5lXqNEm8YdOuuyV-ykLYPz99EPokvHZ-lzauNFwdsORh1zXeslVe3hEQlFobKtRaPCX6_SS2Vr71-AM6P_LDbfeUARa_j-e4rS_02EDisZm0Y-UCMjmvQbK8eQBgCHwj4WvNGhH8zECtokBpi5JtQ_QoitOgpynCDRXqVrP_wmy_OFJL7VfK8kN1WogjfU3HK2xQ_O9Xf8wYRICuQtEPQx-bk0MvY78ErBk0b1Lk99m4MPyfDiAn_04dXGu8vfvuoLefcmHMlvIqaaE4mDm-oL4wWCB-MxIR0-bUDVWdHCbdOC-RWX3k_lM93zCU6wdsDTzUxmvsQ7qdvwoLXevDPFvEDZjysgrIF7yDe8clorHsb4mX01ygt_I0XrGmwfqsKe38C__1WEgWB-2d5jcfIrtVnELMNSCkVzs45YsR2EEV1ZE_2LWJEylggeNCXBOpR-dOKUvIVRkgWjExr468mShXHpmZl3HKo8Ti7ORB-GgbtStRMZPvTbNNBE8ofW5ZC12PuW1uXg-Ca5IB8MjDGNCu6JaS3rxcqyqP4IzX7iX3hmZuwuxwvde39ZeZsXlJz3HD8a2QYnUSS0mdhnggOfhPXeR4GbUqIT7HNt55lBSco7qn5IE3xvH_v1oRpOMIDCl2hqgjsE9MUoWDhZzCW2T6wL0hpgcbNaS9TmwAKSfOF3bNGnaF03v-f27tkmNSqFDOsGN7UUEvQuP5T
,
Apr 13 2017
,
Apr 13 2017
Alexei, I am puzzled by the results of the latter two perf-bots. I split the params into two disjoint groups and tried what happens if I remove each of the groups - https://codereview.chromium.org/2817923002/ - https://codereview.chromium.org/2808763011/ For both of the groups I can cite your earlier assessment from #19: In the perf try run results, we can see the regression from 11.1MB to 11.7MB in "memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size"." While this is of course possible that both of them trigger the same regression and thus the effect on memory consumption is not additive, my brain fails to believe that it is the case. The two sets of params are read from different files, having absolutely orthogonal effect. Do you see any explanation for this phenomenon?
,
Apr 13 2017
On a related note: I failed to run the perf-tests locally, so far. The param "--browser android-chromium" does not work for me. My list of browsers is: Android device TA9290B3E6 desktop beta dev stable system How do I need to compile clankium so that the apk is found by the script? Alternatively, is there a way to print debug output in the perfbot and observe it after it finishes the job on the server? I would like to see what the scheduler actually does during the test...
,
Apr 18 2017
,
Apr 19 2017
One way to run with the local build is to specify: --browser=exact --browser-executable=out_android/Release/apks/Chrome.apk Another thing you may want to try here is since the java heap is affected, you could grab a heap dump of the java heap with/without the experiment parameters and compare. How to get heap dump: https://developer.android.com/studio/profile/investigate-ram.html#HeapDump For comparison: use go/ahat
,
May 2 2017
Taking back for now, since we're not certain of my earlier Finch config bisects being correct. Will try to do some more to understand the problem.
,
May 8 2017
An update: Ned & Simon helped me make sense of some the perf try bot output. I've kicked of some new perf try runs at the revision the original CL landed (thanks Simon for that support!) to see if I can reproduce the results and binary search through the field trial configs from there. I've also separately started a run to reproduce the original thing I found - which is the NTPArticles config's masking of a memory regression on TOT.
,
May 9 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8980049769329360128
,
May 10 2017
=== Auto-CCing suspected CL author skobes@chromium.org === Hi skobes@chromium.org, the bisect results pointed to your CL, please take a look at the results. === BISECT JOB RESULTS === Perf regression found with culprit Suspected Commit Author : skobes Commit : 56451775d90238d9def5e9663771de8d0ca7ef6f Date : Tue Mar 28 18:46:38 2017 Subject: Feed ScrollableArea::showOverlayScrollbars into ScrollbarAnimationController. Bisect Details Configuration: android_nexus6_perf_bisect Benchmark : memory.top_10_mobile Metric : memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size_avg/foreground/https_www_google_co_uk_hl_en_q_science Change : 1.32% | 18407196.4444 -> 18650453.3333 Revision Result N chromium@460084 18407196 +- 221737 9 good chromium@460150 18432000 +- 191660 6 good chromium@460167 18432853 +- 185468 6 good chromium@460175 18408789 +- 147910 9 good chromium@460177 18470912 +- 665447 14 good chromium@460178 18740907 +- 715482 9 bad <-- chromium@460179 18578091 +- 208414 9 bad chromium@460183 18763776 +- 919112 9 bad chromium@460215 18650453 +- 201885 6 bad Please refer to the following doc on diagnosing memory regressions: https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests memory.top_10_mobile Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8980049769329360128 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5887561458778112 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
May 10 2017
Let me file a separate bug for the regression bisected in comment 36. As I mentioned, two separate regressions got lumped together in this bug, so I've split that one off to: crbug.com/720361
,
May 10 2017
Assigning this one back to me since the issue with skobes@'s CL has been split out to a separate bug.
,
May 11 2017
Some more results from perf try bots. I'm running with Simon's infra to run the patches against revision d47c7180be6c7531200d034d36c470fb5980f0e4 - which is where my original CL landed. So they should show exactly the effect. The candidate field trial configs that use params and features are the following: AutofillCreditCardDropdownVariations AutofillCreditCardSigninPromo BackgroundVideoOptimizations DisallowFetchForDocWrittenScriptsInMainFrame ExpensiveBackgroundTimerThrottling NTPArticleSuggestions NTPBookmarkSuggestions NTPPersonalizedSectionRanking NTPRecentForeignTabs NetworkTimeQueries NoStatePrefetchValidation PermissionPromptUIAndroid PersistentHistograms SafeBrowsingThreatDomDetailsTagAttributes My binary is as follows: Split into two groups: A [ AutofillCreditCardDropdownVariations AutofillCreditCardSigninPromo BackgroundVideoOptimizations DisallowFetchForDocWrittenScriptsInMainFrame ExpensiveBackgroundTimerThrottling NTPArticleSuggestions NTPBookmarkSuggestions ] B[ NTPPersonalizedSectionRanking NTPRecentForeignTabs NetworkTimeQueries NoStatePrefetchValidation PermissionPromptUIAndroid PersistentHistograms SafeBrowsingThreatDomDetailsTagAttributes ] Then: AA [ AutofillCreditCardDropdownVariations AutofillCreditCardSigninPromo BackgroundVideoOptimizations ] AB[ DisallowFetchForDocWrittenScriptsInMainFrame ExpensiveBackgroundTimerThrottling NTPArticleSuggestions NTPBookmarkSuggestions ] And then I dove into each of the AA ones individually. The results on memory:chrome:all_processes:reported_by_os:system_memory:java_heap:proportional_resident_size -> https_www_google_co_uk_hl_en_q_science are as follows: A => +180k B => +110k AA => +130k AB => -90k AA_AutofillCreditCardDropdownVariations => 0 AA_AutofillCreditCardSigninPromo => +130k AA_BackgroundVideoOptimizations => +130k So from those results, it seems the data is not stable enough to effectively bisect at this granularity? i.e. the first split into A & B showed both groups regressing. But then when I split A into AA and AB, the result was +130k & -90k and those two don't add up to +180k which was the result for A. Similarly, for AA (+130k) being split into the three groups - it showed two *different* things showing a +130k change *each*. So basically, it doesn't seem the binary search method through these is working. The data just doesn't seem stable? Here are the CLs in question for reference: A: https://codereview.chromium.org/2871733002/ B: https://codereview.chromium.org/2868743003/ AA: https://codereview.chromium.org/2870023004/ AB: https://codereview.chromium.org/2872923003/ AA_AutofillCreditCardDropdownVariations: https://codereview.chromium.org/2870413003/ AA_AutofillCreditCardSigninPromo: https://codereview.chromium.org/2874853002/ AA_BackgroundVideoOptimizations: https://codereview.chromium.org/2876463005/ So until we can make sense with the above, I'm not sure there's more progress I can do here. I guess the question I have - how confident are we in the try run results? i.e. do they do multiple runs to get statistical significance? Do we have any confidence intervals?
,
May 12 2017
Wow, great work Alexei! From some of my earlier investigations, I share your feelings about instability of results :|
,
May 12 2017
perezju, hjd, primiano: can one of you take a look at the results in #39 and help make sense of them?
,
May 15 2017
Re #39: So, the metric for proportional resident size will be noisy and it is expected. That is why we created metrics for each allocator to observe more stable metrics on benchmarks. My question is: which process do we expect the memory to increase because of the suspected CL? I think the answer is browser. Then we should be looking at browser_process:proportional_resident_size and not all_process. If it is browser process, is the memory allocated from native heap or java heap? I think the answer is native. So, we should look at either browser_process:reported_by_os:private_dirty or browser_process:reported_by_chrome:malloc. If we are able to spot the regression by looking at these metrics locally, then is it easier to narrow down. If these metrics are still noisy, we can turn on heap profiling for the native heap on browser. This would give us a component wise split of memory of browser native heap. But, first let's look at the above metrics and see how noisy they are. Note: I just assumed the regression is on browser native. Looking at the traces in the regression on the links in comment #1: the regression seems to be because of all components like renderer blink, renderer partition alloc and browser malloc regressed by small amounts totaling to 2MB. This is very strange.
,
Aug 14 2017
Alexei, this is a pretty old issue now. Should it still be considered P1? The target milestone is also obsolete.
,
Aug 14 2017
Closing. Unfortunately, we weren't able to make much progress here despite earnest effort (several weeks) trying run perf try bots to figure out which field trial testing config was the cause. It was not clear from the perf try bots whether the deltas were due to to just variance or real effects. |
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by lanwei@chromium.org
, Mar 30 2017