Issue metadata
Sign in to add a comment
|
48.7%-71.8% regression in system_health.memory_desktop at 410080:410419 |
||||||||||||||||||||
Issue descriptionSee the link to graphs below.
,
Aug 9 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9004795970942841504
,
Aug 10 2016
=== Auto-CCing suspected CL author rsesek@chromium.org === Hi rsesek@chromium.org, the bisect results pointed to your CL below as possibly causing a regression. Please have a look at this info and see whether your CL be related. ===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : Revert of [Mac] Only enable the CFBundleBlocker in the browser process. (patchset #1 id:1 of https://codereview.chromium.org/2199313002/ ) Author : rsesek Commit description: Reason for revert: Speculative revert for perf regression. BUG= https://crbug.com/634917 Original issue's description: > [Mac] Only enable the CFBundleBlocker in the browser process. > > In child processes, which are sandboxed in most cases, the sandbox will prevent > access to the filesystem locations where the potentially blocked bundles are > stored. Furthermore, on macOS 10.11 and higher, the Google Chrome build is > codesigned in such a way where bundle loading is blocked by SIP. > > This reduces some of the "triggered DYLD shared region unnest for map" messages. > > BUG= 428858 > R=mark@chromium.org > > Committed: https://crrev.com/65e732d3b4dc595512b731143fd49d372acc1a87 > Cr-Commit-Position: refs/heads/master@{#409244} TBR=mark@chromium.org,thakis@chromium.org # Not skipping CQ checks because original CL landed more than 1 days ago. BUG= 428858 Review-Url: https://codereview.chromium.org/2218163002 Cr-Commit-Position: refs/heads/master@{#410113} Commit : 3026ec1f3bd67bb2cac5e42fe6d1a1298efcd049 Date : Fri Aug 05 18:22:47 2016 ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@410079 44525384 0.0 5 good chromium@410101 44525384 0.0 5 good chromium@410112 44525384 0.0 5 good chromium@410113 76411003 147876 5 bad <-- chromium@410114 76212245 147001 5 bad chromium@410115 76343931 180226 5 bad chromium@410117 82046485 13006664 5 bad chromium@410122 76343931 180226 5 bad chromium@410164 82569851 13995365 5 bad chromium@410249 76343931 180226 5 bad chromium@410419 82702459 14104744 5 bad Bisect job ran on: mac_retina_perf_bisect Bug ID: 635918 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_desktop Test Metric: load_search-memory:chrome:all_processes:reported_by_chrome:cc:effective_size_avg/load_search_baidu Relative Change: 85.74% Score: 99.9 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1558 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9004795970942841504 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5802975524552704 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 10 2016
Don't think that's right. That CL was a revert for a different perf regression.
,
Aug 10 2016
,
Aug 10 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9004700693610793536
,
Aug 10 2016
The result of the bisect seems very clear, but I'm running another bisect.
,
Aug 11 2016
===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : Revert of [Mac] Only enable the CFBundleBlocker in the browser process. (patchset #1 id:1 of https://codereview.chromium.org/2199313002/ ) Author : rsesek Commit description: Reason for revert: Speculative revert for perf regression. BUG= https://crbug.com/634917 Original issue's description: > [Mac] Only enable the CFBundleBlocker in the browser process. > > In child processes, which are sandboxed in most cases, the sandbox will prevent > access to the filesystem locations where the potentially blocked bundles are > stored. Furthermore, on macOS 10.11 and higher, the Google Chrome build is > codesigned in such a way where bundle loading is blocked by SIP. > > This reduces some of the "triggered DYLD shared region unnest for map" messages. > > BUG= 428858 > R=mark@chromium.org > > Committed: https://crrev.com/65e732d3b4dc595512b731143fd49d372acc1a87 > Cr-Commit-Position: refs/heads/master@{#409244} TBR=mark@chromium.org,thakis@chromium.org # Not skipping CQ checks because original CL landed more than 1 days ago. BUG= 428858 Review-Url: https://codereview.chromium.org/2218163002 Cr-Commit-Position: refs/heads/master@{#410113} Commit : 3026ec1f3bd67bb2cac5e42fe6d1a1298efcd049 Date : Fri Aug 05 18:22:47 2016 ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@410079 32359536 1025801 5 good chromium@410101 32359536 1025801 5 good chromium@410112 32359536 1025801 5 good chromium@410113 48793507 280.434 5 bad <-- chromium@410114 48711638 112974 5 bad chromium@410115 51636029 6355900 5 bad chromium@410117 48670115 112828 5 bad chromium@410122 48629053 92047.3 5 bad chromium@410164 48670115 112594 5 bad chromium@410249 48670115 112594 5 bad chromium@410419 48752906 92254.1 5 bad Bisect job ran on: mac_retina_perf_bisect Bug ID: 635918 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_desktop Test Metric: load_search-memory:chrome:all_processes:reported_by_chrome:gpu:effective_size_avg/load_search_baidu Relative Change: 50.66% Score: 99.9 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1561 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9004700693610793536 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5795554492153856 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 11 2016
resek: Another bisect (on a different metric) confirmed that the revert (r410113) indeed caused the regression (+15.7 MiB in GPU effective size, +30.4 MiB in CC effective size). Please figure out whether this was expected and if anything needs to be done. I can see on the charts (https://chromeperf.appspot.com/group_report?bug_id=635918), that the regression followed a symmetrical improvement between r409580 and r409624 (http://test-results.appspot.com/revision_range?start=409580&end=409624). Is any of the patches in that range related to your revert (r410113)?
,
Aug 11 2016
No, my patch originally landed at 409244, which doesn't show as the starting point for the improvement. So either the starting point for the improvement is wrong, there's an issue with the bisect, or there's an issue with the test. 7490bda13d8646a6225b45bb45126be71e19358c and 8ee20259fc28430498df2b22cb4a506d215d26dd could be candidates, but it does not align with the graph. I'm inclined to WontFix.
,
Aug 12 2016
This is a +15.7 (+30.4) MiB regression in the effective size of GPU (cc), so I'd really like to get to the bottom of this. I'll bisect the improvement to figure out what caused it and hopefully it will shed some light. I don't think that the bisect is wrong here (given that it repeatedly blamed the same patch).
,
Aug 12 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9004527254386230368
,
Aug 12 2016
===== BISECT JOB RESULTS ===== Status: completed === Bisection aborted === The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression. Please contact the the team (see below) if you believe this is in error. === Warnings === The following warnings were raised by the bisect job: * Bisect failed to reproduce the regression with enough confidence. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@409579 44525384 0.0 12 good chromium@409624 44525384 0.0 8 bad Bisect job ran on: mac_retina_perf_bisect Bug ID: 635918 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_desktop Test Metric: load_search-memory:chrome:all_processes:reported_by_chrome:cc:effective_size_avg/load_search_baidu Relative Change: 0.00% Score: 0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1565 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9004527254386230368 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5790290741297152 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 15 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9004268388312819344
,
Aug 16 2016
=== Auto-CCing suspected CL author dmazzoni@chromium.org === Hi dmazzoni@chromium.org, the bisect results pointed to your CL below as possibly causing a regression. Please have a look at this info and see whether your CL be related. ===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : Add aboxhall as owner of modules/accessibility in Blink Author : dmazzoni Commit description: BUG=none Review-Url: https://codereview.chromium.org/2204593006 Cr-Commit-Position: refs/heads/master@{#409246} Commit : 8b1118e99d2b589b523ab2f97b3c6cc500f31df1 Date : Tue Aug 02 18:17:58 2016 ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@409210 76434632 116422 8 good chromium@409236 82635387 14143037 5 good chromium@409243 76410082 147345 5 good chromium@409245 76478792 2827.61 5 good chromium@409246 44525384 0.0 5 bad <-- chromium@409249 44525384 0.0 5 bad chromium@409262 48519240 11296331 8 bad chromium@409314 44525384 0.0 5 bad chromium@409417 44525384 0.0 5 bad chromium@409624 44525384 0.0 5 bad Bisect job ran on: mac_retina_perf_bisect Bug ID: 635918 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_desktop Test Metric: load_search-memory:chrome:all_processes:reported_by_chrome:cc:effective_size_avg/load_search_baidu Relative Change: 41.73% Score: 99.9 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1566 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9004268388312819344 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5887771248427008 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 16 2016
It's obviously not my change either, that's just a change to an OWNERS file, it doesn't affect Chrome at all. No idea what's going on, but it's troublesome that when the test fails it has no standard deviation.
,
Aug 16 2016
I agree that this is quite worrying. Here are the traces for the two revisions: r409245 (cc=72.9 MiB): https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_6-2016-08-15_17-19-42-28832.html r409246 (cc=42.5 MiB): https://console.developers.google.com/m/cloudstorage/b/chrome-telemetry-output/o/trace-file-id_6-2016-08-15_16-01-13-25061.html [Note that we were bisecting an *improvement*] Clearly, the numbers must be wrong because r409246 just adds an owner. The only rational explanation I can think of is that there must be a difference between the two builds: r409245: gs://chrome-perf/Mac Builder/full-build-mac_39bef912185b65331bb3f86c7f035eeb34b18792.zip r409246: gs://chrome-perf/Mac Builder/full-build-mac_8b1118e99d2b589b523ab2f97b3c6cc500f31df1.zip Unfortunately, the builder runs corresponding to these builds have already been deleted. To figure out if there's any difference between the two, I downloaded and unzipped them. I diffed the resulting folders (diff -Nqr r409245 r409246 | sed 's/^Files r409245\/\(.*\) and r409246\/\(.*\) differ$/\1/g' | sort). Firstly, the following groups of files seem irrelevant: ===== CRASHPAD & BREAKPAD [there were no crashes] ===== crashpad_handler crashpad_handler-54.0.2812.0.breakpad crashpad_handler-54.0.2813.0.breakpad crashpad_handler-54.0.2814.0.breakpad crashpad_handler-54.0.2815.0.breakpad crashpad_handler-54.0.2816.0.breakpad crashpad_handler-54.0.2817.0.breakpad Google Chrome.app/Contents/Versions/54.0.2817.0/Google Chrome Framework.framework/Helpers/crashpad_handler Google Chrome Framework-54.0.2812.0.breakpad Google Chrome Framework-54.0.2813.0.breakpad Google Chrome Framework-54.0.2814.0.breakpad Google Chrome Framework-54.0.2815.0.breakpad Google Chrome Framework-54.0.2816.0.breakpad Google Chrome Framework-54.0.2817.0.breakpad Google Chrome Framework.framework/Helpers/crashpad_handler ===== TEXT FILES WITH REVISION INFO ===== FULL_BUILD_REVISION Google Chrome.app/Contents/Info.plist Google Chrome.app/Contents/Versions/54.0.2817.0/Google Chrome Framework.framework/Resources/Info.plist Google Chrome Framework.framework/Resources/Info.plist ===== MOJO [no difference after unzipping] ===== irt_x64/gen/ipc/ipc.mojom.srcjar irt_x64/gen/mojo/public/interfaces/bindings/interface_control_messages.mojom.srcjar irt_x64/gen/mojo/public/interfaces/bindings/pipe_control_messages.mojom.srcjar ===== PROTO ===== pyproto/chrome/common/safe_browsing/download_file_types_pb2.pyc pyproto/google/__init__.pyc pyproto/google/protobuf/descriptor_database.pyc pyproto/google/protobuf/descriptor_pb2.pyc pyproto/google/protobuf/descriptor_pool.pyc pyproto/google/protobuf/descriptor.pyc pyproto/google/protobuf/__init__.pyc pyproto/google/protobuf/internal/api_implementation.pyc pyproto/google/protobuf/internal/containers.pyc pyproto/google/protobuf/internal/decoder.pyc pyproto/google/protobuf/internal/encoder.pyc pyproto/google/protobuf/internal/enum_type_wrapper.pyc pyproto/google/protobuf/internal/__init__.pyc pyproto/google/protobuf/internal/message_listener.pyc pyproto/google/protobuf/internal/python_message.pyc pyproto/google/protobuf/internal/type_checkers.pyc pyproto/google/protobuf/internal/well_known_types.pyc pyproto/google/protobuf/internal/wire_format.pyc pyproto/google/protobuf/message.pyc pyproto/google/protobuf/reflection.pyc pyproto/google/protobuf/symbol_database.pyc pyproto/google/protobuf/text_encoding.pyc pyproto/google/protobuf/text_format.pyc pyproto/google/third_party/six/six.pyc The following differences seemed potentially relevant: ===== BINARY FILES ===== Google Chrome.app/Contents/Versions/54.0.2817.0/Google Chrome Framework.framework/Google Chrome Framework Google Chrome.dSYM.tar.bz2 [The two contained files below differ] crashpad_handler.dSYM/Contents/Resources/DWARF/crashpad_handler Google Chrome Framework.dSYM/Contents/Resources/DWARF/Google Chrome Framework Google Chrome Framework.framework/Google Chrome Framework performance_browser_tests Of these files, the following differ in size: Google Chrome.dSYM.tar.bz2: 443484 -> 443448 Google Chrome Framework.dSYM/Contents/Resources/DWARF/Google Chrome Framework: 1917072 -> 1917096 irt_x64/obj/ppapi/proxy/libppapi_proxy.a: 42372 -> 42368 ===== NINJA FILES ===== build.ninja toolchain.ninja After sorting, the build.ninja files are identical. There are only 5 different lines between the toolchain.ninja files. Specifically, TOOL_VERSIONs of filter_libtool.py, linker_driver.py and compile_xcassets.py have changed: r409245 r409246 build/toolchain/mac/filter_libtool.py: 1468923233 -> 1468923090 build/toolchain/mac/linker_driver.py: 1468932295 -> 1468933041 build/toolchain/mac/compile_xcassets.py: 1468923233 -> 1468923090 these timestamps translate to 19th July 2016 with the following UTC times (yes, the first and the third one seem to go backwards): build/toolchain/mac/filter_libtool.py: 10:13:53 -> 10:11:30 build/toolchain/mac/linker_driver.py: 12:44:55 -> 12:57:21 build/toolchain/mac/compile_xcassets.py: 10:13:53 -> 10:11:30 Looking at the git history of the three files, the following two patches match the region: r406110 by rsesek@ landing on 18th July 2016 @ 21:59:17 UTC r406253 by sdefresne@ landing on 19th July 2016 @ 12:43:59 UTC rsesek,sdefresne: Is there any chance either of your changes could have led to such a dramatic improvement in memory usage? It seems unlikely that this could be due to a hardware/software change on the builder. The modification timestamps of the "Google Chrome Framework.framework/Google Chrome Framework" file are just 3 minutes apart: r409245: 2nd August 2016 @ 11:53 r409246: 2nd August 2016 @ 11:56 In fact, these two seem to be too close to come from the same device. Perhaps the two builds were generated by different slaves. sullivan,robertocn,dtu: Could there be a difference in build configuration across build slaves of a single bot? Any other ideas how the two builds could differ?
,
Aug 16 2016
We did at one point have a problem where most of the Mac builders were running 10.11 but one was running 10.9, but that was fixed in bug 608233 long before the revision range of this regression. For the perf builders, you can see some data on the sizes of the binaries they're building, and which revisions they built originally, by looking at the sizes chart: https://chromeperf.appspot.com/report?sid=24d6dcc57f09b4bed2677c35f7c66ebf2a8125bb0a558d55496af4d4af30c55c&start_rev=409225&end_rev=409300 It looks like they are building every revision, as expected. Does anything look strange in the sizes to you? I am not sure how to figure out: * Whether any revisions the bisects used were built by mac_perf_bisect_builder * Whether mac_perf_bisect_builder's slaves are all set up as expected. https://build.chromium.org/p/tryserver.chromium.perf/builders/mac_perf_bisect_builder Dave, do you know?
,
Aug 18 2016
In the past we have seen sizes differences across different builders despite them running the exact same hardware and OS version: https://chromeperf.appspot.com/report?sid=0b68579269a9a5a4402ed6ac605e79dc9b793a1ae695f94b6ec0b94117e9fede&start_rev=407068&end_rev=407282 I can confirm that all the Mac Builders and mac_perf_bisect_builders are running OS X 10.11.5 (15F34): https://docs.google.com/spreadsheets/d/1LTOSY9y1_sdDiL94XQTZrXmSzokn4p44hlgHQvLnEt4/preview
,
Aug 18 2016
rsesek,sdefresne: ping (#17)
,
Aug 18 2016
No, I don't think either of those CLs would have an effect on memory usage.
,
Aug 19 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9003899331866169952
,
Aug 19 2016
rsesek: Thanks! This is really really weird. I'll try to bisect the improvement once again with the "Use archive" option unchecked.
,
Aug 19 2016
,
Aug 19 2016
===== BISECT JOB RESULTS ===== Status: completed === Bisection aborted === The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression. Please contact the the team (see below) if you believe this is in error. === Warnings === The following warnings were raised by the bisect job: * Bisect failed to reproduce the regression with enough confidence. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@409579 73760627 9207606 12 good chromium@409624 84466568 14791277 8 bad Bisect job ran on: mac_retina_perf_bisect Bug ID: 635918 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_desktop Test Metric: load_search-memory:chrome:all_processes:reported_by_chrome:cc:effective_size_avg/load_search_baidu Relative Change: 27.47% Score: 0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1589 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9003899331866169952 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5344056775802880 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 19 2016
OK, I really don't understand this. The bisects in #15 and #26 were on the same platform and the same value. Yet they reported completely different numbers for r409624: Revision Mean Std Dev N Good? #15: chromium@409624 44525384 0.0 5 bad #26: chromium@409624 84466568 14791277 8 bad This is a complete enigma to me :-\
,
Aug 19 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9003878490358401616
,
Aug 19 2016
Furthermore, both bisects ran on the same slave a few days apart: #15: https://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1566 (Monday, August 15th) #26: https://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1589 (Friday, August 19th) and had the same configuration (except for a different "good" revision): #15: https://chromeperf.appspot.com/buildbucket_job_status/9004268388312819344 #26: https://chromeperf.appspot.com/buildbucket_job_status/9003899331866169952 I'm running another bisect with the same config as in #15.
,
Aug 20 2016
===== BISECT JOB RESULTS ===== Status: failed ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@409210 77676929 6094741 27 good chromium@409262 75282267 6147158 27 bad chromium@409314 76477740 2577.84 18 bad chromium@409417 76480840 0.0 5 bad chromium@409624 80409160 11121104 8 bad Bisect job ran on: mac_retina_perf_bisect Bug ID: 635918 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_desktop Test Metric: load_search-memory:chrome:all_processes:reported_by_chrome:cc:effective_size_avg/load_search_baidu Relative Change: 8.60% Score: 0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1592 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9003878490358401616 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5285414902956032 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 20 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9003786817281587280
,
Aug 20 2016
===== BISECT JOB RESULTS ===== Status: failed ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@409210 77676929 6094741 27 good chromium@409262 75282267 6147158 27 bad chromium@409314 76477740 2577.84 18 bad chromium@409417 76480840 0.0 5 bad chromium@409624 80409160 11121104 8 bad Bisect job ran on: mac_retina_perf_bisect Bug ID: 635918 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_desktop Test Metric: load_search-memory:chrome:all_processes:reported_by_chrome:cc:effective_size_avg/load_search_baidu Relative Change: 8.60% Score: 0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1592 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9003878490358401616 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5285414902956032 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 21 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9003694712405398560
,
Aug 21 2016
===== BISECT JOB RESULTS ===== Status: completed === Bisection aborted === The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression. Please contact the the team (see below) if you believe this is in error. === Warnings === The following warnings were raised by the bisect job: * Bisect failed to reproduce the regression with enough confidence. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@409210 76393757 149107 12 good chromium@409624 76436040 117011 8 bad Bisect job ran on: mac_retina_perf_bisect Bug ID: 635918 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_desktop Test Metric: load_search-memory:chrome:all_processes:reported_by_chrome:cc:effective_size_avg/load_search_baidu Relative Change: 0.00% Score: 0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1595 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9003878490358401616 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5285414902956032 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 22 2016
,
Aug 22 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9003624442702882688
,
Aug 22 2016
===== BISECT JOB RESULTS ===== Status: completed === Bisection aborted === The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression. Please contact the the team (see below) if you believe this is in error. === Warnings === The following warnings were raised by the bisect job: * Bisect failed to reproduce the regression with enough confidence. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@409579 46692080 5191918 12 good chromium@409624 46510576 5590339 8 bad Bisect job ran on: mac_retina_perf_bisect Bug ID: 635918 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_desktop Test Metric: load_search-memory:chrome:all_processes:reported_by_chrome:gpu:effective_size_avg/load_search_baidu Relative Change: 0.97% Score: 0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1599 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9003624442702882688 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5778642873876480 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 23 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9003542269757162832
,
Aug 23 2016
===== BISECT JOB RESULTS ===== Status: completed === Bisection aborted === The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression. Please contact the the team (see below) if you believe this is in error. === Warnings === The following warnings were raised by the bisect job: * Bisect failed to reproduce the regression with enough confidence. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@409210 48725019 101278 12 good chromium@409624 48779088 19901.5 8 bad Bisect job ran on: mac_retina_perf_bisect Bug ID: 635918 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_desktop Test Metric: load_search-memory:chrome:all_processes:reported_by_chrome:gpu:effective_size_avg/load_search_baidu Relative Change: 0.14% Score: 0 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1602 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9003542269757162832 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=6444331263590400 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 24 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/9003435607012180608
,
Aug 25 2016
===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : Revert of [Mac] Only enable the CFBundleBlocker in the browser process. (patchset #1 id:1 of https://codereview.chromium.org/2199313002/ ) Author : rsesek Commit description: Reason for revert: Speculative revert for perf regression. BUG= https://crbug.com/634917 Original issue's description: > [Mac] Only enable the CFBundleBlocker in the browser process. > > In child processes, which are sandboxed in most cases, the sandbox will prevent > access to the filesystem locations where the potentially blocked bundles are > stored. Furthermore, on macOS 10.11 and higher, the Google Chrome build is > codesigned in such a way where bundle loading is blocked by SIP. > > This reduces some of the "triggered DYLD shared region unnest for map" messages. > > BUG= 428858 > R=mark@chromium.org > > Committed: https://crrev.com/65e732d3b4dc595512b731143fd49d372acc1a87 > Cr-Commit-Position: refs/heads/master@{#409244} TBR=mark@chromium.org,thakis@chromium.org # Not skipping CQ checks because original CL landed more than 1 days ago. BUG= 428858 Review-Url: https://codereview.chromium.org/2218163002 Cr-Commit-Position: refs/heads/master@{#410113} Commit : 3026ec1f3bd67bb2cac5e42fe6d1a1298efcd049 Date : Fri Aug 05 18:22:47 2016 ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@410079 50849608 14141395 5 good chromium@410101 44525384 0.0 5 good chromium@410112 44525384 0.0 5 good chromium@410113 76344341 180599 5 bad <-- chromium@410114 76409877 147231 5 bad chromium@410115 76476027 457.947 5 bad chromium@410117 76345058 181266 5 bad chromium@410122 76277986 180039 5 bad chromium@410164 76279112 181592 5 bad chromium@410249 82701742 14105711 5 bad chromium@410419 76345058 181266 5 bad Bisect job ran on: mac_retina_perf_bisect Bug ID: 635918 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests system_health.memory_desktop Test Metric: load_search-memory:chrome:all_processes:reported_by_chrome:cc:effective_size_avg/load_search_baidu Relative Change: 50.14% Score: 99.9 Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1607 Job details: https://chromeperf.appspot.com/buildbucket_job_status/9003435607012180608 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5772976805380096 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Aug 25 2016
The metric is bimodal and maybe even trimodal. It's never going to be able to bisect. If you look at the individual values of the last run, even multiple iterations that ran consecutively produced bimodal values. chromium@410079 [44525384, 76146504, 44525384, 44525384, 44525384] chromium@410249 [76476232, 107933512, 76146504, 76476232, 76476232] I've attached plots of the results from comment 37 and comment 41. They both show these kinds of bimodal outliers.
,
Aug 25 2016
Is it possible that some kind of external factor could affect this metric? If, for example, some other process starting and stopping on the device could affect the metric, that could lead to this kind of bimodal behavior.
,
Aug 30 2016
,
Sep 16 2016
Temporarily declaring bankruptcy on the *desktop* system health benchmark. The number of alerts became unmanageable and the overall process needs to be improved to make it sustainable. The alerts have been turned off and I'm archiving the outstanding regressions. Note: this is just about desktop, the mobile system health stays up. |
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by petrcermak@chromium.org
, Aug 9 2016