Relative change set to "None" when bisect fails to reproduce |
||||||||
Issue descriptionExample: https://bugs.chromium.org/p/chromium/issues/detail?id=664191#c8 "Relative Change: None" Bisect used to print out the relative change in this case and I found it very useful. For example, if the original regression is 10%, and the relative change is 0.001% it means the regression is really not happening on the bisector. But if the regression is 10% and the relative change is 10%, either there is a lot of noise or there is a problem with the bisect.
,
Nov 14 2016
,
Nov 14 2016
,
Nov 16 2016
,
Nov 17 2016
,
Nov 18 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/7c338b8134cf3e966cd66c4ef9fc72c969ca77c3 commit 7c338b8134cf3e966cd66c4ef9fc72c969ca77c3 Author: Simon <simonhatch@chromium.org> Date: Thu Nov 17 17:51:05 2016 [bisect] - Compute relative change even when bisect fails to reproduce. Previously relative change was computed after checking for confidence. If the initial gathering references step fails, this is never called and the relative change is listed as null. BUG= 664743 Change-Id: I74ac02a3ea5bbfc1f38caf7f94818ab7b82a5ce8 Reviewed-on: https://chromium-review.googlesource.com/412300 Reviewed-by: Roberto Carrillo <robertocn@chromium.org> Commit-Queue: Simon Hatch <simonhatch@chromium.org> [modify] https://crrev.com/7c338b8134cf3e966cd66c4ef9fc72c969ca77c3/scripts/slave/recipe_modules/auto_bisect_staging/bisector.py [modify] https://crrev.com/7c338b8134cf3e966cd66c4ef9fc72c969ca77c3/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/basic_win32_bisect.json [modify] https://crrev.com/7c338b8134cf3e966cd66c4ef9fc72c969ca77c3/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/basic_win64_bisect.json [modify] https://crrev.com/7c338b8134cf3e966cd66c4ef9fc72c969ca77c3/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/no_repro.json [modify] https://crrev.com/7c338b8134cf3e966cd66c4ef9fc72c969ca77c3/scripts/slave/recipe_modules/auto_bisect_staging/local_bisect.py
,
Nov 18 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995637913191610784
,
Nov 18 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995637817363340864
,
Nov 18 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995637739555403280
,
Nov 18 2016
===== BISECT JOB RESULTS =====
Status: failed
=== Bisection aborted ===
The bisect was aborted because Step('Compare samples') failed with return_code 1
Please contact the the team (see below) if you believe this is in error.
===== TESTED REVISIONS =====
Revision Mean Std Dev N Good?
chromium@431022 N/A N/A 0 good
chromium@431075 N/A N/A 0 bad
Bisect job ran on: staging_android_nexus5X_perf_bisect
Bug ID: 664743
Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http.search.yahoo.com.search..ylt.p.google memory.top_10_mobile
Test Metric: memory:chrome:renderer_processes:reported_by_chrome:v8:effective_size_avg/background/after_http_search_yahoo_com_search__ylt_p_google
Relative Change: None
Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/staging_android_nexus5X_perf_bisect/builds/55
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995637913191610784
Not what you expected? We'll investigate and get back to you!
https://chromeperf.appspot.com/bad_bisect?try_job_id=5785289899900928
| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
| X | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 18 2016
===== BISECT JOB RESULTS =====
Status: failed
=== Bisection aborted ===
The bisect was aborted because Step('Compare samples') failed with return_code 1
Please contact the the team (see below) if you believe this is in error.
===== TESTED REVISIONS =====
Revision Mean Std Dev N Good?
chromium@431022 N/A N/A 0 good
chromium@431075 N/A N/A 0 bad
Bisect job ran on: staging_android_nexus5X_perf_bisect
Bug ID: 664743
Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http.search.yahoo.com.search..ylt.p.google memory.top_10_mobile
Test Metric: memory:chrome:renderer_processes:reported_by_chrome:v8:effective_size_avg/background/after_http_search_yahoo_com_search__ylt_p_google
Relative Change: None
Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/staging_android_nexus5X_perf_bisect/builds/56
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995637739555403280
Not what you expected? We'll investigate and get back to you!
https://chromeperf.appspot.com/bad_bisect?try_job_id=5821814301786112
| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
| X | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 18 2016
,
Nov 18 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@431022 4607604 592926 27 good chromium@431075 4652496 812111 27 bad Bisect job ran on: android_nexus7_perf_bisect Bug ID: 664743 Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http.search.yahoo.com.search..ylt.p.google memory.top_10_mobile Test Metric: memory:chrome:renderer_processes:reported_by_chrome:v8:effective_size_avg/background/after_http_search_yahoo_com_search__ylt_p_google Relative Change: None Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/3499 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995637817363340864 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5828461837418496 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 18 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995630209572955264
,
Nov 18 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995630170650145120
,
Nov 18 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995630049696338064
,
Nov 18 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995629598083722880
,
Nov 18 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995629212583629552
,
Nov 18 2016
Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/staging_android_nexus5X_perf_bisect/builds/58 Failure reason: the build has failed due to infrastructure failure.
,
Nov 18 2016
Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/staging_android_nexus5X_perf_bisect/builds/59 Failure reason: the build has failed due to infrastructure failure.
,
Nov 18 2016
===== BISECT JOB RESULTS =====
Status: failed
=== Bisection aborted ===
The bisect was aborted because Step('Compare samples') failed with return_code 1
Please contact the the team (see below) if you believe this is in error.
===== TESTED REVISIONS =====
Revision Mean Std Dev N Good?
chromium@431022 N/A N/A 0 good
chromium@431075 N/A N/A 0 bad
Bisect job ran on: staging_android_nexus5X_perf_bisect
Bug ID: 664743
Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http.search.yahoo.com.search..ylt.p.google memory.top_10_mobile
Test Metric: memory:chrome:renderer_processes:reported_by_chrome:effective_size_avg/background/after_http_search_yahoo_com_search__ylt_p_google
Relative Change: None
Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/staging_android_nexus5X_perf_bisect/builds/57
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995630209572955264
Not what you expected? We'll investigate and get back to you!
https://chromeperf.appspot.com/bad_bisect?try_job_id=6473338828881920
| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
| X | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 18 2016
===== BISECT JOB RESULTS =====
Status: failed
=== Bisection aborted ===
The bisect was aborted because Step('Compare samples') failed with return_code 1
Please contact the the team (see below) if you believe this is in error.
===== TESTED REVISIONS =====
Revision Mean Std Dev N Good?
chromium@430986 N/A N/A 0 good
chromium@431126 N/A N/A 0 bad
Bisect job ran on: staging_android_nexus5X_perf_bisect
Bug ID: 664743
Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http.m.intl.taobao.com.group.purchase.html memory.top_10_mobile
Test Metric: memory:chrome:all_processes:reported_by_os:system_memory:native_heap:proportional_resident_size_avg/foreground/http_m_intl_taobao_com_group_purchase_html
Relative Change: None
Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/staging_android_nexus5X_perf_bisect/builds/61
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995629212583629552
Not what you expected? We'll investigate and get back to you!
https://chromeperf.appspot.com/bad_bisect?try_job_id=4926412404817920
| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
| X | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 18 2016
===== BISECT JOB RESULTS =====
Status: failed
=== Bisection aborted ===
The bisect was aborted because Step('Compare samples') failed with return_code 1
Please contact the the team (see below) if you believe this is in error.
===== TESTED REVISIONS =====
Revision Mean Std Dev N Good?
chromium@431022 N/A N/A 0 good
chromium@431075 N/A N/A 0 bad
Bisect job ran on: staging_android_nexus5X_perf_bisect
Bug ID: 664743
Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http.search.yahoo.com.search..ylt.p.google memory.top_10_mobile
Test Metric: memory:chrome:renderer_processes:reported_by_chrome:effective_size_avg/background/after_http_search_yahoo_com_search__ylt_p_google
Relative Change: None
Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/staging_android_nexus5X_perf_bisect/builds/60
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995629598083722880
Not what you expected? We'll investigate and get back to you!
https://chromeperf.appspot.com/bad_bisect?try_job_id=5299668953595904
| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
| X | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 18 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995627058699909584
,
Nov 18 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@431803 1380.59 26.6008 25 good chromium@431804 1379.78 28.8881 25 bad Bisect job ran on: staging_linux_perf_bisect Bug ID: 664743 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=textarea.parsing.html blink_perf.parser Test Metric: textarea-parsing/textarea-parsing.html Relative Change: 0.06% Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/staging_linux_perf_bisect/builds/39 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995627058699909584 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5820740291526656 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Dec 2 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/bfafef31482c11bfdf10fc56a00ce85623e020de commit bfafef31482c11bfdf10fc56a00ce85623e020de Author: Simon <simonhatch@chromium.org> Date: Tue Nov 22 17:08:11 2016 Pushing "[bisect] - Compute relative change even when bisect fails to reproduce." to prod. Change: https://chromium-review.googlesource.com/c/412300/ BUG= 664743 Change-Id: Id4e07d121890c177e478bf97f4fc6ba66ec26f5b Reviewed-on: https://chromium-review.googlesource.com/413119 Reviewed-by: Prasad Vuppalapu <prasadv@chromium.org> Commit-Queue: Simon Hatch <simonhatch@chromium.org> [modify] https://crrev.com/bfafef31482c11bfdf10fc56a00ce85623e020de/scripts/slave/recipe_modules/auto_bisect/bisector.py [modify] https://crrev.com/bfafef31482c11bfdf10fc56a00ce85623e020de/scripts/slave/recipe_modules/auto_bisect/example.expected/basic_win32_bisect.json [modify] https://crrev.com/bfafef31482c11bfdf10fc56a00ce85623e020de/scripts/slave/recipe_modules/auto_bisect/example.expected/basic_win64_bisect.json [modify] https://crrev.com/bfafef31482c11bfdf10fc56a00ce85623e020de/scripts/slave/recipe_modules/auto_bisect/example.expected/no_repro.json [modify] https://crrev.com/bfafef31482c11bfdf10fc56a00ce85623e020de/scripts/slave/recipe_modules/auto_bisect/local_bisect.py
,
Dec 2 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/bfafef31482c11bfdf10fc56a00ce85623e020de commit bfafef31482c11bfdf10fc56a00ce85623e020de Author: Simon <simonhatch@chromium.org> Date: Tue Nov 22 17:08:11 2016 Pushing "[bisect] - Compute relative change even when bisect fails to reproduce." to prod. Change: https://chromium-review.googlesource.com/c/412300/ BUG= 664743 Change-Id: Id4e07d121890c177e478bf97f4fc6ba66ec26f5b Reviewed-on: https://chromium-review.googlesource.com/413119 Reviewed-by: Prasad Vuppalapu <prasadv@chromium.org> Commit-Queue: Simon Hatch <simonhatch@chromium.org> [modify] https://crrev.com/bfafef31482c11bfdf10fc56a00ce85623e020de/scripts/slave/recipe_modules/auto_bisect/bisector.py [modify] https://crrev.com/bfafef31482c11bfdf10fc56a00ce85623e020de/scripts/slave/recipe_modules/auto_bisect/example.expected/basic_win32_bisect.json [modify] https://crrev.com/bfafef31482c11bfdf10fc56a00ce85623e020de/scripts/slave/recipe_modules/auto_bisect/example.expected/basic_win64_bisect.json [modify] https://crrev.com/bfafef31482c11bfdf10fc56a00ce85623e020de/scripts/slave/recipe_modules/auto_bisect/example.expected/no_repro.json [modify] https://crrev.com/bfafef31482c11bfdf10fc56a00ce85623e020de/scripts/slave/recipe_modules/auto_bisect/local_bisect.py
,
Dec 2 2016
,
Feb 3 2017
,
Mar 22 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8984378618558380768
,
Mar 24 2017
=== BISECT JOB RESULTS ===
Bisect failed unexpectedly
Bisect was aborted with the following:
Step('Compare samples') failed with return_code 1
Bisect Details
Configuration: staging_android_nexus5X_perf_bisect
Benchmark : memory.top_10_mobile
Metric : memory:chrome:all_processes:reported_by_os:system_memory:native_heap:proportional_resident_size_avg/foreground/http_m_intl_taobao_com_group_purchase_html
Please refer to the following doc on diagnosing memory regressions:
https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md
To Run This Test
src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http.m.intl.taobao.com.group.purchase.html memory.top_10_mobile
Debug Info
https://chromeperf.appspot.com/buildbucket_job_status/8984378618558380768
Is this bisect wrong?
https://chromeperf.appspot.com/bad_bisect?try_job_id=4926412404817920
| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
| X | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982647698511014656
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982624158536067104
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982611719741649600
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982592381189601184 |
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by sullivan@chromium.org
, Nov 12 2016