Bisect result sample |
|||||||||||||||||||
Issue description🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬 for https://bugs.chromium.org/p/chromium/issues/detail?id=634917#c3 Perf regression found ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ Suspected commit [Mac] Only enable the CFBundleBlocker in the browser process. by rsesek https://codereview.chromium.org/2199313002 (r409244) Bisect details Configuration: mac_hdd_perf_bisect Benchmark: startup.warm.blank_page Metric: open_tabs_time/open_tabs_time Change: 4.3% | 1.8σ | 493.0 → 514.3 ms To run this test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests startup.warm.blank_page Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5878870750789632 Debug info https://chromeperf.appspot.com/buildbucket_job_status/9005160320731443728 Revision Result N chromium@409230 487.7 ± 10.0 5 good chromium@409238 500.5 ± 27.2 8 good chromium@409242 498.6 ± 10.5 8 good chromium@409243 493.0 ± 14.1 8 good chromium@409244 514.3 ± 9.1 8 bad ⬅ chromium@409246 510.9 ± 9.0 8 bad 🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬 for https://bugs.chromium.org/p/chromium/issues/detail?id=580232#c19 No perf regression found ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ Bisect details Configuration: android_nexus9_perf_bisect Benchmark: blink_perf.pywebsocket Metric: fetch-receive-text-window-async-verify/fetch-receive-text-window-async-verify Change: +0.10% | 0.15σ | 1334.5 → 1335.9 ms To run this test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests blink_perf.pywebsocket Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5880400228909056 Debug info https://chromeperf.appspot.com/buildbucket_job_status/9004499378821125488 Revision Result N chromium@369728 1334.5 ± 12.7 12 good chromium@369774 1335.9 ± 6.2 8 bad 🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬 for https://bugs.chromium.org/p/chromium/issues/detail?id=631398#c5 Perf regression round, but unable to narrow commit range ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ Suspected commit range 2 commits in range http://test-results.appspot.com/revision_range?start=407017&end=407018 Error message Infrastructure error while checking sample difference. Bisect details Configuration: android_nexus9_perf_bisect Benchmark: smoothness.gpu_rasterization.polymer Metric: input_event_latency_discrepancy/input_event_latency_discrepancy Change: +32% | +4.8σ | 99.5 → 131.4 ms To run this test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests smoothness.gpu_rasterization.polymer Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5870255256109056 Debug info https://chromeperf.appspot.com/buildbucket_job_status/9006084735086028704 Revision Result N chromium@407004 98.0 ± 4.3 6 good chromium@407013 96.7 ± 3.9 6 good chromium@407016 99.5 ± 6.4 5 good chromium@407018 131.4 ± 6.8 4 bad chromium@407022 129.9 ± 3.8 5 bad chromium@407040 123.4 ± 7.5 4 bad chromium@407076 125.5 ± 5.4 5 bad 🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬🐬 for https://bugs.chromium.org/p/chromium/issues/detail?id=608476#c14 Bisect failed ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ ✂️ Error message The benchmark run passed, but didn't produce the metric we're looking for. Bisect details Configuration: android_s5_perf_bisect Benchmark: cc_perftests.polymer Metric: build_raster_task_graph/32_1 To run this test src/build/android/test_runner.py gtest --release -s cc_perftests Debug info https://chromeperf.appspot.com/buildbucket_job_status/9004869311766773632
,
Aug 16 2016
,
Aug 16 2016
,
Aug 16 2016
,
Aug 16 2016
,
Aug 18 2016
,
Aug 18 2016
,
Aug 18 2016
,
Aug 19 2016
,
Aug 19 2016
,
Aug 19 2016
,
Aug 19 2016
,
Aug 24 2016
I like the proposal. Here are a few more ideas:
1. I assume that this is just a raw bisect result that would accompanied by
the usual "Hi <EMAIL>, the bisect results pointed to your CL ...". In my
opinion, there should also be more narrative throughout the results, e.g.
"The bisect's confidence that patch X caused the regression was Y% (high).
However, if you believe that the bisect is wrong, please click on the link
below".
2. Add more links to the bisects where possible (non-experts don't know where
to find these):
* Link to failing step in case the bisect failed.
* Link to the chart of the bisected metric.
* Link to the benchmark source code.
* Link to the metric source code.
* Links to the traces for each of the tested revisions (or a link to a
step that lists them).
3. Mention who the benchmark and metric owners are (so that the people know
who to contact if they're lost).
4. Add a warning if the bisect results are completely different from the
dashboard ones (and an option to file a bug against Speed Infra).
5. Remove "--upload-results" from "To Run This test".
6. Explain how to run the benchmark on a trybot (almost no one knows how to
do this and it's very useful if you don't have access to the relevant
platform).
7. Add a link to restart the bisect (with an option to modify the bisect
config). In my opinion, this would be one of the MOST USEFUL things to
have in the current sheriffing workflow.
,
Aug 25 2016
Re #13, great ideas and I agree except: For comments 1 and 2, I think there's a big tradeoff between having more info and having a wall of text. I wonder if maybe it'd be better to link to a dashboard page with deep debugging details? For comment 5, I think --upload-results should stay in there, because the person whose CL caused the regression can then link us to the traces etc in cloud if they are confused.
,
Aug 25 2016
+jparent, jrobbins, agable for monorail: looks like the trick dtu used to bold text doesn't render on android (screenshot attached). We aren't sure of the timing for https://bugs.chromium.org/p/monorail/issues/detail?id=768, any opinions on how bisect could better format its bug text?
,
Aug 25 2016
Markdown support is not currently planned for an upcoming Milestone, so, not soon :( We have other promised features to deliver first. That said ... Monorail is fully open source, and we would *love* to have contributions. If someone on your team was able to tackle it, we'd fully support, do speedy code reviews, etc.
,
Aug 25 2016
Thanks, Julie. Will follow up on the monorail bug. Any idea if there are any issues with the way the text is bolded in the first comment?
,
Aug 25 2016
sullivan: tracking that in issue monorail:1709 .
,
Aug 26 2016
,
Aug 26 2016
,
Aug 26 2016
,
Aug 26 2016
,
Aug 31 2016
,
Aug 31 2016
8fdfb52f84f4f9c28084ed5bb620a9746f76c0e
,
Nov 22 2016
|
|||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||
Comment 1 by dtu@chromium.org
, Aug 16 2016