Issue metadata
Sign in to add a comment
|
5.1%-15.1% regression in tracing.tracing_with_debug_overhead at 433427:433451 |
||||||||||||||||||||
Issue descriptionSee the link to graphs below.
,
Nov 21 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995368823296436400
,
Nov 21 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@433427 30577.5 136328 45 good chromium@433437 30967.7 144550 45 bad Bisect job ran on: android_nexus9_perf_bisect Bug ID: 667326 Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests tracing.tracing_with_debug_overhead Test Metric: Max number of events per second_min/Max number of events per second_min Relative Change: None Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus9_perf_bisect/builds/2273 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995368823296436400 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5822279097778176 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 21 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995363029239150544
,
Nov 21 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995363014566084096
,
Nov 21 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@433427 27772.0 134120 45 good chromium@433451 30291.8 141552 45 bad Bisect job ran on: android_nexus9_perf_bisect Bug ID: 667326 Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests tracing.tracing_with_debug_overhead Test Metric: Max number of events per second_max/Max number of events per second_max Relative Change: None Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus9_perf_bisect/builds/2275 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995363014566084096 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5779554105294848 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 21 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@433427 27911.5 134492 45 good chromium@433451 29065.6 139231 45 bad Bisect job ran on: android_nexus9_perf_bisect Bug ID: 667326 Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests tracing.tracing_with_debug_overhead Test Metric: Max number of events per second_avg/Max number of events per second_avg Relative Change: None Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus9_perf_bisect/builds/2274 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995363029239150544 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5799773569810432 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 22 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8995271967651721824
,
Nov 22 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@433427 27208.5 130701 45 good chromium@433451 30123.2 140137 45 bad Bisect job ran on: android_nexus9_perf_bisect Bug ID: 667326 Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests tracing.tracing_with_debug_overhead Test Metric: Max number of events per second_sum/Max number of events per second_sum Relative Change: None Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus9_perf_bisect/builds/2280 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995271967651721824 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5622732501286912 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Nov 29 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8994628386916809280
,
Nov 29 2016
===== BISECT JOB RESULTS ===== Status: completed ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@433427 22791.7 15947.8 18 good chromium@433433 23385.1 5924.17 18 good chromium@433436 22792.2 13835.1 18 good chromium@433437 24054.1 20877.6 27 unknown chromium@433438 24382.3 18135.6 27 unknown chromium@433439 25035.9 15635.4 18 bad chromium@433451 25847.7 10213.8 12 bad Bisect job ran on: android_nexus9_perf_bisect Bug ID: 667326 Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http...www.ask.com. tracing.tracing_with_debug_overhead Test Metric: Max number of events per second_avg/http___www.ask.com_ Relative Change: 14.28% Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus9_perf_bisect/builds/2288 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994628386916809280 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=6744210235981824 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982611777959419968
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982592590297140304
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982580424068338064
,
Apr 11 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: android_nexus9_perf_bisect Benchmark : tracing.tracing_with_debug_overhead Metric : Max number of events per second_sum/Max number of events per second_sum Revision Result N chromium@433427 27208.5 +- 130701 45 good chromium@433451 30123.2 +- 140137 45 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests tracing.tracing_with_debug_overhead Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982611777959419968 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5622732501286912 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 11 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: android_nexus9_perf_bisect Benchmark : tracing.tracing_with_debug_overhead Metric : Max number of events per second_max/Max number of events per second_max Revision Result N chromium@433427 27772.0 +- 134120 45 good chromium@433451 30291.8 +- 141552 45 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests tracing.tracing_with_debug_overhead Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982592590297140304 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5779554105294848 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 11 2017
=== BISECT JOB RESULTS === NO Perf regression found Bisect Details Configuration: android_nexus9_perf_bisect Benchmark : tracing.tracing_with_debug_overhead Metric : Max number of events per second_avg/Max number of events per second_avg Revision Result N chromium@433427 27911.5 +- 134492 45 good chromium@433451 29065.6 +- 139231 45 bad To Run This Test src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests tracing.tracing_with_debug_overhead Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982580424068338064 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=5799773569810432 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 12 2017
cc-ing the test owners re what to do here: Most of the 28 regressions here are valid but in the last 3+ months many of them has seen even bigger regressions later on. Do we care about the ones in this bug?
,
Apr 12 2017
Zhen, Oysteine: I think we need to declare that the tracing benchmarks is a failure: it keeps producing unbisectable regressions & the breakdown data is not good enough for us to address the cause. Given the fact that this is not producing positive value in term of regression prevention, I think we should turn down the benchmarks & retry a better approach once we have more bandwidth. Wdyt?
,
Apr 12 2017
,
Apr 12 2017
Reassign to Zhen & Oysteine to answer to #19
,
Apr 12 2017
Is that because the trace size depends on how long it takes to run a story, which could vary from run to run?
,
Apr 12 2017
Unclear, but IIRC, there are still some bug in the current implementation of max number events per second metric that make it incorrect in the case of clock sync.
,
Apr 12 2017
What kind of bug is it? Oystein, have you guys found the tracing benchmark useful to tracing? I haven't followed that thread much.
,
Apr 12 2017
Zhen: it was https://bugs.chromium.org/p/chromium/issues/detail?id=646521#c22 My CL was adding around 200 trace events at total to the combined trace, but it regressed the metric by 34% (number of Chrome's trace events is tens of thousands)
,
May 15 2017
I just chatted with Oystein offline. Can we turn off the alert for this until either of us has more time digging into it more?
,
May 15 2017
#26: Why not just turn off the benchmark entirely then? Otherwise I think we are just wasting bot time if we don't have bandwidth to address its issues :-)
,
May 15 2017
Oystein, do we need to keep the benchmark running? I guess any new trace event additions can use this benchmark locally for manual verification. If not, I can prepare a disable CL.
,
May 15 2017
For locally run benchmark, you can move it to "tools/perf/contrib/". Benchmarks there can be run locally & on tryserver. THey are also not scheduled on perf waterfall, so there is no need to add @Decorators.Disabled('...').
,
Aug 16 2017
Do we still want this benchmark? If not, who can move it to contrib?
,
Aug 16 2017
Ping oysteine@: I would be ok with moving this to tools/perf/contrib/
,
Aug 17 2017
I am fine with the move. Oystein?
,
Aug 17 2017
Yep that sounds reasonable, since no one is really in a position to maintain this at the moment.
,
Aug 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/65c8da95cbc89501e08ba28cd3bc9d308f20a026 commit 65c8da95cbc89501e08ba28cd3bc9d308f20a026 Author: Zhen Wang <zhenw@chromium.org> Date: Fri Aug 18 19:05:33 2017 Move tracing.tracing_with_debug_overhead to tools/perf/contrib/ Bug: 667326 Change-Id: I030d157aa1faa0082979942c1d4a0242ffe92d8c Reviewed-on: https://chromium-review.googlesource.com/619872 Commit-Queue: Zhen Wang <zhenw@chromium.org> Reviewed-by: Ned Nguyen <nednguyen@google.com> Cr-Commit-Position: refs/heads/master@{#495639} [modify] https://crrev.com/65c8da95cbc89501e08ba28cd3bc9d308f20a026/testing/buildbot/chromium.perf.fyi.json [modify] https://crrev.com/65c8da95cbc89501e08ba28cd3bc9d308f20a026/testing/buildbot/chromium.perf.json [modify] https://crrev.com/65c8da95cbc89501e08ba28cd3bc9d308f20a026/tools/perf/benchmark.csv [modify] https://crrev.com/65c8da95cbc89501e08ba28cd3bc9d308f20a026/tools/perf/benchmarks/tracing.py [add] https://crrev.com/65c8da95cbc89501e08ba28cd3bc9d308f20a026/tools/perf/contrib/tracing/OWNERS [add] https://crrev.com/65c8da95cbc89501e08ba28cd3bc9d308f20a026/tools/perf/contrib/tracing/__init__.py [add] https://crrev.com/65c8da95cbc89501e08ba28cd3bc9d308f20a026/tools/perf/contrib/tracing/tracing.py
,
Aug 18 2017
|
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by mustaq@chromium.org
, Nov 21 2016