Issue metadata
Sign in to add a comment
|
10.2%-38.8% regression in webrtc.stress at 433524:434584 |
||||||||||||||||||||
Issue descriptionSee graphs below.
,
Dec 2 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8994384604094035152
,
Dec 2 2016
Patrik, I'm having trouble figuring out what could be causing these regressions in CPU utilization. It would be great if there were corresponding FYI graphs to compare with.
,
Dec 2 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: No values were found while testing the reference range. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@434241 N/A N/A 0 good chromium@434307 N/A N/A 0 bad Bisect job ran on: linux_perf_bisect Bug ID: 670657 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=cpu.utilization.browser webrtc.peerconnection Test Metric: cpu_utilization/cpu_utilization_browser Relative Change: None Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6873 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994384604094035152 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=4970136304353280 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Dec 2 2016
But there is a WebRTC roll very close, in the build before: 2016-11-21 sakal@webrtc.org Correctly pass drawn frame size when layout aspect ratio is used in EglRenderer. 2016-11-21 danilchap@webrtc.org RtpPacket::payload() return rtc::ArrayView instead of raw pointer 2016-11-21 kjellander@webrtc.org iOS: Cleanup buildbot JSON files + bump iOS version to 10.0 2016-11-19 solenberg@webrtc.org Remove unused files linux.cc/.h and linuxfdwalk.c/.h. 2016-11-18 zijiehe@chromium.org Add more logging in ScreenCapturerIntegrationTest 2016-11-18 honghaiz@webrtc.org Revert of Remove unused HttpClient class. (patchset #1 id:1 of https://codereview.webrtc.org/2511883005/ ) 2016-11-18 solenberg@webrtc.org Remove unused HttpClient class. 2016-11-18 solenberg@webrtc.org Remove unused dbus.cc/.h and related things. 2016-11-18 nisse@webrtc.org Move FirewallSocketServer to test code. 2016-11-18 ehmaldonado@webrtc.org Changed the way we find the ProjectRootPath. 2016-11-18 ehmaldonado@webrtc.org Modify audio_processing_unittest to use ResourcePath instead of ProjectRootPath. 2016-11-18 nisse@webrtc.org Delete WindowPicker class and subclasses. 2016-11-18 aleloi@webrtc.org Changed the interface AudioMixer::RemoveSource to have a void return type. 2016-11-18 danilchap@webrtc.org Introduce ArrayView::subview function to return portion of the original view 2016-11-18 magjed@webrtc.org Reland of Stop using hardcoded payload types for video codecs (patchset #1 id:1 of https://codereview.webrtc.org/2513633002/ ) 2016-11-18 magjed@webrtc.org Revert of Move smoothing filter to common audio. (patchset #3 id:60001 of https://codereview.webrtc.org/2484153002/ )
,
Dec 2 2016
One thing that makes it really hard here is that we can't bisect into WebRTC rolls. One possible solution is to make the telemetry tests run on the WebRTC FYI bots.
,
Dec 5 2016
+robertocn Adding webrtc support to bisect shouldn't be that difficult if this is something that would be handy.
,
Dec 6 2016
That sounds promising, we've had bug 656974 for a while and it seems the Bisect team was overloaded in Q4. I should however try to find time to take a look at the suggestions in that bug myself...
,
Dec 6 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8994004065807318768
,
Dec 6 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build.git/+/900a8675c27039ff701b98d085936ef6c82bf8e9 commit 900a8675c27039ff701b98d085936ef6c82bf8e9 Author: Roberto Carrillo <robertocn@google.com> Date: Tue Dec 06 19:55:11 2016 [bisect] Adding webrtc to bisectable repos. Also sorting the keys of a dictionary output to make it deterministic and not break expectations. TBR=simonhatch@chromium.org BUG= 670657 Change-Id: I9839f62dbcf73935d517a3ba01e9a5abc52de540 Reviewed-on: https://chromium-review.googlesource.com/417182 Reviewed-by: Roberto Carrillo <robertocn@chromium.org> Commit-Queue: Roberto Carrillo <robertocn@chromium.org> [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/bisector.py [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/depot_config.py [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/basic_bisect_other_direction.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/basic_buildbot_bisect.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/basic_linux_bisect.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/basic_mac_bisect.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/basic_win32_bisect.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/basic_win64_bisect.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/failed_build.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/failed_build_inconclusive_1.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/failed_build_inconclusive_11.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/failed_buildbucket_get.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/gathering_references_no_values.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/no_repro.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/no_values.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/retest_bisect.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/return_code.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/v8_roll_bisect.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipe_modules/auto_bisect_staging/example.expected/v8_roll_bisect_bis.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipes/bisection/android_bisect_staging.expected/local_basic_recipe_basic_device.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipes/bisection/android_bisect_staging.expected/local_basic_recipe_disconnected_device.json [modify] https://crrev.com/900a8675c27039ff701b98d085936ef6c82bf8e9/scripts/slave/recipes/bisection/android_bisect_staging.expected/local_basic_recipe_failed_device.json
,
Dec 6 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8993994189190762448
,
Dec 6 2016
=== PERF REGRESSION === === Auto-CCing suspected CL author jif@chromium.org === Hi jif@chromium.org, the bisect results pointed to your CL, please take a look at the results. ===== BISECT JOB RESULTS ===== Status: completed ===== SUSPECTED CL(s) ===== Subject : Explicitely pass ownership of ReadingListWebStateObserverUserDataWrapper Author : jif Commit description: BUG=None Review-Url: https://codereview.chromium.org/2513403003 Cr-Commit-Position: refs/heads/master@{#433821} Commit : 34145bb8726255a7da92275c2bb7bc40dda1ec14 Date : Tue Nov 22 09:22:41 2016 ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@433523 1.64647 0.325265 18 good chromium@433739 1.62341 0.119755 18 good chromium@433793 1.62226 0.119784 18 good chromium@433820 1.61401 0.0764671 18 good chromium@433821 1.57866 0.0433308 5 bad <-- chromium@433822 1.58203 0.0264355 5 bad chromium@433824 1.574 0.0307051 5 bad chromium@433827 1.5838 0.0370612 5 bad chromium@433834 1.59797 0.106551 27 bad chromium@433847 1.59887 0.0716433 18 bad chromium@433955 1.57946 0.0285173 8 bad chromium@434386 1.5963 0.0935493 12 bad Bisect job ran on: linux_perf_bisect Bug ID: 670657 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests webrtc.peerconnection Test Metric: cpu_utilization/cpu_utilization_browser Relative Change: 3.05% Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6893 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994004065807318768 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5229518405500928 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Dec 6 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: No values were found while testing the reference range. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@433559 N/A N/A 0 good chromium@434584 N/A N/A 0 bad Bisect job ran on: staging_linux_perf_bisect Bug ID: 670657 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=cpu.utilization.browser webrtc.stress Test Metric: cpu_utilization/cpu_utilization_browser Relative Change: 0.00% Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/staging_linux_perf_bisect/builds/48 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8993994189190762448 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=6444132371791872 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Dec 7 2016
robertocn@: thanks for https://chromium-review.googlesource.com/417182 Looking at the bisect job comment in #13 - does that mean it wasn't able to bisect into the roll for some reason?
,
Dec 7 2016
Bisector incorrectly assigned bug to me (CL mentioned in #12 only concerns iOS, where webRTC does not run). Assigning to ivoc@
,
Dec 12 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8993450325294056416
,
Dec 12 2016
re: #c14 The bisect is complaining there were no actual values produced, so it can't proceed. My guess is the use of --story-filter, the default value is invalid and we're not running any tests. I'll try rekicking this with no filter. If it comes back, we should log a bug against the default filter for being invalid.
,
Dec 12 2016
,
Dec 12 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@433559 7.3146 2.16058 21 good chromium@434584 7.04942 1.69233 21 bad Bisect job ran on: staging_linux_perf_bisect Bug ID: 670657 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests webrtc.stress Test Metric: cpu_utilization/cpu_utilization_browser Relative Change: 3.63% Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/staging_linux_perf_bisect/builds/54 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8993450325294056416 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5263476899971072 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Dec 12 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8993446045747445264
,
Dec 12 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@433523 1.64241 0.326819 21 good chromium@434386 1.61779 0.188082 21 bad Bisect job ran on: staging_linux_perf_bisect Bug ID: 670657 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests webrtc.peerconnection Test Metric: cpu_utilization/cpu_utilization_browser Relative Change: 1.50% Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/staging_linux_perf_bisect/builds/55 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8993446045747445264 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5291502366883840 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Dec 30 2016
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8991857145763570224
,
Dec 30 2016
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: No values were found while testing the reference range. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? Bisect job ran on: staging_linux_perf_bisect Bug ID: 670657 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=cpu.utilization.browser webrtc.peerconnection Test Metric: cpu_utilization/cpu_utilization_browser Relative Change: 0.00% Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/staging_linux_perf_bisect/builds/73 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8991857145763570224 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5813468148531200 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Dec 30 2016
Just FYI: I fired off this job as part of bug 656974 ... It fails in a similar way (which is expected). I was mostly checking that it worked to fire the staging bots. According to Simon, perf bisection support is only implemented at the staging bot so far. I assume we'll get it to the real ones soon?
,
Jan 2 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8991568413390201568
,
Jan 2 2017
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: No values were found while testing the reference range. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? Bisect job ran on: linux_perf_bisect Bug ID: 670657 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=cpu.utilization.browser webrtc.stress Test Metric: cpu_utilization/cpu_utilization_browser Relative Change: 0.00% Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisect/builds/6963 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8991568413390201568 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=6409386405134336 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Jan 2 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8991566330522805856
,
Jan 2 2017
re: #24 Yeah looks like my guess in #17 is right, story filter is invalid and filtering out all the tests. I kicked it off again with the story filter cleared. Yeah this is only on staging right now, would ideally like to confirm it's working before pushing it to all the bots. Filed a bug here: https://github.com/catapult-project/catapult/issues/3107
,
Jan 2 2017
Looked a bit closer at the graph and realized the ref moved at the same time: https://chromeperf.appspot.com/report?sid=905cf9e0cf2bcd2ffa6b3869f0d13755faf8790a431baa418d0f8e7720b07b8e So this regression is probably a false alarm.
,
Jan 2 2017
===== BISECT JOB RESULTS ===== Status: failed === Bisection aborted === The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence. Please contact the the team (see below) if you believe this is in error. ===== TESTED REVISIONS ===== Revision Mean Std Dev N Good? chromium@433523 1.62654 0.377014 21 good chromium@434386 1.59149 0.100313 21 bad Bisect job ran on: staging_linux_perf_bisect Bug ID: 670657 Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests webrtc.peerconnection Test Metric: cpu_utilization/cpu_utilization_browser Relative Change: 2.15% Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/staging_linux_perf_bisect/builds/74 Job details: https://chromeperf.appspot.com/buildbucket_job_status/8991566330522805856 Not what you expected? We'll investigate and get back to you! https://chromeperf.appspot.com/bad_bisect?try_job_id=5771619195158528 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Tests>AutoBisect. Thank you!
,
Jan 4 2017
ivoc: as mentioned in #29: if both the test and the ref metric moved at the same time, we should be able to close this, right?
,
Jan 5 2017
Agreed, I'll close this.
,
Apr 10 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982667472838309504
,
Apr 10 2017
=== BISECT JOB RESULTS === NO Perf regression found, tests failed to produce values Bisect Details Configuration: linux_perf_bisect Benchmark : webrtc.peerconnection Metric : cpu_utilization/cpu_utilization_browser To Run This Test src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=cpu.utilization.browser webrtc.peerconnection Debug Info https://chromeperf.appspot.com/buildbucket_job_status/8982667472838309504 Is this bisect wrong? https://chromeperf.appspot.com/bad_bisect?try_job_id=4970136304353280 | O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq | X | for more information addressing perf regression bugs. For feedback, | / \ | file a bug with component Speed>Bisection. Thank you!
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982632316337039840
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982624293935451408
,
Apr 11 2017
Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8982607786903219024 |
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by ivoc@chromium.org
, Dec 2 2016