New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 632652 link

Starred by 2 users
Status: WontFix
Owner:
█ alexclarke@chromium.org
Last visit > 30 days ago
Closed: Aug 2017
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression



Sign in to add a comment

11%-87.2% regression in webrtc.datachannel at 407925:407983

Project Member Reported by alexclarke@chromium.org, Jul 29 2016 
See the link to graphs below.
Project Member

Comment 3 by 42576172...@developer.gserviceaccount.com, Jul 29 2016 


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N   Good?
chromium@407938  679964  141811   18  good
chromium@407967  646429  70100.6  18  bad

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 632652

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.datachannel
Test Metric: vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total
Relative Change: 3.24%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1462
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005812486923450800


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5814587270627328

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 5 by alexclarke@chromium.org, Jul 29 2016 

Trying a bisect on ChromiumPerf/chromium-rel-mac-retina/webrtc.peerconnection / vm_resident_set_size_final_size_total  which seems to show a very clear regression.
Project Member

Comment 7 by 42576172...@developer.gserviceaccount.com, Jul 29 2016 


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N  Good?
chromium@407700  352758  2029.28  5  good

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 632652

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.peerconnection
Test Metric: vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total
Relative Change: None
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1469
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005806395393289680


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6471664653565952

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 9 by 42576172...@developer.gserviceaccount.com, Jul 30 2016 


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N   Good?
chromium@407938  352525  1602.93  18  good
chromium@407967  353435  1703.45  18  bad

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 632652

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.peerconnection
Test Metric: vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total
Relative Change: 0.04%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1482
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005806395393289680


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6471664653565952

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 11 by alexclarke@chromium.org, Aug 1 2016 

Trying again with a lot of iterations.
Project Member

Comment 13 by 42576172...@developer.gserviceaccount.com, Aug 1 2016 


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N  Good?
chromium@407500  352439  1349.72  5  good

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 632652

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.peerconnection
Test Metric: vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total
Relative Change: None
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1490
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005541682094766016


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5337583035351040

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 16 by alexclarke@chromium.org, Aug 1 2016 

Trying yet again.
Project Member

Comment 18 by 42576172...@developer.gserviceaccount.com, Aug 3 2016 


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N   Good?
chromium@407938  353284  1772.77  18  good
chromium@407967  352921  1675.12  18  bad

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 632652

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.peerconnection
Test Metric: vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total
Relative Change: 0.28%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1512
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005541682094766016


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5337583035351040

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 19 by 42576172...@developer.gserviceaccount.com, Aug 3 2016 


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N  Good?
chromium@407800  352810  1523.04  5  good

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 632652

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.peerconnection
Test Metric: vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total
Relative Change: None
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1502
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005518539610181776


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5771994818150400

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 20 by 42576172...@developer.gserviceaccount.com, Aug 3 2016 


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N  Good?
chromium@407500  703979  137399   5  good

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 632652

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.datachannel
Test Metric: vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total
Relative Change: None
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1501
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005518556993296112


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5861459255361536

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 23 by 42576172...@developer.gserviceaccount.com, Aug 3 2016 


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N  Good?
chromium@407800  352810  1523.04  5  good

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 632652

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.peerconnection
Test Metric: vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total
Relative Change: None
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1502
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005518539610181776


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5771994818150400

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 24 by 42576172...@developer.gserviceaccount.com, Aug 3 2016 


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N  Good?
chromium@407500  703979  137399   5  good

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 632652

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.datachannel
Test Metric: vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total
Relative Change: None
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1501
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005518556993296112


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5861459255361536

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 27 by 42576172...@developer.gserviceaccount.com, Aug 4 2016 


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N   Good?
chromium@407938  352924  1991.99  18  good
chromium@407967  351690  1374.85  18  bad

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 632652

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.peerconnection
Test Metric: vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total
Relative Change: 0.41%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1532
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005518539610181776


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5771994818150400

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 28 by 42576172...@developer.gserviceaccount.com, Aug 5 2016 


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N   Good?
chromium@407938  353084  1964.81  18  good
chromium@407967  352809  1559.72  18  bad

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 632652

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.peerconnection
Test Metric: vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total
Relative Change: 0.16%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1534
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9005518556993296112


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5861459255361536

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 29 by qyears...@chromium.org, Aug 18 2016 

Perf sheriff ping: reminder to follow up on possible performance issues

Comment 30 by sullivan@chromium.org, Oct 6 2016

Owner: ----
Perf sheriff fixit: unsetting owners where reporter==owner, to clarify the current member of the rotation is responsible for triage.

Comment 31 by benhenry@google.com, Nov 10 2016

Owner: alexclarke@chromium.org
Alex - you ran bisect lots and lots of times. It looks like there were just two results: completed by not able to find bad CL, or failed to complete. For the failed builds, are there common threads? Could it be that the failure and the regression are the same thing?

Comment 33 by alexclarke@chromium.org, Nov 11 2016 

I'm wondering if there really was a regression here, the graph looks like there was but there's no ref and sometimes something unrelated to chromium patches is the culprit.   I kicked off one last bisect on the mac retina bot - that seems to have the most clear cut regression which the bisects will find if it's real.

If the bisect fails we should close this and file a bug asking to get a ref for these metrics.
Project Member

Comment 34 by 42576172...@developer.gserviceaccount.com, Nov 11 2016 


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean  Std Dev  N  Good?
chromium@407000  N/A   N/A      0  good

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 632652

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests webrtc.peerconnection
Test Metric: vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1813
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8996297073416573728


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5229799046381568

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 37 by 42576172...@developer.gserviceaccount.com, Nov 12 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N   Good?
chromium@407924  352623  8283.01  27  good
chromium@407967  352820  10508.9  27  bad

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 632652

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests webrtc.peerconnection
Test Metric: vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1816
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8996200736752906560


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5245872760160256

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 40 by 42576172...@developer.gserviceaccount.com, Apr 11 2017 


=== BISECT JOB RESULTS ===
Perf regression found but unable to continue

Bisect was stopped because a commit couldn't be classified as either
good or bad.


Bisect Details
  Configuration: mac_retina_perf_bisect
  Benchmark    : webrtc.peerconnection
  Metric       : vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total


To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests webrtc.peerconnection

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982651607059694736

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5229799046381568


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
Project Member

Comment 41 by 42576172...@developer.gserviceaccount.com, Apr 11 2017 


=== BISECT JOB RESULTS ===
NO Perf regression found

Bisect Details
  Configuration: mac_retina_perf_bisect
  Benchmark    : webrtc.peerconnection
  Metric       : vm_resident_set_size_final_size_total/vm_resident_set_size_final_size_total

Revision             Result                 N
chromium@407924      352883 +- 7956.96      21      good
chromium@407967      352646 +- 8582.49      21      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests webrtc.peerconnection

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982640083947313072

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5245872760160256


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Comment 42 by sullivan@chromium.org, Aug 16 2017

Status: WontFix (was: Assigned)
Only happened on one bot, some of the metrics have been deprecated, and no bisects could reproduce. WontFix-ing.
►

Sign in to add a comment