New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 595719 link

Starred by 0 users

Issue metadata

Status: WontFix
Owner:
Closed: Apr 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression



Sign in to add a comment

44.1% regression in webrtc.datachannel at 378150:378150

Project Member Reported by tdres...@chromium.org, Mar 17 2016

Issue description

See the link to graphs below.
 
All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=595719

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgIDAn4nErwoM


Bot(s) for this bug's original alert(s):

chromium-rel-win7-dual
Project Member

Comment 2 by 42576172...@developer.gserviceaccount.com, Mar 17 2016


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision                Mean Value  Std. Dev.   Num Values  Good?
chromium@378149         214113.11111115836.16571618          good
chromium@378150         207622.22222217470.18333718          bad

Bisect job ran on: win_perf_bisect
Bug ID: 595719

Test Command: python src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --also-run-disabled-tests webrtc.datachannel
Test Metric: vm_working_set_final_size_total/vm_working_set_final_size_total
Relative Change: 4.92%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/6412
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9017933062939000288


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with label Cr-Tests-AutoBisect.  Thank you!
Project Member

Comment 3 by 42576172...@developer.gserviceaccount.com, Mar 18 2016


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision                Mean Value  Std. Dev.   Num Values  Good?
chromium@378130         216280.44444416974.11703718          good
chromium@378150         222142.0    24691.19849918          bad

Bisect job ran on: win_perf_bisect
Bug ID: 595719

Test Command: python src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --also-run-disabled-tests webrtc.datachannel
Test Metric: vm_working_set_final_size_total/vm_working_set_final_size_total
Relative Change: 13.83%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/6415
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9017847658119501232


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with label Cr-Tests-AutoBisect.  Thank you!
Project Member

Comment 4 by 42576172...@developer.gserviceaccount.com, Mar 18 2016


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision                Mean Value  Std. Dev.   Num Values  Good?
chromium@378149         211252.44444418137.94903218          good
chromium@378150         213552.66666714334.31097318          bad

Bisect job ran on: win_perf_bisect
Bug ID: 595719

Test Command: python src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --also-run-disabled-tests webrtc.datachannel
Test Metric: vm_working_set_final_size_total/vm_working_set_final_size_total
Relative Change: 4.05%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/6416
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9017834214202336688


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with label Cr-Tests-AutoBisect.  Thank you!
Cc: phoglund@chromium.org
Adding test owner.

phoglund@, any idea what's going on there? There has been a _lot_ of movement in this metric recently.
I don't know. It looks like the metric holds together pretty well, but I can't find any reasonable culprits when it goes up nor reverts when it goes down. Could be memory metrics are too noisy for this particular test.
Project Member

Comment 7 by 42576172...@developer.gserviceaccount.com, Mar 21 2016


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision                Mean Value  Std. Dev.   Num Values  Good?
chromium@378149         230508.444444101415.00811918          good
chromium@378150         219446.44444427576.30065718          bad

Bisect job ran on: win_perf_bisect
Bug ID: 595719

Test Command: python src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --also-run-disabled-tests webrtc.datachannel
Test Metric: vm_working_set_final_size_total/vm_working_set_final_size_total
Relative Change: 2.28%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/6427
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9017572150307129200


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with label Cr-Tests-AutoBisect.  Thank you!
Project Member

Comment 10 by 42576172...@developer.gserviceaccount.com, Mar 29 2016


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision                Mean Value  Std. Dev.   Num Values  Good?
chromium@378145         230408.0    13033.55632918          good
chromium@378160         234723.77777878442.68596 18          bad

Bisect job ran on: win_perf_bisect
Bug ID: 595719

Test Command: python src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --also-run-disabled-tests webrtc.datachannel
Test Metric: vm_working_set_final_size_total/vm_working_set_final_size_total
Relative Change: 22.28%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/6440
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9016826369490048864


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=595719

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Owner: sullivan@chromium.org
Re #6, sounds like we should definitely not be monitoring. Annie, can you make that change?
Re #12: no, it appears to be a waste of time unfortunately.
Status: WontFix (was: Assigned)
Thanks, Ryan, looks like every test that reports old style memory metrics is monitored; I filed bug 602283 to track. Closing this one.

Sign in to add a comment