New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 623835 link

Starred by 2 users

Issue metadata

Status: WontFix
Owner:
please use my google.com address
Closed: Jul 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression



Sign in to add a comment

61.7% regression in webrtc.datachannel at 402068:402068

Project Member Reported by lanwei@google.com, Jun 28 2016

Issue description

See the link to graphs below.
 

Comment 1 by lanwei@google.com, Jun 28 2016

All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=623835

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgotH9uAoM


Bot(s) for this bug's original alert(s):

chromium-rel-win7-dual
Project Member

Comment 2 by 42576172...@developer.gserviceaccount.com, Jun 28 2016


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N   Good?
chromium@402067  259042  62158.2  18  good
chromium@402068  277728  25839.5  18  bad

Bisect job ran on: win_perf_bisect
Bug ID: 623835

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.datachannel
Test Metric: vm_working_set_final_size_total/vm_working_set_final_size_total
Relative Change: 5.39%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/6626
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9008647923223317936


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5832786984828928

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 3 by sheriffbot@chromium.org, Jun 28 2016

Labels: Hotlist-Google
Project Member

Comment 4 by sheriffbot@chromium.org, Jul 2 2016

Labels: -M-53 M-54 MovedFrom-53
Moving this nonessential bug to the next milestone.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Lan, can you follow up on this?

From the instructions (https://chromium.googlesource.com/chromium/src/+/master/tools/perf/docs/perf_regression_sheriffing.md):
"After your shift, please try to follow up on the bugs you filed weekly. Kick off new bisects if the previous ones failed, and if the bisect picks a likely culprit follow up to ensure the CL author addresses the problem. If you are certain that a specific CL caused a performance regression, and the author does not have an immediate plan to address the problem, please revert the CL."
Trying a bisect with more repeats.
Project Member

Comment 8 by 42576172...@developer.gserviceaccount.com, Jul 22 2016


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N   Good?
chromium@402067  198464  80695.3  18  good
chromium@402068  240739  72995.3  18  bad

Bisect job ran on: win_perf_bisect
Bug ID: 623835

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests webrtc.datachannel
Test Metric: vm_working_set_final_size_total/vm_working_set_final_size_total
Relative Change: 40.71%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/6725
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9006406387947215872


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5227047731855360

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Owner: roc...@chromium.org
Hi could this be due to https://codereview.chromium.org/2099043002?  

I note the regression has since recovered so we may be able to close this.

Seems unlikely that it could be related
Cc: tdres...@chromium.org
Does look like it has recovered, but the graph also does seem to strongly implicate https://codereview.chromium.org/2099043002. As you can see though, that CL is trivial and extremely unlikely to have had any impact on this metric.

Is it worth understanding where the spike came from if not that CL or is this sort of noise common?
Status: WontFix (was: Assigned)
In general, if something like this has recovered, we don't spent the effort to diagnose it – there are plenty of regressions that haven't recovered yet. 

Marking WontFix.

Sign in to add a comment