New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 666316 link

Starred by 2 users

Issue metadata

Status: WontFix
Owner:
Closed: Jul 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression



Sign in to add a comment

26.2% regression in tracing.tracing_with_debug_overhead at 432185:432258

Project Member Reported by mustaq@chromium.org, Nov 17 2016

Issue description

See the link to graphs below.
 
Project Member

Comment 3 by 42576172...@developer.gserviceaccount.com, Nov 17 2016


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev   N   Good?
chromium@432184  5613350  8726954   45  good
chromium@432258  6361977  17372368  45  bad

Bisect job ran on: winx64_zen_perf_bisect
Bug ID: 666316

Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests tracing.tracing_with_debug_overhead
Test Metric: Max event size in bytes per second_max/Max event size in bytes per second_max
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/604
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995735679255812048


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5247375377956864

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 5 by mustaq@chromium.org, Nov 21 2016

 Issue 666315  has been merged into this issue.
Project Member

Comment 7 by 42576172...@developer.gserviceaccount.com, Nov 21 2016


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev   N   Good?
chromium@432184  5610053  8864740   45  good
chromium@432258  6438982  17134264  45  bad

Bisect job ran on: winx64_zen_perf_bisect
Bug ID: 666316

Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests tracing.tracing_with_debug_overhead
Test Metric: Max event size in bytes per second_max/Max event size in bytes per second_max
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/606
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995363367719420048


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5213948788342784

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 8 by 42576172...@developer.gserviceaccount.com, Nov 21 2016


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@432184  21190.4  35732.6  45  good
chromium@432258  24492.9  70412.5  45  bad

Bisect job ran on: winx64_zen_perf_bisect
Bug ID: 666316

Test Command: src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests tracing.tracing_with_debug_overhead
Test Metric: Max number of events per second_max/Max number of events per second_max
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/607
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995363190430294656


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6055591519191040

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 10 by 42576172...@developer.gserviceaccount.com, Nov 22 2016

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64_zen_perf_bisect/builds/609
Failure reason: the build has failed.
Additional errors:
Bisect aborted early for lack of confidence.
Either of the initial "good" or "bad" revisions failed to be tested or built.

Project Member

Comment 13 by 42576172...@developer.gserviceaccount.com, Apr 11 2017

Cc: mal...@yandex-team.ru
Owner: mal...@yandex-team.ru

=== Auto-CCing suspected CL author malets@yandex-team.ru ===

Hi malets@yandex-team.ru, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : malets
  Commit : 08374f2157484efc50451130a41eff6467507694
  Date   : Tue Nov 15 14:11:46 2016
  Subject: Add JS exception details to EvaluateException.

Bisect Details
  Configuration: winx64_zen_perf_bisect
  Benchmark    : tracing.tracing_with_debug_overhead
  Metric       : Max event size in bytes per second_max/Max event size in bytes per second_max
  Change       : 19.14% | 5674923.7037 -> 6761240.40741

Revision                                 Result                 N
chromium@432184                          5674924 +- 250770      6      good
chromium@432189                          5700242 +- 396189      6      good
chromium@432192                          5683632 +- 173049      6      good
chromium@432193                          5757361 +- 191402      6      good
chromium@432193,catapult@08374f2157      6461372 +- 245172      6      bad       <--
chromium@432194                          6746120 +- 174854      6      bad
chromium@432203                          6644735 +- 219011      6      bad
chromium@432221                          6657187 +- 211031      6      bad
chromium@432258                          6761240 +- 214678      6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests tracing.tracing_with_debug_overhead

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982651809380098944

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5213948788342784


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
Project Member

Comment 14 by 42576172...@developer.gserviceaccount.com, Apr 11 2017


=== BISECT JOB RESULTS ===
Bisect was unable to run to completion

Error: BUILD_FAILURE

The bisect was able to narrow the range, you can try running with:
  good_revision: ad7c7f0de20e3ad160f063f1990c69ce97f7e16f
  bad_revision : a03d6a549ad14b4a4d10eb81aa3eb55b438eae15

If failures persist contact the team (see below) and report the error.


Bisect Details
  Configuration: winx64_zen_perf_bisect
  Benchmark    : tracing.tracing_with_debug_overhead
  Metric       : Max event size in bytes per second_max/Max event size in bytes per second_max
  Change       : 28.12% | 5320539.83333 -> 6816912.87037

Revision             Result                 N
chromium@432184      5320540 +- 394335      6      good
chromium@432189      5687531 +- 386106      6      good
chromium@432192      5667066 +- 153563      6      good
chromium@432193      5659760 +- 287872      6      good
chromium@432194      6806083 +- 205596      6      bad
chromium@432203      6753464 +- 350869      6      bad
chromium@432221      6773205 +- 505195      6      bad
chromium@432258      6816913 +- 314357      6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests tracing.tracing_with_debug_overhead

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982636303987634704

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5247375377956864


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
Status: Assigned (was: Untriaged)
Explictly assigning. A CL you landed tripped one of the speed metrics we measure in the lab. If this is the first time this has happened to one of your CLs, or if it's been a while, please read: https://chromium.googlesource.com/chromium/src/+/master/docs/speed/addressing_performance_regressions.md

We're looking for one of the following:
1. Justification via explanation
2. Plan to revert or fix
3. Angry rage throwing of equipment at my head

Just be aware that I'm trained in trumpet playing and First Aid and am not afraid to use it.

Note: This was a bulk edit message and not very personal.
Status: WontFix (was: Assigned)
First of all, the change in question was in Catapult repo and not in Chrome itself, thus it did not have a chance to degrade Chrome's performance.

Considering the impact of the change on the metric, I would rather conclude that more debug information may cause increased trace messages size and there's nothing to fix here: the JS exception details in Telemetry log are definitely useful and worth of debug overhead price we pay for them.
Cc: zh...@chromium.org oysteine@chromium.org nedngu...@google.com
Adding Ned, Zhen, and Oystein, owners of tracing.tracing_with_debug_overhead benchmark, as FYI: is it okay to WontFix?

Sign in to add a comment