New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 601792 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Last visit > 30 days ago
Closed: Jul 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression



Sign in to add a comment

2.2%-117.1% regression in performance_browser_tests at 385569:385654

Project Member Reported by rsch...@chromium.org, Apr 8 2016

Issue description

See the link to graphs below.
 
All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=601792

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6MyUsAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6ObmtgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6LLivQkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6JaepAsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6MzyqQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6IvMvAsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6MzuuAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6IvM_AgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgqI3X5goM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6Ob85wgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6IrCpQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6M2xvAkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6Ivl6woM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgqOvI-QgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6I-1uQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6PG8vgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6Ivl6wkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6LbDuAkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6PS05AgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6I-rpAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6I-rpAkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6MSc9goM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6IrWtAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6PG8vgkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgmJa-uwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgqIPb4AsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgmMjXpQoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6IeEuwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgmKi_ugoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgmMyTpwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6KXh8wgM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6JnH5AkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgmMiA7ggM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgmOjEpwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgmPbGrwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6If2twkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgmNLGogsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6PSguQsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6PXtuwoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgmOHnoQkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgmJa-uwsM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6PX7pgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6LPwswoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6LSC7woM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgmMyTpwkM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6MfauAoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgmOjStgoM,agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICg6LvZsQsM


Bot(s) for this bug's original alert(s):

chromium-rel-win7-dual
chromium-rel-win7-gpu-ati
chromium-rel-win7-gpu-nvidia
chromium-rel-win7-single

===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision                Mean Value  Std. Dev.   Num Values  Good?
chromium@385616         19.844749   0.44046     18          good
chromium@385631         19.700073   0.317633    18          bad

Bisect job ran on: win_perf_bisect
Bug ID: 601792

Test Command: .\src\out\Release\performance_browser_tests.exe --test-launcher-print-test-stdio=always --enable-gpu
Test Metric: CastV2Performance_gpu_30fps_fast/capture_duration
Relative Change: 2.36%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/6451
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9015937875199100672


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=601792

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Some of these regressions recovered, some didn't. We're likely dealing with multiple regressions.

Submitted more bisects.
Project Member

Comment 4 by 42576172...@developer.gserviceaccount.com, Apr 15 2016


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision                Mean Value  Std. Dev.   Num Values  Good?
chromium@385397         105.013827  5.216789    18          good
chromium@385402         104.005262  3.358495    18          bad

Bisect job ran on: winx64ati_perf_bisect
Bug ID: 601792

Test Command: python src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --also-run-disabled-tests page_cycler.morejs
Test Metric: warm_times/page_load_time
Relative Change: 0.75%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64ati_perf_bisect/builds/1323
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9015282726717894096


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=601792

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 5 by 42576172...@developer.gserviceaccount.com, Apr 16 2016


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision                Mean Value  Std. Dev.   Num Values  Good?
chromium@385616         18.28441    0.129076    18          good
chromium@385653         18.282943   0.150417    18          bad

Bisect job ran on: winx64ati_perf_bisect
Bug ID: 601792

Test Command: .\src\out\Release_x64\performance_browser_tests.exe --test-launcher-print-test-stdio=always --enable-gpu
Test Metric: TabCapturePerformance_comp_gpu_webrtc/CaptureSucceeded
Relative Change: 0.10%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64ati_perf_bisect/builds/1322
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9015282729569522160


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=601792

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Hrmph. More of these recovering, still trying more bisects.
Previous bisect jobs failed due to error in bisect recipe: crbug.com/606281
Re-launched another bisect:https://chromeperf.appspot.com/buildbucket_job_status/9012758313767151632
Cc: pras...@chromium.org
prasadv@: The bisect in #8 seemed to succeed but didn't post anything on the bug, and I can't find the results in the buildbot link. Do you know where the results are?
Labels: -performance-sheriff Performance-Sheriff
Ping Prasa: can you answer rmcilroy's question in #9?
Still not recovered.  Kicked off another bisect.
Project Member

Comment 13 by 42576172...@developer.gserviceaccount.com, Jul 16 2016

Cc: w...@chromium.org
Owner: w...@chromium.org

=== Auto-CCing suspected CL author wez@chromium.org ===

Hi wez@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Patch to try dump-on-DCHECK.
Author  : wez
Commit description:
  
This patch does two things:

1. Adds a flag to switch DCHECK from logging, dumping, and then crashing the process, to only uploading a crash dump, and only on the first failed DCHECK in each process.

2. Forces that flag, and DCHECK_ALWAYS_ON, on in Windows official builds.

All non-debug e.g. CHECK behaviours remain unchanged; the intended effect is for DCHECKs to switch from no-ops to uploading dumps without crashing, in Windows official builds.

Note that this CL is intended to be landed, a branch cut to release from, and then immediately reverted; it is not intended to be landed in Chromium for any longer period.

BUG=596231

Committed: https://crrev.com/6436ac7ddec4b2b3aba4ee38aabe7dffe238a077
Cr-Commit-Position: refs/heads/master@{#383894}

Committed: https://crrev.com/bac26c8a840909a679a5a74557fa6f4f60ae9e07
Cr-Commit-Position: refs/heads/master@{#384011}

Committed: https://crrev.com/16502cb143f737bafad5d035b8ed6d76aabce288
Cr-Commit-Position: refs/heads/master@{#384675}

Review URL: https://codereview.chromium.org/1814423002

Cr-Commit-Position: refs/heads/master@{#385413}
Commit  : b80aa8f5c269c9ee4f3a00b03840ea3df68f77d1
Date    : Wed Apr 06 09:13:42 2016


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@385390  120.168  1.21779  5   good
chromium@385404  121.732  2.21756  5   good
chromium@385409  121.356  1.43411  5   good
chromium@385411  121.009  2.52168  12  good
chromium@385412  123.688  6.91187  12  good
chromium@385413  129.745  1.72946  8   bad    <--
chromium@385420  130.828  3.15989  5   bad

Bisect job ran on: win_8_perf_bisect
Bug ID: 601792

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --also-run-disabled-tests page_cycler.morejs
Test Metric: warm_times/page_load_time
Relative Change: 8.87%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_8_perf_bisect/builds/2041
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9007037833921935888


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5824636831399936

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 14 by w...@chromium.org, Jul 18 2016

Owner: rsch...@chromium.org
crrev.com/1814423002 was a one-canary experiment, expected to impact performance adversely.  It was reverted on Apr 7th by crrev.com/1870633003.

Did performance not recover after the revert?
On this particular test it looks like it came down on April 22. So I'm not sure what happened there - but we're good on that test.

Many of the performance_browser_tests haven't recovered or haven't recovered fully. I'll try bisecting on those.
Cc: -w...@chromium.org
Project Member

Comment 19 by 42576172...@developer.gserviceaccount.com, Jul 19 2016


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev   N   Good?
chromium@385568  19.7775  0.297232  18  good
chromium@385631  19.8096  0.253373  17  bad

Bisect job ran on: win_perf_bisect
Bug ID: 601792

Test Command: .\src\out\Release\performance_browser_tests.exe --test-launcher-print-test-stdio=always --enable-gpu
Test Metric: CastV2Performance_gpu_30fps_fast/capture_duration
Relative Change: 0.93%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/builds/6713
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9006701361830643760


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5863113363030016

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 20 by 42576172...@developer.gserviceaccount.com, Jul 19 2016

Cc: x...@chromium.org
Owner: x...@chromium.org

=== Auto-CCing suspected CL author xjz@chromium.org ===

Hi xjz@chromium.org, the bisect results pointed to your CL below as possibly
causing a regression. Please have a look at this info and see whether
your CL be related.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Remove --tab-capture-upscale/downscale-quality.
Author  : xjz
Commit description:
  
Historically three scalers were designed for tab-capture upscaling
and downscaling: fast, good, best. Experiments show that "good" or
"best" down-scalers may cause downstream encoders to perform much
worse, as the detail they maintain adds excessive entropy to the
images.

This change removes the command-line switches and always uses
"fast" for video down-scaling and "best" for upscaling.

BUG= 350135 

Review URL: https://codereview.chromium.org/1690013002

Cr-Commit-Position: refs/heads/master@{#385626}
Commit  : 9000b88d353b4929a21b4cc5fcb2b87abadd37a1
Date    : Thu Apr 07 02:28:51 2016


===== TESTED REVISIONS =====
Revision         Mean     Std Dev    N  Good?
chromium@385616  7.4951   0.152772   5  good
chromium@385625  7.47794  0.0761176  5  good
chromium@385626  10.5866  0.210105   5  bad    <--
chromium@385627  10.5622  0.148259   5  bad
chromium@385628  10.522   0.174821   5  bad
chromium@385630  10.5646  0.225541   5  bad
chromium@385634  10.4706  0.0761088  5  bad
chromium@385653  10.6022  0.14883    5  bad

Bisect job ran on: winx64ati_perf_bisect
Bug ID: 601792

Test Command: .\src\out\Release_x64\performance_browser_tests.exe --test-launcher-print-test-stdio=always --enable-gpu
Test Metric: CastV2Performance_gpu_24fps/capture_duration
Relative Change: 41.45%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/winx64ati_perf_bisect/builds/1438
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9006701315147526192


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5789075643039744

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 21 by x...@chromium.org, Jul 20 2016

Cc: m...@chromium.org
Status: WontFix (was: Assigned)
The CL removes the command-line switches for tab capture scalers and sets default up/down scalers. Now the down-scaling uses the same scaler as before, but the up-scaling changes to use the "best" scaler from the "fast" one. So in up-scaling cases, we sacrifice the performance for higher quality.

Sign in to add a comment