New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 648884 link

Starred by 4 users
Status: WontFix
Owner:
perezju@chromium.org
Closed: Dec 2016
Cc:
perezju@chromium.org
█ pmeenan@chromium.org
kouhei@chromium.org
█ robert...@chromium.org
nedngu...@google.com
benhenry@google.com
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression

Blocked on:
issue 644865
issue 653519


Sign in to add a comment

102.6%-147% regression in page_cycler_v2.basic_oopif at 418785:419531

Project Member Reported by toyoshim@chromium.org, Sep 21 2016 
See the link to graphs below.
Project Member

Comment 3 by 42576172...@developer.gserviceaccount.com, Sep 21 2016 

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1011
Failure reason: the build has failed.
Additional errors:
The metric was not found in the test output.
Either of the initial "good" or "bad" revisions failed to be tested or built.

Comment 4 by sullivan@chromium.org, Sep 21 2016

Blockedon: 644865

Comment 5 by toyoshim@chromium.org, Sep 22 2016 

will rerun within the minimum range 418822:418829
Project Member

Comment 7 by 42576172...@developer.gserviceaccount.com, Sep 22 2016 

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1019
Failure reason: the build has failed.
Additional errors:
The metric was not found in the test output.
Either of the initial "good" or "bad" revisions failed to be tested or built.

Comment 9 by toyoshim@chromium.org, Sep 26 2016 

Kicked another bisect with another test since the test on zenbook failed.

Comment 10 by toyoshim@chromium.org, Sep 26 2016 

Issue 648906 has been merged into this issue.

Comment 11 by toyoshim@chromium.org, Sep 26 2016 

Issue 648911 has been merged into this issue.
Project Member

Comment 12 by 42576172...@developer.gserviceaccount.com, Sep 26 2016 


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N  Good?
chromium@418821  770.381  16.2519  5  good

Bisect job ran on: android_s5_perf_bisect
Bug ID: 648884

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests page_cycler_v2.intl_ar_fa_he
Test Metric: timeToFirstMeaningfulPaint_avg/pcv1-warm/http___msn.co.il_
Relative Change: None
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1034
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9000477927522418032


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5874076860547072

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 15 by 42576172...@developer.gserviceaccount.com, Sep 28 2016 


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N  Good?
chromium@418821  771.483  8.36911  5  good
chromium@418822  770.997  5.86723  5  good
chromium@418823  1606.03  102.343  5  bad
chromium@418824  1620.1   177.864  5  bad
chromium@418827  1638.23  171.068  5  bad
chromium@418833  1587.21  136.504  5  bad
chromium@418844  1625.38  285.049  5  bad
chromium@418866  1769.98  269.472  5  bad
chromium@418910  1585.88  152.062  5  bad
chromium@418999  1668.84  158.288  5  bad
chromium@419176  1666.39  140.162  5  bad
chromium@419531  1622.17  112.593  5  bad

Bisect job ran on: android_s5_perf_bisect
Bug ID: 648884

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests page_cycler_v2.intl_ar_fa_he
Test Metric: timeToFirstMeaningfulPaint_avg/pcv1-warm/http___msn.co.il_
Relative Change: 110.27%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1039
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9000365390816790144


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6348738056945664

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 16 by sullivan@chromium.org, Sep 28 2016

Cc: robert...@chromium.org
Owner: nednguyen@chromium.org
Ned, looks like r418823 is a catapult roll with 'Revert of "[telemetry] Wire TsProxy through telemetry's network stack."' as the only CL. regression expected?

Roberto, why didn't the bisect complete correctly?
Project Member

Comment 17 by 42576172...@developer.gserviceaccount.com, Sep 29 2016 


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N  Good?
chromium@418821  770.532  14.3593  5  good
chromium@418822  764.544  9.89505  5  good
chromium@418823  1653.72  131.542  5  bad
chromium@418824  1768.18  281.357  5  bad
chromium@418827  1685.19  152.605  5  bad
chromium@418833  1649.43  265.45   5  bad
chromium@418844  1596.16  115.767  5  bad
chromium@418866  1709.53  158.594  5  bad
chromium@418910  1712.77  263.243  5  bad
chromium@418999  1583.99  106.885  5  bad
chromium@419176  1611.85  124.815  5  bad
chromium@419531  1634.73  155.637  5  bad

Bisect job ran on: android_s5_perf_bisect
Bug ID: 648884

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests page_cycler_v2.intl_ar_fa_he
Test Metric: timeToFirstMeaningfulPaint_avg/pcv1-warm/http___msn.co.il_
Relative Change: 112.16%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1042
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9000312902926573120


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5872542684807168

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 18 by sullivan@chromium.org, Sep 29 2016

Status: WontFix (was: Assigned)
This is: "2016-09-15 perezju Revert of "[telemetry] Wire TsProxy through telemetry's network stack." (patchset #1 id:1 of https://codereview.chromium.org/2335403003/ )"

The change is expected.

Comment 19 by nedngu...@google.com, Sep 29 2016

Cc: pmeenan@chromium.org kouhei@chromium.org
Owner: nedngu...@google.com
Status: Available (was: WontFix)
Hmhh, I didn't expect the pass through mode of ts_proxy cause such a huge regression on page_cycler_v2 benchmarks.

Note that if ts_proxy change is the root cause, it should affect all the pages, not just some page. So I suspect that this is something else.
bzTCpC2DUqg.png
135 KB View Download

Comment 21 by nedngu...@google.com, Sep 29 2016 

I kick off another bisect.
Project Member

Comment 22 by 42576172...@developer.gserviceaccount.com, Sep 30 2016 


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@418821  786.959  64.2763  12  good
chromium@418866  797.324  16.3207  8   bad
chromium@418910  807.402  15.9994  5   bad
chromium@418999  796.568  17.8617  12  bad
chromium@419176  797.43   13.5155  12  bad
chromium@419531  797.99   12.4004  5   bad

Bisect job ran on: android_s5_perf_bisect
Bug ID: 648884

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests page_cycler_v2.intl_ar_fa_he
Test Metric: timeToFirstMeaningfulPaint_avg/pcv1-cold/http___msn.co.il_
Relative Change: 4.01%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1044
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9000181005169101568


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5021721340411904

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 24 by 42576172...@developer.gserviceaccount.com, Oct 1 2016 


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N  Good?
chromium@418821  766.42   10.9427  8  good
chromium@418827  812.515  16.3361  5  bad
chromium@418833  800.717  15.0464  8  bad
chromium@418844  794.859  13.5693  5  bad
chromium@418866  800.941  23.8462  8  bad
chromium@418910  792.53   13.4046  8  bad
chromium@418999  798.237  18.2304  8  bad
chromium@419176  803.551  20.3369  5  bad
chromium@419531  806.948  12.2866  5  bad

Bisect job ran on: android_s5_perf_bisect
Bug ID: 648884

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests page_cycler_v2.intl_ar_fa_he
Test Metric: timeToFirstMeaningfulPaint_avg/pcv1-cold/http___msn.co.il_
Relative Change: 5.60%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1048
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9000181005169101568


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5021721340411904

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 25 by nedngu...@google.com, Oct 6 2016

Blockedon: 653519

Comment 26 by benhenry@google.com, Nov 17 2016 

I don't understand - is this as Annie says in comment #18? Why was this reopened?

Comment 27 by nedngu...@google.com, Nov 17 2016

Cc: benhenry@google.com
Ben: this was kept opened so we can study & confirm whether this is actually due to TsProxy change. Assuming it's, if we improve ts_proxy, it's possible that doing so would fix this regression.

Comment 28 by benhenry@google.com, Dec 20 2016

Status: Assigned (was: Available)

Comment 29 by benhenry@google.com, Dec 21 2016

Status: WontFix (was: Assigned)
These will all be invalid as they are specific to Samsung S5s, which we do not trust to give us a signal and have subsequently removed from the waterfalls. If you think this regression is real, please re-open.
Project Member

Comment 31 by 42576172...@developer.gserviceaccount.com, Apr 11 2017

Cc: perezju@chromium.org
Owner: perezju@chromium.org

=== Auto-CCing suspected CL author perezju@chromium.org ===

Hi perezju@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Perf regression found with culprit

Suspected Commit
  Author : perezju
  Commit : 3cc40b37372927435ed69f91aec2117488b0825c
  Date   : Thu Sep 15 09:25:34 2016
  Subject: Revert of "[telemetry] Wire TsProxy through telemetry's network stack." (patchset #1 id:1 of https://codereview.chromium.org/2335403003/ )

Bisect Details
  Configuration: winx64_high_dpi_perf_bisect
  Benchmark    : page_cycler_v2.basic_oopif
  Metric       : timeToFirstContentfulPaint_avg/pcv1-cold/http___www.fifa.com_
  Change       : 251.41% | 319.709333333 -> 1123.49683333

Revision                                 Result                  N
chromium@418789                          319.709 +- 74.4992      6      good
chromium@418806                          302.714 +- 15.2795      6      good
chromium@418815                          310.928 +- 37.9786      6      good
chromium@418819                          332.657 +- 156.085      6      good
chromium@418821                          299.656 +- 19.9238      6      good
chromium@418822                          310.782 +- 38.9596      6      good
chromium@418822,catapult@3cc40b3737      1203.15 +- 195.973      6      bad       <--
chromium@418823                          1123.5 +- 75.8724       6      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release_x64 --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http...www.fifa.com. page_cycler_v2.basic_oopif

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982659729961739200

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5021721340411904


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
Project Member

Comment 32 by 42576172...@developer.gserviceaccount.com, Apr 11 2017

Cc: nedngu...@google.com
 Issue 647605  has been merged into this issue.
Project Member

Comment 33 by 42576172...@developer.gserviceaccount.com, Apr 11 2017 

Issue 649983 has been merged into this issue.
►

Sign in to add a comment