New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 648893 link

Starred by 2 users

Issue metadata

Status: WontFix
Owner:
Last visit > 30 days ago
Closed: Dec 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression

Blocked on:
issue 644865



Sign in to add a comment

114.2% regression in page_cycler_v2.intl_es_fr_pt-BR at 419111:419159

Project Member Reported by toyoshim@chromium.org, Sep 21 2016

Issue description

See the link to graphs below.
 
All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=648893

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICguYmLtAoM


Bot(s) for this bug's original alert(s):

android-galaxy-s5
Cc: toyoshim@chromium.org
 Issue 648889  has been merged into this issue.
Project Member

Comment 4 by 42576172...@developer.gserviceaccount.com, Sep 21 2016

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1014
Failure reason: the build has failed.
Additional errors:
The metric was not found in the test output.
Either of the initial "good" or "bad" revisions failed to be tested or built.

Blockedon: 644865
Project Member

Comment 8 by 42576172...@developer.gserviceaccount.com, Sep 22 2016


===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@419110  898.151  280.186  18  good
chromium@419159  1096.38  217.963  18  bad

Bisect job ran on: android_s5_perf_bisect
Bug ID: 648893

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests page_cycler_v2.intl_es_fr_pt-BR
Test Metric: timeToFirstContentfulPaint_avg/pcv1-warm/http___www.ebay.es_sch_i.html?_sacat_382__trkparms_clkid%3D6548971389060485883__qi_RTM1381637
Relative Change: 42.82%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1018
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9000870900387577616


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5840096287260672

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Project Member

Comment 9 by 42576172...@developer.gserviceaccount.com, Sep 22 2016

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1021
Failure reason: the build has failed.
Additional errors:
The metric was not found in the test output.
Either of the initial "good" or "bad" revisions failed to be tested or built.

Regression is likely happened in the graph, but Std Dev wasn't narrow enough in the bisect. I kicked another bisect with a large N, 40.
Still running

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@419110  948.768  185.271  12  good
chromium@419135  866.666  192.739  5   good
chromium@419159  1254.32  112.274  8   bad
Project Member

Comment 13 by 42576172...@developer.gserviceaccount.com, Sep 27 2016


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@419110  948.768  185.271  12  good
chromium@419135  866.666  192.739  5   good
chromium@419159  1254.32  112.274  8   bad

Bisect job ran on: android_s5_perf_bisect
Bug ID: 648893

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests page_cycler_v2.intl_es_fr_pt-BR
Test Metric: timeToFirstContentfulPaint_avg/pcv1-warm/http___www.ebay.es_sch_i.html?_sacat_382__trkparms_clkid%3D6548971389060485883__qi_RTM1381637
Relative Change: 29.83%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1035
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9000477583037353616


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5873003856920576

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Kicked another bisect within 419135:419159.
Also, this might be recovered around r419884 that is the last build at this point.
Project Member

Comment 16 by 42576172...@developer.gserviceaccount.com, Sep 29 2016


===== BISECT JOB RESULTS =====
Status: failed


===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@419135  881.591  215.518  12  good
chromium@419159  1191.3   189.176  18  bad

Bisect job ran on: android_s5_perf_bisect
Bug ID: 648893

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests page_cycler_v2.intl_es_fr_pt-BR
Test Metric: timeToFirstContentfulPaint_avg/pcv1-warm/http___www.ebay.es_sch_i.html?_sacat_382__trkparms_clkid%3D6548971389060485883__qi_RTM1381637
Relative Change: 10.28%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1041
Job details: https://chromeperf.appspot.com/buildbucket_job_status/9000313304104328592


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5213129913401344

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Cc: sullivan@chromium.org
cc: sullivan, perfbot sheriff

Could you help me to succeed to run a bisect?
> "failure_reason": "INFRA_FAILURE"
Cc: dtu@chromium.org
Owner: robert...@chromium.org
Roberto, I can't find the error message in the output (I no error in stdio for "Working on revision chromium@419147.Performance Test 4 of 6 (2)". Can you clarify the error?

Dave, how far along is the story-filter work? Could we re-run these bisects with story filters?
This job timed out after 24 hrs.
Perf sheriff ping
Project Member

Comment 22 by 42576172...@developer.gserviceaccount.com, Oct 20 2016

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_bisect/builds/1116
Failure reason: the build has failed.
Additional errors:
The metric was not found in the test output.
Either of the initial "good" or "bad" revisions failed to be tested or built.

Project Member

Comment 24 by 42576172...@developer.gserviceaccount.com, Nov 17 2016


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@419110  621.127  1312.18  27  good
chromium@419159  717.424  1479.56  27  bad

Bisect job ran on: android_nexus5X_perf_bisect
Bug ID: 648893

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=http...www.ebay.es.sch.i.html..sacat.382..trkparms.clkid.3D6548971389060485883..qi.RTM1381637 page_cycler_v2.intl_es_fr_pt-BR
Test Metric: timeToFirstContentfulPaint_avg/pcv1-warm/http___www.ebay.es_sch_i.html?_sacat_382__trkparms_clkid%3D6548971389060485883__qi_RTM1381637
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus5X_perf_bisect/builds/880
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995785690798052640


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5231048730869760

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!
Status: WontFix (was: Assigned)
Closing as the s5 bot has been decommissioned.
Project Member

Comment 28 by 42576172...@developer.gserviceaccount.com, Apr 11 2017


=== BISECT JOB RESULTS ===
Perf regression found but unable to narrow commit range

Build failures prevented the bisect from narrowing the range further.


Bisect Details
  Configuration: android_s5_perf_bisect
  Benchmark    : page_cycler_v2.intl_es_fr_pt-BR
  Metric       : timeToFirstContentfulPaint_avg/pcv1-warm/http___www.ebay.es_sch_i.html?_sacat_382__trkparms_clkid%3D6548971389060485883__qi_RTM1381637

Suspected Commit Range
  1 commits in range
  https://chromium.googlesource.com/chromium/src/+log/325c64a962cb5c007368ef431dfa863c26c7847b..d527839060fe162c8cf2e9841098b19f5963d6e8


Revision             Result                  N
chromium@419135      881.591 +- 215.518      12      good
chromium@419159      1191.3 +- 189.176       18      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --also-run-disabled-tests page_cycler_v2.intl_es_fr_pt-BR

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982651822800608480

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5213129913401344


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Sign in to add a comment