New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 667810 link

Starred by 2 users
Status: WontFix
Owner:
mstarzinger@chromium.org
Closed: Dec 2016
Cc:
perezju@chromium.org
nzolghadr@chromium.org
█ robert...@chromium.org
█ nednguyen@chromium.org
sullivan@chromium.org
mstarzinger@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug-Regression

Blocked on:
issue 667816


Sign in to add a comment

86.4% regression in system_health.memory_mobile at 427317:427350

Project Member Reported by nzolghadr@chromium.org, Nov 22 2016 
See the link to graphs below.

Comment 1 by nzolghadr@chromium.org, Nov 22 2016 

All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=667810

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgm4DVqgoM


Bot(s) for this bug's original alert(s):

android-nexus7v2
Project Member

Comment 3 by 42576172...@developer.gserviceaccount.com, Nov 22 2016 

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/3510
Failure reason: the build has failed due to infrastructure failure.

Comment 4 by sullivan@chromium.org, Nov 22 2016

Blockedon: 667816
Project Member

Comment 6 by 42576172...@developer.gserviceaccount.com, Nov 22 2016 

Bisect failed: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/3512
Failure reason: the build has failed due to infrastructure failure.
Project Member

Comment 8 by 42576172...@developer.gserviceaccount.com, Nov 22 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N   Good?
chromium@427316  446085  893955   27  good
chromium@427350  684338  2168334  27  bad

Bisect job ran on: android_nexus7_perf_bisect
Bug ID: 667810

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.media.imgur system_health.memory_mobile
Test Metric: memory:chrome:all_processes:reported_by_chrome:skia:effective_size_avg/browse_media/browse_media_imgur
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/3513
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995267017993704480


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5003447810654208

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 9 by nzolghadr@chromium.org, Nov 24 2016

Cc: nednguyen@chromium.org
nednguyen@ the graph seems very noisy and it seems that the regression took its effect over time with a lot of noise. Bisect seems to fail to find anything. How should we proceed?

Comment 10 by nedngu...@google.com, Nov 24 2016

Owner: perezju@chromium.org
Status: Assigned (was: Untriaged)

Comment 12 by perezju@chromium.org, Nov 25 2016 

Let's see if bisecting over the stdev works ...
Project Member

Comment 13 by 42576172...@developer.gserviceaccount.com, Nov 25 2016 


===== BISECT JOB RESULTS =====
Status: failed


=== Bisection aborted ===
The bisect was aborted because Bisect cannot identify a culprit: Bisect failed to reproduce the regression with enough confidence.
Please contact the the team (see below) if you believe this is in error.

===== TESTED REVISIONS =====
Revision         Mean    Std Dev  N   Good?
chromium@427316  513026  1494306  27  good
chromium@427350  751458  2312064  27  bad

Bisect job ran on: android_nexus7_perf_bisect
Bug ID: 667810

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.media.imgur system_health.memory_mobile
Test Metric: memory:chrome:all_processes:reported_by_chrome:skia:effective_size_avg/browse_media/browse_media_imgur
Relative Change: None

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/3515
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8995031242109492240


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5834788156473344

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 14 by perezju@chromium.org, Nov 25 2016

Cc: sullivan@chromium.org robert...@chromium.org
Well, it didn't. Annie, Roberto, any help? Looks like a pretty clear regression (both on mean and stdev) but the bisect "failed to reproduce with enough confidence".

Comment 15 by nedngu...@google.com, Nov 25 2016

Cc: perezju@chromium.org
Owner: dtu@chromium.org
Reassign to Dave since this seems like a bisect numerical comparison problem.

Comment 16 by simonhatch@chromium.org, Nov 29 2016 

re: #c12

Was this bisect supposed to be over stdev? Because it looks like it's still over the metric values directly. Is stdev bisect still supported?

https://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/3515/steps/Re-testing%20reference%20range.Compare%20samples%20%288%29/logs/json.output

Comment 17 by sullivan@chromium.org, Nov 29 2016 

Bisect stddev is no longer supported, dashboard dialog needs an update :(

Comment 18 by simonhatch@chromium.org, Nov 29 2016 

Ok logged a bug to remove that: https://github.com/catapult-project/catapult/issues/3047

Comment 20 by perezju@chromium.org, Dec 5 2016 

Trying again a regular bisect.
Project Member

Comment 21 by 42576172...@developer.gserviceaccount.com, Dec 5 2016

Cc: mstarzinger@chromium.org
Owner: mstarzinger@chromium.org

=== PERF REGRESSION ===


=== Auto-CCing suspected CL author mstarzinger@chromium.org ===

Hi mstarzinger@chromium.org, the bisect results pointed to your CL, please take a look at the
results.


===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Revert of [compiler] Prepare for partially shipping Ignition. (patchset #1 id:1 of https://codereview.chromium.org/2443573002/ )
Author  : mstarzinger
Commit description:
  
Reason for revert:
Causes regressions: https://bugs.chromium.org/p/chromium/issues/detail?id=658711

Original issue's description:
> [compiler] Prepare for partially shipping Ignition.
>
> This prepares the code-base so that Ignition can be enabled on a certain
> subset of compilations without setting the {FLAG_ignition} flag (which
> enables Ignition on all compilations). We should not check the flag in
> question explicitly anywhere outside of the compiler heuristics.
>
> R=mvstanton@chromium.org

BUG=chromium:658711
TBR=mvstanton@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.

Review-Url: https://codereview.chromium.org/2448443002
Cr-Commit-Position: refs/heads/master@{#40534}
Commit  : 6dd0587be31034b958762d60f7aa6ada66b8a3b8
Date    : Mon Oct 24 13:02:29 2016


===== TESTED REVISIONS =====
Revision                       Mean     Std Dev  N   Good?
chromium@427316                445082   926679   27  good
chromium@427318                463887   897097   18  good
chromium@427318,v8@cc448ff0f0  489124   884637   12  good
chromium@427318,v8@6dd0587be3  892649   2392923  27  bad    <--
chromium@427319                930254   1933284  18  bad
chromium@427321                820566   1935561  18  bad
chromium@427325                955164   1552176  12  bad
chromium@427333                1151643  820408   5   bad
chromium@427350                926192   1961310  18  bad

Bisect job ran on: android_nexus7_perf_bisect
Bug ID: 667810

Test Command: src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.media.imgur system_health.memory_mobile
Test Metric: memory:chrome:all_processes:reported_by_chrome:skia:effective_size_avg/browse_media/browse_media_imgur
Relative Change: 108.09%

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/3531
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8994117788357615776


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6182781061693440

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Comment 22 by pmeenan@chromium.org, Dec 13 2016 

Perf sheriff ping

Comment 23 by perezju@chromium.org, Dec 19 2016

Status: WontFix (was: Assigned)
This appears to have gotten fixed by something in the range:
http://test-results.appspot.com/revision_range?start=435898&end=436036
Project Member

Comment 25 by 42576172...@developer.gserviceaccount.com, Apr 10 2017 


=== BISECT JOB RESULTS ===
NO Perf regression found

Bisect Details
  Configuration: android_nexus7_perf_bisect
  Benchmark    : system_health.memory_mobile
  Metric       : memory:chrome:all_processes:reported_by_chrome:skia:effective_size_avg/browse_media/browse_media_imgur

Revision             Result                 N
chromium@427316      446085 +- 893955       27      good
chromium@427350      684338 +- 2168334      27      bad

Please refer to the following doc on diagnosing memory regressions:
  https://chromium.googlesource.com/chromium/src/+/master/docs/memory-infra/memory_benchmarks.md

To Run This Test
  src/tools/perf/run_benchmark -v --browser=android-chromium --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=browse.media.imgur system_health.memory_mobile

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8982667337166136432

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5003447810654208


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
►

Sign in to add a comment