All graphs for this bug:
  https://chromeperf.appspot.com/group_report?bug_id=658501

Original alerts at time of bug-filing:
  https://chromeperf.appspot.com/group_report?keys=agxzfmNocm9tZXBlcmZyFAsSB0Fub21hbHkYgICgnaSqoQkM


Bot(s) for this bug's original alert(s):

chromium-rel-mac-retina

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8998132736003980032

===== BISECT JOB RESULTS =====
Status: completed


===== SUSPECTED CL(s) =====
Subject : Revert of cc: Remove frame queuing from the scheduler. (patchset #14 id:400001 of https://codereview.chromium.org/2339633003/ )
Author  : sunnyps
Commit description:
  
Reason for revert:
Highly likely that this is causing the flakes we are seeing in  http://crbug.com/645736 

Original issue's description:
> Reland of cc: Remove frame queuing from the scheduler. (patchset #1 id:1 of https://codereview.chromium.org/2336493002/ )
>
> Reason for revert:
> Reland after fixing screenshot grabber test and perf issues.
>
> Original issue's description:
> > Revert of cc: Remove frame queuing from the scheduler. (patchset #3 id:40001 of https://codereview.chromium.org/2323063004/ )
> >
> > Reason for revert:
> > Broke ChromeScreenshotGrabberTest.TakeScreenshot on  Linux ChromiumOS Tests (dbg)(1): https://build.chromium.org/p/chromium.chromiumos/builders/Linux%20ChromiumOS%20Tests%20%28dbg%29%281%29/builds/18015.
> >
> > Original issue's description:
> > > cc: Remove frame queuing from the scheduler.
> > >
> > > CC scheduler has a frame queuing mechanism called "retro frames". This
> > > has been responsible for a lot of complexity and hard to fix bugs. The
> > > original intent for adding retro frames was to allow the scheduler to
> > > handle multiple frames in flight but that goal doesn't seem feasible or
> > > even desirable any more. This CL removes the retro frames queue and
> > > instead makes the Scheduler end the previous frame when it receives a
> > > BeginFrame message.
> > >
> > > One surprising behavior was that SyntheticBFS MISSED frames would be
> > > queued as retro frames and this would convert the synchronous begin
> > > frame call (inside Scheduler::ProcessScheduledActions) to an
> > > asynchronous retro frame PostTask. To work around this the Scheduler
> > > keeps track of a single CancelableClosure that's used for MISSED frames.
> > >
> > > The above behavior was also causing the BeginFramesNotFromClient tests
> > > to work even though there was an extra MISSED frame in the queue. This
> > > is more elegantly solved in another way by using frame number to advance
> > > the task runner instead of just running pending tasks.
> > >
> > > Lastly SchedulerStateMachine is modified so that it's possible to end
> > > the previous frame and still have the same behavior as before in the
> > > commit to active tree (browser compositor) mode.
> > >
> > > R=brianderson@chromium.org,enne@chromium.org,danakj@chromium.org
> > > BUG= 602485 ,  644992 
> > > CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel
> > >
> > > Committed: https://crrev.com/559280b26cc5672f5f054e8ac35281e804c14d78
> > > Cr-Commit-Position: refs/heads/master@{#417764}
> >
> > TBR=enne@chromium.org,brianderson@chromium.org,danakj@chromium.org,sunnyps@chromium.org
> > # Not skipping CQ checks because original CL landed more than 1 days ago.
> > BUG= 602485 ,  644992 
> >
> > Committed: https://crrev.com/95beb47e50065959ee2f5b43cf431431e32e14cd
> > Cr-Commit-Position: refs/heads/master@{#417895}
>
> TBR=enne@chromium.org,brianderson@chromium.org,danakj@chromium.org,sammc@chromium.org
> # Not skipping CQ checks because original CL landed more than 1 days ago.
> BUG= 602485 ,  644992 
> CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel
>
> Committed: https://crrev.com/864a70f6f93a87ff374bf2aea2494d4d7d0150d7
> Cr-Commit-Position: refs/heads/master@{#421268}

TBR=enne@chromium.org,danakj@chromium.org,brianderson@chromium.org
BUG= 602485 ,  644992 
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_precise_blink_rel

Review-Url: https://codereview.chromium.org/2386183003
Cr-Commit-Position: refs/heads/master@{#422595}
Commit  : 5a28cd7dd0ac324e02428174ed3333dbe88edfcd
Date    : Mon Oct 03 23:32:22 2016


===== TESTED REVISIONS =====
Revision         Mean     Std Dev   N  Good?
chromium@422340  16.6048  0.388322  8  good
chromium@422494  16.7639  0.416567  8  good
chromium@422571  16.5918  0.422226  4  good
chromium@422590  16.903   0.81077   8  good
chromium@422593  17.0059  0.618785  8  good
chromium@422594  17.3065  0.878097  5  good
chromium@422595  18.2365  0.388591  8  bad    <--
chromium@422600  17.9305  0.573572  8  bad
chromium@422609  17.6574  0.433069  5  bad
chromium@422648  18.2137  1.07674   5  bad
chromium@422956  18.0013  0.57593   5  bad
chromium@423572  17.9338  1.13197   8  bad

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 658501

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests battor.power_cases
Test Metric: idle:power_avg/idle:power_avg
Relative Change: 4.62%
Score: 99.9

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1761
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8998132736003980032


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5849230390853632

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Reopening, the other bug has been closed but the regression is still there for this metric.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8997795702915931008

===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev   N   Good?
chromium@421845  17.0715  0.883701  18  good
chromium@421875  17.2119  0.634546  18  bad

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 658501

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests battor.power_cases
Test Metric: idle:power_avg/idle:power_avg
Relative Change: 1.44%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1764
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8997795702915931008


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5796829441032192

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

Looks like there was an improvement prior to this regression (Point ID: 421875 Time added: 2016-09-30T00:07:12.000Z) because of an environmental issue affecting battor (the reference build also showed the same improvement) which is confirmed by the latest bisect above not being able to reproduce the improvement.

The commit range with the regression has both my revert and reland CLs so it's impossible for my CLs to cause this. Also there was a similar regression on the reference build at the same time so that may be a battor issue too.

The initial bisect seems to have detected a very small regression (~1W) compared to the large regression seen on the bot. And the reland of my CL should have fixed that in any case.

Also the commit range for the build with the regression (Time added: 2016-10-07T22:35:58.000Z - 5 days since last build) and the one after that (Time added: 2016-10-21T16:12:54.000Z - 14 days
since last build) are too large so that does not rule out that there was a real regression due to some other CL.

perf-sheriffs discussion: https://groups.google.com/a/chromium.org/d/msg/perf-sheriffs/DBElv-0IvIg/D0N2Ph8VBwAJ

I'm closing this as WontFix because this isn't actionable.

I'm going to do some more work on this and see if I can find out more about the regression by looking at the traces.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8997724986515669040

===== BISECT JOB RESULTS =====
Status: completed


=== Bisection aborted ===
The bisect was aborted because The metric values for the initial "good" and "bad" revisions do not represent a clear regression.
Please contact the the team (see below) if you believe this is in error.

=== Warnings ===
The following warnings were raised by the bisect job:

 * Bisect failed to reproduce the regression with enough confidence.

===== TESTED REVISIONS =====
Revision         Mean     Std Dev  N   Good?
chromium@422648  18.3982  1.68697  18  good
chromium@423572  19.0369  1.23143  18  bad

Bisect job ran on: mac_retina_perf_bisect
Bug ID: 658501

Test Command: src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=about.blank battor.power_cases
Test Metric: idle:power_avg/about_blank
Relative Change: 10.51%
Score: 0

Buildbot stdio: http://build.chromium.org/p/tryserver.chromium.perf/builders/mac_retina_perf_bisect/builds/1767
Job details: https://chromeperf.appspot.com/buildbucket_job_status/8997724986515669040


Not what you expected? We'll investigate and get back to you!
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5896456039301120

| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Tests>AutoBisect.  Thank you!

rnephew@, just to make sure I understand:

the timeline looks like

A-------B--------C---------D

A=start of revision range
B=sunnyps's CL lands
C=sunnyps's CL is reverted
D=end of revision range

We did a bisect from A to D, and it came back with C.

Would it be possible to kick off bisects from A to B-1, B+1 to C-1, and C+1 to D and see what comes back?

Yep, the C+1 to D is what I just ran.I'll kick off the other ones.

I doubt we will make progress on this at this point. What should we do with this bug?

I think closing as Archived makes sense. We probably should have followed up more with this, but following up with a regression from Oct 26 seems likely to be fruitless at this point.