Here is a list of CLs:

https://chromium.googlesource.com/chromium/src/+log/fe16437ea76d582b4a931e219cd452756e2c6498%5E..f0d8d6ac72c8c8e0bf7447abe2ba455caf12d8cf?pretty=fuller

It's not obvious to me what's going wrong, a finer bisect or a backtrace with line numbers would help a lot.

Started bisect job https://chromeperf.appspot.com/buildbucket_job_status/8986738297224767296

Investigating...

=== BISECT JOB RESULTS ===
Test failure found with culprit

Suspected Commit
  Author : staraz
  Commit : 9d4e3df6e621c6bd80dfaee7008b3e33d0c4fe79
  Date   : Sat Feb 18 15:16:39 2017
  Subject: DirectCompositorFrameSink Uses CompositorFrameSinkSupport

Bisect Details
  Configuration: win_x64_perf_bisect
  Benchmark    : performance_browser_tests
  Metric       : CastV2Performance_gpu_novsync_24fps/capture_duration

Revision             Exit Code      N
chromium@451457      0 +- N/A       20      good
chromium@451461      0 +- N/A       20      good
chromium@451462      1 +- N/A       20      bad       <--
chromium@451463      1 +- N/A       20      bad
chromium@451465      1 +- N/A       20      bad
chromium@451472      1 +- N/A       20      bad
chromium@451486      1 +- N/A       20      bad

To Run This Test
  .\src\out\Release_x64\performance_browser_tests.exe --test-launcher-print-test-stdio=always --enable-gpu

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8986738297224767296

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6712164296949760


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Reviving, as the other bug was cut before staraz@'s change. This bug was cut related to a recent alert.

Note: There was some discussion on this in  bug 687284 .

I don't understand what's going on here. The bots are purple not red. Is this is a real perf regression?

Sorry it's confusing. We migrated the bots to swarming, and the purple is because some tasks are timing out. But performance_browser_tests is failing, not timing out. Example failed step:
https://build.chromium.org/p/chromium.perf/builders/Win%207%20Perf/builds/435/steps/performance_browser_tests%20on%20%28102b%29%20GPU%20on%20Windows%20on%20Windows-2008ServerR2-SP1

I was about to complain about the problem being Windows-only when I realized these tests got dropped for Mac somehow (filed bug 697195).

I ran on my Linux desktop, and the tests were fine. So, perhaps the problem is Windows-only. I am making a build on my Windows desktop to double-check.

staraz: In the meantime, I'd recommend `ninja performance_browser_tests` on your Windows desktop and run `out/Release/performance_browser_tests --enable-gpu` to see what is causing the failures (timeout, or assertion, or other?). I'll see if anything obvious pops-out on my own desktop.

Since both the TabCapturePerformanceTests and CastV2Streaming perf tests broke, and given that r451462 is known to have caused it; that means there's a problem w/ tab capture. So, let's just focus on TabCapturePerformanceTests to find the root cause.

Test code: https://cs.chromium.org/chromium/src/chrome/browser/extensions/api/tab_capture/tab_capture_performancetest.cc

From the latest logs, the failures are:

[1104:4016:0228/063730.625:ERROR:tab_capture_performancetest.cc(117)] Not enough events of type Capture found (1).
[1104:4016:0228/063730.626:ERROR:tab_capture_performancetest.cc(117)] Not enough events of type CaptureSucceeded found (0).

These events are generated via the tracing infrastructure. Their source is here:

https://cs.chromium.org/chromium/src/media/capture/content/thread_safe_capture_oracle.cc?rcl=ebdab4190ceb21a9ba4a5d8e719df9ab94b2716e&l=215

and

https://cs.chromium.org/chromium/src/media/capture/content/thread_safe_capture_oracle.cc?rcl=ebdab4190ceb21a9ba4a5d8e719df9ab94b2716e&l=224

So, DidCaptureFrame() is not being called, which means that the copy requests to the compositor are not running their result callbacks (see DelegatedFrameHost for where the CopyOutputRequests are initiated).

Therefore, something about r451462 has caused CopyOutputRequests to either: 1) be ignored; 2) be piling up somewhere, never executing. My suspicion is that there is something about the calls to SubmitCompositorFrame() or ForceReclaimResources() that has broken the processing of CopyOutputRequests. Perhaps there is a timing issue (order-of-operations). Also, I noticed the new code has left DirectCompositorFrameSink::WillDrawSurface() unimplemented. Perhaps that could be the cause?

FYI, I built and ran TabCapturePerformanceTest on a Win7 machine and got this:
https://paste.googleplex.com/5693109828583424

TabCapturePerformanceTests failed a CHECK when I ran it on Linux. Stack trace: https://paste.googleplex.com/6209892641669120

miu@: What gn args are you building with on Linux? I'm using 
is_debug = false
use_goma = true

And nothing else.

I'll look into the methods you mentioned.

staraz: Looks like I was wrong about Linux. Some tests work, some do not. When I run all the TabCapturePerformanceTests, the following fail (4 out of 8):

    TabCapturePerformanceTest.Performance/2 (../../chrome/browser/extensions/api/tab_capture/tab_capture_performancetest.cc:193)
    TabCapturePerformanceTest.Performance/3 (../../chrome/browser/extensions/api/tab_capture/tab_capture_performancetest.cc:193)
    TabCapturePerformanceTest.Performance/6 (../../chrome/browser/extensions/api/tab_capture/tab_capture_performancetest.cc:193)
    TabCapturePerformanceTest.Performance/7 (../../chrome/browser/extensions/api/tab_capture/tab_capture_performancetest.cc:193)

All of these have one thing in common: They disable vsync:

INSTANTIATE_TEST_CASE_P(
    ,
    TabCapturePerformanceTest,
    testing::Values(
/*0*/        0,
/*1*/        kUseGpu | kForceGpuComposited,
/*2*/        kDisableVsync,
/*3*/        kDisableVsync | kUseGpu | kForceGpuComposited,
/*4*/        kTestThroughWebRTC,
/*5*/        kTestThroughWebRTC | kUseGpu | kForceGpuComposited,
/*6*/        kTestThroughWebRTC | kDisableVsync,
/*7*/        kTestThroughWebRTC | kDisableVsync | kUseGpu | kForceGpuComposited));

...and I ran all the CastV2PerformanceTests. Only .../3 fails. And, you guessed it: that is the only one that disables vsync.

It sounds like this bug has the same cause as https://bugs.chromium.org/p/chromium/issues/detail?id=480361#c26.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d9bbb44814c924c58e80bb0e8b64a9a634fd2198

commit d9bbb44814c924c58e80bb0e8b64a9a634fd2198
Author: eseckler <eseckler@chromium.org>
Date: Thu Mar 02 22:12:14 2017

[cc] Forward BeginFrameAcks through DirectCFS and CFSSupport.

Also updates unit tests to check that acks are indeed forwarded.

BUG= 697086 ,646774,  696030 ,  480361 
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_trusty_blink_rel

Review-Url: https://codereview.chromium.org/2727553004
Cr-Commit-Position: refs/heads/master@{#454400}

[modify] https://crrev.com/d9bbb44814c924c58e80bb0e8b64a9a634fd2198/cc/surfaces/compositor_frame_sink_support.cc
[modify] https://crrev.com/d9bbb44814c924c58e80bb0e8b64a9a634fd2198/cc/surfaces/compositor_frame_sink_support.h
[modify] https://crrev.com/d9bbb44814c924c58e80bb0e8b64a9a634fd2198/cc/surfaces/compositor_frame_sink_support_unittest.cc
[modify] https://crrev.com/d9bbb44814c924c58e80bb0e8b64a9a634fd2198/cc/surfaces/direct_compositor_frame_sink.cc
[modify] https://crrev.com/d9bbb44814c924c58e80bb0e8b64a9a634fd2198/cc/surfaces/direct_compositor_frame_sink.h
[modify] https://crrev.com/d9bbb44814c924c58e80bb0e8b64a9a634fd2198/cc/surfaces/direct_compositor_frame_sink_unittest.cc
[modify] https://crrev.com/d9bbb44814c924c58e80bb0e8b64a9a634fd2198/cc/test/fake_compositor_frame_sink_client.cc
[modify] https://crrev.com/d9bbb44814c924c58e80bb0e8b64a9a634fd2198/cc/test/fake_compositor_frame_sink_client.h

Assigning to myself for verification. The graphs didn't update in the couple of runs made after the "fix" landed, so I'm not confident the issue is fixed.

However, now there's bug 698757. It's unclear whether the infra issue was present during those 2 runs.

FWIW we got fresh numbers for these two metrics in bug 692600, so maybe these two tests are fine now?

Verified. Thanks! :)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6a9efe9bb5bce08d48578cda5497045c4d33b6ec

commit 6a9efe9bb5bce08d48578cda5497045c4d33b6ec
Author: eseckler <eseckler@chromium.org>
Date: Fri Mar 24 16:29:56 2017

[cc] Fix CompositorFrameSinkSupport BeginFrameAck interface.

This replaces CFSSupport::DidFinishFrame with BeginFrameDidNotSwap and
uses SubmitCompositorFrame for BeginFrameAcks with damage instead.

This change also requires that all CompositorFrame submitters actually
set the BeginFrameAck on their CompositorFrames, so that we avoid
 crbug.com/696030  (and so that we don't hit the newly added
DLOG/DCHECKs).

This is work towards unified BeginFrame acknowledgments, see:
Tracking bug:  https://crbug.com/697086 
Design doc: http://bit.ly/beginframeacks

BUG= 697086 , 646774,  696030 
CQ_INCLUDE_TRYBOTS=master.tryserver.blink:linux_trusty_blink_rel;master.tryserver.chromium.linux:linux_site_isolation

Review-Url: https://codereview.chromium.org/2755463002
Cr-Commit-Position: refs/heads/master@{#459451}

[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/android_webview/browser/hardware_renderer.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/android_webview/browser/surfaces_instance.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/android_webview/browser/test/rendering_test.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/ash/laser/laser_pointer_view.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/ash/laser/laser_pointer_view.h
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/cc/ipc/mojo_compositor_frame_sink.mojom
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/cc/output/begin_frame_args.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/cc/output/begin_frame_args.h
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/cc/surfaces/compositor_frame_sink_support.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/cc/surfaces/compositor_frame_sink_support.h
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/cc/surfaces/compositor_frame_sink_support_unittest.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/cc/surfaces/direct_compositor_frame_sink.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/cc/surfaces/direct_compositor_frame_sink_unittest.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/cc/trees/layer_tree_host_impl_unittest.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/components/display_compositor/gpu_compositor_frame_sink.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/components/display_compositor/gpu_compositor_frame_sink.h
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/components/display_compositor/gpu_root_compositor_frame_sink.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/components/display_compositor/gpu_root_compositor_frame_sink.h
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/components/exo/compositor_frame_sink.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/components/exo/compositor_frame_sink.h
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/components/exo/compositor_frame_sink_holder.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/components/exo/surface.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/content/browser/frame_host/render_widget_host_view_child_frame_unittest.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/content/browser/frame_host/render_widget_host_view_guest_unittest.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/content/browser/renderer_host/delegated_frame_host.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/content/browser/renderer_host/offscreen_canvas_compositor_frame_sink.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/content/browser/renderer_host/offscreen_canvas_compositor_frame_sink.h
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/content/renderer/android/synchronous_compositor_frame_sink.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/services/ui/public/cpp/client_compositor_frame_sink.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/services/ui/ws/display_client_compositor_frame_sink.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/services/ui/ws/window_tree_client_unittest.cc
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/third_party/WebKit/Source/core/offscreencanvas/OffscreenCanvas.cpp
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/third_party/WebKit/Source/platform/graphics/OffscreenCanvasFrameDispatcherImpl.cpp
[modify] https://crrev.com/6a9efe9bb5bce08d48578cda5497045c4d33b6ec/third_party/WebKit/Source/platform/graphics/OffscreenCanvasFrameDispatcherImpl.h