Can you give a link to the site you are debugging? We try and monitor startup speed and if what you say is a regression we want to fix it, but need more context to reproduce it.

Thanks!

First I thought it was only on local websites, but I discovered now that this page http://atomic.run has the same issues. I know that the particles here need processing power, but before the update, Chrome was able to handle this without a blink. It happens on many more local running sites.

Is there anyway of rolling back to Chrome 59 without it constantly updating again for the time being, till this is resolved? Because otherwise I'll have to change to some other Browser for developing.

BTW: I'm on a 6core MacPro with a Cinema 4k Display

Thank you for providing more feedback. Adding requester "allada@chromium.org" to the cc list and removing "Needs-Feedback" label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Able to reproduce the issue on mac 10.12.6 using chrome reported version #61.0.3163.100 and latest canary #63.0.3233.0.
Issue is specific to OS-Mac.

Bisect Information:
=====================
Good build: 61.0.3154.0  Revision(485485)
Bad Build : 61.0.3155.0  Revision(485784)

Change Log URL: 
https://chromium.googlesource.com/chromium/src/+log/11e48b29d00fdd714c3ff3f8ebe0d51a2c4ce61c..63fbdcc12a7ef762e0652c972986fe74835ba00e

From the above change log suspecting below change
Change-Id: Iac99671e887b18e0bdf32d192d13e83fb0b1e55c
Reviewed-on: https://chromium-review.googlesource.com/566083

sunnyps@ - Could you please check whether this is caused with respect to your change, if not please help us in assigning it to the right owner.
Note: Adding label ReleaseBlock-Stable as it seems to be a recent regression. Please feel free to remove the same if not appropriate.

Thanks...!!

I can reproduce on Canary (63.0.3230.0) but not on ToT (r505193).

I don't know why I can't repro on ToT but I think that's a red herring.

I got a trace from Canary 63.0.3234.0 and it's clear that this is a gpu scheduler fairness issue. The devtools renderer waits for an absurdly long time to initialize in WaitForGetOffset. This might be a problem on mac only because the backpressure mechanism seems to be insufficient. There's lots of blocking in commands as well as the swap fence client wait.

Maybe the short term solution is to upgrade the priority of contexts with client waits. piman@ WDYT?

Deleted: trace_inspector_hang.json.gz 7.9 MB

trace_inspector_hang.json.gz 7.9 MB Download

Upgrading the priority to highest doesn't work because the UI builds up too much backpressure as soon as it's given higher priority after the client wait completes.

CL for raising priority on client wait (doesn't fix the regression): https://chromium-review.googlesource.com/#/c/chromium/src/+/706129

What seems to work (other than disabling the scheduler) is to set all priorities equal, and fall back to using order number only (similar to scheduler disabled).

ccameron@ any ideas?

Deleted: trace_equal_priorities.json.gz 5.3 MB

trace_equal_priorities.json.gz 5.3 MB Download

Also, to clear up earlier confusion, I can repro on ToT now. I had dchecks enabled before (dcheck_always_on = true), which slows down WebGL significantly and was probably hiding the regression by giving devtools more time.

sunnyps@, The same issue exists in current stable#60.0.3163.100 as well. So do you think it should block next M62 stable release? I am removing 'RB-Stable' as of now, however please feel free to add if you think otherwise.

Thank you!

On Mac there are a handful of complications

Mac has global glFlush ordering -- work is executed on the GPU in the order that it was flushed to the GPU, regardless of the fences/dependencies issued. This is true across contexts and across processes .

Mac has its compositing done by the WindowServer, which we can easily starve. The client wait exists to attempt to "play nice" with the WindowServer.

Re #9, the "fall back to using order number only" solution sounds like it is likely to play nicer with the Mac constraints.

Marking as RBS, we need to merge the fix into M62.

1) Visit http://atomic.run
2) Open inspector

inspector lags to even open, console and elements are unusable.

#12: Can you confirm this is happening on M60 as described in #14? Inspector is a bit slow with the gpu scheduler disabled but doesn't hang. I'm guessing it doesn't hang on M60 as well.

And isn't current stable 61.0.3163.100 (not 60.0.3163.100 as in #12).

#15,16:
I can reproduce this issue (per c#14) on current stable#61.0.3163.0 as well as on Latest Canary#63.0.3236.0 for OS X 10.12.6. It's not a complete hang, however this particular tab is consuming lot of CPU memory along with 'gpu process'. The impact is severe in Canary compared to Stable and at one point of time DevTools is unusable (Kind of a hung).

For M60#60.0.3112.113, i do not see much lag in Dev tools though the tab (http://atomic.run) is consuming little more CPU memory than usual.

I am attaching chrome://tracing for all the above scenarios here (https://goto.google.com/oiahn) along with chrome://gpu.

PS: The above behavior (Dev tools hang) is considerably varies w.r.t no. of open tabs in the browser.

And in #12 it was a typo. It used to be chrome#61.0.3160.0.

Thank you!

Deleted: chrome_gpu.rtf 91.0 KB

chrome_gpu.rtf 91.0 KB Download

I think there are two separate bugs here:
1. Slowness / lag when devtools is docked within the Chrome window. This seems to be because of starvation of the devtools renderer context because display compositor hogs up all the time. I have a fix for this (upgrade context priority when client is waiting).
2. Both the page and devtools stop rendering and hang when devtools is opened in its own window. I've been able to repro this with the gpu scheduler turned on and off.

This page is special because most of the GPU work is done in the display compositor (probably blurs and filters) as opposed to canvas context. Since the display compositor has highest priority it can starve the devtools context.

Also, adding a DescheduleUntilFinished to canvas prepare mailbox didn't do anything, probably because canvas context isn't doing much in the first place.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c98596763f36fd5fcc0c984be67b59f3775b35ac

commit c98596763f36fd5fcc0c984be67b59f3775b35ac
Author: Sunny Sachanandani <sunnyps@chromium.org>
Date: Wed Oct 11 23:19:16 2017

gpu: Prioritize contexts with client waits.

Sequences with client waits are prioritized the same way as sync token
releases. The UI context is given the same priority so that it can't
starve contexts with client waits. This provides some fairness and fixes
indefinite blocking of renderer contexts when the GPU process is loaded.

R=piman
BUG= 770146 
TEST=gpu_unittests SchedulerTest.ClientWaitIsPrioritized

Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: If5298a9bcdae13a6882b1297bc81708109b7922c
Reviewed-on: https://chromium-review.googlesource.com/706129
Reviewed-by: Antoine Labour <piman@chromium.org>
Commit-Queue: Sunny Sachanandani <sunnyps@chromium.org>
Cr-Commit-Position: refs/heads/master@{#508166}
[modify] https://crrev.com/c98596763f36fd5fcc0c984be67b59f3775b35ac/content/common/gpu_stream_constants.h
[modify] https://crrev.com/c98596763f36fd5fcc0c984be67b59f3775b35ac/gpu/command_buffer/common/scheduling_priority.cc
[modify] https://crrev.com/c98596763f36fd5fcc0c984be67b59f3775b35ac/gpu/command_buffer/common/scheduling_priority.h
[modify] https://crrev.com/c98596763f36fd5fcc0c984be67b59f3775b35ac/gpu/command_buffer/service/scheduler.cc
[modify] https://crrev.com/c98596763f36fd5fcc0c984be67b59f3775b35ac/gpu/command_buffer/service/scheduler.h
[modify] https://crrev.com/c98596763f36fd5fcc0c984be67b59f3775b35ac/gpu/command_buffer/service/scheduler_unittest.cc
[modify] https://crrev.com/c98596763f36fd5fcc0c984be67b59f3775b35ac/gpu/ipc/service/gpu_channel_unittest.cc
[modify] https://crrev.com/c98596763f36fd5fcc0c984be67b59f3775b35ac/gpu/ipc/service/gpu_command_buffer_stub.cc

abdulsyed@ This is marked as M62 release blocker. I've landed a fix which will roll into next canary. Can you decide if this indeed a release blocker and if we should merge the fix to 62?

#17: Can you please verify the fix on  63.0.3238.0?

Seems like things are improved and able to navigate within dev tools without much lag on Latest Canary#63.0.3238.0 for Mac OS X 10.12.6.

[Auto-generated comment by a script] We noticed that this issue is targeted for M-61; it appears the fix may have landed after branch point, meaning a merge might be required. Please confirm if a merge is required here - if so add Merge-Request-61 label, otherwise remove Merge-TBD label. Thanks.

abdulsyed@ I've requested a merge, please decide if the bug is serious enough to warrant a a merge this late in the cycle.

This bug requires manual review: We are only 3 days from stable.
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), bhthompson@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

I tested Canary 63.0.3239 and it seems to be solved (couldn't test it very throughly yet) but tested also some local running pages which I experienced the issues in chrome and on all of them, the inspector in Canary doesn't seem to lag anymore. 

Thanks for the fix. My recommendation is to wait until M63 since M62 is only a few days from stable. This does not seem to be critical enough to warrant a last minute merge. If you disagree, please provide a detailed explanation for why this is critical. 

I agree that this doesn't warrant a merge this late in the cycle.

- Do we have an idea on how often this is going to happen?
- Is there a simpler fix we can push into stable? With security update?

What I'm seeing at http://atomic.run is a true stable blocker. If we expect it to happen on more than a handful of sites in the internet, we need to fix in M62.

This bug requires manual review: We are only 3 days from stable.
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), bhthompson@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

I expect this to happen on pages that tax the GPU a lot. Normal pages shouldn't show this behavior.

Basically games? If this means DevTools is not available on the game sites, we have a serious issue.

So far, I've only seen the hang on the atomic.run page. We see absurdly long (50-100 ms) gpu backpressure on that page every alternate frame. Most of the gpu work is from the display compositor (lots of renderpasses maybe?). There is a canvas on that page but that also hangs.

Other pages I've tried:
1. http://webglsamples.org/aquarium/aquarium.html (lots of draw calls, not really gpu taxing)
2. fishgl.com (few draw calls, somewhat gpu taxing)

Both these cases seem to work fine wrt dev tools.

If you know of other pages that show this issue, please let me know.

Sunnyps@ - thanks for more feedback. Can you also comment on how safe/complex this merge is overall? Looking at the CL, seems like it's modifying some common classes, which makes me a bit nervous. Our biggest goal right now is to ensure that we minimize risk of introducing new regressions as we move M62 to Stable. Can you please comment on how safe this merge is overall,  it's overall complexity, and automated test coverage?

Re #23: No further M61 releases. Hence, removing "Merge-TBD" label. 

Confirmed with sunnyps@. This is not a simple fix, and due to it's very low prevalence, recommend not merging to M62. However, if this does become more prevalent, we can aim to include this in next M62 respin. 

This bug is also present in M61. 

 issue 760834  is apparently a duplicate (with very simple reproduction steps)

> On Mac there are a handful of complications

While  issue 760834  looks very much like a duplicate, the symptoms seem different indeed:

ChromeOS 63.0.3236.0
- Aquarium 4000 hangs inspector
- http://atomic.run does NOT

MacOS 62.0.3202.62
- Aquarium 4000 does NOT hang inspector
- http://atomic.run does

http://webglsamples.org/aquarium/aquarium.html

Note Aquarium 2000 on ChromeOS just makes things slower while Aquarium 4000 causes complete starvation and hangs.

(BTW  issue 760834  was filed one month earlier than this one)