Browser hang in M67 with excessive download attempts |
||||||||||
Issue descriptionSimilar to issue 809775 , the browser hangs when the attached repro is run. As mentioned @ crbug.com/809775#c39 , this is a new instance of browser process DoS. The long-term solution will be to manage incoming requests on BrowserThread::IO. In the meantime hotfixes will be required. On M67-M69, the issue with this specific repro is that the download UI goes nuts trying to refresh and is no longer able to process input as a result (thus the hang is unstoppable). See attached flame graph (even though the warning dialog for too many downloads comes up, chrome still goes into an infinite queue of processing download messages). Even killing the bad renderer results in minutes of 100% CPU processing queued requests. @qinmin as download/OWNERS
,
Jul 4
From robliao@ offline : """ I did some targeted lightspeeds (e.g. no-oping selected areas) in the downloads code on the UI thread on where we spend a bit of time and the resulting effect seems to be to make more time for the incoming requests with limited improvement in responsiveness. Rate-limiting the downloads as well as splitting the incoming tasks to the UI thread would also help in preventing this hang. """
,
Jul 4
The ultimate fix can't be strictly in the renderer IMO as our security model is for the browser to assume the renderer has gone rogue. Having SequenceManager on the UI and IO threads will allow us to solve this nicely IMO. If there isn't anything we can do to hotfix downloads, feel free to reassign to me.
,
Jul 5
The examples I have seen are using the download attribute, which could mean it was probably fixed by https://chromium-review.googlesource.com/758236 and reintroduced by https://chromium-review.googlesource.com/992322
,
Jul 5
,
Jul 5
it didn't cause this, but as carewolf@ said reintroduced. I wonder whether https://chromium-review.googlesource.com/c/chromium/src/+/899666 is related? At this point, this extra hop is no longer required /cc mek
,
Jul 5
i don't think there are any hotfixes we can do at download side. DownloadResourceThrottle already provides such protection by allowing only one download to take place from the page. However, if user allows multiple download to happen on the page, then the UI thread will become swamped by all the DownloadShelf updates.
,
Jul 5
@qinmin : did you run the repro? It hangs even if user blocks multiple downloads. The whole UI is hung, even the page unresponsive dialog won't draw (see it as a transparent window in the taskbar).
,
Jul 5
Yes, I run it on my linux desktop. If user blocks multiple downloads, i can switch to another tab and open settings etc. If user don't do anything, a pop up dialog will be shown, and the whole browser become unresponsive and switching tabs are not possible. I can still go to terminal to kill the browser process though. If user allows multiple download, the download shelf will go crazy and the whole machine starts becoming sluggish. Shutting down the browser process also becomes a uneasy task. If we don't want too much stress on the UI thread, we have to move a lot DownloadRequestLimiter work onto the IO thread. But since content settings/webcontents/NavigationHandle are all accessed on the UI thread, there is a lot of stuff needs to be reworked. I don't think a short term hotfix will be possible though.
,
Jul 6
Ok, would some sort of backpressure system on the download IPC work? Le jeu. 5 juill. 2018 16 h 54, qinmin via monorail < monorail+v2.951229368@chromium.org> a écrit :
,
Jul 9
Yes, if we can rate limit calls to RenderFrameMessageFilter::DownloadUrl(), that would definitely free things up on the UI thread
,
Jul 9
Agreed. The easiest solution at the moment is to impose a rate limit (akin to a QPS limit) on multiple download attempts.
,
Jul 10
ok, assigning this to me as i will work on a temporary rate limit fix
,
Jul 11
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a13cb2a1020b933c070d9db84fbc48dae5b0e1c6 commit a13cb2a1020b933c070d9db84fbc48dae5b0e1c6 Author: Min Qin <qinmin@chromium.org> Date: Wed Jul 11 04:53:36 2018 Throttle url download requests from a RenderFrame RenderFrameImpl::DownloadUrl() can be abused to trigger a large number of download to overwhelm the browser process. This CL rate limits the number of download to a maximum of 10 per second. Bug: 860045 Change-Id: I490ae3a99eb59aef709b9f1e6be5eb0f787354f4 Reviewed-on: https://chromium-review.googlesource.com/1132485 Commit-Queue: Min Qin <qinmin@chromium.org> Reviewed-by: John Abd-El-Malek <jam@chromium.org> Cr-Commit-Position: refs/heads/master@{#574078} [modify] https://crrev.com/a13cb2a1020b933c070d9db84fbc48dae5b0e1c6/content/renderer/render_frame_impl.cc [modify] https://crrev.com/a13cb2a1020b933c070d9db84fbc48dae5b0e1c6/content/renderer/render_frame_impl.h [modify] https://crrev.com/a13cb2a1020b933c070d9db84fbc48dae5b0e1c6/content/renderer/render_frame_impl_browsertest.cc
,
Jul 12
tested on an very old windows 7 machine (bought in 2009) with the latest canary build, browser is still responsive even after i clicked accept all download button from the malicious page.
,
Jul 12
This bug requires manual review: We are only 11 days from stable. Please contact the milestone owner if you have questions. Owners: cmasso@(Android), kariahda@(iOS), bhthompson@(ChromeOS), abdulsyed@(Desktop) For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Jul 13
How safe is this merge overall?
,
Jul 13
Should be quite safe, i think. If there are pages that issuing a lot of downloads within a second, they will be affected anyways when M69 is released.
,
Jul 16
Approving merge to M68. Branch:3440
,
Jul 17
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/4d44d2be0eb693d9b477b476072011cc3b852a78 commit 4d44d2be0eb693d9b477b476072011cc3b852a78 Author: Min Qin <qinmin@chromium.org> Date: Tue Jul 17 20:41:16 2018 Throttle url download requests from a RenderFrame RenderFrameImpl::DownloadUrl() can be abused to trigger a large number of download to overwhelm the browser process. This CL rate limits the number of download to a maximum of 10 per second. Bug: 860045 Change-Id: I490ae3a99eb59aef709b9f1e6be5eb0f787354f4 Reviewed-on: https://chromium-review.googlesource.com/1132485 Commit-Queue: Min Qin <qinmin@chromium.org> Reviewed-by: John Abd-El-Malek <jam@chromium.org> Cr-Original-Commit-Position: refs/heads/master@{#574078}(cherry picked from commit a13cb2a1020b933c070d9db84fbc48dae5b0e1c6) Reviewed-on: https://chromium-review.googlesource.com/1140457 Reviewed-by: Min Qin <qinmin@chromium.org> Cr-Commit-Position: refs/branch-heads/3440@{#703} Cr-Branched-From: 010ddcfda246975d194964ccf20038ebbdec6084-refs/heads/master@{#561733} [modify] https://crrev.com/4d44d2be0eb693d9b477b476072011cc3b852a78/content/renderer/render_frame_impl.cc [modify] https://crrev.com/4d44d2be0eb693d9b477b476072011cc3b852a78/content/renderer/render_frame_impl.h [modify] https://crrev.com/4d44d2be0eb693d9b477b476072011cc3b852a78/content/renderer/render_frame_impl_browsertest.cc
,
Jul 17
|
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by qin...@chromium.org
, Jul 4