New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 765605 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug-Security

Blocking:
issue 765371



Sign in to add a comment

Security: ble adv flooding: kernel panics/crashes

Project Member Reported by mnissler@chromium.org, Sep 15 2017

Issue description

This is a spin-off from  issue 765371 

There's evidence of spontaneous reboots when the BT stack gets flooded with BLE advertisements for thousands of devices.

We'll need to repro and check the nature of the spontaneous reboots, i.e. look at syslogs and pstore (/dev/pstore ? - I don't remember exactly). Grab any stack traces you can get hold of and post them here.

snanda@, groeck@, can you find someone to work with dmitrygr@ to gather details on the panics?
 
Here's one idea to help speed up analysis and potentially allow regression testing: We could maybe use hci_vhci + hciemu to inject advertisements into the kernel? Not sure whether we'd be able to crash the kernel, but it's definitely possible to create observable symptoms. While at it, we might use this to also hook up a fuzzer?
Project Member

Comment 2 by sheriffbot@chromium.org, Sep 15 2017

Labels: ReleaseBlock-Beta
This is a critical security issue. If you are not able to fix this quickly, please revert the change that introduced it.

If this doesn't affect a release branch, or has not been properly classified for severity, please update the Security_Impact or Security_Severity labels, and remove the ReleaseBlock label. To disable this altogether, apply ReleaseBlock-NA.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 3 by sheriffbot@chromium.org, Sep 15 2017

Labels: -Pri-1 Pri-0
Project Member

Comment 4 by sheriffbot@chromium.org, Sep 15 2017

Status: Assigned (was: Unconfirmed)
Labels: -ReleaseBlock-Beta -Pri-0 Pri-1
This is ongoing internal research on a potentially-critical security issue. No need to hold releases at this time.

Comment 6 by r...@chromium.org, Sep 15 2017

This seems like a fairly straightforward case of overloading the kernel's dispatch mechanisms to a point that it is doing nothing but handling incoming advertisements.

RCE seems unlikely but we definitely should investigate it.

Re comment #6: If the kernel were just overloaded, how do you explain the spontaneous reboots? My bet is on a kernel OOM situation (and that would make it a DoS only), but better assume the worst ;-)

Comment 8 by groeck@google.com, Sep 15 2017

Depends what "overload" means. 800pps should not overload the overall kernel, but who knows what happens in the bluetooth stack. It is quite likely not designed to handle 800pps of advertisements. If each of those creates a udev event, you have a likely explanation for any hang (and a mitigation might be to limit the rate of such udev events).
Other possibilities: There might be a memory leak, some locking issue, or even memory use after free. At this point everything is just a wild guess, though - we'll need a ramoops log after a crash/reboot.

Cc: mrjain@google.com
Owner: dmitrygr@google.com
Each adv gets processed in the kernel (including possibly some AES operations to resolve private resolvable addresses, which will get slower the more devices are paired) and then sent to userspace, where an actual dbus object is created
Project Member

Comment 12 by sheriffbot@chromium.org, Sep 16 2017

Labels: ReleaseBlock-Beta
This is a critical security issue. If you are not able to fix this quickly, please revert the change that introduced it.

If this doesn't affect a release branch, or has not been properly classified for severity, please update the Security_Impact or Security_Severity labels, and remove the ReleaseBlock label. To disable this altogether, apply ReleaseBlock-NA.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Labels: -ReleaseBlock-Beta ReleaseBlock-NA
dmitrygr@, groeck@: Can you folks work together to repro and collect kernel crash information?
So far I have not seen a kernel crash. Chrome and thus the UI locks up reliably.  After flooding it with requests for maybe a minute, it never recovers even if the source is removed (and chrome stays at 100% CPU forever). bluetoothd seems to recover. The driver either stops receiving data after some time, or the firmware hangs. So far I have not seen a kernel crash or traceback.

Note: This was observed with chromeos-4.12 running on eve.

Blocking: 765371
Project Member

Comment 16 by sheriffbot@chromium.org, Sep 30 2017

dmitrygr: Uh oh! This issue still open and hasn't been updated in the last 14 days. This is a serious vulnerability, and we want to ensure that there's progress. Could you please leave an update with the current status and any potential blockers?

If you're not the right owner for this issue, could you please remove yourself as soon as possible or help us find the right one?

If the issue is fixed or you can't reproduce it, please close the bug. If you've started working on a fix, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 17 by sheriffbot@chromium.org, Oct 15 2017

dmitrygr: Uh oh! This issue still open and hasn't been updated in the last 29 days. This is a serious vulnerability, and we want to ensure that there's progress. Could you please leave an update with the current status and any potential blockers?

If you're not the right owner for this issue, could you please remove yourself as soon as possible or help us find the right one?

If the issue is fixed or you can't reproduce it, please close the bug. If you've started working on a fix, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 18 by sheriffbot@chromium.org, Oct 15 2017

Labels: Deadline-Exceeded
We commit ourselves to a 30 day deadline for fixing for critical severity vulnerabilities, and have exceeded it here. If you're unable to look into this soon, could you please find another owner or remove yourself so that this gets back into the security triage queue?

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 19 by sheriffbot@chromium.org, Oct 18 2017

Labels: -M-61 M-62
this cannot really be debugged till the chrome issue is resolved (as otherwise chrome dies first) (see crbug/765371)
Labels: -Security_Severity-Critical -Deadline-Exceeded -M-62 Security_Severity-High
Project Member

Comment 22 by sheriffbot@chromium.org, Nov 1 2017

Labels: M-62
Comment #20 still holds
Project Member

Comment 24 by sheriffbot@chromium.org, Nov 14 2017

Labels: Deadline-Exceeded
We commit ourselves to a 60 day deadline for fixing for high severity vulnerabilities, and have exceeded it here. If you're unable to look into this soon, could you please find another owner or remove yourself so that this gets back into the security triage queue?

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 25 by sheriffbot@chromium.org, Dec 7 2017

Labels: -M-62 M-63
A fix for 765371 has landed in M64. Is this issue now unblocked?
Project Member

Comment 27 by sheriffbot@chromium.org, Jan 25 2018

Labels: -M-63 M-64
Cc: awhalley@chromium.org jorgelo@chromium.org kerrnel@chromium.org
This is our second-oldest High severity vulnerability (137 days). We need to resolve it ASAP; it looks like the blocking issue has been resolved. dmitrygr, if you're not the right person to handle this bug, can you please reassign to someone who is? Thanks!

+awhalley, jorgelo, kerrnel FYI
this is actually resolved. rkc@'s team took care of it. Now such a flood only slows down a chromebook, and when it is stopped, the chromebook recovers within a minute.
Owner: r...@chromium.org
rkc@ can you mark this as Fixed if it's so? Thanks.

Comment 31 by r...@chromium.org, Jan 30 2018

Owner: xiaoyinh@chromium.org
This should no longer be an issue. The kernel itself seems to be fine, only the UI hangs, making it an effective DoS attack. The UI hang has been fixed by Sarah Hu.

Re-assigning to Sarah to confirm that this is actually fixed.

Status: Fixed (was: Assigned)
UI has been fixed to better handle the advertisement flood. With this fix, chrome will recover shortly after the advertisement stops.  I'm not familiar with kernel code, but looking at the /var/log/messages during the advertisement flood I didn't find something that are obviously wrong about kernel. So looks like this is resolved.
Project Member

Comment 33 by sheriffbot@chromium.org, Feb 8 2018

Labels: -Restrict-View-SecurityTeam Restrict-View-SecurityNotify
Project Member

Comment 34 by sheriffbot@chromium.org, Feb 8 2018

Labels: Merge-Request-65
Project Member

Comment 35 by sheriffbot@chromium.org, Feb 9 2018

Labels: -Merge-Request-65 Merge-Review-65 Hotlist-Merge-Review
This bug requires manual review: M65 has already been promoted to the beta branch, so this requires manual review
Please contact the milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), bhthompson@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Labels: -Hotlist-Merge-Review -Merge-Review-65
The fix(see  crbug.com/765371  comment#47) is initially landed in 64.0.3281.0. No need to merge.
Project Member

Comment 37 by sheriffbot@chromium.org, Mar 27 2018

Labels: -M-64 M-65
Project Member

Comment 38 by sheriffbot@chromium.org, May 9 2018

Labels: -Restrict-View-SecurityNotify allpublic
This bug has been closed for more than 14 weeks. Removing security view restrictions.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Sign in to add a comment