Monorail Project: project-zero Issues People Development process History Sign in
New issue
Advanced search Search tips
Issue 1139 cloudflare: Cloudflare Reverse Proxies are Dumping Uninitialized Memory
Starred by 50 users Project Member Reported by taviso@google.com, Feb 19 Back to list
Status: Fixed
Owner:
Closed: Feb 23
Cc:

Restricted
  • Only users with EditIssue permission may comment.


Show other hotlists

Hotlists containing this issue:
Hotlist-1
Hotlist-1


Sign in to add a comment
(It took every ounce of strength not to call this issue "cloudbleed")

Corpus distillation is a procedure we use to optimize the fuzzing we do by analyzing publicly available datasets. We've spoken a bit about this publicly in the past, for example:

https://security.googleblog.com/2011/08/fuzzing-at-scale.html
http://taviso.decsystem.org/making_software_dumber.pdf#page=11

On February 17th 2017, I was working on a corpus distillation project, when I encountered some data that didn't match what I had been expecting. It's not unusual to find garbage, corrupt data, mislabeled data or just crazy non-conforming data...but the format of the data this time was confusing enough that I spent some time trying to debug what had gone wrong, wondering if it was a bug in my code. In fact, the data was bizarre enough that some colleagues around the Project Zero office even got intrigued.

It became clear after a while we were looking at chunks of uninitialized memory interspersed with valid data. The program that this uninitialized data was coming from just happened to have the data I wanted in memory at the time. That solved the mystery, but some of the nearby memory had strings and objects that really seemed like they could be from a reverse proxy operated by cloudflare - a major cdn service.

A while later, we figured out how to reproduce the problem. It looked like that if an html page hosted behind cloudflare had a specific combination of unbalanced tags, the proxy would intersperse pages of uninitialized memory into the output (kinda like heartbleed, but cloudflare specific and worse for reasons I'll explain later). My working theory was that this was related to their "ScrapeShield" feature which parses and obfuscates html - but because reverse proxies are shared between customers, it would affect *all* Cloudflare customers.

We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major cloudflare-hosted sites from other users. Once we understood what we were seeing and the implications, we immediately stopped and contacted cloudflare security.

This situation was unusual, PII was actively being downloaded by crawlers and users during normal usage, they just didn't understand what they were seeing. Seconds mattered here, emails to support on a friday evening were not going to cut it. I don't have any cloudflare contacts, so reached out for an urgent contact on twitter, and quickly reached the right people.

https://twitter.com/taviso/status/832744397800214528

After I explained the situation, cloudflare quickly reproduced the problem, told me they had convened an incident and had an initial mitigation in place within an hour.

"You definitely got the right people. We have killed the affected services"


This bug is subject to a 90 day disclosure deadline. After 90 days elapse
or a patch has been made broadly available, the bug report will become
visible to the public.

 
Project Member Comment 1 by taviso@google.com, Feb 19
I worked with cloudflare over the weekend to help clean up where I could. I've verified that the original reproduction steps I sent cloudflare no longer work.

We're discussing some remaining issues and Cloudflare are still working on their investigation. From our data, we believe the issue has been present for some time, I've given cloudflare all the information we have.  

This is an unusual type of vulnerability disclosure for Project Zero, and due to the third party nature of the data involved I'm reluctant to be as transparent as we normally are on our issue tracker. We've destroyed the samples we collected during analysis.

Cloudflare have assured me they will prepare a detailed postmortem for their customers once the issue is resolved. They have an excellent reputation for transparency, so that's good enough for me.

Project Member Comment 2 by taviso@google.com, Feb 19
We've been trying to help clean up cached pages inadvertently crawled at Google. This is just a bandaid, but we're doing what we can. Cloudflare customers are going to need to decide if they need to rotate secrets and notify their users based on the facts we know.

I don't know if this issue was noticed and exploited, but I'm sure other crawlers have collected data and that users have saved or cached content and don't realize what they have, etc. We've discovered (and purged) cached pages that contain private messages from well-known services, PII from major sites that use cloudflare, and even plaintext API requests from a popular password manager that were sent over https (!!).

Really impressed with Cloudflare's quick response, and how dedicated they are to cleaning up from this unfortunate issue. I told them I hoped they were able to get some sleep during the incident, and an engineer responded:

"Its going to be a long weekend for us, but this is really important to us.  We need to protect as many people as we can and we need to make sure this can never happen again on our watch.".
Project Member Comment 3 by taviso@google.com, Feb 19
Cc: taviso@google.com
Issue 1138 has been merged into this issue.
Project Member Comment 4 by taviso@google.com, Feb 19
Labels: CCProjectZeroMembers
Project Member Comment 5 by taviso@google.com, Feb 19
We keep finding more sensitive data that we need to cleanup. I didn't realize how much of the internet was sitting behind a Cloudflare CDN until this incident.

The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.

My current theory is that they had some code in their "ScrapeShield" feature that did something like this:

int Length = ObfuscateEmailAddressesInHtml(&OutputBuffer, CachedPage);

write(fd, OutputBuffer, Length);

But they weren't checking if the obfuscation parsers returned a negative value because of malformed HTML. This would explain the data I'm seeing.

Project Member Comment 6 by taviso@google.com, Feb 20
I had a call with Cloudflare, they reassured me they're planning on complete transparency and believe they can have a customer notification ready this week.

Project Member Comment 7 by taviso@google.com, Feb 20
I'm satisfied cloudflare are committed to doing the right thing, they've explained their current plan for disclosure and their rationale.

We're still working on identifying data that needs to be purged from caches. Here's a heavily redacted random example of the data we're finding and purging, many major internet companies are using cloudflare.

cloudflareexample.png
85.5 KB View Download
Project Member Comment 8 by taviso@google.com, Feb 21
Update from Cloudflare, they're confident they can get their notification ready by EOD Tuesday (Today) or early Wednesday.

We're cleaning up some last remaining artifacts.
Comment 9 Deleted
Project Member Comment 10 by taviso@google.com, Feb 21
Processing more removals this morning, and some further discussion on disclosure.

Project Member Comment 11 by taviso@google.com, Feb 22
Here's another random redacted example from today.
redacted.png
118 KB View Download
Project Member Comment 12 by taviso@google.com, Feb 22
Cloudflare told me that they couldn't make Tuesday due to more data they found that needs to be purged.

They then told me Wednesday, but in a later reply started saying Thursday.

I asked for a draft of their announcement, but they seemed evasive about it and clearly didn't want to do that. I'm really hoping they're not planning to downplay this. If the date keeps extending, they'll reach our "7-day" policy for actively exploited attacks.

https://security.googleblog.com/2013/05/disclosure-timeline-for-vulnerabilities.html

If an acceptable notification is not released on Thursday, we'll decide how we want to proceed.
Comment 13 Deleted
Project Member Comment 14 by taviso@google.com, Feb 23
Labels: -Reported-2016-Feb-17 Reported-2017-Feb-17
Project Member Comment 15 by taviso@google.com, Feb 23
Cloudflare sent a list of sites with more PII, but the list contained many well-known domains that were not using cloudflare (e.g. blogspot).

I let them know, and then tried to manually clean their list of obviously invalid requests to avoid any further delays. I filtered the remaining requests through a quick script I wrote  to verify they were reasonable, which removed another 20 incorrect entries.

I'm sure this was a good faith mistake from a team rushing to protect users before publication today.
Project Member Comment 16 by taviso@google.com, Feb 23
I had a call with cloudflare, and explained that I was baffled why they were not sharing their notification with me.

They gave several excuses that didn't make sense, then asked to speak to me on the phone to explain. They assured me it was on the way and they just needed my PGP key. I provided it to them, then heard no further response.

I sent the following mail:

Thanks for the call John, I still haven't received anything.

Can we commit to getting this out the door today? We're discussing whether we need to apply our "7-day" policy for actively exploited attacks, and while we were hoping the notification was going to be smooth sailing we don't know how to interpret the lack of cooperation here. I've been working with Marc this morning on more cleaning due to some incorrect data you sent us, I'm confident this will be done within hours.

Let's commit to getting this out by 2PM PST. 

Tavis.

Here is another sample from the data we cleaned today, a private message from a popular dating site. Sigh.
red.png
88.5 KB View Download
Project Member Comment 17 by taviso@google.com, Feb 23
Cloudflare pointed out their bug bounty program, but I noticed it has a top-tier reward of a t-shirt.

https://hackerone.com/cloudflare

Needless to say, this did not convey to me that they take the program seriously.
Project Member Comment 18 by taviso@google.com, Feb 23
Cloudflare explained that they pushed a change to production that logged malformed pages that were requested, and then sent me the list of URLs to double check.

Many of the logged urls contained query strings from https requests that I don't think they intended to share. Nevertheless, we will keep the data safe and destroy it once we've checked if they match any cached pages.

I can already see that the (huge) list does not contain many pages that still have cached content from before the bug was fixed.

Project Member Comment 19 by taviso@google.com, Feb 23
Cloudflare did finally send me a draft. It contains an excellent postmortem, but severely downplays the risk to customers.

They've left it too late to negotiate on the content of the notification. 

Let's hope that their notification in combination with the details from this issue will be adequate explanation of what happened. I think we're waiting for cached links to start expiring, and then we're publishing whether they're ready or not.



Project Member Comment 20 by taviso@google.com, Feb 23
Labels: -Restrict-View-Commit
Status: Fixed
OK, this is as done as it's going to be.
>(It took every ounce of strength not to call this issue "cloudbleed")

too late...
cloudbleed.png
48.3 KB View Download
Our (Cloudflare's) blog post with the complete details are here: https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/
Comment 23 Deleted
Comment 24 Deleted
Comment 25 Deleted
Comment 26 Deleted
Comment 27 Deleted
Comment 28 Deleted
Comment 29 Deleted
Comment 30 Deleted
Sign in to add a comment