I'd like to weigh in on this and provide some views from my perspective.

"we have been experiencing difficulty explaining to developers at scale, why they should fix the bugs even when the browser says the attack was stopped" - because not all browsers have auditors and the underlying bug needs to be fixed. This is a last line of defense to save you when you miss something, not a solution!

"we surveyed all internal XSS bugs that triggered the XSSAuditor and were able to find bypasses to all of them" Is this a reason to remove the auditor or improve it? Perhaps there are attackers out there who aren't sufficiently skilled or motivated to bypass the auditor?

"we've surveyed security pentesters and found out some do not report vulnerabilities unless they can find a bypass of the XSSAuditor" As a former penetration tester this is grossly negligent on behalf of the tester. An XSS vulnerability has been found in the application and that should be reported, this is not a test of the auditor. Referring back to my previous point that not all browsers have auditors, not reporting this finding is an awful idea.

On top of the above points the XSS Auditor can also send reports [1] to a reporting endpoint which provides the opportunity for a website operator to know almost immediately if suspicious activity is taking place that has triggered the auditor. The report often contains the payload used by the attacker which could also aid in remediation of the vulnerability. Site operators could lose access to valuable information if the auditor is removed.

Full disclosure: I operate a reporting service [2] that is capable of collecting XSS Auditor reports.

[1] https://scotthelme.co.uk/introducing-xss-reporting-to-report-uri/
[2] https://report-uri.com/

Hi Scott

Thanks for chiming in, do you have any metrics you can share regarding the number of times the XSSAuditor triggers for the sites your service manages?

Out of our total volume the % of XSS Auditor reports is miniscule, but only because CSP is really effective at drowning everything else out in that regard.

What I feel is a really important metric is that we filter almost none of the XSS Auditor reports we receive but, on average, almost 70% of CSP reports. The signal:noise ratio on XSS Auditor reports is exceptional, I feel it'd be a shame to lose them.

Could you quantify a bit the "miniscule", and "almost none" in some way?

As in, we get approximately 10 reports per day for YouTube, and all of the reports are false positives.

Chrome's position at the moment is:

1. We don't buy the argument that developers/pentesters are unaware that they need to fix/report XSS ... internally, we've remarked that this is analogous to how "having an airbag doesn't mean that you can drive with your eyes closed." The problem lies with the developers, not the XSSAuditor.

2. We have yet to see a bypass for the simplest case of XSS, e.g. a page containing nothing more than <?php echo $_GET['q']?>. So even though there may be bypasses in more complex situations, we are stopping at least some XSS. If you can give us a reliable bypass for this case, which we can't subsequently fix, then we will have to reconsider all of this.

3. Chrome *shows* you exactly where the XSS occurs when you go to view-source on a page that was blocked, highlighted in RED.  Rather than making it hard to find the XSS because one has to manually parse a page in one's head, XSSAuditor makes it easier.

Re C5 - Google/Youtube statistics are unlikely to be representative of all of the web, since these are high-quality sites, with large competent security teams backing them up, and I would be reluctant to make generalizations about the rest of the web based upon them.

I'm happy to share my own numbers for the sites I manage as they are representative of the service as a whole but I can't share specifics on other users of the service.

CSP
Accepted - 245,833
Filtered - 650,483

XSS Auditor
Accepted - 118
Filtered - 0

@scott.helme, thanks! that's useful.

@tsepez, There is a bypass on a file <?php echo $_GET['q']?> that XSSAuditor can't fix. That's one of the problems (ex: b/117931435 b/117841343 b/115970273 b/115305545 b/114489286). But that's not what I'm arguing.

The argument being made is that it's more harmful than useful, since it triggers on XSS it can't effectively stop, which unfortunately humans/developers misunderstand. This might be a problem that could be solved with UX (see alternatives considered above).

More internal links for the b/ links above.
 - http://go/content-sniffing#security-issues-related-to-serving-non-html-documents
 - http://g/content-sniff-research is where this is discussed

For those that aren't googlers: https://www.alchemistowl.org/pocorgtfo/pocorgtfo12.pdf has some details on 12:4

Note however, that this is completely irrelevant, since content sniffing is out of scope of the XSSAuditor.

I really like the idea of improving the warning strings. @tsepez, please file a bug for that. (Do you recall if that will need UI sign-off, or we can you just land the string change?)

More broadly, this bug report sure seems like an argument in favor of keeping the XSSAuditor with some tweaks. Yes, there are obviously vulnerability patterns it will never be able to block—particularly with a dedicated attacker targeting an environment as complex as Google's—but the data is also very clear that the XSSAuditor is effective both for many (most?) simple XSS patterns and as a general purpose logging and debugging tool (and even more so with some tweaks to the strings).

Given that, I'm closing this WontFix and we'll get the warning improved.

Regarding bypasses, please file reports as new issues in this tracker as they are found, otherwise there is no hope of improvement.

https://crbug.com/898284 is the follow-on.

jschuh, could you clarify what you mean by "but the data is also very clear that the XSSAuditor is effective both for many (most?) simple XSS".

as in, what would be such data, and what would qualify as a simple XSS and what does "many" or "most" mean?

I ask because the data I've seen on our vendor's and acquisitons XSS (which have very simple XSS) shows that the XSS auditor is effective in 0% of the simple XSS.

To qualify my "simple xss", here's 5 examples of what I mean:
b/116175361
  <iframe src="$XSSHERE"> -> Bypass: javascript:alert(1)

b/115729152
  https://drive.google.com/file/d/177Qvdw6vCqbDaVq7YCr4uDMwfKHWWlbu/view has a demo of the XSS, summary is that it has multiple injections

b/115384772
  multiple injection (arbitrary number of parts)

b/115353909
  innerHTML=XSS injection

b/115300921
  double injection, with new line and trash in the middle


We also have a ton of other XSS that trigger on the <?php echo $_GET['xss']; ?>. Is there a way we can qualify what vulnerabilities XSSAuditor actually protects against? Since I'm struggling to find one, and I would be very curious to see one!

evn@ - I note with interest and some vexation that none of the bugs you reference are crbug.com issues filed against XSSAuditor.

If you know of auditor bypass issues but are not reporting them to us then yes there is indeed a big problem here, but it's not with XSSAuditor.

Andrew, none of those bypasses are in scope of XSSAuditor. They are by design.

awhalley@ for your reference:
https://chromium.googlesource.com/chromium/src/+/master/docs/security/faq.md#are-xss-filter-bypasses-considered-security-bugs

"There will also be a number of infrequently occurring reflected XSS corner case in an HTML context that it will never be able to cover. Among these are:
 - Multiple unsanitized variables injected into the page.
 - Unexpected server side transformation or decoding of the payload."

3 out of the 5 examples I provided are multiple injections. The other two are also out of scope, which I'm happy to pull out crbug.com references for you if you prefer.

Right, so the criteria for reporting to us is if you can break a page containing no content other than <?php echo $GET_['q']?>.

@tsepez does https://bugs.chromium.org/p/chromium/issues/detail?id=898081#c11 count?

I think the data that is missing is what number of actual XSS vulnerabilities that occur on the web does auditor stop (i.e. they are of interest to the maintainers because of being in scope and the exploit doesn't use a new bypass that keep being found). Based on my experience, t's a really small subset. For example, most pages that have a one reflected injection point, have multiple ones. Or the attacker controls the whole response. 

I have not seen any data suggesting that the auditor, with its limitations and historical scoping is an effective xss-protection for the current web. I am genuinely curious to look at such dataset.

> If you know of auditor bypass issues but are not reporting them to us then yes there is indeed a big problem here

Bypasses in XSS auditor are not security issues, yet if I'm reading the tone right, you want them to be treated like ones (and more so - sitting on one is ethically wrong). Why?

Because fixing functional issues are important, too, and we want to make it as effective as possible.

kkotowicz@ & evn@ - apologies for my tone. I misunderstood the line of reasoning and assumed you were providing examples of thing you thought were in scope.

Nope, we don't track them as security bugs, per the policy quoted in comment 19. Though that also contains "We do appreciate reports of XSSAuditor bypasses, and endeavor to close them". It's also a good way to help bring the issue you raise to our attention, and is genuinely appreciated :-)

Yea, I understand :). If I found proper bypasses we would report them, but we are just bypassing the auditor with "working as intended" design decisions.

That said, I know of pentesters that don't report them because they would rather have an XSS PoC to show to their customers.

Yes that's a thing I see repeatedly -https://twitter.com/brcrwilliams/status/1054704145184153600?s=09

The thing is - if auditor is tailored to only stop bugs that don't happen a lot in practice, and at the same time it has other properties (like the sop bypasses, or unintended consequence of discouraging reporting and fixing the underlying XSS), it might just be net negative. All the data and anecdotes I have access to suggest this might just be the case :/

So would you be willing to say that XSSAuditor is completely effective for N% of sites, for some nonzero value of N, but it may be a small value?

I've been looking for one for a while, so far it's 0. But happy to be proven wrong!

@evn, I don't find that response accurate or helpful. Yes, Google has a very complex environment, which unfortunately means that the XSSAuditor generally isn't useful as a mitigation. However, that is an entirely different assertion than the claim that it's not generally useful in simpler environments, or as an analysis and debugging tool. And the comments from @scott.helme are just one example of the feedback we get regarding the value of XSSAuditor for real-world sites. 

More generally, I think it's important to appreciate that the perspective from working inside Google services isn't going to match the Web at large. So, while we absolutely appreciate constructive, expert feedback from all corners, we need to base our decisions on the Web as a whole.

Now, I've already closed the bug and I don't think the continued back-and-forth here is helpful, so I would appreciate if we not continue this further.