Reposting with corrected setspn commands.

Setting kerberos SPNs for the CNAMES against that computer account seems to help the situation some. 

So running the following:
setspn -S HTTP/cname.domain.com servername
setspn -S HTTP/cname servername
setspn -S HOST/cname.domain.com servername
setspn -S HOST/cname servername

However, this only seems to fix the fqdn version of the cname, using the shortname (http://cname/) in the browser still fails to login.

Sorry but I was impatient, it looks like setting the SPNs fixes it for the shortnames as well. I just needed to wait longer for things to percolate.

@comment #3: is this issue resolved then?

Let's wait for the reporter to respond.

Setting the SPN to the CNAME works to allow access to a site which uses Kerberos for authentication with the current Chrome Beta and Canary.

The only issue with doing this is that this is different behaviour from version 68 and earlier and differs from the descriptions of how Kerberos authentication is claimed to work at https://www.chromium.org/developers/design-documents/http-authentication (this is the only description of how it works I can find).

Also if this is a change do you need the policy at https://dev.chromium.org/administrators/policy-list-3#DisableAuthNegotiateCnameLookup 

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

As the issue is not reproducible at TE end, it is related to "web server setup with kerberos authentication" it requires (Oracle VDI hosts prior to setting up the user directory in the Oracle VDI Manager.).Hence adding 'TE-NeedsTriageHelp' label and requesting the Internals>Network>Auth team to look into the issue and help in further triaging.

Thanks.!  

 Issue 873955  has been merged into this issue.

Reporter or anyone running into this: Could you give me a net-internals log as described in https://dev.chromium.org/for-testers/providing-network-details ?

I'd like to see what's going on during the CNAME lookup phase.

Also, make sure the DisableAuthNegotiateCnameLookup is not set, or set explicitly to False. We want to get the browser to perform a CNAME lookup.

I have attached the net internals log for this for you now.

Deleted: chrome-net-export-log.json 85.1 KB

chrome-net-export-log.json 85.1 KB View Download

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Affects us as well, The option "DisableAuthNegotiateCnameLookup" seems to have no impact with the new version "Version 69.0.3497.81 (Offizieller Build) (64-Bit)". Adding the cname as a SPN is not a option in our case.

If you need more information just let me know.

Deleted: chrome-net-export-log.json 112 KB

chrome-net-export-log.json 112 KB View Download

This is affecting us as well. I was able to reproduce this in the older version of chrome by setting DisableAuthNegotiateCnameLookup to true. This is really affecting SSO for all our internal application since it is falling back to basic auth. Can we confirm this as bug ? or can we get any workaround ? 

We are also facing the same issue.

Chrome 69 does not provide anymore Kerberos authentication to our web applications with url myapp.mydomain.com.
However, the behaviour is as expected if we use URL myapp.internal.local
We have both *.internal.local and *.mydomain.com in Intranet Zone.
The domain mydomain.com is resolvable from the Internet.

We have also digged a little :

We noticed with TCPDUMP that when we use myapp.mydomain.com URL, Chrome 69 provide CNAME instead of ARecord for requesting the Kerberos Ticket. Search in our Active Directory failed because the cooresponding SPN does not exist.
If we use myapp.internal.local URL, ARecord is provided as expected, and authentication succeed.

Please, can you confirm the issue and suggest a workaround ?

Kind regards

+mef: Is there a network service experiment on M69?

Can anyone provide a Chrome 68 msi download link?  This issue has caused a huge outage for us and we're desperately trying to download our fleet.

Ok, that's not good. I didn't see anything that looks different in the net-internals logs. In both logs (i.e. #13, and #15) I see that a CNAME lookup was performed. Is that not the intention?

Let's say you have the following RRs:

  www.example.com.	1000	IN	CNAME	server1.example.com.
  server1.example.com. 1000	IN	A	10.10.10.10

Then:

A. With DisableAuthNegotiateCnameLookup unset or set to false, the SPN for www.example.com is HTTP/server1.example.com@YOURREALM.COM.

B. Woth DisableAuthNegotiateCnameLookup set to true, the SPN for www.example.com is HTTP/www.example.com@YOURREALM.COM.

From #0, #2, and #15, it sounds like B doesn't work.

From #16 is sounds like A doesn't work.

Can reporters confirm that chrome:policy correctly shows the policy you need?

I'll see what changed on the DNS side for M69.

(Also, if B doesn't work, then it's not DNS).

Also ruling out Network Service which is reportedly only experimenting on Canary (M71) at the moment. M69 shouldn't be affected.

+Enterprise, +robertshield@ FYI. This is on stable.

M69 TPM FYI

Our policy DisableAuthNegotiateCnameLookup is unset in our case, but CNAME is provided by M69.
Following your example, HTTP/www.example.com@YOURREALM.COM is provided instead of HTTP/server1.example.com@YOURREALM.COM

We have DisableAuthCnameLookup unset.

With you example domain names when trying to go to www.example.com then the SPN that is required is for HTTP/www.example.com@YOURREALM.COM

When Chrome 69 was in Beta I was able to test it side by side with Chrome 68 while changing the DisableAuthCnameLookup values.  For Chrome 69 I was not finding a difference in how Kerberos authentication was working when modifying the DisableAuthCnameLookup value.  With Chrome 68 the DisableAuthCnameLookup value was having the effect I would expect it to have.  Chrome 69 was performing as if DisableAuthCnameLookup was set to true regardless of the status of the flag.

#26: For completeness, could you send me a net-internals log as well? Instructions are at https://dev.chromium.org/for-testers/providing-network-details . You can email the log directly to me if you aren't comfortable attaching it to a public issue. I just want to distinguish between the DNS lookup not working vs. CNAME lookup being skipped entirely.

So you sound like you are in the "A doesn't work" camp (as laid out in #20).

#20: The default behavior has been and should be "HTTP/server1.example.com@YOURREALM.COM", so at least in our case A does not work. (But I am sure that this concerns all the described problems here.)

+more enterprise folk FYI

69 broke our SSO as well. Disabling/Enabling lookup shows no effect in 69 (but does work as expected in 68). 

every website with cname does not work anymore. when the spn is an a record it works. we use one config per reverse proxy which holds the a record of the proxy host and the apps are all cname aliases which point to the proxy. 

Setting custom spn is not an option, also having a config per app is not an option. Please advise. 

Adding RBS

Same problem here, SSO is broken when using CNAME records

If possible, can an impacted user try running a bisect to determine when exactly this issue started occurring? There's a simplified bisect tool at:

https://github.com/jay0lee/chrome-bisect#chrome-bisect

if you are able to run, please grab the generated log file and add here.

Here is log file for Bisect when I have run it

Deleted: chrome-bisect-2018-09-05-19-45-29-952000x6c02_.log 5.4 KB

chrome-bisect-2018-09-05-19-45-29-952000x6c02_.log 5.4 KB View Download

Many thanks alun@

Based on the changelist from bisect of:

https://chromium.googlesource.com/chromium/src/+log/bddfeda54e0f2d1d3cb4a3b1bed1ec3d28e1c1b2..b67c0a8eaa84c767041a00d5f0033e95e179e2e0

this change seems suspect:
https://chromium.googlesource.com/chromium/src/+/b67c0a8eaa84c767041a00d5f0033e95e179e2e0

+ mmenke for his thoughts on if this is related.

 Issue 880846  has been merged into this issue.

Our enterprise (20'000 clients) is also affected.

Not sure how that CL could have broken CNAME lookup - I hooked up CNAME configuration to the network service APIs in https://chromium-review.googlesource.com/c/chromium/src/+/1089661, but the bisect shows multiple phases where that CL was marked as good, so it's pretty clearly not that one that caused the issue.  I can't see how an incorrectly configured HostResolver would cause this.

Same issue here, but I also noticed that Chrome 69 now sends NTLMSSP tokens for Negotiate auth when up until 68 it used to send Kerberos. This happens regardless of the policy settings (whitelist, CNAME, auth mechanisms).

We have applications on Apache with mod_auth_gssapi, none of which work now.

Chrome does not attempt to acquire a service ticket, instead it directly uses NTLM.

#43: With an incorrect SPN, the ticket acquisition will fail. Windows defaults to NTLM in those cases.

OK, if I have to specifically say it, then I'll say it.

Nothing but the Chrome version has changed since it last worked, with Chrome 68.

the cname lookup is clearly broken with 69, how can’t you reproduce this? 🙄 

Hi all, we are investigating this as P0 and we believe we've identified the regression above. Please hold off on any further commentary that's not constructive so we can focus efforts here.

mmenke@ is reverting this change an option? This is P0 and blocking stable at this point.

Reverting on trunk isn't really an option.  We may be able to revert on M69, but it may well not be a clean revert, given the amount of work done in that area.

Note that the Kerberos SSP maintains a negative cache of failed TGS-REQ SPNs.  If it got a principal does not exist error recently, it will immediately fail without putting a new request on the wire.  That may be why you’re observing it go straight to NTLM when looking at a packet capture.  The cache lives for 15 or 20 minutes, klist purge flushes it. 

Updating affected platforms.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/cc704901b1876e8cab502bac46b955d78ea0198a

commit cc704901b1876e8cab502bac46b955d78ea0198a
Author: Asanka Herath <asanka@chromium.org>
Date: Thu Sep 06 03:50:14 2018

[net] Force use of system host resolver for CNAME lookups.

Async host resolver currently doesn't resolve CNAMEs. Thus use of the
async resolver is not currently compatible with the needs of
HttpAuthHandlerNegotiate where correct CNAME lookup is required.

Due to an unfortunate oversight, b67c0a8eaa84c767041a00d5f0033e95e179e2
enabled the async DNS code path by default.

This CL addresses both. The updates to the HttpAuthHandlerNegotia tests
verify that the SYSTEM source is forced for CNAME lookups.

Bug:  872665 
Change-Id: I520f88f01084ef8ac12c377a50bba6c7f09ec3c1
Reviewed-on: https://chromium-review.googlesource.com/1208421
Reviewed-by: Matt Menke <mmenke@chromium.org>
Commit-Queue: Asanka Herath <asanka@chromium.org>
Cr-Commit-Position: refs/heads/master@{#589097}
[modify] https://crrev.com/cc704901b1876e8cab502bac46b955d78ea0198a/chrome/browser/net/system_network_context_manager.cc
[modify] https://crrev.com/cc704901b1876e8cab502bac46b955d78ea0198a/net/dns/host_resolver.cc
[modify] https://crrev.com/cc704901b1876e8cab502bac46b955d78ea0198a/net/http/http_auth_handler_negotiate_unittest.cc

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ec59d3235f25748c8a83d6b54c5a6da479fa7fb0

commit ec59d3235f25748c8a83d6b54c5a6da479fa7fb0
Author: Asanka Herath <asanka@chromium.org>
Date: Thu Sep 06 03:59:51 2018

[net] Force use of system host resolver for CNAME lookups.

Async host resolver currently doesn't resolve CNAMEs. Thus use of the
async resolver is not currently compatible with the needs of
HttpAuthHandlerNegotiate where correct CNAME lookup is required.

Due to an unfortunate oversight, b67c0a8eaa84c767041a00d5f0033e95e179e2
enabled the async DNS code path by default.

This CL addresses both. The updates to the HttpAuthHandlerNegotia tests
verify that the SYSTEM source is forced for CNAME lookups.

Bug:  872665 
Change-Id: I520f88f01084ef8ac12c377a50bba6c7f09ec3c1
Reviewed-on: https://chromium-review.googlesource.com/1208421
Reviewed-by: Matt Menke <mmenke@chromium.org>
Commit-Queue: Asanka Herath <asanka@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#589097}(cherry picked from commit cc704901b1876e8cab502bac46b955d78ea0198a)
Reviewed-on: https://chromium-review.googlesource.com/1208901
Reviewed-by: Krishna Govind <govind@chromium.org>
Cr-Commit-Position: refs/branch-heads/3544@{#3}
Cr-Branched-From: 0da7dce9a83cbad841f925002a4ddfee8f2afcf7-refs/heads/master@{#589076}
[modify] https://crrev.com/ec59d3235f25748c8a83d6b54c5a6da479fa7fb0/chrome/browser/net/system_network_context_manager.cc
[modify] https://crrev.com/ec59d3235f25748c8a83d6b54c5a6da479fa7fb0/net/dns/host_resolver.cc
[modify] https://crrev.com/ec59d3235f25748c8a83d6b54c5a6da479fa7fb0/net/http/http_auth_handler_negotiate_unittest.cc

I'm triggering new canary from branch 3544 with merge listed at #55.

Canary version 71.0.3544.2 currently building includes this merge. Thank you.

Pls update bug with canary result tomorrow.

Yay for auto updates. Good thing I have those disabled in my organization. Yeah, we've been hit too. Good thing I test the enterprise build first before distributing it so I caught it early. I was meaning to register a bug, but it turns out it already exists :)

Are we getting a patched 69 build or do we need to wait for 70/71 for this? I need to tell (security) management something to explain why we can't upgrade yet and when we can expect a solution.

I have just tried accessing a location using Kerberos authentication with a CNAME using Canary version 71.0.3544.2 and it appears to work properly now.

71.0.3544.2 Works as expected! Thank you for providing it! 

CNAME gets resolved again and the same usecase which did not work with 69 or 71.0.3544.0 works after installation of .2. 

works fine for me too!

The NextAction date has arrived: 2018-09-06

 Issue 881220  has been merged into this issue.

Thanks for the confirmations everyone. We'll queue this for merge into M69 and then M70.

This bug requires manual review: Request affecting a post-stable build
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), kariahda@(iOS), cindyb@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

We are having the same problem. Thanks to everyone who reported this. Looks like a fix has been identified.

When can we expect the fix to be pushed out in an update? I need to be able to provide a timeline to my company.

Thanks!

Approving merge to M69 branch 3497 based on comment #65 and offline group chat. Pls merge. 

Also pls request a merge to M70. Thank you.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/547588635540eff0c0998e5bace388c38190e038

commit 547588635540eff0c0998e5bace388c38190e038
Author: Asanka Herath <asanka@chromium.org>
Date: Thu Sep 06 18:01:38 2018

[net] Force use of system host resolver for CNAME lookups.

Async host resolver currently doesn't resolve CNAMEs. Thus use of the
async resolver is not currently compatible with the needs of
HttpAuthHandlerNegotiate where correct CNAME lookup is required.

Due to an unfortunate oversight, b67c0a8eaa84c767041a00d5f0033e95e179e2
enabled the async DNS code path by default.

This CL addresses both. The updates to the HttpAuthHandlerNegotia tests
verify that the SYSTEM source is forced for CNAME lookups.

TBR=asanka@chromium.org

(cherry picked from commit cc704901b1876e8cab502bac46b955d78ea0198a)

Bug:  872665 
Change-Id: I520f88f01084ef8ac12c377a50bba6c7f09ec3c1
Reviewed-on: https://chromium-review.googlesource.com/1208421
Reviewed-by: Matt Menke <mmenke@chromium.org>
Commit-Queue: Asanka Herath <asanka@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#589097}
Reviewed-on: https://chromium-review.googlesource.com/1211084
Reviewed-by: Asanka Herath <asanka@chromium.org>
Cr-Commit-Position: refs/branch-heads/3497@{#896}
Cr-Branched-From: 271eaf50594eb818c9295dc78d364aea18c82ea8-refs/heads/master@{#576753}
[modify] https://crrev.com/547588635540eff0c0998e5bace388c38190e038/chrome/browser/net/system_network_context_manager.cc
[modify] https://crrev.com/547588635540eff0c0998e5bace388c38190e038/net/dns/host_resolver_impl.cc

And M70.

+abdulsyed: ping for M70 merge?

Hi asanka@chromium.org do you have any ERD to communicate for the merge to M69 branch 3497?

Thanks,

When will the new release come out, correcting this bug?

asanka@chromium.org this was fixed 24 hours ago in M71.  The merge to M69 is approved so when will the update be available?  Please advise ASAP.

please handle this serious error with increased priority. In our environment, the update of client installations is delayed by 2 weeks. The fix must be included in M69 before 09-18.

The fix has been verified on 69.0.3497.87.

We need an urgend hotfix release. 
We coudn't work for our customer because in our dev/qa/prod system we have SSO as authentification.

asanka@chromium.org after checking for updates in Help->About Chrome x64 (Windows) the version is still 69.0.3497.81.  Please advise.

Please also note that https://enterprise.google.com/chrome/chrome-browser/ is still showing v 69.0.3497.81.

Hi all, we've verified the fix on internal build 69.0.3497.87 but still need to further qualify the build. We anticipate pushing 69.0.3497.87 or later sometime next week.

Until then, if you are in need of the 68 MSIs to downgrade, please file a support ticket with Google and request the file. Please reference this  crbug.com/872665  in your request.

Your change meets the bar and is auto-approved for M70. Please go ahead and merge the CL to branch 3538 manually. Please contact milestone owner if you have questions.
Owners: benmason@(Android), kariahda@(iOS), geohsu@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f058f99fa7ae6d0a48509af652c5a8f9f9b92883

commit f058f99fa7ae6d0a48509af652c5a8f9f9b92883
Author: Asanka Herath <asanka@chromium.org>
Date: Fri Sep 07 19:32:48 2018

[net] Force use of system host resolver for CNAME lookups.

Async host resolver currently doesn't resolve CNAMEs. Thus use of the
async resolver is not currently compatible with the needs of
HttpAuthHandlerNegotiate where correct CNAME lookup is required.

Due to an unfortunate oversight, b67c0a8eaa84c767041a00d5f0033e95e179e2
enabled the async DNS code path by default.

This CL addresses both. The updates to the HttpAuthHandlerNegotia tests
verify that the SYSTEM source is forced for CNAME lookups.

Bug:  872665 
Change-Id: I520f88f01084ef8ac12c377a50bba6c7f09ec3c1
Reviewed-on: https://chromium-review.googlesource.com/1208421
Reviewed-by: Matt Menke <mmenke@chromium.org>
Commit-Queue: Asanka Herath <asanka@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#589097}(cherry picked from commit cc704901b1876e8cab502bac46b955d78ea0198a)
Reviewed-on: https://chromium-review.googlesource.com/1214122
Reviewed-by: Asanka Herath <asanka@chromium.org>
Cr-Commit-Position: refs/branch-heads/3538@{#151}
Cr-Branched-From: 79f7c91a2b2a2932cd447fa6f865cb6662fa8fa6-refs/heads/master@{#587811}
[modify] https://crrev.com/f058f99fa7ae6d0a48509af652c5a8f9f9b92883/chrome/browser/net/system_network_context_manager.cc
[modify] https://crrev.com/f058f99fa7ae6d0a48509af652c5a8f9f9b92883/net/dns/host_resolver.cc
[modify] https://crrev.com/f058f99fa7ae6d0a48509af652c5a8f9f9b92883/net/http/http_auth_handler_negotiate_unittest.cc

 Issue 881265  has been merged into this issue.

Any News for us?

 Issue 882484  has been merged into this issue.

Verified the fix on 69.0.3497.92

69.0.3497.92 is now pushing to stable channel with this fix and MSI should be downloadable soon also. 

Worked for me. 

Worked for me as well. Thank you.

Worked for me. Thanks.

 Issue 882222  has been merged into this issue.

This has worked for me as well.  Thank you.

Have you tried this also with https? In my case with http it worked, but with https not.

It works with HTTP and HTTPS for us. Thanks

+ imranal@ (Chrome Test Lead)as FYI

Thanks govind, is there a plan to get kerberos tests (integrated or manual) added? 

imranal@ (Chrome Test Lead) probably a right person to answer #97. Thank you.

Update: I am working with bustamante@ to add manual kerberos tests to our regression test passes.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/541a90cc3bf250ce66fdcc31feae3e78370765b6

commit 541a90cc3bf250ce66fdcc31feae3e78370765b6
Author: Eric Orth <ericorth@chromium.org>
Date: Thu Jan 17 20:11:46 2019

Coerce more canonicalname resolves to use ProcTask

DnsClient doesn't handle cannonname very well (see
 https://crbug.com/872665 ), so some logic was added a while back to force
ProcTask for such cases by setting HostResolverSource::SYSTEM when the
HOST_RESOLVER_CANONNAME flag is set. But this didn't affect the newer
HostResolver::CreateRequest() API, as that has callers directly set the
source and doesn't directly use ProcTask flags.

Bug:  872665 
Change-Id: I709959911ec299118369f1f995ea5802df7a92f9
Reviewed-on: https://chromium-review.googlesource.com/c/1418350
Reviewed-by: Asanka Herath <asanka@chromium.org>
Commit-Queue: Eric Orth <ericorth@chromium.org>
Cr-Commit-Position: refs/heads/master@{#623815}
[modify] https://crrev.com/541a90cc3bf250ce66fdcc31feae3e78370765b6/net/dns/host_resolver_impl.cc
[modify] https://crrev.com/541a90cc3bf250ce66fdcc31feae3e78370765b6/net/dns/host_resolver_impl_unittest.cc