New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 530482 link

Starred by 31 users

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: All
Pri: 2
Type: Bug

Blocked on:
issue 344685



Sign in to add a comment

Can't reach private IPv6 hosts on networks without global IPv6 connectivity

Reported by bkans...@firstgaming.com, Sep 11 2015

Issue description

UserAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36

Example URL:
http://h.searx.info/

Steps to reproduce the problem:
1. Chromium 45, 46
2. --enable-ipv6 switch
3. Private IPv6 network without global IPv6 connectivity
4. URL with domain pointing to IPv6 address in private network

What is the expected behavior?
URL is opening or something like ERR_CONNECTION_REFUSED

What went wrong?
ERR_NAME_NOT_RESOLVED

Did this work before? N/A 

Chrome version: 45.0.2454.85  Channel: stable
OS Version: 4.1.0-1-amd64
Flash Version: Shockwave Flash 18.0 r0

Worked on Chromium 44.0.2403.89. On Iceweasel and Firefox it works.
URl with literal ipv6 address also works.
 
44.0.2403.89.net-internals-log.json
274 KB Download
45.0.2454.85.net-internals-log.json
55.2 KB Download
Cc: mmenke@chromium.org
Owner: rtenneti@chromium.org
Status: Assigned
Will investigate asap. Thanks for the report.

Comment 3 by mmenke@chromium.org, Sep 11 2015

We removed the command line flag.  See  issue 344685 .
Blockedon: chromium:344685
Status: WontFix
Thanks mmenke.
So there is no way to use DNS for private IPv6 addresses in Chromium anymore?

Comment 6 by mmenke@chromium.org, Sep 11 2015

Cc: ttuttle@chromium.org pauljensen@chromium.org eroman@chromium.org
Labels: -Cr-Internals-Network Cr-Internals-Network-DNS
No, there's not.  I think it's worth asking if we care about this case, and if so, how we should support it.  Relying on command line flags for it seems rather bad, but we obviously don't want to slow down DNS resolutions for everyone on IPv4-only networks just to support this one case.
Is this an issue with Chrome's internal resolver?  Does disabling Chrome's internal resolver address the issue?  I'm never quite sure how AI_ADDRCONFIG reacts (Does it ignore loopback addresses? I think yes.  Does it ignore link-local addresses? I think no.).

Comment 8 by mmenke@chromium.org, Sep 11 2015

Cc: rtenneti@chromium.org
Owner: ----
Status: Available
Hrm...I was thinking the issue was that our IPv6 probe determines there's no IPv6 support, since it just checks support with a remote address.  That would affect both the internal and system resolver.
True, our IPv6 probe will fail without global IPv6 connectivity and then mask out any IPv6 results too.

Comment 10 by Deleted ...@, Oct 13 2015

Other solution would be to allow this address to be configurable.

So in case of local network you can specify IPv6 availability DNS server.
Labels: -OS-Linux OS-All Cr-Internals-Network-Connectivity
There's a pretty high bar for adding settings UI to Chrome, unfortunately, and I don't think this meets the bar.

We could try IPv6 if IPv4 fails, even if we don't think we have IPv6 connectivity.  That could significantly delay time until error pages are seen for people for whom IPv6 is badly broken, but I think that's the only downside (Other than extra code complexity).

Comment 12 by pmarks@google.com, Oct 24 2015

If a DNS A request responds with "no data" (which is distinct from NXDOMAIN), the resolver could try again with AAAA, even if global IPv6 connectivity is unavailable.

This should have zero impact on IPv4-only users, until they try to contact a server without an IPv4 address.

(Alternatively, restarting a connection with IPv6 enabled, after any kind of DNS+TCP failure, would catch more edge cases but incur longer timeouts.)
There's no way to tell if the response is "no data" while using the system resolver, to the extent of my knowledge.  The async resolver is basically disabled everywhere buy ChromeOS, since it doesn't quite match behavior of the system resolver.
Cc: -ttuttle@chromium.org juliatut...@chromium.org
Summary: Can't reach private IPv6 hosts on networks without global IPv6 connectivity (was: --enable-ipv6 switch is broken on Chromium 45)
 Issue 584028  has been merged into this issue.
Owner: pauljensen@chromium.org
Status: Assigned (was: Available)
Can this: https://cs.chromium.org/chromium/src/net/dns/host_resolver_impl.cc?l=2367 check just be removed and let Happy Eyeballs deal with misbehaving IPv6 connectivity?
I believe the reason for this is to save us the IPv6 resolve, which may (?) take a long time on non-IPv6 networks. 
I'm not sure why a AAAA query over IPv4 transport would take a long time?
Because IPv6-unaware dns servers can leave much to be desired, particularly those in home routers or captive portals.
Err...AAAA-unaware, rather.
Surely networks that are using recursors with that broken behaviour, are not handing out IPv6 GUA or ULA addresses, so the point is moot?  
(If the end host doesn't have a non-link-local IPv6 address, it shouldn't send a AAAA query)
@richard: The check you wanted to remove in comment #18 is the one that prevents Chrome from sending AAAA queries when Chrome detects no IPv6 connectivity, which if I'm not mistaken is what you proposed in comment #23.
Could we fix this bug by detecting the special case where the host has only site-local (private) (fec0::/48) IPv6 addresses and disabling the IPv6 connectivity probe?  Seems like DnsConfigService could set DnsConfig::use_local_ipv6 in this case to accomplish disabling the probe.
I'm not sure.

I also wonder if determining if we have only site-local addresses is a blocking operation or not - "connecting" UDP sockets presumably doesn't tend to block (much?), so we can do it pretty cheaply.  Getting local IP addresses is generally done off thread, I believe, implying it can be a blocking operation.
@paul Sorry you're right, the reference was too high-level.  Removing the dummy socket creation here: https://cs.chromium.org/chromium/src/net/dns/host_resolver_impl.cc?rcl=1484267005&l=231 is probably better suited as that's what breaks when there's no default route. Leave the link-local and teredo checking below it intact. 

However then the function name is a bit misleading and should change.
@mmenke: My proposal was "DnsConfigService could set DnsConfig::use_local_ipv6 in this case", so it would be calculated very occasionally when DnsConfigService detected an IP address change (IIRC DnsConfigService already recalculates on OnIPAdddressChanged) and recomputes the DnsConfig, not at every resolution time like our IPv6 connectivity probe.  That being said, getting the local IP addresses is a also essentially non-blocking (just a few syscalls like our IPv6 connectivity probe (side note: on Android these syscalls are actually IPCs to netd...)); I believe libc's getaddrinfo() implementation calls getifaddrs() to implement AI_ADDRCONFIG.

@richard: When you connect a UDP socket on linux it checks that there is a route from the local machine to the peer; I think the kernel runs a dummy packet through the routing logic.  So the result of the Connect() call is the main component of our IPv6 connectivity check; it answers the question "do we have a global IPv6 route?".  I don't think we can remove it without breaking that function's functionality.
@paul:  That's exactly what we want to remove.  At least for this "Should I do a AAAA query or not" check. 

If after it receives a valid AAAA response, then Chrome's Happy Eyeballs implementation should make that "do I have global connectivity" decision by sending SYNs over both IPv6 and v4 simultaneously, not a presumptuous dummy UDP socket check to an arbitrary hardcoded IPv6 address.  

If this function is used elsewhere, perhaps the functionality can be split out into two?  One for "Do I have a non-link-local IPv6 address", and another for "Do I have an IPv6 default route" (if that's indeed needed)

Comment 30 by pmarks@google.com, Jan 13 2017

> "do I have global connectivity" decision by sending SYNs over both IPv6 and v4 simultaneously, not a presumptuous dummy UDP socket

You're describing Happy Eyeballs, which should never be *required* unless the network is broken (e.g. it advertises a route to ::/0 but blackholes the packets).  The primary "do I have a route?" decision occurs during the RFC6724 sort, which works by UDP connect()ing to each address listed in DNS.

I think it sounds reasonable to enable AAAA lookups whenever a non-{::1,fe80::,Teredo} address is listed in getifaddrs(), instead of looking for a default route.
> You're describing Happy Eyeballs, which should never be *required* unless the network is broken (e.g. it advertises a route to ::/0 but blackholes the packets).

Or if IPv6 is drastically more latent.

> The primary "do I have a route?" decision occurs during the RFC6724 sort, 

Yup

> which works by UDP connect()ing to each address listed in DNS.

and *before* it even attempts to do a DNS query, when this: https://goo.gl/WI52ZA connect()s to the hardcoded Google Public DNS recursor: https://goo.gl/XRte7E   Which there-in lies the problem.

> I think it sounds reasonable to enable AAAA lookups whenever a non-{::1,fe80::,Teredo} address is listed in getifaddrs(), instead of looking for a default route.

Excellent, that sounds like a valid solution.  Will still require adjusting the above though.


If DNS returns an AAAA record, and doesn't return an A record, the application should attempt a connection to the AAAA result, and inform the user that it couldn't connect if that fails, rather than informing the user that DNS lookup failed, which is false. The current behavior is extremely confusing when debugging, because it points a finger at DNS when DNS is fine. Don't lie to the user, and don't give up without trying.

I'm on a v4 connection right now, with a v6-enabled VPN tunnel, and I can't get to sites that I have a perfectly valid AAAA record for and route to. Trusting the network stack would fix this. Happy Eyeballs would fix this. There's an obviously correct behavior here that doesn't require testing anything.
Note that DNS doesn't return a AAAA records in this case, since we only do A lookups when we think AAAA is unavailable.  AAAA lookups on some intolerant networks just timeout, so if we delay A lookups on those networks until AAAA is complete, that can lead to performance issues.
In that case, if A lookup fails, you should try an AAAA lookup, in case there's a partial v6 configuration or your test endpoint is firewalled. That would allow correct behavior on correct partial v6 setups like VPNs with no impact on performance for the broken configurations you're attempting to accommodate.

Comment 35 by pmarks@google.com, Mar 22 2017

csnook: That's similar to what I suggested in Comment 12, but with "no data" replaced by "any failure".

That means IPv4-only users navigating to nonexistent domain names will incur an extra DNS round trip, which doesn't sound too bad to me.

Comment 36 by vyl...@gmail.com, Aug 15 2017

So now that I understand what the problem actually is, how about first fixing the error reporting.

Here's how I got here:

* local domain .dev didn't resolve, but works in all other tools and browsers
* changed it to .test since that is what I should be using in the first place
* no change in behavior
* tcpdump on port 53 on loopback interface => chksum errors. Research that first. Red herring: other lookups also display chksum errors but answers are accepted by chrome
* changed CNAME record to AAAA record to simplify the setup
* finally noticed when I isolated the request/response cycle in tcpdump that there was no AAAA lookup tried
* searched and got here via earlier bug report

So, my use case is that I simply run local development on [::1] (it's shorter, sockstat -6l lists all dev stuff, there's a plethora of reasons for it). This whole mechanism of testing for ipv6 routing is bogus to begin with as the vast majority *nix machines out there will have [::1] whether or not global or neighborhood IPv6 connectivity exists.

It's disturbing to see that code complexity is posted as a negative for fixing this, while instead of simply trusting the OS to the right thing and use `getaddrinfo(3)` and friends, complex code has been introduced to do things differently. I do understand the performance issues, but they are not actually that severe and in total save me less time then this whole chase into the rabbit hole has cost me, for a setup that is in fact completely correct!

So, the proposed fix:

> I think it sounds reasonable to enable AAAA lookups whenever a non-{::1,fe80::,Teredo} address is listed in getifaddrs()

would still bite me, unless I put up a bogus interface with a bogus address that falls outside above scopes. Or switch to 127.0.0.1 (except that I'm not the one with the flawed setup).
Owner: ----
Status: Available (was: Assigned)
We're having the same issue in our organisations IPv6-only VPN. We don't route all traffic through the VPN but only have a route into our internal network. All our internal devices have DNS names with only AAAA records.
Clients which don't have IPv6 connectivity through their Internet connection trigger this issue in Chrome. Accessing internal sites via Firefox and using Samba shares works just fine due to them using the OS's resolver. A solution like mentioned in Comment #34 would solve this without any negative impacts.

Our current workaround is to push a route to 2001:4860:4860::/112 to the VPN client to fool Chrome into thinking it has full IPv6 connectivity. It doesn't cause any noticeable slowdowns so far but we'd still prefer a cleaner solution.
Cc: mge...@chromium.org
Adding Miriam who may be interested in this DNS issue.
Cc: -juliatut...@chromium.org

Comment 41 by hkba...@gmail.com, Jan 15 2018

I can't use chrome/chromium in my ULA IPv6 network without global connectivity because of this, and I just refuse to add a fake route to cover for an issue that is made up in the first place! 
This is of course not a show stopper in any way as Edge and Firefox works flawlessly, but I am disappointed to realize that I must keep away from chrome/chromium for IPv6 networks when I thought google was actually encouraging IPv6 deployments, offline or not.

Everything else on my computer resolves A and AAAA records in parallell and I get that this browser is used by an insane amount of people that knows nothing about computers, so you optimize for that, but I find it strange that chromium is the only thing not handling this without massive time delays.

My vote goes to completely removing this feature. It is not logical because nothing else behaves like this, it slows down working IPv6 deployments by encuraging bad or non-functional setups, and I see no real reason to mess this up and not behave like every other browser in this aspect. Also it wastes time for some of us!
Is there any progress on this one? I frequently get "X's server IP address could not be found." even though I am on an IPv6 network (dual stack or ipv6 only) and the aaaa entry resolves just fine.

+1 on this. Having a v6-only or dualstack VPN in an otherwise v4-only environment is a perfectly acceptable use case. The current behaviour renders Chrome completely useless, i have to use Firefox for this.
+100 for this. We have here the same problem company VPN ipv6 only but no default gateway. User complaints that they can't reach internal ipv6 services.

We have now started to remove all chrome browsers from company systems. This makes chrome unusable.
Hello google? Anyone there? This is a serious problem and everyone in the IPv6 land is waiting for a fix.
comments #42, #43, #44, #45: can you explain more about why your configuration does not have a IPv6 global route?
One scenario exhibiting this bug is as follows:

(1.) There's a dual-stack enterprise network (with default v4 and v6 routes for machines on that network).  There's a firewall that protects internal services.  Some internal services are IPv6-only (i.e., have a DNS name with a AAAA record but no A record).

(2.) There's a VPN for employees to access internal services (i.e., bypass the firewall) when they're at home or otherwise remote.  Since the goal of the VPN is to allow employees to access internal services, VPN clients get routes to the specific v4 and v6 prefixes of the enterprise network, but don't get default v4 or v6 routes via the VPN.

(3.) Some employees connect to the VPN from a remote network that is v4-only (e.g., their ISP at home does not provide v6).

The employee machines from step 3 above lack a default v6 route, but have routes for specific v6 prefixes via the VPN.  If these employees run Chrome, they cannot access v6-only internal enterprise services using the DNS name of that service.
Here it is the same our internal systems are all ipv6 only. Employees get VPN access via dual stack VPN server. Only routes to our networks are running over the VPN tunnel.

So if the employee is in an ipv4 only network he has no ipv6 default routes but ipv6 routes to our ipv6 only internal systems via VPN.
Hey do you exclude VMware network devices? They can also be a bridge via  VirtualMachine to the outside World?

What interfaces are described in EXCLUDE_HOST_SCOPE_VIRTUAL_INTERFACES? Can this match against all type of tunnel interfaces?
I've got a USB RNDIS instrument running Linux with systemd-resolved for LLMNR. There is no IPv4 configuration on the instrument's interface and the Windows interface gets a IPv4 zeroconf address after DHCP times out. There is an IPv6 link-local address on both ends (EUI64 on the instrument and random on Windows).

systemd-resolved resolves AAAA queries to the instrument's link-local address and Firefox, Edge, Ping and whatnot work with the instrument's hostname.

Chrome doesn't, though. I can see in Wireshark that Chrome doesn't send any AAAA queries to the instruments ever, so Chrome never gains the knowledge of the instrument name.

Are the proposed changes going to fix this?

Sign in to add a comment