New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 518673 link

Starred by 7 users

Issue metadata

Status: Archived
Owner: ----
Closed: Jun 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 2
Type: Bug



Sign in to add a comment

Broken IPv6 can cause very slow DNS resolution on Linux

Reported by evan.mar...@gmail.com, Aug 9 2015

Issue description

UserAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36

Example URL:

Steps to reproduce the problem:
I'm in an airbnb in Prague right now and Chrome is slow.
Net log attached.  It appears to take ~five seconds for DNS to resolve.

What is the expected behavior?

What went wrong?
In the attached log, I attempt to open bradfitz.com.  It takes five seconds to resolve.

Did this work before? N/A 

Chrome version: 44.0.2403.125  Channel: n/a
OS Version: 
Flash Version: 

I'm not sure ipv6 is implicated, but it's a little funny maybe:

$ dig bradfitz.com A
takes 183ms

$ dig bradfitz.com A
takes 1048ms

Using @8.8.8.8 makes both queries take ~200ms.

$ cat /etc/resolv.conf
# Generated by resolvconf
search Home
nameserver 10.0.0.138
nameserver fe80::1%wlp2s0

Some other info that might be relevant:

$ ip addr show dev wlp2s0
2: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1280 qdisc mq state UP group default qlen 1000
    link/ether 34:02:86:a2:71:91 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.74/24 brd 10.0.0.255 scope global dynamic wlp2s0
       valid_lft 83009sec preferred_lft 83009sec
    inet6 2a00:1028:838a:8d1e:3602:86ff:fea2:7191/64 scope global noprefixroute dynamic 
       valid_lft 172781sec preferred_lft 86381sec
    inet6 fe80::3602:86ff:fea2:7191/64 scope link 
       valid_lft forever preferred_lft forever
 
net-internals-log.json
361 KB Download
(To be clear, resolving ip addresses outside of Chrome using "host" or "dig", both with the default DNS server and with 8.8.8.8, are relatively fast.  It seems only Chrome has a problem.)
Cc: rnimmagadda@chromium.org
Labels: Needs-Feedback
Unable to repro this issue on Ubuntu Trusty (14.04) for Google Chrome Stable Version - 43.0.2357.130

Screen-recording is attached.

@petrnedved1995: Could you please re-test the same on a clean profile [chrome://settings -> Add Person] and let us know your observations, which would help us in triaging it further.

Also, update your Google Chrome to Latest Stable Version - 44.0.2403.130

Thank you.
518673.ogv
1.6 MB Download
This only reproduces on the specific network configuration I'm using.  Chrome has worked fine elsewhere on this trip -- it's only when I am using this particular wifi that Chrome is slow.  You will not be able to reproduce this locally.  That is why I included the network trace.

(And I'll also restate that other networking probes, those outside of Chrome, work fine.  It's only Chrome that is having DNS problems.)

Comment 5 by eroman@chromium.org, Aug 10 2015

Cc: ttuttle@chromium.org
Labels: Cr-Internals-Network-DNS
From the log it is something of a black box as to why getaddrinfo() is taking 5 seconds to complete.

 * It probed IPv6 as being supported (maybe disabling IPv6 would have sped things up)
 * It passes the AI_ADDRCONFIG

Unfortunately the command-line flag to turn off IPv6 was removed, and the latter is not configurable. So you can't easily experiment with either of these settings.

The easiest thing might be to run a test program which modulates those parameters calling getaddrinfo() directly, and see there is a setting causing it to run slowly. (Let me see if I can throw something together, assuming you are still able to reproduce).

Also curious if you have you restarted Chrome since this issue (in case res_ninit() is failing to reload any changes to resolv.conf)

@tuttle would be the DNS expert and might have other ideas.
I screwed up my above paste.  Here's a shorter summary:

$ time -p dig bradfitz.com A > /dev/null
real 0.04
$ time -p dig bradfitz.com AAAA > /dev/null
real 1.81

...ah, here's something interesting.  dig has a -6 option to force ipv6 (I was thinking the above AAAA query would do that but of course it doesn't).

When I use "dig -6 bradfitz.com A" it takes 15 seconds then times out.

When I try AAAA it has an odd behavior:

$ time -p dig -6 bradfitz.com AAAA
; <<>> DiG 9.10.2-P3 <<>> -6 bradfitz.com AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41544
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;bradfitz.com.			IN	AAAA

;; AUTHORITY SECTION:
bradfitz.com.		778	IN	SOA	ns-640.awsdns-16.net. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

;; Query time: 133 msec
;; SERVER: fe80::1%2#53(fe80::1%2)
;; WHEN: Tue Aug 11 00:58:04 CEST 2015
;; MSG SIZE  rcvd: 122

real 5.14

That's saying the query took 133ms but the end-to-end "dig" call took 5.14s for some reason?

Comment 7 by eroman@chromium.org, Aug 11 2015

Could you run this test program, to try some different parameters to getaddrinfo()?

$ g++ getaddrinfo_experiment.cc
$ ./a.out bradfitz.com

And then attach the output.
getaddrinfo_experiment.cc
5.7 KB Download
Wow, it seems pretty screwed.
It's weird that "host bradfitz.com" is fast, given how many of these look slow.


$ ./a.out bradfitz.com
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints=NULL):
  Duration: 5.07227 seconds
  Succeeded with:
    0: 50.112.116.235
    1: 50.112.116.235
    2: 50.112.116.235
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_UNSPEC, ai_flags=0}):
  Duration: 10.0092 seconds
  Succeeded with:
    0: 50.112.116.235
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_UNSPEC, ai_flags=AI_CANONNAME}):
  Duration: 5.00461 seconds
  Succeeded with:
    0: 50.112.116.235
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_UNSPEC, ai_flags=AI_ADDRCONFIG}):
  Duration: 5.00496 seconds
  Succeeded with:
    0: 50.112.116.235
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_UNSPEC, ai_flags=AI_ADDRCONFIG | AI_CANONNAME}):
  Duration: 5.00421 seconds
  Succeeded with:
    0: 50.112.116.235
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_UNSPEC, ai_flags=AI_ADDRCONFIG | AI_V4MAPPED}):
  Duration: 5.00474 seconds
  Succeeded with:
    0: 50.112.116.235
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_INET, ai_flags=0}):
  Duration: 0.019135 seconds
  Succeeded with:
    0: 50.112.116.235
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_INET, ai_flags=AI_CANONNAME}):
  Duration: 0.00205684 seconds
  Succeeded with:
    0: 50.112.116.235
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_INET, ai_flags=AI_ADDRCONFIG}):
  Duration: 0.00188088 seconds
  Succeeded with:
    0: 50.112.116.235
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_INET, ai_flags=AI_ADDRCONFIG | AI_CANONNAME}):
  Duration: 0.00187302 seconds
  Succeeded with:
    0: 50.112.116.235
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_INET, ai_flags=AI_ADDRCONFIG | AI_V4MAPPED}):
  Duration: 0.00197697 seconds
  Succeeded with:
    0: 50.112.116.235
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_INET6, ai_flags=0}):
  Duration: 20.0158 seconds
  ERROR: -2: Name or service not known
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_INET6, ai_flags=AI_CANONNAME}):
  Duration: 20.0175 seconds
  ERROR: -2: Name or service not known
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_INET6, ai_flags=AI_ADDRCONFIG}):
  Duration: 20.0183 seconds
  ERROR: -2: Name or service not known
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_INET6, ai_flags=AI_ADDRCONFIG | AI_CANONNAME}):
  Duration: 20.0115 seconds
  ERROR: -2: Name or service not known
-------------------------------------------------------
getaddrinfo(host="bradfitz.com", hints={ai_family=AF_INET6, ai_flags=AI_ADDRCONFIG | AI_V4MAPPED}):
  Duration: 20.21 seconds
  Succeeded with:
    0: ::ffff:50.112.116.235

Comment 9 by eroman@chromium.org, Aug 18 2015

Oops sorry, I forgot to star this issue and didn't see you had posted the log already!

Anyway looking at the log, it is pretty clear that this is an IPv6 problem with your network.

  * When requesting specifically IPv6 addresses (AF_INET6) it takes 20 seconds (this is probably hitting a timeout at 20 seconds somewhere)

  * When requesting either IPv4 or IPv6 (AF_UNSPEC) it takes 5 seconds. Note this is the configuration Chrome uses and consistent with your performance

  * When requesting IPv4 only (AF_INET) it takes very little time and works as expected.


Fundamentally this is a problem with the system DNS implementation (digg/host look to be using different implementations than getaddrinfo()). But given this what could Chrome be doing better?

  * Chrome probed IPv6 as supported because it could create a socket. Really it should have probed IPv6 as broken and forcefully disabled it by passing AF_INET

  * Chrome no longer has a command line to disable IPv6, so you can't do much about this

  * Chrome has its own DNS implementation that you could try enabling instead. I don't have the option to enable it on Mac (tuttle would know more), but maybe we expose it on about:flags on Linux.

  * Disable IPv6 at a system level. In fact I think you can simply remove "options inet6" from /etc/resolv.conf.
Cc: cbentzel@chromium.org
FYI cbentzel (there are still situations where --disable-ipv6 would be useful)
When I first started suspecting ipv6 I tried to disable it systemwide but I couldn't get it to work.  But I only tried a few minutes.

I'm no longer staying at the airbnb with the network problems so I can no longer experiment with it.  Feel free to close this bug if you feel you have all the lessons learned you could want.
Cc: pauljensen@chromium.org
#10: Thanks. We were using Chrome's built-in resolver on Linux until a few months ago because there was at least one case where we weren't accurately handling some options in either nsswitch.conf or resolv.conf (i forget which). 

It's possible that would make things better, but seems like the real issue is that we are doing AF_UNSPEC and not backing off to IPv4 only.
Cc: -ttuttle@chromium.org
Owner: ttuttle@chromium.org
Status: Assigned
> seems like the real issue is that we are doing AF_UNSPEC and not backing off to IPv4 only.

If I'm reading the "ip addr" output correctly, it looks like the device in question has a valid IPv6 globally-routable address, in which case we should be doing AF_UNSPEC queries under normal operation.  Are you suggesting we add specialized behavior to better detect and handle broken DNS servers?
I was suggesting that we should add the specialized behavior to backoff
AF_UNSPEC to AF_IPV4 if DNS resolution is routinely very slow, even in
presence of globally routable IPv6 space.

The other options are:

a) Do nothing in Chrome. This may be a really infrequent case, and adding
additional complexity doesn't make sense - and fixes should be in network.

b) Put logic in async resolver. It's even possible that the async resolver
wouldn't have run into this issue since retry limits are generally better
than default (and right now async resolver is disabled on Linux).

c) Never issue AF_UNSPEC resolution - always do A and AAAA resolution
separately. This is part of rethinking our happy eyeballs approach, and is
a bit closer to what Apple is doing.
Labels: -Needs-Feedback Cr-Internals-Network-Connectivity
Not waiting for feedback on this one.
Labels: Hotlist-Recharge
This issue likely requires triage.  The current issue owner may be inactive (i.e. hasn't fixed an issue in the last 30 days or commented in this particular issue in the last 90 days).  Thanks for helping out!

-Anthony
Owner: juliatut...@chromium.org
Owner: ----
Status: Available (was: Assigned)
Owner: mge...@chromium.org
Summary: Broken IPv6 can cause very slow DNS resolution on Linux (was: Slow DNS resolution only within Chrome(?))
Owner: ----
Status: Archived (was: Available)

Sign in to add a comment