@atwilson: could you help us find the right owner for this bug?

I'm not sure - Paul, who on the networking team can look at DNS issues like this?

These errors aren't coming from the net/ stack.  The errors are coming from shill.  Someone from shill should investigate.  Assigning to a shill owner:
https://chromium.googlesource.com/aosp/platform/system/connectivity/shill/+/master/OWNERS

Hi Chromium team,
This is Rifel from Google Cloud PSO, I'm helping customer Solvay as a TAM. Let me know if you need any additional info or context that may help on this issue.
I can as well organize a call/troubleshooting demo with Solvay.
Regards,
Rifel

Hello team, it's been a month this issue is transferred to Chrome support.
Is there anything we can do to accelerate the investigations/resolutions?
Adding abhishekbh@ in the Cc to help triage.

Adding brian to CC. Brian, who has looked at DNS last ?

Maybe Matthew? He's definitely looking at portal/DNS stuff somewhere along the way.

But I took a glance, and it's easy to reproduce a problem -- just fill in only 1 or 2 of the 4 DNS server boxes in the settings UI. It looks like the c-ares library doesn't like seeing empty strings in its list (e.g., "192.168.1.1,,,").

Did it ever work to only fill in some of the 4? We probably need to do a better job at sanitizing things here.

For the reporter: IIUC a workaround would be to fill all 4 DNS entries, e.g., with duplicates.

The values are passed from Chrome -> Shill as an Array, so any string cleanup, i.e. "192.168.1.1,,," -> "192.168.1.1" would need to happen in Shill.

That said, we could fix this in Chrome by duplicating entries as suggested. The easiest place to do that is in the JS here:
https://cs.chromium.org/chromium/src/chrome/browser/resources/settings/internet_page/internet_detail_page.js?sq=package:chromium&dr=C&q=internet_detail&g=0&l=992

Will put in a patch to fix this.

Besides the DNS entry sanitization, is it expected that if we fail to set up DNS parameters, we fall back to the previous configuration? It looks like that's what's causing $subject behavior ("continue to access Google's DNS"). Should we be able to propagate the failure back to the UI somehow?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/a8ba568e95d1f4338bbcf36bddd1e367bae8561c

commit a8ba568e95d1f4338bbcf36bddd1e367bae8561c
Author: Matthew Wang <matthewmwang@chromium.org>
Date: Sun Jan 13 21:52:18 2019

shill: fix string join for DNS servers

When a network was configured with custom DNS servers, shill would
join empty strings into the list of servers. For example, the DNS
server string could look like this: "8.8.8.8,,," when it should omit
the trailing commas. This causes some parsing errors in the c-ares
library.

Fix this by checking for an empty string before appending the comma
separator.

BUG= chromium:912955 
TEST=Logged the server_addresses variable to make sure it was fixed,
     made sure the portal detection failures were no longer appearing,
     `FEATURES=test emerge-caroline shill`

Change-Id: I63fb041dee692513dabc13220aec2b09f4b52b42
Reviewed-on: https://chromium-review.googlesource.com/1403898
Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com>
Tested-by: Matthew Wang <matthewmwang@chromium.org>
Reviewed-by: Abhishek Bhardwaj <abhishekbh@chromium.org>
Reviewed-by: Ben Chan <benchan@chromium.org>
Reviewed-by: Brian Norris <briannorris@chromium.org>

[modify] https://crrev.com/a8ba568e95d1f4338bbcf36bddd1e367bae8561c/shill/dns_client_test.cc
[modify] https://crrev.com/a8ba568e95d1f4338bbcf36bddd1e367bae8561c/shill/dns_client.h
[modify] https://crrev.com/a8ba568e95d1f4338bbcf36bddd1e367bae8561c/shill/dns_client.cc

So IIUC, #15 should fix the reporter's problems.

Is there anything else needed here? I guess a few blank DNS entries isn't exactly something that the settings UI needs to complain about. But I still see a few problems when playing around myself:

(1) We still allow all blank entries, with no warning anywhere. This has odd fallbacks, as we still end up using Google DNS for some stuff (portal checks?)
(2) We don't report any warnings back to the user in *any* case -- even if I put total junk in these fields.
(3) Resolver::SetDNSFromLists() still seems to complain about "Malformed nameserver IP" for empty fields.

For (1) and (2): Do we expect any sort of validation/feedback in the settings UI? I guess that's probably in stevenjb's court?

For (3): This seems to be mostly benign, since we still end up skipping over blank entries. But it feels like maybe we're still doing the sanitization in the wrong place.

stevenjb, WDYT about giving the user feedback when the DNS servers are somehow malformed?

It would be pretty straightforward to enforce 4 entries in the UI. We would probably want to add a 'save' button in that case (like we do with proxies) and show an error if there are less than 4 entries. Alternately we could allow blank entries and repeat the last one (which would get reflected in the UI after saving.)

If someone wants to write that up I can try to find someone to work on that, or help anyone who wants to work on that, or try to make some time myself.

Validating entries would be trickier. I don't know if Shill provides any feedback for invalid entries. It might be simplest just to do basic translation / validation (which someone would have to research and document) in the UI. Either way, that's a bigger task.

Opened https://bugs.chromium.org/p/chromium/issues/detail?id=922219 for further UI work. Marking this as fixed for now.