DNS at systemd's 127.0.0.53 is ignoring some lookups

Question

The systemd's DNS lovated at 127.0.0.53 appears to be working except when I query for local machines by name. But if I query for them and specifically specify the local DNS server (my router) then I get the proper reply. But the config file says it is also using the router as the search address. Any thoughts?

I am running Ubuntu 18.04 on my Dell laptop.

Incorrect Results:

$ nslookup web1

Server:     127.0.0.53
Address:    127.0.0.53#53

** server can't find web1: SERVFAIL

Also Fails

$ nslookup -i wlp3s0 web1
nslookup: couldn't get address for 'web1': not found

Correct Results:

$ nslookup web1 192.168.1.1

Server:     192.168.1.1
Address:    192.168.1.1#53

Name:   web1
Address: 192.168.1.107

Configuration Info systemd-resolve

$ systemd-resolve --status

Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 3 (wlp3s0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.1.1
          DNS Domain: wp.comcast.net

Link 2 (enp2s0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Configuration Info NetworkManager

$ cat /etc/NetworkManager/NetworkManager.conf
[main]
plugins=ifupdown,keyfile

[ifupdown]
managed=false

[device]
wifi.scan-rand-mac-address=no

So how do I get nslookup to return the correct answer? Link 3 appears to be the correct information (my wifi connection) and my DNS on the router is returning the correct answer but the local cache never tries to look up the address (or so it seems).

Check out this post and see if that solve your problem: https://askubuntu.com/questions/1034064/diagnosing-dns-lookup-problems-possible-bad-nsswitch-conf-or-other-problem/1039063#1039063 — , May 26 '18 at 13:32
I don't have dns=dnsmasq in my config file. I am updating my question to show this. — schworak, May 26 '18 at 13:52
Which version of Ubuntu are you running and can you also update your post with the IP configuration? — , May 26 '18 at 14:13

score 36 · Accepted Answer · edited Oct 13 '21 at 01:55

36

I found the fix that worked for me.

my resolv.conf file was pointing to the wrong place. This seems like a bug in Ubuntu as it happened on my laptop (the machine I first noticed this issue on) and on a fresh install of Ubuntu 18.04 Server.

The Default

$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Apr 26 12:07 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

I deleted this and pointed to the correct file. After rebooting, this solved my issue. And I was even able to switch networks on my laptop and the DNS switched correctly. Of course when on external networks I can't resolve any of my local machines but that is expected. As soon as I switch back to my local network, all the local machines resolve correctly because my router is the DNS.

The Fix

$ sudo unlink /etc/resolv.conf
$ sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf
$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 32 May 29 08:48 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf
$ sudo reboot

After that, everything worked as I expected and 127.0.0.53 is no longer being used at all.

The Correct Results

$ nslookup web1
Server:     192.168.1.1
Address:    192.168.1.1#53
Name:   web1
Address: 192.168.1.107
$ nslookup google.com
Server:     192.168.1.1
Address:    192.168.1.1#53
Non-authoritative answer:
Name:   google.com
Address: 172.217.7.174
Name:   google.com
Address: 2607:f8b0:4004:80e::200e

edited Oct 13 '21 at 01:55

Marcelo Guedes

113

answered May 29 '18 at 16:08

schworak

871

Please report this bug using ubuntu-bug resolvconf. – Chai T. Rex May 29 '18 at 17:57
When I start to send it says resolvconf (not installed). Is resolvconf and systemd-resolve the same thing? – schworak May 31 '18 at 00:28
systemd-resolve is provided by the systemd package, so please try ubuntu-bug systemd instead. – Chai T. Rex May 31 '18 at 00:45
Thanks! I had never used that bug reporting feature before. Very nice. – schworak Jun 01 '18 at 12:30
2

I had the same problem and this solved it. I didn't reboot the laptop only restarted the network with sudo /etc/init.d/networking restart – matteol Nov 03 '18 at 09:02
2

Wow, this is crazy. Thank you. Did this bug ever get fixed? Is it a Docker-specific bug? I think the resolv.conf is set up this way for docker bridge network DNS right? – void.pointer Dec 18 '18 at 00:25
I've had this problem today. Does anyone have a link the bug? – mafrosis Mar 21 '20 at 21:19
1

just raised it as I could not find it: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1872015 – Nicolas Penin Apr 10 '20 at 07:19
This is not a Docker-specific bug as my laptop nor server run Docker. – schworak Apr 02 '21 at 13:11
1

Definitely not the answer. – jbarlow Jan 17 '22 at 06:48
Most definitely exactly the answer. Had this problem today on a fresh install of Ubuntu 18.04.6. It couldn't resolve host1.my-domain.local. Worked perfectly fine on my older Ubuntu, and on all other hosts in the network. Changing the symlink fixed the problem immediately. Thanks a lot! – marc.guenther Oct 18 '22 at 14:54
I had that problem also on a ubuntu 22.04, and this is the fix – roman204 Nov 14 '23 at 16:32

Laurence Gonsalves · Answer 2 · 2018-09-05T00:17:52.917

15

Your resolv.conf file was not pointing to the wrong place — ../run/systemd/resolve/stub-resolv.conf is where it's supposed to point at by default.

The problem is that systemd-resolved does not pass non-dotted names onto DNS. Apparently this is working "as designed". See this github issue which states that "resolved will never allow single-label lookups to leak onto unicast DNS".

Whether or not you agree with the reasoning in that github issue, there is a way to fix this. It doesn't even require making any changes to the default setup on your Ubuntu machine:

First, your LAN's DNS must have a domain name.

If you're using dnsmasq, add the following to /etc/dnsmasq.conf on your DNS server:
```
expand-hosts
domain=your-domain # replace "your-domain" with domain of your choice
```
You should now be able to resolve LAN hostnames if your add the domain:
```
nslookup web1.your-domain
```
Second, make sure the name for your LAN's domain is also set in your DHCP server if it's different from your DNS server. On my DHCP server (my router), this setting is just called "Domain Name".

If you then renew your DHCP lease on your Ubuntu box, you should see a search directive appear in /run/systemd/resolve/stub-resolv.conf:
```
nameserver 127.0.0.53
search your-domain
```

Now looking up web1 will expand it to web1.your-domain, which will then resolve using DNS.

$ nslookup web1
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   web1.your-domain
Address: 192.168.1.107

Note that if you use dig instead of nslookup, dig doesn't use the search path by default — use its +search option to enable that.

edited Sep 05 '18 at 00:17

answered Sep 04 '18 at 20:49

Laurence Gonsalves

889

Before rebooting it looked up web1.mydomain.com just fine. But of course looking up just web1 didn't work. So i rebooted and for the life of me I have no idea where it is picking up the Comcast domain but now if I lookup web1 it responds with the correct IP but shows the comcast domain instead of my domain. It resolves so I am not too worried but what the heck???? – schworak Sep 05 '18 at 11:52
@schworak Weird! Is your DHCP server also your Comcast modem? Do you see that domain showing up in /etc/resolv.conf or in the output of either nmcli -g all or systemd-resolve --status? Maybe try looking at what's in your DHCP lease? – Laurence Gonsalves Sep 05 '18 at 16:31
It is not the comcast modem. I have a SysLink router with DDWRT running. The comcast settings are totally replaced. The comcast name shows up in the resolv.conf file which is auto-generated at boot up. I am not too worried about it but it is weird. – schworak Sep 06 '18 at 01:27
@LaurenceGonsalves Thanks a ton for the github issue link. I had found workarounds to the problem but this actually helped me figure out the root issue. – Gregory Arenius Mar 15 '19 at 22:07
This is the correct answer. In case you came here to look for a solution where you use OpenWrt to set some CNAME for another local host, what you need to do is to set both the CNAME and target to be fully qualified. That is, assuming the default OpenWrt domain to be lan, the CNAME should be mycname.lan and the target mytarget.lan. Once you do that, you can resolve the CNAME as both mycname and mycname.lan. – Nicolay Doytchev Jun 27 '20 at 00:30
/etc/dnsmasq.conf doesn't exist on current versions of ubuntu – jbarlow Jan 17 '22 at 06:49
@jbarlow the mention of /etc/dnsmasq.conf is not referring to your Ubuntu machine, but to your LAN's DNS server. You need to set the domain name for your LANs DNS. Instructions on how to do this with dnsmasq are provided for convenience, but if you're using something else for your LAN's DNS you'll have to figure out how to set its domain. – Laurence Gonsalves Jan 18 '22 at 00:02

score 5 · Answer 3 · answered Aug 10 '21 at 12:24

Sometimes removing the symlink at /etc/resolv.conf is the right thing to do.

I recently migrated our authoritative DNS server to a newly built host, which (unlike the previous one) had systemd installed.

The results were terrible.

Although the rest of the world would see the correct results, the systemd resolver would serve stale results results to local services, and that would mess up those other services, in particular the internal mail forwarder.

dig @127.0.0.53 would show it bumping the TTL back up from time to time, so its cache would never expire; I suspect it was effectively querying itself, since the NS points at this host.

Running systemd-resolve --flush-caches did nothing.

Editing /etc/systemd/resolved.conf and setting DNSStubListener=no eventually stopped the resolver running, though it seems to pop up for a few seconds every now and then.

At least now the traditional services would actually get the right answers, but the ones built to perform name lookups over DBUS are irredeemably broken.

Now I understand why so many people hated systemd; it interferes other subsystems that it has no reason to touch, and there's no effective way to stop it from doing so. Even NIS wasn't broken as badly as this; why did systemd need to change it and make it worse?

DNS at systemd's 127.0.0.53 is ignoring some lookups

3 Answers3

Linked