Netplan DNS configuration is overridden UBUNTU 22.04

Question

I have just installed ubuntu 22.04 from scratch on a new physical server. One thing that has been a thorn in my side with Ubuntu since about 14.04, is trying to wrap my head around the complexities it puts over DNS configuration. Today is no exception. There's no shortage of questions and problems related to this (so here's one more - I'm very sorry!), and that should tell somebody there's a problem here. Things were so much simpler and so much more reliable before netplan/NetworkManager and that ilk. dnsmasq was my favorite tool; I could easily set up my whole home network in one confifuation - but alas! I cannot restore the past.

Anyway, what I have is

• An inflexible router, provided by my ISP, that will not allow me to specify DNS settngs and will not allow me to specify fixed addresses for connected devices
• A network consisting of around 60 devices, most of which intercommunicate (so internal DNS is critical here)
• A single network (eth) interface

The configuration file in /etc/netplan/01_wired.yaml is:

network:
  renderer: networkd
  ethernets:
    eth_lan0:
      dhcp4: no
      addresses: [192.168.1.12/24]
      gateway4: 192.168.1.254
      #routes:
      #  - to: default
      #    via: 192.168.1.254
      nameservers:
        addresses:  [192.168.1.2]
        search: [domainname.com]
      optional: true
      match:
        macaddress: 90:2b:34:36:ae:bc
      set-name: eth0

I actually have a dnsmasq system running on a server on the network (on rPi raspbian since ubuntu makes this such a pain), and its address is 192.168.1.2, the DNS I've told netplan to use. And all server names are hostname.domainname.com (or a subdomain of same) (domainname.com represents a network name managed by me.)

Even though there is no place in the system that I've indicated to use DHCP, it seems that DHCP is used, aS referenced in /etc/resolv.conf:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.
nameserver 127.0.0.53
search lovelady.com attlocal.net

(Note: it's that 'attlocal.net' in there that tells me it's referencing the DHCP server.) Also, resolvectl status returns this:

root@r2d2:~# resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: foreign
      DNS Domain: attlocal.net lovelady.com
Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 2600:1700:5950:3240::1
       DNS Servers: 191.168.1.2 2600:1700:5950:3240::1
        DNS Domain: attlocal.net lovelady.com
Link 3 (eno1)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
root@r2d2:~# resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: foreign
      DNS Domain: attlocal.net lovelady.com
Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 2600:1700:5950:3240::1
       DNS Servers: 191.168.1.2 2600:1700:5950:3240::1
        DNS Domain: attlocal.net lovelady.com
Link 3 (eno1)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

(Note that resolvectl puts attlocal.net before lovelady.com (!!!!) and I don't even want attlocal.net in the configuration. At all!

Anyway, resolvectl is not telling the accurate story becuase it implies that lookups will be sent to 192.168.1.2 (which does work great) but on this system, correct IP addresses are not being returned. This pair of lookups shows the issue:

username@r2d2:~# dig velmicro
; <<>> DiG 9.18.18-0ubuntu0.22.04.1-Ubuntu <<>> velmicro
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 59512
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;velmicro.                      IN      A
;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Wed Dec 06 00:45:20 UTC 2023
;; MSG SIZE  rcvd: 37
username@r2d2:~# dig @192.168.1.2 velmicro
; <<>> DiG 9.18.18-0ubuntu0.22.04.1-Ubuntu <<>> @192.168.1.2 velmicro
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53926
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;velmicro.                      IN      A
;; ANSWER SECTION:
velmicro.               0       IN      A       192.168.1.6
;; Query time: 3 msec
;; SERVER: 192.168.1.2#53(192.168.1.2) (UDP)
;; WHEN: Wed Dec 06 00:45:46 UTC 2023
;; MSG SIZE  rcvd: 53

If I don't tell dig what server to use, it uses the default, which returns no address for velmicro. But if I tell dig to use 192.168.1.2, it returns the correct address.

I've spent a whole afternoon reading article after article about resolutions for this. The vast majority of resolutions I've found say to make just the configuration that I've made, and they stop there. Most of the rest have to do with Network Manager, which isn't (or doesn't seem to be - who can tell?) involved in this configuration. Then there are those who say to just modify /etc/resolv.conf ... and then we get into resolveconf and other configuration tools, each of which adds more complexity. All I want, is for the system to use the configuration I set manually. Doesn't seem unreasonable to me.

So I'm here to add yet another voice to the cries in the dark looking for a DNS solution on recent UBUNTU. Can you help?

UPDATE at @upboden's request:

root@r2d2:~# ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 29 Dec  5 23:17 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
root@r2d2:~# more /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.
nameserver 127.0.0.53
search lovelady.com attlocal.net

UPDATE 2

I have removed the package resolvconf (which I added in hopes of resolving this problem). Removal of the package restored the link, but has not solve the issue.

root@r2d2:~# sudo apt remove resolveconf
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package resolveconf
root@r2d2:~# sudo apt remove resolvconf
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  resolvconf
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 203 kB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 87264 files and directories currently installed.)
Removing resolvconf (1.84ubuntu1) ...
resolvconf.postrm: Reboot recommended
Processing triggers for man-db (2.10.2-1) ...
root@r2d2:~# ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Dec  6 01:51 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
root@r2d2:~# cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0 trust-ad
search lovelady.com attlocal.net

EDIT 3

I have added a dhcp6: no line to the /etc/netplan/01_wired.yaml file, and the problem was resolved. New content is:

network:
  renderer: networkd
  ethernets:
    eth_lan0:
      dhcp4: no
      dhcp6: no
      addresses: [192.168.1.12/24]
      gateway4: 192.168.1.254
      #routes:
      #  - to: default
      #    via: 192.168.1.254
      gateway4: 192.168.1.254
      nameservers:
        addresses:  [192.168.1.2]
        search: [lovelady.com]
      optional: true
      match:
        macaddress: 90:2b:34:36:ae:bc
      set-name: eth0

resolvectl status (note the absence of IP6 spec in output):

> resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.1.2
       DNS Servers: 192.168.1.2 2600:1700:5950:3240::1
        DNS Domain: attlocal.net lovelady.com
Link 3 (eno1)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

dig output annoyingly still looks wrong (no answer shown from default) :

> dig velmicro
; <<>> DiG 9.18.18-0ubuntu0.22.04.1-Ubuntu <<>> velmicro
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 41096
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;velmicro.                      IN      A
;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Wed Dec 06 14:47:54 UTC 2023
;; MSG SIZE  rcvd: 37
dennis@r2d2:/home/dennis   12/06 14:47:54
> dig @192.168.1.2 velmicro
; <<>> DiG 9.18.18-0ubuntu0.22.04.1-Ubuntu <<>> @192.168.1.2 velmicro
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1145
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;velmicro.                      IN      A
;; ANSWER SECTION:
velmicro.               0       IN      A       192.168.1.6
;; Query time: 0 msec
;; SERVER: 192.168.1.2#53(192.168.1.2) (UDP)
;; WHEN: Wed Dec 06 14:48:07 UTC 2023
;; MSG SIZE  rcvd: 53

But the practial/important thing is that adding dhcp6: no seems to have worked.

For completeness, the answer to slangasek's question about other netplan files is: Only 00-installer-config.yaml is present, and its contents are:

# This is the network config written by 'subiquity'
network:
  ethernets:
    eno1:
      optional: true
      dhcp4: true
    enp6s0:
      optional: true
      dhcp4: true
  version: 2

Note that search attlocal.net remains in the configuration (from where, I cannot tell). It's in the /etc/resolv.conf file:

# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the 
# internal DNS stub resolver of systemd-resolved. This file lists all 
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0 trust-ad
search lovelady.com attlocal.net

but of course, I did not put it there.

So current status is that DNS seems to be working (for now) but some mysteries remain, and mysteries in IT are rarely a good thing.

I let my frustration show in the original message. I still feel strongly that UBUNTU is going a wrong direction w.r.t. what should be a simple matter, and hope that perhaps one day those involved will realize the value of simplicity. But I should not let it get to me; UBUNTU is, all in all, the best of those available for my needs.

Answer 1

I share your frustration Dennis and agree. Below is the simple fix for your issue. Note: this answer is based on Dennis like of the "old" days. This is not the current recommended fix.

Despite all the frustrating things systemd, NetworkManager etc does with DNS, they all follow the design rule, or not modifiying resolv.conf directly, when they install, they link /etc/resolv.conf to a location unique to their service, and they modify that file.

So to fix your issue, remove the symbolic link at /etc/resolv.conf. Create the file the way you want it. eg:

nameserver 192.168.1.2
search local.domain
<any other options>

You should be good to go now, the file /etc/resolv.conf is the only file that matters for dns, and you now have control of it. In my experience systemd and Network Manager wont change the file, now the symbolic link is removed. However if you want to make sure that is the case, change ownership to your user, and give only read access to everyone else. Then it is safe. You just need to edit /etc/resolv.conf if you make changes to your setup.

After feedback from Dennis and some more research:

To disable Network Manager from modifying resolv.conf edit /etc/NetworkManager/NetworkManager.conf adding under [main]:

dns=none
rc-manager=unmanaged

Alternativly if you want to disable NetworkManager completely these are the instructions on the Ubuntu site and they reference it back to How do I disable network manager permanently?

Stop network manager
sudo systemctl stop NetworkManager.service
sudo systemctl stop NetworkManager-wait-online.service
sudo systemctl stop NetworkManager-dispatcher.service
sudo systemctl stop network-manager.service
Disable network manager (permanently) to avoid it restarting after a reboot
sudo systemctl disable NetworkManager.service
sudo systemctl disable NetworkManager-wait-online.service
sudo systemctl disable NetworkManager-dispatcher.service
sudo systemctl disable network-manager.service