How can I identify detailed DNS behavior with systemd?

Question

I am trying to diagnose issues with a DNS server, in an environment with multiple DNS servers set by DHCP.

In Ubuntu Server 20.04.2 (with no changes to DNS behavior) I need to find which DNS server(s) are actually being used to resolve a specific query. The DNS servers are set by DHCP.

Essentially, I need to replicate the behavior of the dig command or nslookup that modern Ubuntu versions break (showing a localhost address for DNS server, instead of the address for the "real" server.)

systemd-resolve / resolvectl does not appear to be of any use, as it does not provide info for how it resolves a specific query.

systemd-resolve --status / resolvectl status is not useful, as it only provides "DNS Servers" and "Current DNS Server" at a point in time (not in relation to a specific DNS query).

nmcli does not appear to be relevant.

It is possible that my question would be (indirectly) answered by answering "How do I view the 'feature level information' referenced in the manpage of resolvectl?" This would depend on the kind of information available there.

Since this is only a first step of a process, it would be helpful if the method provided insight into why one DNS server is used instead of others (for a specific query.), or what process is used to determine that.

Relevant links that don't actually address my question:

https://stgraber.org/2012/02/24/dns-in-ubuntu-12-04/

How do I list DNS server order in systemd-resolve?

https://unix.stackexchange.com/questions/328131/how-to-troubleshoot-dns-with-systemd-resolved

systemd-resolve does not forward DNS requests to stated DNS server

Edit

To make more explicit what (I thought) was perfectly clear in the original question:

I am aware that the query is resolved by a real DNS server at localhost. The question is specifically about the two (or more) DNS servers assigned by DHCP. The use of the local proxy obscures which of the two other systems on the network is being used to resolve a specific query.

does dig +trace google.com help you? Its part of the dnsutils — kanehekili, Apr 08 '22 at 18:05
dig or nslookup show localhost as DNS server, because it is actually the DNS server used to resolve names. systemd-resolved acts as a proxy DNS server, forwarding the request to the server(s) shown by resolvectl status. So the actual query is resolved against localhost. — raj, Apr 08 '22 at 18:25
Read man resolved.conf and man systemd-resolved.service; all your questions get answered. — paladin, Apr 08 '22 at 20:22
systemd priorizes on-link DNS server over global DNS server over global fallback DNS servers in its default settings. — paladin, Apr 08 '22 at 20:24
"With no changes to DNS behavior"... does that mean you don't want to alter the way dns works on that machine? If not you could use another dns service that will make it easier to track traffic. You could install dnmasq and disable systemd-resolved, then replacing locahost53 with your dns servers in /etc/resolv.conf. Then you should be able to monitor which server is being hit with something like tcpdump... I cant tell youre trying to figure out where traffic is going or rather why it is going there? But this would eliminate the "showing a localhost address for a DNS server" — WU-TANG, Apr 09 '22 at 01:50

muru · Accepted Answer · 2022-04-08T18:14:46.707

You can set the log level for systemd-resolved, run a query using resolvectl query, and then check the logs of systemd-resolved. For example:

% sudo resolvectl log-level debug
% resolvectl query example.com
example.com: 93.184.216.34                     -- link: enp6s0
             2606:2800:220:1:248:1893:25c8:1946 -- link: enp6s0
-- Information acquired via protocol DNS in 14.0ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network

And checking the logs of systemd-resolved:

% journalctl -u systemd-resolved
...
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.1142 path=n/a interface=n/a member=n/a cookie=36 reply_cookie=62 signature=u error-name=n/a error-message=n/a
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: D-Bus hostname resolution request from client PID 194160 (n/a) with UID 4294967295
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Looking up RR for example.com IN A.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Looking up RR for example.com IN AAAA.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=AddMatch cookie=63 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=GetNameOwner cookie=64 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.1142 path=n/a interface=n/a member=n/a cookie=38 reply_cookie=64 signature=s error-name=n/a error-message=n/a
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Cache miss for example.com IN A
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Firing regular transaction 61323 for <example.com IN A> scope dns on */* (validate=yes).
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Using feature level UDP+EDNS0 for transaction 61323.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Using DNS server 8.8.8.8 for transaction 61323.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Announcing packet size 1472 in egress EDNS(0) packet.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Emitting UDP, link MTU is 1500, socket MTU is 1500, minimal MTU is 40
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Sending query packet with id 61323 of size 40.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Cache miss for example.com IN AAAA
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Firing regular transaction 17323 for <example.com IN AAAA> scope dns on */* (validate=yes).
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Using feature level UDP+EDNS0 for transaction 17323.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Using DNS server 8.8.8.8 for transaction 17323.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Announcing packet size 1472 in egress EDNS(0) packet.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Emitting UDP, link MTU is 1500, socket MTU is 1500, minimal MTU is 40
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Sending query packet with id 17323 of size 40.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Cache miss for example.com IN AAAA
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Firing regular transaction 7013 for <example.com IN AAAA> scope dns on enp6s0/* (validate=yes).
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Using feature level UDP+EDNS0 for transaction 7013.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Using DNS server 8.8.8.8 for transaction 7013.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Announcing packet size 1472 in egress EDNS(0) packet.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Emitting UDP, link MTU is 1500, socket MTU is 0, minimal MTU is 40
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Sending query packet with id 7013 of size 40.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Cache miss for example.com IN A
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Firing regular transaction 32950 for <example.com IN A> scope dns on enp6s0/* (validate=yes).
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Using feature level UDP+EDNS0 for transaction 32950.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Using DNS server 8.8.8.8 for transaction 32950.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Announcing packet size 1472 in egress EDNS(0) packet.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Emitting UDP, link MTU is 1500, socket MTU is 0, minimal MTU is 40
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Sending query packet with id 32950 of size 40.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.1142 path=n/a interface=n/a member=n/a cookie=37 reply_cookie=63 signature=n/a error-name=n/a error-message=n/a
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Match type='signal',sender='org.freedesktop.DBus',path='/org/freedesktop/DBus',interface='org.freedesktop.DBus',member='NameOwnerChanged',arg0=':1.1170' successfully installed.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Received dns UDP packet of size 56, ifindex=2, ttl=0, fragsize=0, sender=8.8.8.8, destination=192.168.0.2
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Processing incoming packet of size 56 on transaction 61323 (rcode=SUCCESS).
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Added positive unauthenticated non-confidential cache entry for example.com IN A 7200s on enp6s0/INET/8.8.8.8
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Regular transaction 61323 for <example.com IN A> on scope dns on */* now complete with <success> from network (unsigned; non-confidential).
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Received dns UDP packet of size 68, ifindex=2, ttl=0, fragsize=0, sender=8.8.8.8, destination=192.168.0.2
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Processing incoming packet of size 68 on transaction 7013 (rcode=SUCCESS).
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Added positive unauthenticated non-confidential cache entry for example.com IN AAAA 1277s on enp6s0/INET/8.8.8.8
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Regular transaction 7013 for <example.com IN AAAA> on scope dns on enp6s0/* now complete with <success> from network (unsigned; non-confidential).
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Received dns UDP packet of size 68, ifindex=2, ttl=0, fragsize=0, sender=8.8.8.8, destination=192.168.0.2
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Processing incoming packet of size 68 on transaction 17323 (rcode=SUCCESS).
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Added positive unauthenticated non-confidential cache entry for example.com IN AAAA 7200s on enp6s0/INET/8.8.8.8
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Regular transaction 17323 for <example.com IN AAAA> on scope dns on */* now complete with <success> from network (unsigned; non-confidential).
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Freeing transaction 61323.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Freeing transaction 7013.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Freeing transaction 32950.
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Added socket 26 to graveyard
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Sent message type=method_return sender=n/a destination=:1.1170 path=n/a interface=n/a member=n/a cookie=65 reply_cookie=2 signature=a(iiay)st error-name=n/a error-message=n/a
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=RemoveMatch cookie=66 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Apr 09 02:59:49 cthulhu systemd-resolved[193385]: Freeing transaction 17323.
...

I can't explain the behaviour, but you can see the behaviour in detail.

Also check man systemd-resolved.service, which has IMO a fairly detailed explanation of the algorithm used:

   •   Queries for multi-label names are routed via unicast DNS on
       local interfaces that have a DNS server configured, plus the
       globally configured DNS servers if there are any. Which
       interfaces are used is determined by the routing logic based
       on search and route-only domains, described below. [...]
If lookups are routed to multiple interfaces, the first
   successful response is returned (thus effectively merging the
   lookup zones on all matching interfaces). If the lookup failed on
   all interfaces, the last failing response is returned.
[...]
The following query routing logic applies for unicast DNS lookups
   initiated by systemd-resolved.service:
•   If a name to look up matches (that is: is equal to or has as
       suffix) any of the configured routing domains (search or
       route-only) of any link, or the globally configured DNS
       settings, "best matching" routing domain is determined: the
       matching one with the most labels. The query is then sent to
       all DNS servers of any links or the globally configured DNS
       servers associated with this "best matching" routing domain.
       (Note that more than one link might have this same "best
       matching" routing domain configured, in which case the query
       is sent to all of them in parallel).
   In case of single-label names, when search domains are
   defined, the same logic applies, except that the name is
   first suffixed by each of the search domains in turn. Note
   that this search logic doesn't apply to any names with at
   least one dot. Also see the discussion about compatibility
   with the traditional glibc resolver below.


•   If a query does not match any configured routing domain
       (either per-link or global), it is sent to all DNS servers
       that are configured on links with the DefaultRoute= option
       set, as well as the globally configured DNS server.
•   If there is no link configured as DefaultRoute= and no global
       DNS server configured, one of the compiled-in fallback DNS
       servers is used.
•   Otherwise the unicast DNS query fails, as no suitable DNS
       servers can be determined.

systemd priorizes on-link DNS server over global DNS server over global fallback DNS servers in its default settings. — paladin, Apr 08 '22 at 20:24

How can I identify detailed DNS behavior with systemd?

1 Answers1