Ubuntu Server 18.04 locking up

Question

I am running Server 18.04 as the attached shows there are 2 Kernels available. 4.15.0-72.81 Kernel uses the default Realtek r8169 Ethernet driver. While 4.15.0.76.86 kernel has the Realtek r8168 driver installed.

4.15.0-72.81 after a period of inaction will stop talking to the router and thus the LAN and sometimes the server will lock up in either case the fix is to reboot.

4.15.0.76.86 will always after a few minutes regardless of activity will just lock, ignore the keyboard and need a reboot.

The following is the result of several checks run in 4.15.0-72.81 after it no longer connects with either the Ethernet or the Wi-Fi interface. I'm looking for suggestions on how to debug this further.

$ apt list --installed | grep image-4.15.0
linux-image-4.15.0-72-generic/bionic-security,bionic-updates,now 4.15.0-72.81 amd64 [installed,automatic]
linux-image-4.15.0-76-generic/bionic-security,bionic-updates,now 4.15.0-76.86 amd64 [installed,automatic]

$ ip -s a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast
    12289968   12864    0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    12289968   12864    0       0       0       0

2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 94:de:80:28:a2:71 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.200/24 brd 192.168.1.255 scope global enp5s0
       valid_lft forever preferred_lft forever
    inet6 fe80::96de:80ff:fe28:a271/64 scope link
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast
    0          0        0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    0          0        0       0       0       0

3: wlp4s6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:14:a5:cc:49:b3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.201/24 brd 192.168.1.255 scope global wlp4s6 
       valid_lft forever preferred_lft forever
    inet6 fe80::214:a5ff:fecc:49b3/64 scope link
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast
    1366996    8848     0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    1656237    2776     0       0       0       0

$ lspci -knn | grep Eth -a3
    Subsystem: Gigabyte Technology Co., Ltd 88SE9172 SATA 6Gb/s Controller [1458:b000]
    Kernel driver in use: ahci
    Kernel modules: ahci
04:06.0 Ethernet controller [0200]: Qualcomm Atheros AR2413/AR2414 Wireless Network Adapter [AR5005G(S) 802.11bg] [168c:001a] (rev 01)
    Subsystem: Gemtek Technology Co., Ltd DX-WGDTC 802.11bg Wireless PCI Adapter [17f9:0018]
    Kernel driver in use: ath5k
    Kernel modules: ath5k
--  
    Subsystem: Gigabyte Technology Co., Ltd GA-7VT600-1394 Motherboard [1458:1000]
    Kernel driver in use: firewire_ohci
    Kernel modules: firewire_ohci
05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd.  RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06)
    Subsystem: Gigabyte Technology Co., Ltd Onboard Ethernet [1458:e000]
    Kernel driver in use: r8169
    Kernel modules: r8168
06:00.0 USB controller [0c03]: Etron Technology, Inc. EJ168 USB 3.0 Host Controller [1b6f:7023] (rev 01)

$ lshw -C network
  *-network
       description: Wireless interface
       product: AR2413/AR2414 Wireless Network Adapter [AR5005G(S) 802.11bg]
       vendor: Qualcomm Atheros
       physical id: 6
       bus info: pci@0000:04:06.0
       logical name: wlp4s6
       version: 01
       serial: 00:14:a5:cc:49:b3
       width: 32 bits
       clock: 33MHz
       capabilities: pm bus_master cap_list ethernet physical wireless
       configuration: broadcast=yes driver=ath5k driverversion=4.15.0-72-generic firmware=N/A ip=192.168.1.201 latency=168 link=yes maxlatency=28 mingnt=10 multicast=yes wireless=IEEE 802.11
       resources: irq:20 memory:fe200000-fe20ffff  
  *-network
       description: Ethernet interface
       product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 0
       bus info: pci@0000:05:00.0
       logical name: enp5s0
       version: 06
       serial: 94:de:80:28:a2:71
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=2.3LK-NAPI duplex=full firmware=rtl8168e-3_0.0.4 03/27/12 ip=192.168.1.200 latency=0 link=yes multicast=yes port=MII speed=1Gbit/s
       resources: irq:16 ioport:b000(size=256) memory:d2104000-d2104fff memory:d2100000-d2103fff

$ networkctl status enp5s0:
● 2: enp5s0
       Link File: /lib/systemd/network/99-default.link
    Network File: /run/systemd/network/10-netplan-enp5s0.network 
            Type: ether
           State: routable (configured)
            Path: pci-0000:05:00.0
          Driver: r8169
          Vendor: Realtek Semiconductor Co., Ltd.
           Model: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (Onboard Ethernet)
      HW Address: 94:de:80:28:a2:71 (GIGA-BYTE TECHNOLOGY CO.,LTD.)
         Address: 192.168.1.200
                  fe80::96de:80ff:fe28:a271
         Gateway: 192.168.1.1
             DNS: 8.8.8.8
                  8.8.4.4

$ networkctl status wlp4s6:
● 3: wlp4s6  
       Link File: /lib/systemd/network/99-default.link
    Network File: /run/systemd/network/10-netplan-wlp4s6.network
            Type: wlan
           State: routable (configured)
            Path: pci-0000:04:06.0
          Driver: ath5k
          Vendor: Qualcomm Atheros
           Model: AR2413/AR2414 Wireless Network Adapter [AR5005G(S) 802.11bg] (DX-WGDTC 802.11bg Wireless PCI Adapter)
      HW Address: 00:14:a5:cc:49:b3 (Gemtek Technology Co., Ltd.)
         Address: 192.168.1.201
                  fe80::214:a5ff:fecc:49b3
         Gateway: 192.168.1.1 (Actiontec Electronics, Inc)
             DNS: 8.8.8.8
                  8.8.4.4

Next morning it was locked and I got this:

tail syslog | grep error:

Feb 10 23:44:00 timberoakspetspa systemd-resolved[1160]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 10 23:44:00 timberoakspetspa systemd-resolved[1160]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 10 23:46:58 timberoakspetspa systemd-resolved[1160]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 10 23:46:59 timberoakspetspa systemd-resolved[1160]: message repeated 2 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Feb 10 23:50:32 timberoakspetspa systemd-resolved[1160]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 10 23:50:33 timberoakspetspa systemd-resolved[1160]: message repeated 3 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Feb 11 00:23:34 timberoakspetspa systemd-resolved[1160]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 00:23:35 timberoakspetspa systemd-resolved[1160]: message repeated 3 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Feb 11 02:14:02 timberoakspetspa systemd-resolved[1160]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 02:14:05 timberoakspetspa systemd-resolved[1160]: message repeated 8 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Feb 11 08:49:20 timberoakspetspa ntpd[3430]: error resolving pool 0.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Feb 11 08:49:36 timberoakspetspa ntpd[3430]: error resolving pool 1.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Feb 11 08:49:51 timberoakspetspa snapd[3273]: stateengine.go:150: state ensure error: cannot refresh snap-declaration for "core": Get https://api.snapcraft.io/api/v1/snaps/assertions/snap-declaration/16/99T7MUlRhtI3U0QFgl5mXXESAiSwt776?max-format=3: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Feb 11 08:49:56 timberoakspetspa ntpd[3430]: error resolving pool 2.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Feb 11 08:50:16 timberoakspetspa ntpd[3430]: error resolving pool 3.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Feb 11 08:50:37 timberoakspetspa ntpd[3430]: error resolving pool ntp.ubuntu.com: Temporary failure in name resolution (-3)
Feb 11 08:50:57 timberoakspetspa ntpd[3430]: error resolving pool 0.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Feb 11 08:51:17 timberoakspetspa ntpd[3430]: error resolving pool 1.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Feb 11 08:51:37 timberoakspetspa ntpd[3430]: error resolving pool 3.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3)
Feb 11 08:53:18 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:18 timberoakspetspa systemd-resolved[1162]: message repeated 2 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Feb 11 08:53:18 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:18 timberoakspetspa systemd-resolved[1162]: message repeated 3 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Feb 11 08:53:18 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:18 timberoakspetspa systemd-resolved[1162]: message repeated 2 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Feb 11 08:53:22 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:23 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:23 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:23 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:24 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:24 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:26 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:26 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:28 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:28 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:31 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:31 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:33 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:33 timberoakspetspa systemd-resolved[1162]: message repeated 3 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Feb 11 08:53:34 timberoakspetspa systemd-resolved[1162]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 11 08:53:40 timberoakspetspa systemd-resolved[1162]: message repeated 5 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]

Please edit your unreadable Question and format the computer text with the {} tool. — waltinator, Feb 07 '20 at 23:48
If I knew what that meant, I would gladly do so. As it is that's an unreadable answer telling me to use a tool where I can find no list of available tools or what thet do. I have entered a list of commands I tried before the system locked and the responses they elicited. many of those responses were reformatted here after posting. — MickG, Feb 09 '20 at 18:14
I found a guide page which let me format better but it gave no reference to a {} tool. — MickG, Feb 09 '20 at 18:25

score 0 · Answer 1 · answered Feb 11 '20 at 04:14

Thanks to whoever reformatted my original post to make the admins happier.
I finally tried what I saw as a risky strategy, since little was really working:
I rebooted and when the splash screen appeared I typed &ltesc&gt until I got a menu offering GNU GRUB options.
I chose 'Advanced options for Ubuntu.
This offered both available kernels in normal and (Recovery Mode) I chose Recovery Mode for the latest kernel, the one that froze after a few seconds.
I was now taken to a screen offering several choices. (see attachment).
I used: fsck - Check all file systems, then grub - Update grub boot loader, then dpkg - Repair broken packages. Followed by resume - Resume normal boot which booted to the server but not normally so I rebooted completely with sudo shutdown -r now. This b0oted the latest kernel: 4.15.0-76.86 and thus far it has worked without issue.
Hopefully this will help other idiots like myself facing issues that could really hurt the business they are using the server for.

Added next morning syslog to original. – MickG Feb 11 '20 at 14:40 — MickG, Feb 11 '20 at 14:40

Ubuntu Server 18.04 locking up

1 Answers1