Kubuntu 22.04 - wifi freeze after hwe kernel upgrade to 6.5 (was fine on 6.2)

Question

Since my Kubuntu 22.04 got the HWE kernel update from the 6.2 series(6.2.0-39) to the 6.5 series(6.5.0-14), I'm experiencing short ~30s WiFi hangs. Disconnecting and reconnecting from the network usually fixes them, and if I don't reconnect to the network, they resolve by themself after a while.

There is usually one hang ~5-10 minutes after booting up, and more randomly spaced hangs later on. The hangs are especially noticeable if I switch from an offline activity to the browser and try to open a website, only to find it not loading. For example, the "Similar questions" section randomly stopped loading while I was typing the question. I've not yet experienced the issue when playing real-time online games, leading me to think the issue might be driven by slow low-power mode exit in my wifi card. How do I investigate that?

Looking at dmesg, the odd things I can see is

[  226.169003] warning: `kded5' uses wireless extensions which will stop working for Wi-Fi 7 hardware; use nl80211
[  745.934898] workqueue: pm_runtime_work hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND
[ 1482.423020] workqueue: pm_runtime_work hogged CPU for >10000us 16 times, consider switching to WQ_UNBOUND

I'm digging through journalctl -b -u NetworkManager output but there is nothing obvious. Further debugging suggestions are welcome. Were there any significant changes in mt7921e power management between 6.2 and 6.5?

Not seeing anything obvious in https://www.phoronix.com/review/linux-65-features/2, https://www.phoronix.com/review/linux-64-features or https://www.phoronix.com/review/linux-63-features/2. The most suspicious change is BIG TCP enablement in 6.3, but I'm not sure how to check if it's enabled.

The symptoms correspond to https://community.frame.work/t/tracking-unstable-and-unreliable-wlan-rz616-mt7922-fw13-amd-diy/40316 and https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/2049220

System specifications:

lizsquir@PiMutant:~$ inxi -Fz
System:
  Kernel: 6.5.0-14-generic x86_64 bits: 64 Desktop: KDE Plasma 5.24.7
    Distro: Ubuntu 22.04.3 LTS (Jammy Jellyfish)
Machine:
  Type: Laptop System: ASUSTeK product: ROG Strix G513IH_G513IH v: 1.0
    serial: <superuser required>
  Mobo: ASUSTeK model: G513IH v: 1.0 serial: <superuser required>
    UEFI: American Megatrends LLC. v: G513IH.301 date: 01/27/2021
Battery:
  ID-1: BAT0 charge: 44.3 Wh (100.0%) condition: 44.3/56.0 Wh (79.0%)
CPU:
  Info: 8-core model: AMD Ryzen 7 4800H with Radeon Graphics bits: 64
    type: MT MCP cache: L2: 4 MiB
  Speed (MHz): avg: 1211 min/max: 400/4300 cores: 1: 1396 2: 1801 3: 1397
    4: 400 5: 1439 6: 400 7: 400 8: 400 9: 1520 10: 3432 11: 1524 12: 400
    13: 3680 14: 400 15: 400 16: 400
Graphics:
  Device-1: NVIDIA TU117M [GeForce GTX 1650 Mobile / Max-Q] driver: nvidia
    v: 535.146.02
  Device-2: AMD Renoir driver: amdgpu v: kernel
  Display: x11 server: X.Org v: 1.21.1.4 driver: X:
    loaded: amdgpu,ati,nvidia unloaded: fbdev,modesetting,nouveau,vesa
    gpu: amdgpu resolution: 1920x1080~144Hz
  OpenGL: renderer: RENOIR (renoir LLVM 15.0.7 DRM 3.54 6.5.0-14-generic)
    v: 4.6 Mesa 23.0.4-0ubuntu1~22.04.1
Audio:
  Device-1: NVIDIA driver: snd_hda_intel
  Device-2: AMD Renoir Radeon High Definition Audio driver: snd_hda_intel
  Device-3: AMD Raven/Raven2/FireFlight/Renoir Audio Processor driver: N/A
  Device-4: AMD Family 17h HD Audio driver: snd_hda_intel
  Sound Server-1: ALSA v: k6.5.0-14-generic running: yes
  Sound Server-2: PulseAudio v: 15.99.1 running: yes
  Sound Server-3: PipeWire v: 0.3.48 running: yes
Network:
  Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
    driver: r8169
  IF: enp2s0 state: down mac: <filter>
  Device-2: MEDIATEK MT7921 802.11ax PCI Express Wireless Network Adapter
    driver: mt7921e
  IF: wlp3s0 state: up mac: <filter>
Bluetooth:
  Device-1: IMC Networks Wireless_Device type: USB driver: btusb
  Report: hciconfig ID: hci0 state: up address: <filter> bt-v: 3.0
Drives:
  Local Storage: total: 476.94 GiB used: 154.59 GiB (32.4%)
  ID-1: /dev/nvme0n1 vendor: Samsung model: MZVLQ512HBLU-00B00
    size: 476.94 GiB
Partition:
  ID-1: / size: 234.88 GiB used: 86.01 GiB (36.6%) fs: btrfs
    dev: /dev/nvme0n1p2
  ID-2: /boot/efi size: 511 MiB used: 23.1 MiB (4.5%) fs: vfat
    dev: /dev/nvme0n1p1
  ID-3: /home size: 234.88 GiB used: 86.01 GiB (36.6%) fs: btrfs
    dev: /dev/nvme0n1p2
Swap:
  ID-1: swap-1 type: partition size: 7.18 GiB used: 764.5 MiB (10.4%)
    dev: /dev/nvme0n1p3
Sensors:
  System Temperatures: cpu: 52.0 C mobo: N/A gpu: amdgpu temp: 48.0 C
  Fan Speeds (RPM): cpu: 0
Info:
  Processes: 415 Uptime: 3h 3m Memory: 7.17 GiB used: 4.28 GiB (59.6%)
  Shell: Bash inxi: 3.3.13

Answer 1

Seems like there is an upstream fix, and Ubuntu is backporting it. You can find the proposed package in https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/2049220 and install it manually if you don't want to wait for the backport being released. It did solve my WiFi freezing issues.

The fix is to update the firmware - if your firmware build time shows the 2nd half of 2023, you're good:

lizsquir@PiMutant:~$ journalctl -k -o short-precise -b 0 | grep mt
Feb 17 09:44:20.551798 PiMutant kernel: mt7921e 0000:03:00.0: enabling device (0000 -> 0002)
Feb 17 09:44:20.559799 PiMutant kernel: mt7921e 0000:03:00.0: ASIC revision: 79610010
Feb 17 09:44:20.651779 PiMutant kernel: mt7921e 0000:03:00.0: HW/SW Version: 0x8a108a10, Build Time: 20231109190918a
Feb 17 09:44:20.915856 PiMutant kernel: mt7921e 0000:03:00.0: WM Firmware Version: ____010000, Build Time: 20231109190959

The firmware issue triggered by newer kernels can be seen in firmware built in 2022:

lizsquir@PiMutant:~$  journalctl -k -o short-precise -b 10 | grep mt
Jun 14 19:33:38.444998 PiMutant kernel: mt7921e 0000:03:00.0: enabling device (0000 -> 0002)
Jun 14 19:33:38.464979 PiMutant kernel: mt7921e 0000:03:00.0: ASIC revision: 79610010
Jun 14 19:33:38.553002 PiMutant kernel: mt7921e 0000:03:00.0: HW/SW Version: 0x8a108a10, Build Time: 20220209150832a
Jun 14 19:33:38.813494 PiMutant kernel: mt7921e 0000:03:00.0: WM Firmware Version: ____010000, Build Time: 20220209150915
Jun 14 19:33:39.660991 PiMutant kernel: mt7921e 0000:03:00.0 wlp3s0: renamed from wlan0

journalctl -k -o short-precise is just another way of executing dmesg, except it doesn't require sudo rights. The value after -b is which boot you want to view logs from - 0 is current boot (default value), positive values are boots since the system was installed, negative values are boots relative to current.

Edit: The updated package got released last week