TL;DR: Looks like it might be an issue with X11 and external primary displays.
I have to give credit here to both @Tiago and @hadarS as it was both of their answers combined that led me to a solution.
I'm using an MSI Bravo 15 (AMD Ryzen 7 and Radeon RX 5500M) with a (relatively) fresh install of KDE Neon 5.21. The current 20.04 LTS kernel doesn't support my driver so I've updated to the mainline kernel 5.10 and installed the AMD 20.45 firmware. Yet, I was still finding that my external display (4K) was horrendously slow at doing some of the most basic tasks while doing the same task on the main display was as responsive as ever.
@hadarS mentioned changing their primary display back to the laptop screen so I checked that and, lo and behold, the 4k screen was set as the primary. Switched it to the laptop screen and immediately the external display became usable. Great! Fixed it! But why does it happen?
This is where @Tiago's answer came in handy (as well as your information in your question, OP).
- The obvious info is that in cases where there was slow performance, an external monitor is used, in some cases without the internal one. That's a given.
- OP is using an Nvidia card, I'm using an AMD card, @Tiago is using an external AMD card. Same issue in all three cases so we can rule out both the card manufacturer and the bus for the GPU.
- @Tiago mentioned GNOME specifically, I'm assuming OP is using GNOME as well. I'm using KDE Plasma 5. There goes the DE as a source of the issue.
So what's left?
Well, because Wayland doesn't seem to recognise my external display, I'm using X11. C'est la vie. Without knowing whether everyone else is also using X11 - I'm assuming so since AFAIK X11 is still the default display server for Ubuntu 20.04 - I'm going to have to simply assume that this is our common factor. It seems that X11 just doesn't like having an externally connected display set as the primary display.
I have another laptop with a "fresh" Neon install and an Nvidia GPU. I'll test this theory later tonight if I remember.