While my PC is a pretty decent one and it came up with Ubuntu preinstalled the overheating problem was getting out of hand. I tried so many things like tlp
thermald
but none worked.
Turns out when NVIDIA is set on Performance Mode in NVIDIA X Server settings it causes the PC to try and use the GPU resources too much. And it was set to Performance Mode by default.
Inspired by this post I could fix it, now I don't even hear the fans and I have no issue running GPU intensive stuff (Not games thought)
For anyone out there having a similar issue please try this, it might save you a ton of time!