After sluggish performance and freezing for two weeks I finally managed to nail down the culprit to Compiz, I found out it's vital to Unity and is basically an OpenGL wrapper for Ubuntu and Xorg.
How do I fix this? There's plenty of bug reports, but none of them give an indication on how to do a 'quick fix' before they fix the memory issue.
I also get 'System Error' messages during startup and the details show that Xorg is crashing.
Should I do a clean install?