You first need to define what considers "break my system". A system without a desktop is not broken as is.
Besides that: the safest way is what you did and to take a look at this by "category" from Ubuntu Software Center and the "installed" group.
Accessories
Books & Magazines
Developer Tools
Education
Fonts
Games
Graphics
Internet
Medicine
Office
Science & Engineering
Sound & Video
Themes & Tweaks
Universal Access
Besides "accessories" any of these should be safe to remove and will not remove anything important since they all point to applications. Remove them 1 by 1 and do pay attention to the list of packages it is offering to delete just in case.
And when in doubt check the packages online for what is depending.
1 more comment: what you did was probably overkill; to get those icons back in system settings all you had to do is re-install that specific packages. This should have been enough to fix it:
sudo apt-get remove unity-control-center
sudo apt-get install unity-control-center