Baseline
"Installing" a program is a broad term. The actual definition is
Place or fix (equipment or machinery) in position ready for use.
So basically, just having something in the right position and setup to be able to function and be used properly. This is the definition of the term that I'll be using throughout this answer.
Process
APT (Advanced Packaging Tool)
The actual "under-the-hood" process of apt-get is explained very well by muru in this post. This basically covers what the command does and what it accesses. I'm linking to his answer because I don't want to duplicate the same answer on a different question, I just want to reference it.
Basic installation concept
Theoretically, just moving the executable file to the bin folder and having your bash environment point to that folder as its PATH, as well as moving any additional files to another directory (usually /var/) and referenced by the executable should be fine to "install" something. But, the things that are downloaded and moved by a package varies greatly between different packages as I mention later in this answer. So, this process isn't especially useful unless you're installing either your own package that you know where to put the files (even then, you should probably package it into a .deb file for autonomy) or a very basic package.
Extra Files
APT (Advanced Packaging Tool)
This varies heavily for each package installed. This mostly deals with dependencies as these are the things that you aren't explicitly telling Ubuntu to install, but it needs to in order to properly install the package you specified. If you want to remove these files then, in theory, you would have to manually go through each dependency installed and remove them. But, there are two ways to deal with these files without doing that. apt-get comes with two useful commands,
apt-get remove --purge [package]apt-get autoremove
The first command tells Ubuntu to remove all configuration/extra files along with the core files of a specific package. However, this does not remove the dependencies that were installed with the package. To do that, we have to use our second command.
The second command tells Ubuntu to remove all orphaned packages. An orphaned package is a package that was installed as a dependency, but is no longer needed by any other packages. Doing both of these commands will remove all files that were initially downloaded/installed from the original installation.
There may be other minor things that are specific to the package installed that need to be manually reverted such as additional configuration files or default application associations, but those should be easy to reverse.
Basic removal concept
As stated in the Basic installation concept section above, you really just have to move all required files to the correct paths referenced by your bash and system environment. To rid the system of these files, just reverse the process by going back to whatever directory they were placed in and remove them from there. This usually doesn't apply to most packages installed, but it is the basic concept.
