I would like all of my windows to open in "cascade" or "un-maximize" mode. Seemingly randomly, the VLC window, document viewer, image viewer, and others will open and take up my entire screen. (perhaps it has something to do with the size of media I am viewing) How can I limit a maximum windows size the can open with?
Is there a better term for un-maximized windows?
By default, windows of a large size will automatically maximize indeed. This is controlled by a setting of the mutter window manager. You can turn that effect off with the command
gsettings set org.gnome.mutter auto-maximize false
That should solve your issue where you find that windows are maximized even if you did not explicitly do that yourself.
There is unfortunately little control on the default window size of a created window. This is in first instance determined by the application itself. Many applications "remember" sizes, Evince remembers sizes (annoyingly according to me) on a per document basis, others do not remember size and position and open each time the same way with some default size and a placement determined by the "smart" algorithm of mutter, or "centered" if you turn that option on in Gnome Tweaks.
There are old tools, i.e., devilspie and devilspie2, that allow to define rules so that newly created windows are sized and placed where you determine, but they are not that easy to work with.
Anyway, likely the configuration setting will already solve most of your current issue.
