I've had it happen a few times -- e.g. while running a python program that parses a large file in memory, or using Firefox to share a screen while a lot of docker containers were running -- that Gnome would get to the state where I can't do anything and I have to reboot the computer.
This "state where I can't do anything" isn't a crash -- I can move the mouse around, and if there was audio running the audio keeps running -- but I can't open a new terminal, or type anything on an existing terminal.
Is there something I can set that forces Ubuntu to leave enough memory to be able to run Gnome (e.g. open terminals, kill other processes) even if another process is attempting to consume all the memory on the machine? I'd prefer the other process to get an out of memory error (which it can handle itself), even if it hasn't consumed literally all the memory.
This system has 8GB of RAM and 1GB of swap. The swap is a swap partition, not a file.
An answer to the question OOM killer not working suggests a possible cause and partial fix: if vm.oom_kill_allocating_task is not set to 1, scanning the system to try to find a process to kill may take too long to succeed, making the system unusable. Comments to that answer suggest vm.admin_reserve_kbytes should do what I want if it worked, but unfortunately it apparently does not.
What I'm seeing is apparently a bug fixed in 2014 that has resurfaced.
