Prevent Ubuntu from freezing, even if system memory is low

Question

Sometimes I work with huge dumps of data I want to keep in memory for processing. Sometimes I miscalculate the amount of memory my program will produce, or a debugger multiplies the memory usage by a factor that exceeds my available memory.

Whenever I start a memory-hungry process, this is what I'd expect from a sane operating system: try to eat all free memory, then ask some other non-essential processes nicely to give up some memory they don't need, then write to swap.

Here's what Ubuntu does for me: eat all fre memory, then ask the operating system to swap all essential services (gnome session, terminal, keyboard), then freeze and wait for me to pull the power plug.

Two questions:

1. How can an operating system assume, that anything could be so important that it is ok to stop listening to user input?
2. How can I tell Ubuntu to never swap essential services and always react to user input, even if some stupid process tries to eat up more resources than the system provides.

Answer 1

I still don't have a solution for the problem, but I can offer two workarounds which might be of interest to others:

1) earlyoom

That's a service that watches memory usage and kills the process that consumes the most memory when a certain threshold is reached (see also this and this question regarding the OOM killer in the linux kernel)

I've tested it with a demo process that indefinetly requests memory in small chunks. Here's my first impression: When I start the rogue process, it quickly eats up all of my RAM. Then swapping begins, the system becomes inresponsive. A few seconds later the system is back online. The log of earlyoom shows that it killed the memory eating process after both memory and swap usage reached 90%.

There is still the annoying lag when swapping begins and after the process was killed, some portions of other processes usually remain in swap until they are requested, but it's a start.

2) just disable swap

I know this is a controversial topic, but for the purpose of desktop systems and especially development machines where it can happen from time to time that a process tries to eat up all your memory, it makes sense: Without swap, the OOM killer just works as intended. When you run out of memory, it finds the best process to kill and gets rid of it. No lag, no delay.

You can disable swap for the current session with sudo swapoff -a or make the change permanent.

---

The proper solution for the problem would of course be that the system stays responsive when the main memory is depleted and it starts to swap memory like there's no tomorrow, but that doesn't seem to be happening any time soon.

Answer 2

From +22.04 LTS

There's a service called systemd-oomd that will automatically monitor memory usage and attempt to kill processes when out of memory and it is installed by default as a part of systemd. To view the current status type oomctl. Example output:

Dry Run: no
Swap Used Limit: 90.00%
Default Memory Pressure Limit: 60.00%
Default Memory Pressure Duration: 20s
System Context:
        Memory: Used: 4.1G Total: 15.5G
        Swap: Used: 1.0G Total: 3.9G
Swap Monitored CGroups:
        Path: /
                Swap Usage: (see System Context)
Memory Pressure Monitored CGroups:
        Path: /user.slice/user-1000.slice/user@1000.service
                Memory Pressure Limit: 50.00%
                Pressure: Avg10: 0.00 Avg60: 0.02 Avg300: 0.00 Total: 9s
                Current Memory Usage: 3.9G
                Memory Min: 0B
                Memory Low: 0B
                Pgscan: 23663084
                Last Pgscan: 23663084

By default, start reclaiming at 90% usage of both memory and swap. It uses memory pressure on user slices control groups (i.e. user@1000.service) which is where most application spawn, but not on system.slice control group, which spawns the most crucial components of the system such as NetworkManager and Gnome Display Manager. In short, oomd will free user processes before system ones. To view the current configuration of systemd-oomd, type:

systemd-analyze cat-config systemd/oomd.conf

Refer to this page for a reference on what they do. To edit the OOM parameters of a specific systemd unit, add or edit the variables specified here in the unit file.

Answer 3

Try one of two things:

1) change the swappiness setting from its default setting of 60, to 10, ie: add vm.swappiness = 10 to /etc/sysctl.conf (in terminal, type sudo gedit /etc/sysctl.conf), then reboot the system. Search here for swappiness for more info about it.

2) If swappiness doesn't help... even though you may not want to... increase the size of your swapfile to 1.5x16G and see if that helps.

Keep me posted. Cheers, Al