5

My Ubuntu 18.04 (Bionic Beaver) installation crashed because of a problem with out of memory.

I have messages like this:

Jan 31 16:56:53 adam-Lenovo-YOGA-720-15IKB kernel: [47980.843963] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=chrome,pid=16144,uid=1000
Jan 31 16:56:53 adam-Lenovo-YOGA-720-15IKB kernel: [47980.843992] Out of memory: Killed process 16144 (chrome) total-vm:1229048kB, anon-rss:84920kB, file-rss:0kB, shmem-rss:140kB
Jan 31 16:56:53 adam-Lenovo-YOGA-720-15IKB kernel: [47980.895646] oom_reaper: reaped process 16144 (chrome), now anon-rss:0kB, file-rss:0kB, shmem-rss:140kB
Jan 31 17:07:20 adam-Lenovo-YOGA-720-15IKB kernel: [    0.000000] microcode: microcode updated early to revision 0xca, date = 2019-10-03
Jan 31 17:07:20 adam-Lenovo-YOGA-720-15IKB kernel: [    0.000000] Linux version 5.3.0-28-generic (buildd@lcy01-amd64-009) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020 (Ubuntu 5.3.0-28.30~18.04.1-generic 5.3.13)
Jan 31 17:07:20 adam-Lenovo-YOGA-720-15IKB kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.3.0-28-generic root=UUID=e8808971-420e-45c4-96d7-2f490862bf41 ro quiet splash vt.handoff=1
Jan 31 17:07:20 adam-Lenovo-YOGA-720-15IKB kernel: [    0.000000] KERNEL supported cpus:

I have all the times a lot of programs running (Firefox with multiple windows and tabs, Chromium, Chrome, GIMP, Emacs, LibreOffice, and VMware Player for Windows 7).

Today I have 8 GB of RAM and 8 GB of swap file:

adam@adam-Lenovo-YOGA-720-15IKB:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.5G        3.7G        351M        692M        3.4G        2.8G
Swap:          8.0G          0B        8.0G

Can we have some dynamic swap on Linux instead of limiting it to a preallocated size?

Swappiness:

adam@adam-Lenovo-YOGA-720-15IKB:~$ sysctl vm.swappiness
vm.swappiness = 10
Peter Mortensen
  • 884
London Smith
  • 276
  • Edit your question and show me sysctl vm.swappiness. – heynnema Jan 31 '20 at 17:02
  • @heynnema I have seen a video on YT and changed it actually: https://youtu.be/BLVtxpm5c2A?t=660 – London Smith Jan 31 '20 at 17:17
  • 2
    the suggestion is to enable zram. It's much faster than normal disk swap. See also Does Ubuntu support dynamic swap file sizing?. And it's GB or GiB, not Gb which means Gigabit and is only 1/8 of the GB – phuclv Feb 01 '20 at 04:00
  • I've made a minor correction in my answer. Please note. – heynnema Feb 01 '20 at 13:42
  • When you hit an OOM-killer condition, was most of your swap space full? Linux shouldn't OOM kill anything until it runs out of swap space, for any non-zero swappiness. (Did you accept heynnema's answer just for (correctly) saying you need more RAM to run your workload, or did the swappiness change really make a difference?) And BTW, the log you showed didn't "crash Ubuntu", it crashed one big program running under Ubuntu. Do you sometimes find your X server or window manage getting OOM killed? – Peter Cordes Feb 01 '20 at 15:56
  • @PeterCordes with vm.swappiness=10, swap would barely be used. – heynnema Feb 01 '20 at 16:00
  • @heynnema ... until you're out of RAM. Try it if you have a system with any swap space. I run low swappiness and my 16GB system definitely will still swap if I run a simple benchmark loop that really uses most of my RAM. – Peter Cordes Feb 01 '20 at 16:02
  • @PeterCordes Correct. I also have 16G RAM and vm.swappiness set at 10. And yes, it will swap if RAM is full. It also reduces wear on my SSD. – heynnema Feb 01 '20 at 16:04
  • Status please. Did you note the minor edit that I did to my answer? Did changing vm.swappiness help your situation at all? – heynnema Feb 05 '20 at 13:52
  • Status please... – heynnema Feb 07 '20 at 16:04

3 Answers3

7

You watched a YouTube video that discussed the parameter vm.swappiness, and then set your vm.swappiness=10, which is totally wrong for your system with 8G RAM. That's why you're "running out of memory".

Set vm.swappiness=80 (based on 8G RAM and 8G SWAP), this way...

sudo -H gedit /etc/sysctl.conf # edit this file

Search for an existing vm.swappiness= entry...

CTRL+f vm.swappiness

  • If found, edit it to say vm.swappiness=80

  • If not found, add vm.swappiness=80 at the end of the file

Save your edits and quit gedit

sudo sysctl -p

Note: Based on what apps you're running, you might also be best served by adding more RAM, and changing vm.swappiness to a different value. To get the speed advantage of memory interleaving, do it in equal size RAM stick pairs, so in your case, add another 8G RAM stick (if you have one 8G RAM stick installed now). If you have two 4G sticks installed now, then you'll need two 8G RAM sticks.

heynnema
  • 70,711
  • Your second command would overwrite what the first one did if there's a setting for vm.swappiness somewhere in /etc/sysctl.*. – Ruslan Feb 01 '20 at 07:56
  • @Ruslan The first command writes the value to /etc/sysctl.conf, the second command "Load in sysctl settings from the file specified or /etc/sysctl.conf if none given", as per man sysctl. – heynnema Feb 01 '20 at 12:40
  • No, it never changes the files. What this option (effectively) writes to is /proc/sys/vm/swappiness, which doesn't preserve its values across reboot — only applies to the currently-running kernel. – Ruslan Feb 01 '20 at 12:42
  • @Ruslan Is there another way to get it into /etc/sysctl.conf without a manual edit? – heynnema Feb 01 '20 at 12:59
  • You can create your own setting file without need to edit an existing one. Put it into /etc/sysctl.d/ (you'll find some files already there, take them as examples). Then you can load all via sysctl --system (it's done automatically on boot). – Ruslan Feb 01 '20 at 13:04
  • @Ruslan See if my edit works for you. Thanks for the feedback! – heynnema Feb 01 '20 at 13:17
  • 1
    Do you have any evidence that Linux will invoke the OOM killer instead of swapping if there is swap available, for any non-zero swappiness? That hasn't been my experience; low swappiness just favours keeping anonymous pages of processes over more space for the pagecache. Agreed that 8GB is simply not enough RAM (and more than 8GB of swap might avoid OOM, but not with acceptable performance). But I highly doubt that changing swappiness without changing anything else would avoid the OOM killer, unless there have been massive changes to what it means in the last couple years. – Peter Cordes Feb 01 '20 at 15:53
  • @PeterCordes Low vm.swappiness favors RAM (which was the OP's mistake with 8G RAM), and higher vm.swappiness favors swap, and in this case, stops the out of memory condition (but can increase swapping). 60 is the default, and I set it higher (in this case). – heynnema Feb 01 '20 at 15:56
  • @heynnema: High vm.swappiness favours swapping things when far from OOM out to make room for more pagecache. Low vm.swappiness will still swap out if it has to. See https://www.kernel.org/doc/Documentation/sysctl/vm.txt. It's how aggressive the kernel is at swapping early. Not whether to swap at all. e.g. high swappiness can be good for server throughput, bad for desktop latency. The tradeoff doesn't include more OOM-killer invocations, unless the OOM killer activates sometimes instead of waiting for a lot of pages to swap out. – Peter Cordes Feb 01 '20 at 16:00
  • @PeterCordes re: "Do you have any evidence that Linux will invoke the OOM killer instead of swapping if there is swap available" in reading the OP's error messages, OOM was called even though there was plenty of swap (when vm.swappiness was set at 10). – heynnema Feb 01 '20 at 16:11
  • @heynnema: The OP's free output is either after OOM killed something, or more likely at a totally different time (like after a reboot). If you run the test program from Out of swap - what happens? on a system with low vm.swappiness, you'll see that it will fill up your swap space before OOM kicks in. I think even with swappiness=0. – Peter Cordes Feb 01 '20 at 18:18
  • 1
    I'm with @PeterCordes, this answer is wrong except for the last paragraph. Non-zero swappiness will never result in the OOM killer invoked, it merely controls the eagerness of choosing swap versus RAM. From the OOM point of view, RAM and swap are the same thing, a machine can't "run out of memory" while there's some swap space available. – Sergey Feb 05 '20 at 00:39
  • @Sergey I understand... but then, why was OOM called to kill Chrome? There was 8G RAM and 8G swap. Hard to believe that Chrome couldn't run in that, no? – heynnema Feb 05 '20 at 01:53
  • It wasn't Chrome alone. To quote the OP: "... a lot of programs running (Firefox with multiple windows and tabs, Chromium, Chrome, GIMP, Emacs, LibreOffice, and VMware Player for Windows 7)" - there's nothing strange or unusual that it ran out of memory. VMWare alone was probably using half of the available memory. – Sergey Feb 05 '20 at 02:14
5

An answer from Out of swap - what happens? explains dynamically sizing swap space is perilous and offers a way to test that hypothesis.

There was a package, SwapSpace, which you may run into, but it is no longer maintained.

An alternative dphys-swapfile service, detailed at Dynamically growing swap file on Debian may not be as dynamic as you need.

The solution from heynnema is superior and should be implemented before anything else.

Peter Mortensen
  • 884
K7AAY
  • 17,202
  • If @RonJohn's answer works (creating a swap file with fallocate instead of dd), dynamic swap file creation barely adds any I/O pressure when the system is running low on memory. The answer you linked says that swapping at all is perilous and recommends not using any swap space. That's not an option for the OP unless they add more RAM; they're hitting the OOM killer with 8GB of always-available swap. – Peter Cordes Feb 01 '20 at 15:48
3

Naturally, adding more RAM is the best solution.

However, sometimes that's impractical, as it used to be for me. Because of that, I would sometimes enable a swap file when the system started to get really sluggish. That would give me some time to kill what I wanted killed.

https://linuxize.com/post/create-a-linux-swap-file/

  1. sudo fallocate -l 4G /swapfile # or whatever you want to name it, and wherever you want to place it.
  2. sudo chmod 600 /swapfile
  3. sudo mkswap /swapfile

Then, when the system gets sluggish, run sudo swapon /swapfile. When you deem it not necessary anymore, run sudo swapoff /swapfile.

EDIT: fallocate is know to work on ext4, but fail on XFS. dd if=/dev/zero of=/swapfile bs=1M count=4096 works on XFS.

RonJohn
  • 679
  • When did mkswap / swapon start working on preallocated unwritten extents? It didn't used to; that's a really nice change that makes on-the-fly swap file creation practical. (e.g. auto-managed by a daemon; Dynamically growing swap file on Debian) – Peter Cordes Feb 01 '20 at 15:45
  • @PeterCordes does my answer imply that it does? I don't think so, since Step #1 says to preallocate a swap file, but maybe wasn't clear. – RonJohn Feb 01 '20 at 16:53
  • Yeah, fallocate -l preallocates but doesn't zero the underlying extents. These are what's called "unwritten extents"; they read as zero when read through the FS (like holes), but the kernel doesn't do that after swapon. That's why this didn't used to work. (But of course exposing previous non-zeroed free space contents is not actually a security risk; we're exposing it to the kernel.) Oh, apparently this has worked for a while with EXT4. It was a problem with XFS, maybe still is, see this comment for links to a mailing list post. – Peter Cordes Feb 01 '20 at 18:59
  • @PeterCordes interesting. I just confirmed that while it does work with ext4, sudo swapon with an XFS volume results in swapon: /Data/swapfile: swapon failed: Invalid argument. Thanks. – RonJohn Feb 01 '20 at 19:35
  • After you dd or fallocate your file, chmod it, and mkswap it.. if you want Linux to keep using it after reboot, for a swapfile called /swapfile you can add a line to /etc/fstab like "/swapfile none swap sw 0 0" . It's then turned on automatically during boot. Sometimes I've preferred leaving a swapfile out of fstab, I just put "swapon /swapfile" into /etc/rc.local instead. – hwertz Feb 28 '20 at 02:22