How to identify missing used memory

Question

I am running 22.04, and with the recent systemd-oomd issues 12 I am trying to figure out where my RAM is going. I have 16GB of RAM which should be plenty, but i'm constantly getting Chrome, Gnome Shell, Gnome Terminal's killed by oomd.

My process table (ps -ef) doesn't correspond to my Memory usage from /proc/meminfo, so I am trying to find other allocations of memory i'm not aware of to figure out where my RAM is being used. I am aware of ipcs, slabtop, but the memory allocations do not add up.

Based on my reading of the ps output (below) I'm only using 8.4gb of RAM, but when looking at meminfo I'm using 15+gb.

ps uax | awk '{ print $6 }' | sort -n | awk '{s+=$1} END {print s}'
4551528

slabtop -s c -o | tail -n +7 | awk '{s =s+$7} END {print s}'
1593640

ipcs

------ Message Queues -------- key msqid owner perms used-bytes messages ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x511c029a 196610 ruckc 600 16 1 0xca270022 196611 ruckc 600 65536 1 0x00000000 196632 ruckc 600 524288 2 dest ------ Semaphore Arrays -------- key semid owner perms nsems 0x00105b6b 2 root 600 1 0x00105b6c 3 root 666 2 0xcc270022 8 ruckc 600 1 0xcb270022 9 ruckc 600 1

cat /proc/meminfo

MemTotal: 16088108 kB MemFree: 4183228 kB MemAvailable: 5130228 kB Buffers: 1084 kB Cached: 2270828 kB SwapCached: 22000 kB Active: 620544 kB Inactive: 2647548 kB Active(anon): 138556 kB Inactive(anon): 1932644 kB Active(file): 481988 kB Inactive(file): 714904 kB Unevictable: 900756 kB Mlocked: 72936 kB SwapTotal: 2097148 kB SwapFree: 1967088 kB Dirty: 2856 kB Writeback: 0 kB AnonPages: 1875816 kB Mapped: 562296 kB Shmem: 1072444 kB KReclaimable: 135272 kB Slab: 1624092 kB SReclaimable: 135272 kB SUnreclaim: 1488820 kB KernelStack: 18176 kB PageTables: 35684 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 10141200 kB Committed_AS: 9789584 kB VmallocTotal: 34359738367 kB VmallocUsed: 591532 kB VmallocChunk: 0 kB Percpu: 10848 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 3318960 kB DirectMap2M: 13158400 kB DirectMap1G: 1048576 kB

free -h
total used free shared buff/cache available Mem: 15Gi 9.1Gi 4.0Gi 1.0Gi 2.3Gi 4.9Gi Swap: 2.0Gi 127Mi 1.9Gi

oomd log: journalctl -xeu systemd-oomd.service | grep Killed

Apr 13 10:51:39 laptop systemd-oomd[3562]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-11396.scope due to memory used (16109408256) / total (16474247168) and swap used (1982627840) / total (2147479552) being more than 90.00%
Apr 13 13:09:57 laptop systemd-oomd[3363]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (16195358720) / total (16474247168) and swap used (2053111808) / total (2147479552) being more than 90.00%
Apr 14 13:21:21 laptop systemd-oomd[3248]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-41482.scope due to memory used (16056369152) / total (16474247168) and swap used (1945407488) / total (2147479552) being more than 90.00%
Apr 18 09:58:14 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (16223395840) / total (16474243072) and swap used (2027683840) / total (2147479552) being more than 90.00%
Apr 18 09:58:15 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (16091160576) / total (16474243072) and swap used (2145886208) / total (2147479552) being more than 90.00%
Apr 18 09:58:15 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (16159051776) / total (16474243072) and swap used (2073485312) / total (2147479552) being more than 90.00%
Apr 18 09:58:15 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (16212561920) / total (16474243072) and swap used (2008236032) / total (2147479552) being more than 90.00%
Apr 18 09:58:15 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-org.gnome.Nautilus-0.scope due to memory used (15865094144) / total (16474243072) and swap used (1974681600) / total (2147479552) being more than 90.00%
Apr 20 16:00:06 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (16120066048) / total (16474243072) and swap used (1999306752) / total (2147479552) being more than 90.00%
Apr 20 16:04:18 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (16117116928) / total (16474243072) and swap used (2039713792) / total (2147479552) being more than 90.00%
Apr 26 08:07:19 laptop systemd-oomd[3457]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-code-2704032.scope due to memory used (16189804544) / total (16474243072) and swap used (1975377920) / total (2147479552) being more than 90.00%
May 03 12:17:16 laptop systemd-oomd[3253]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-jetbrains\x2dtoolbox-10574.scope due to memory used (15012593664) / total (16474185728) and swap used (1944969216) / total (2147479552) being more than 90.00%
May 04 22:31:52 laptop systemd-oomd[3253]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-1560985.scope due to memory used (15096467456) / total (16474185728) and swap used (2009534464) / total (2147479552) being more than 90.00%
May 05 12:34:52 laptop systemd-oomd[3354]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-47763.scope due to memory used (16104947712) / total (16474189824) and swap used (1948254208) / total (2147479552) being more than 90.00%
May 09 10:26:05 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (16139558912) / total (16474193920) and swap used (2147348480) / total (2147479552) being more than 90.00%
May 10 19:40:25 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-950608.scope due to memory used (16102866944) / total (16474193920) and swap used (2071912448) / total (2147479552) being more than 90.00%
May 10 19:49:30 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-16c14424-f9a9-4c04-8ff2-0755377a7539.scope due to memory used (16022802432) / total (16474193920) and swap used (1948741632) / total (2147479552) being more than 90.00%
May 11 09:00:04 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-71b574b8-17e5-408b-b91f-431a458d55bb.scope due to memory used (15930691584) / total (16474193920) and swap used (2041720832) / total (2147479552) being more than 90.00%
May 11 10:20:57 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-2201754.scope due to memory used (15575441408) / total (16474193920) and swap used (1939279872) / total (2147479552) being more than 90.00%
May 11 12:05:21 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-24f78600-4612-4805-9b29-a96ca4f432cb.scope due to memory used (15433670656) / total (16474193920) and swap used (1950101504) / total (2147479552) being more than 90.00%
May 11 12:05:51 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-2814868.scope due to memory used (15329484800) / total (16474193920) and swap used (1946755072) / total (2147479552) being more than 90.00%
May 11 12:06:29 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-netbeans-2803190.scope due to memory used (15826632704) / total (16474193920) and swap used (2145792000) / total (2147479552) being more than 90.00%
May 11 14:33:29 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-2909372.scope due to memory used (15613919232) / total (16474193920) and swap used (1933402112) / total (2147479552) being more than 90.00%
May 11 14:56:16 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (16104726528) / total (16474193920) and swap used (1968328704) / total (2147479552) being more than 90.00%
May 12 08:25:14 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-jetbrains\x2dpycharm-3483077.scope due to memory used (15616532480) / total (16474193920) and swap used (1937936384) / total (2147479552) being more than 90.00%
May 12 11:15:23 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-3361444.scope due to memory used (15812100096) / total (16474193920) and swap used (1955221504) / total (2147479552) being more than 90.00%
May 12 17:58:07 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-722172.scope due to memory used (16039403520) / total (16474193920) and swap used (1988845568) / total (2147479552) being more than 90.00%
May 13 11:30:57 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-1218538.scope due to memory used (16068583424) / total (16474193920) and swap used (2087342080) / total (2147479552) being more than 90.00%
May 13 11:30:59 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (16112533504) / total (16474193920) and swap used (1946112000) / total (2147479552) being more than 90.00%
May 18 15:47:46 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-921c3683-819b-4f55-ba40-5bcd85681825.scope due to memory used (16058290176) / total (16474193920) and swap used (2002198528) / total (2147479552) being more than 90.00%
May 20 09:03:27 laptop systemd-oomd[847696]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-1534842.scope due to memory used (15276462080) / total (16474193920) and swap used (1937162240) / total (2147479552) being more than 90.00%
Jun 03 12:15:36 laptop systemd-oomd[3298]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-jetbrains\x2dwebstorm-541749.scope due to memory used (15719739392) / total (16474193920) and swap used (1967783936) / total (2147479552) being more than 90.00%
Jun 08 00:21:43 laptop systemd-oomd[3768]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-google\x2dchrome-16072.scope due to memory used (15445958656) / total (16474218496) and swap used (1942192128) / total (2147479552) being more than 90.00%
Jun 08 07:14:28 laptop systemd-oomd[3768]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-teams-2369963.scope due to memory used (15633932288) / total (16474218496) and swap used (1964466176) / total (2147479552) being more than 90.00%
Jun 08 07:16:38 laptop systemd-oomd[3768]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-jetbrains\x2dgoland-2978322.scope due to memory used (15238217728) / total (16474218496) and swap used (1939480576) / total (2147479552) being more than 90.00%
Jun 13 10:14:46 laptop systemd-oomd[7467]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (15284772864) / total (16474218496) and swap used (1937956864) / total (2147479552) being more than 90.00%

Answer 1

Checking the logs it seems that you are using netbeans, pycharm, etc... and they also got killed due to OOM. It looks like you're using fairly more memory than you have. Java processes for netbeans of pycharm are using probably more memory than you think. You can enlarge or strict the memory reserved for Java for these applications.

• Can you run free -h? We can see easily what available memory you have, what you have in buffers...

• Information about chrome or any process given its PID can be found in /proc/PID/stat (like virtual memory size, or resident set). Explanation of each field in: http://brokestream.com/procstat.html

• htop may come handy, as you can sort by memory usage, group

• you may be interested into checking how often memory is transferred from/to Swap. If processes are competing for memory, it may happen that the system is very busy transferring between/to swap and unable to allocate resources fast enough, ending with OOM. You may check pswpin and pswpout from /proc/vmstat. I've a small program that does that in here: https://blog.carlesmateo.com/2021/05/27/swap-swappiness-servers-not-responding/

• Can you check what plugins are you using for chrome?. 16GB of RAM may sound like a lot but is not that much. Browsers require a lot of memory. Also many times, ads, use a lot of resources.

My advice is to use few tabs with Chrome, specially if you program, as it's easy to get out of memory.

Update 2022-06-23:

• I see from free -h output that there is some swap at use. Please check your swappiness value cat /proc/sys/vm/swappiness

• Swappiness defines the balance between memory used for improving IO versus taking memory from swap for that reason. Basically, if you have a swappiness of 60, you can expect that when 60% of your RAM memory is in use, swap will be used to provide more memory for buffers. That IMO is a huge mistake, as the performance penalty from using swap is higher than the benefits from using cache/buffer. In your case you have almost 5GB of available memory, but your system used (slow) swap. I would set the swappiness to 10 and I expect an improvement of your experience and may be resolution of your problem. sudo sysctl vm.swappiness=10 If you're happy with this new value change it permanently in /etc/sysctl.conf

• You can use iotop --only to see how much bandwidth is wasted writing and reading from/to swap in real time