1

I am using Xubuntu 16.04 on my laptop, installed from a dist-upgrade of a Xubuntu 14.

The laptop has 10 GB RAM and 4 GB Swap.

Recently I upgraded the kernel to version: 4.15.0-51.

The problem is that during normal operation the system is swapping-out over time but never swapping back in. So after hours/days of uptime, the swap space gets full.

Even after closing all the main processes the RAM usage goes to a few hundred MB but the swap remains full. Also, looping over all the /proc/*/stats and grep VmSwap return 0 kB for all processes.

How can I debug why the kernel is not swapping in?

It's like the kernel is losing track of what processes the pages on the swap belong to.

I've already invested a significant amount of time trying to fix this with no success. The only turn around I have for now is a script that is creating and recycling (swapoff ; swapon) a few swap files.

Thank you for any help.

Zanna
  • 70,465
Fininho
  • 21
  • 1
    Possible duplicate of How to empty swap if there is free RAM? – Fabby Jun 19 '19 at 12:50
  • Not exactly the same. I can live with this issue, have a script to run "sudo swapoff -a; sudo swapon -a", in fact, I made a script to manage a few swap files to be sure I don't get caught in the irresponsive world of an "out of RAM and out of Swap". What I want to point out is that I have 3 machines using Xubuntu 16.04 and on two of them if I close firefox, for example, I see the swap being swapped-in, however on the third machine that does not happen, even if it has 1BG used RAM and 9GB free RAM. I really think that this machine has a software issue but I am falling to debug it. – Fininho Jun 19 '19 at 17:26

2 Answers2

1

Today I gave it another try and found the problem.

Started by checking again the meminfo information. Here is the output of cat /proc/meminfo:

MemTotal:       10133276 kB
MemFree:         2617372 kB
MemAvailable:    4660420 kB
Buffers:          503472 kB
Cached:          3825884 kB
SwapCached:        59448 kB
Active:          3852624 kB
Inactive:        3112996 kB
Active(anon):    2688920 kB
Inactive(anon):  2245220 kB
Active(file):    1163704 kB
Inactive(file):   867776 kB
Unevictable:          32 kB
Mlocked:              32 kB
SwapTotal:       6096888 kB
SwapFree:        2644324 kB
Dirty:                52 kB
Writeback:             0 kB
AnonPages:       2576972 kB
Mapped:           770460 kB
Shmem:           2297836 kB
Slab:             391072 kB
SReclaimable:     325960 kB
SUnreclaim:        65112 kB
KernelStack:       15824 kB
PageTables:        53492 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    11163524 kB
Committed_AS:   12259512 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:      2048 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      349640 kB
DirectMap2M:    10039296 kB

Despite the issue of having 4 GB of swap used by nothing, this time the 2GB of shmem used caught my attention.

Searching for that, I found this question with the answer: High SHMem memory usage!

So, in short, having high shmem usage can be related to tmpfs directories. In my case, /dev was using almost 4.8 GB

Here is the output of df -h:

Filesystem      Size  Used Avail Use% Mounted on
udev            4,8G  4,8G     0 100% /dev
tmpfs           990M   34M  956M   4% /run
/dev/sda2       255G  232G   11G  96% /
tmpfs           4,9G  220M  4,7G   5% /dev/shm
tmpfs           5,0M  4,0K  5,0M   1% /run/lock
tmpfs           4,9G     0  4,9G   0% /sys/fs/cgroup
/dev/sda5       333G  208G  126G  63% /media/dados
cgmfs           100K     0  100K   0% /run/cgmanager/fs
tmpfs           990M   88K  990M   1% /run/user/1000

After that I just searched in /dev for the culprit using the du command. As also referred to in the thread I linked above, it was the bootchart command that was storing huge log files inside the /dev directory.

Thanks for the help.

Zanna
  • 70,465
Fininho
  • 21
0

You don't need to worry about it. Have a look for example here. Kernel is tracking it and it can reause space in swap.

marosg
  • 1,303
  • You have enough rep to flag the question as a duplicate.... – Fabby Jun 19 '19 at 12:51
  • That's the point, I have two other machines were that is true, but on this one, I had been caught by out of RAM and Swap too many times. I think that on this machine the kernel is falling to tracking the swap pages, but I don't know how to debug it. Do you know how can I increase the verbose on the service that manages the memory/swap? – Fininho Jun 19 '19 at 17:35