2

I am running a python code on jupyter lab which deals with a large list of dataframes and I need to concatenate, compute averages, etc. It is memory consuming, and I take advantage of swapfiles, as memory size is not sufficient.

The problem I am observing is it crashes when memory usage reaches the end of the first swap file.

It appears I am missing something here... I found references, using two swapfiles are possible here, here and here. And this one even says amounts high as 29 swap files are possible!

  • my first swapfile is at the root (/swapfile).
  • The second swapfile I tried was at home (~/swapfile2). The 'home' is at a secondary device (SSD) mounted as /home.
  • my primary device (/) is a NVMe device (M.2 SSD)
  • if I deactivate/remove the ~/swapfile2, it works fine (if I do not reach the maximum memory+swapfile)
  • In both situations, I see, at system monitor, the equivalent of the first swapfile being used with no problem.
  • I see no error message, it just crashes, and jupyter lab/chrome closes.
  • My system is Ubuntu 22.04.2 LTS
  • I checked, and the priority was for the first swapfile (at M.2), which why I am certain to say the crash happens when the first swapfile is fully used...

I am aware of the differences in the speed, and, though I have more space at SSD, I would prefer having the NVMe device being used, at least until reaching its limits. And, unfortunately, I have no more space available at M.2 to increase that first swapfile.

Are there any requirement of such swapfiles being at similar devices (similar speeds, etc.)? What is the correct way to use two swapfiles?

Thank you in advance,

Update: Regarding how the swapfiles were set:

sudo fallocate -l 50g /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

The same for ~/swapfile2

sudo fallocate -l 50g ~/swapfile2
sudo chmod 600 ~/swapfile2
sudo mkswap ~/swapfile2
sudo swapon ~/swapfile2
echo '~/swapfile2 none swap sw 0 0' | sudo tee -a /etc/fstab

I have tried initially without restarting. After restaring, I saw the 2nd swapfile was deactivated, so I edited the /etc/fstab (sudo nano /etc/fstab), as described here. The problem remains even after restarting again after the edited etc/fstab...

Update 2: I have now managed to release some space on a secondary partition of the same NVMe (same device comprising a partition with the system root '/'). And added a swapfile located there to make a test.

sudo fallocate -l 50g /Data/swapfile2
sudo chmod 600 /Data/swapfile2
sudo mkswap /Data/swapfile2
sudo swapon /Data/swapfile2
echo '/Data/swapfile2 none swap sw 0 0' | sudo tee -a /etc/fstab

With two swapfiles located in different partitions of the same NVMe device, it works fine. I am experiencing the trouble, just when the second swapfile is in the SSD SATA. So I am having more arguments to think it has something with the differences in speed...

All partitions and disks are formated as ext4.

Is there any way to make it work with one (or two) swapfiles at NVMe device and another at SSD SATA?

hamagust
  • 197
  • What is the type of the filesystem mounted on /home? Is it Btrfs maybe? How exactly did you create ~/swapfile2 and configured the OS to use it? – Kamil Maciorowski Jun 29 '23 at 22:53
  • @KamilMaciorowski, thank you for looking at my question. The type of file system is ext4. The swapwile was created with fallocate -l – hamagust Jun 30 '23 at 02:05
  • fallocate -l …. And then what? mkswap? swapon? Did you edit /etc/fstab? Did you reboot? Please [edit] and tell us what exactly you did. – Kamil Maciorowski Jun 30 '23 at 05:28
  • @KamilMaciorowski, thank you so much. I have now edited the question to add more details, and all commands. I am sorry, I was not aware that there were other ways to create and activate swapfiles... – hamagust Jun 30 '23 at 16:57
  • 2
    I hope you don't use ~/swapfile2 in your /etc/fstab, Use the actual path: /home/hamagust/swapfile2. Also, be sure to swapoffcommand before deleting a swap file. Give the system time to migrate off the swapfile. – waltinator Jul 04 '23 at 16:23
  • @waltinator, thanks a lot. That's make all sense, if I use ~ at echo '~/swapfile2 none swap one swap sw 0 0' | sudo tee -a /etc/fstab..., it goes written at /etc/fstab with ~, not the actual path of /home/swapfile2... – hamagust Jul 05 '23 at 00:36
  • So did you use exactly ~/swapfile2 in your /etc/fstab or not? "If I use …" does not tell us what you actually did. And when I asked "how exactly did you create ~/swapfile2 and configure the OS to use it?", you posted exact steps for /swapfile instead. "The same for ~/swapfile2" is kinda ambiguous because sometimes people use ~/… in descriptions to refer to what a shell would expand it to, while using a full path in their code or properly letting their shell to expand the tilde. In other words they expect readers to expand ~ in their minds in order to know what was used. – Kamil Maciorowski Jul 05 '23 at 03:53
  • 1
    @KamilMaciorowski thank you, I am updating the post so it will be made clear – hamagust Jul 05 '23 at 12:29
  • @waltinator, it has a while, but would you like to add the comment above as a solution/response, so I can accept it? The error I made can be possibly naive, but it struggled me a lot, and your comment was so enlightening... I believe your response can be interesting for other new users also... – hamagust Sep 27 '23 at 19:54

1 Answers1

1

I hope you don't use ~/swapfile2 in your /etc/fstab, as ~ is a shell shortcut, Use the actual path: /home/hamagust/swapfile2. Also, be sure to swapoff command before deleting a swap file. Give the system time to migrate off the swapfile.

waltinator
  • 36,399