72

Yesterday I was copying a single, 8 GB file to a USB with a slow, 7 MB/s write speed, while my RAM is 3 GB. While copying the system froze, to the point where I couldn't even move the cursor.

I managed to log into the text console, and ran iotop, it showed that a process named kswapd0 was taking 99.99% of IO.

Are there workarounds so copying a large file doesn't make my system unusable?

Seth
  • 58,122
sashoalm
  • 5,131
  • 20
    This bug is so ridiculous... – king_julien Jul 02 '14 at 19:08
  • Yes, It not only happens with Ubuntu but also in other Debian flavors. I've also seen the same issue in Kali Linux and Parrot OS. Kali has the worst-case scenario, while parrot makes it smooth but even though hangs for very big sizes. I think it's the problem in the Linux Kernel and how it is written. This is and will remain the worst-nightmare of the Linux of all time. – Mohith7548 Nov 04 '19 at 15:32
  • in 19.10 swap management caused the problem . Try disabling the swap usage if you have no intention to use applications like LibeOffice and more than one Browser tabs which requires more ram . it is recommanded not to disable the swap partition completly , try reducing swappiness to 10 or less . – Afsal Dec 11 '19 at 14:15

9 Answers9

42

According to this bug report I solved it adding following lines

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

into /etc/sysctl.conf

and running 

sudo sysctl -p
Philippe Gachoud
  • 5,900
  • 3
  • 43
  • 50
  • 15
    Care to explain what the above lines do? – nsane May 02 '15 at 06:39
  • 3
    @nisargshah95 sorry, but dont have a clue, search for yourself ;-) – Philippe Gachoud Feb 15 '16 at 09:45
  • 4
    @nisargshah95 The details of the problem are explained in the two LWN articles linked in http://unix.stackexchange.com/a/107722/52205 – Rmano May 15 '16 at 15:37
  • 1
    Thank you, I just found that my Ubuntu 16.04 can not write two 1.4 GB files to USB without these two lines, I was getting frozen for hours, this solved problem, who cares what it does, sometimes you just want to copy files and move on. – Mike Aug 10 '16 at 19:29
  • 2
    I had values of 5 and 60. These control the percent of memory used for operations, while dirty_background_bytes and dirty_bytes use absolute bytes values. I've fixed this issue with second answer, but to make it persistent add it to sysctl.conf, see this answer. So when using percent values tweak them when upgrading memory. – PeterM Sep 14 '17 at 13:15
  • I've been having this issue all day, seemingly from nowhere. Has some new update completely broken the file manager? I've tried the various fixes above, but they do nothing more than slow down the initial file transfer speed, the file manager still hangs for hours (quite specifically, hours) after it's supposedly finished the file copy, and, despite memory usage and cpu usage being quite low, the 'load' on the machine is off the scale. It's quite noticeable that this happens with files that are over 1Gb in size. Does anyone have any more ideas for this fix? – JamesBB Feb 20 '19 at 21:32
  • @jamesbb see bug report and what you could do, stackexchange is not done about bug fixing... – Philippe Gachoud Feb 20 '19 at 23:14
  • 1
    This doesn't work with Ubuntu 20.04 – Leszek Oct 31 '20 at 11:25
29

I ran into the similar issue. Mine is 64 bit Ubuntu 14.04. So After a long struggle I found a answer which solves my issue. For easy use I added the commands below used in that above mentioned answer. Check the answer for detailed explanation.

echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes
echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes

After using the above command system started to work normally on copying files.

Thanks goes to @Rmano.

Community
  • 1
Mahendran Sakkarai
  • 481
  • 2
    The ratio settings didn't help on my 12.04 system with a slow NAS share. But after setting bytes directly as suggested here my system is usable again while copying to the NAS. – mivk Dec 24 '15 at 12:04
  • 9
    This question is 3 years old and this is still required to avoid getting an unusable system while copying to pendrive. Some info: if the pendrive is formatted using a Linux fs like ext4 this doesn't happen. When I said "unusable system" I really mean it, mouse pointer becomes unresponsive and you have to insist to move it around the screen, you look at the system monitor and there isn't any abnormal resource usage. Are the kernel people all using 6th gen Intel CPUs and SSD drives? How come they don't notice this while testing. – Hatoru Hansou Feb 22 '17 at 13:03
  • 4
    @HatoruHansou I feel the same, I just installed fresh Debian Stretch and this error is present here also. I know it does not depends on distribution, but on kernel sources, but men, how come this is still not fixed? – Marecky May 19 '17 at 15:20
  • 2
    @Marecky After some reading it seems that the dirty_bytes aren't a usb thing alone. They affect all I/O so after doing the echo thing you are changing them system global, not for pendrives alone. For the current session only, I think. It seems that kernel's current values are tweaked thinking in newer storage devices. Slow pendrives suffer as a side effect. Sorry, no link, but this must be easy to find by googling for it. – Hatoru Hansou May 20 '17 at 01:47
  • This solution did the trick for me when transferring large amounts of data to a remote NFS, but I found that increasing the numbers somewhat gave me around 30% more throughput without bringing back the issue. – Programster Jun 04 '17 at 18:17
  • 4
    See this answer to make it persistent – PeterM Sep 14 '17 at 13:16
  • I awarded bounty to this answer because it is the only one working so far for me, tried to get attention with bounty, but nobody gave an answer in 7 days, this fact should give you some hint about how important is this for linux kernel devs, bug is not only in Nautilus or old PCs, same problem is with copying via command line, affects new kernels, new hardware, I know this is Ubuntu forum, but I use Ubuntu 16.04 and problem persists, I can not cross post this to every forum on internet. – Mike Sep 27 '17 at 15:47
6

I am experiencing a similar issue with the system freezing when copying to a flash drive. I submitted a bug report about it: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1267648

As workaround I found that disabling swap completely eliminates the problem.

sashoalm
  • 5,131
Vladimir Rutsky
  • 1,279
5

I have just had exactly the same problem (in 2019), on ubuntu 19.10, while copying large number of files from USB disk to the SATA disk. Both filesystems are ext4. When I turned off the swap, the problem disappeared. It looks like some bug in memory allocation for disk buffers - apparently, the kernel tries to allocate as much memory for disk buffers, as possible in such situation, which doesn't make sense (making disk buffers in swap...), or it just wrongly calculates the memory size than can be used for caching...

As someone pointed out, setting swappiness to 1 doesn't solve the problem, which is logical, if the files you copy have total size more, than the RAM size...

By the way - can anybody explain, WHY turning off the swap is not recommended? If I have 32GB of RAM, what difference does it make to add another 4GB of swap? I can only think of some obscure applications that actually expect the swap to be there. But I would just stop using such application, since properly written one shouldn't care about swap. The swap should only be managed on the OS level...

chaker
  • 51
  • 1
  • 4
  • 1
    Did any of the solutions here work for you? I started suffering from this issue too after upgrading to 19.10 & in the process updating kernel too. In my case, this happens when I copy a large file from partition A to B of the same SSD. Disabling SWAP "fixes" the problem. – pileofrocks Nov 10 '19 at 10:02
  • 1
    I just keep my swap turned off. I have 32GB of RAM, and I don't see a point in having swap, actually. So far, everything works. – chaker Nov 28 '19 at 11:31
  • 1
    Try not to disable swap completely in 19.10 . you can adjust the swappiness to very small value such as 1 . swappiness determines when to use swap instead of ram . setting swappiness 100 will cause system to allocate mostly to swap and setting a value 1 will cause to utilize swap when there is no ram left . – Afsal Dec 11 '19 at 14:21
  • @pileofrocks This is the only working solution for me . Problem exists only in 19.10 . – Afsal Dec 18 '19 at 16:55
  • 1
    @Afsal Previously I've had it value 20, now tried 1 - but doesn't make any difference to this issue. kswapd0 still uses some cpu, mouse lags and entire system crawls for as long as the file writing is processing. – pileofrocks Dec 18 '19 at 20:21
5

Yes, there are kernel settings you can tweak specifying how much data has to be marked as written before it actually gets written to disk. Look here for a pretty comprehensive description of them. In particular, you'll want to find a value of dirty_ratio that works well for you (it's generally too high for desktop/laptop by default, but there's no one magic number that works for everyone).

carnendil
  • 5,451
Bandrami
  • 151
  • 3
  • 2
    Hey, could you please suggest me what numbers I need to set based on my laptop specifications ? refer http://askubuntu.com/questions/713723/copying-to-usb-pen-drive-slows-everything?noredirect=1#comment1052372_713723 –  Dec 28 '15 at 19:21
3

in Ubuntu 19.10 this problem happens due to problem with swap management . Tinkering with dirty_background broke my nautilus file explorer . Along with freezing when copying large files , i have encountered several performance issues due to bad swap management in 19.10 . Since completely disabling swap is not recommended and we can control the degree of writing into swap partition by setting swappiness .

swappiness=0 tells the kernel to avoid swapping processes out of physical memory for as long as possible . swappiness=100 tells the kernel to aggressively swap processes out of physical memory and move them to swap cache

so , we are going to set swappiness =1 .

To change the swappiness value A temporary change (lost on reboot) with a swappiness value of 1 can be made with

sudo sysctl vm.swappiness=1

To make a change permanent, edit the configuration file with your favorite editor:

sudo gedit /etc/sysctl.conf

Search for vm.swappiness and change its value as desired. If vm.swappiness does not exist, add it to the end of the file like so:

vm.swappiness=1

Save the file and reboot.

This helped me to improve overall performance issues in Ubuntu 19.10 while disabling swap completely eliminated all performance issues that i had with 19.10 but still i won't recommend doing that in normal situations .

Afsal
  • 902
  • 3
  • 9
  • 21
3

As an experiment, I've set sysctl.conf to: 

vm.dirty_background_ratio=1
vm.dirty_ratio=2

and this solved the problem. Previous values 5/10 didn't help. Looks like everybody should play with those params to find the best one. Copying of 12 GB file from SSD to pen flash drive took about 15 min without system hangups (USB 2.0). Probably I'll check values 2/4, etc after several days of testing.

System: Ubuntu 19.10, i7, 8GB Ram, SSD.

Andrei Hryshanovich
  • 56
3

Turning off swap while copying worked for me.

sudo swapoff -a

After the copying is done, I turn on the swap

sudo swapon

Edit: Do note, as Robby1212 mentions in the comment, that swapping off is dangerous and should be done after you have tried out the other answers. I am fortunate that nothing has happened to me when I swap off. I only swap off during copying large files.

Abdulla Masud
  • 51
  • 1
    Turning off the swap is a very dangerous proposition and should not be done without exhausting all other troubleshooting steps. – Robby1212 Apr 06 '20 at 01:35
  • Tried almost everything. Problems are not only when using USB but unpacking large files on disk. Only disabling swap helped although there is some fancy logic behind this :( – DevilaN Mar 31 '21 at 12:36
  • Why is it so dangerous? What's the worst that can happen if my laptop has 16GB RAM? I thought I'd turn it off and not bother turning it back on if that's causing such problems. – Daniel Katz Aug 12 '22 at 11:11
1

I had similar problems when copying files to an exfat drive. I had less trouble using an ext4 filesystem on my USB hard drive .

mace
  • 379
  • 5
  • 15