4

The problem: Thunar claims that it's written my files to a thumb drive, then I spend 15 minutes waiting for the drive to unmount.

I understand it has to do with the write cache. I hope there's a way to get it to do what I'd like, which is for the file manager to show me an accurate progress bar of the remaining write time (with an option to cancel the copy!)

What I'm unclear about is:

  • Does Ubuntu detect some drives as removable and apply optimal write cache settings?
  • If yes, how can I force it for my drive?

And, in general, ie, teach a man to fish:

  • How can I tune the write cache for a drive? The ideal write cache would be a small write-back cache that is written to disk as quickly as possible. Actually, the cache itself need not be small, it just has to permit a small number of dirty pages.
  • How can I have this setting apply each time I use a USB drive?

Full disclosure: I'm on Mint 17.2 Xfce

Aleksandr Dubinsky
  • 731
  • 8
  • 22
  • Is it the Gthumbnailer thats creating thumbnailes for the files on your USB stick? – Ken Mollerup Oct 29 '15 at 12:09
  • @KenMollerup I'm not sure. Would it matter? I have enough RAM that the files will stay in the disk cache so that making thumbnails from them shouldn't cause any I/O. – Aleksandr Dubinsky Oct 29 '15 at 12:39
  • Well on my system whenever i have moved or updated an external disk, and try to unmount or eject, it claims its busy, it turned out it was just creating thumbnails of pictures documents pdfs and such, and that can take a very long time, flushing the cash should be an ongoing process - and not delay you for a quarter of an hour. – Ken Mollerup Oct 29 '15 at 12:47
  • Re. "Thunar claims that it's written my files to a thumb drive, then I spend 15 minutes waiting for the drive to unmount." Could you please explain how you monitor the time taken to unmount? – DK Bose Oct 29 '15 at 13:49
  • @DKBose I click the unmount icon in Thunar. A message pops up similar to "we're writing your data, don't disconnect your drive." I see in my Disk Performance Monitor panel widget that the disk is being written. – Aleksandr Dubinsky Oct 29 '15 at 14:01
  • That seems very unusual. I copied a ~600 MB file from my hardisk to an external USB drive and as soon as the progress bar indicated the copying was done, I could unmount the USB drive. I also checked with df -h that it was indeed unmounted. This is not with Thunar but with PCManFM. – DK Bose Oct 29 '15 at 15:53
  • @DKBose It could be that PCMan is going out of its way to use filesystem calls to make sure that the file is written. Also, the delay depends on 1) how many dirty pages the vm subsystem allows (see my answer) and 2) how fast the external USB drive is. Try writing to a slow USB 2.0 drive. – Aleksandr Dubinsky Oct 29 '15 at 16:39
  • It is a USB 2.0 drive. Can you try another file manager or just the command line? – DK Bose Oct 29 '15 at 16:44
  • @DKBose cd does not seem to exit before the data is written. Similarly Unetbootin waited as well. They must use fsync or similar to flush the cache before quitting. But the problem demonstrates itself in the progress bar going to 99% right away. – Aleksandr Dubinsky Oct 30 '15 at 18:37

3 Answers3

5

The settings that control how many dirty pages are allowed to accumulate before being written to disk are controlled by the virtual memory subsystem.

For systems with a lot of RAM, they usually need to be tweaked. Unfortunately, these settings are system-wide. (Ugh, Linus.)

See the current settings with:

sudo sysctl -a | grep vm.dirty

On my machine:

vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 5
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 10
vm.dirty_writeback_centisecs = 500

Take a look at the documentation for these settings. On my machine, vm.dirty_ratio is set to 10 (i.e., 10% of RAM). This means that the disk cache will swallow up to 10% x 64GB = 6.4GB of data before writing it to disk.

I am going to try tweaking these settings by writing the following into my /etc/sysctl.conf:

vm.dirty_bytes = 67108864 
vm.dirty_background_bytes = 1048576
vm.dirty_expire_centisecs = 30
vm.dirty_writeback_centisecs = 30

I'll update this answer as I learn more, but I think these aggressive settings are fine for my SSD, which has its own large on-drive write cache.

NotTheDr01ds
  • 17,888
Aleksandr Dubinsky
  • 731
  • 8
  • 22
1

I'm not sure if I can answer you what you want to hear. There is the "oldschool" option to disable that buffer - you may use the fstab sync attribute for this.

This will bring you away from the modern way of your device lying to you "your quick computer has already finished" when it hasn't. The reality still is "it's done when it's done" and sync will let your computer show you when it is really done.

If you in fact are annoyed by the "progress bar" (or waiting times in the shell) and want to tweak the buffer size and behaviour you may do that by editing the sysfs/block options. Compare https://unix.stackexchange.com/questions/30286/can-i-configure-my-linux-system-for-more-aggressive-file-system-caching for this. The list of availible options may vary depending on the ioscheduler you(r kernel) use(s).

Now the question is: How to apply this for USB sticks.

AFIAK on Linux Mint udisks handles USB Stick automounting and AFAIK there is no elegant way to teach udisks sensible defaults. You may try udisks-glue to help you.

If there is only a limited number of USB sticks you want to tweak in detail, you may write an udev rule to do set the mount or sysfs options you want. I always look at https://wiki.archlinux.org/index.php/Udev#Writing_udev_rules whenever I'm tinkering with udev.

Hope that helps

Community
  • 1
Nachtexpress
  • 11
  • Thank you for the detailed answer! Would it change in any way for vanilla Ubuntu? Does it also use udisks? – Aleksandr Dubinsky Oct 29 '15 at 11:59
  • I'm trying to follow the advice in your link, but sudo echo 1 > /proc/sys/vm/dirty_background_ratio gives Permission denied. Moreover, this seems like the key setting that I'm looking for, but is there any way to specify it per drive? – Aleksandr Dubinsky Oct 29 '15 at 12:04
  • You may write that variable using sysctl, i.e. sysctl -w vm.dirty_background_ratio=1. Although I don't suggest that - this is a central part of the system and the consequences are not only per drive. In case your system freezes you should be fine after reboot. Just don't have unsaved important work. – Nachtexpress Oct 29 '15 at 14:20
0

User Nachtexpress' advice regarding sync is correct. As to how to implement this dynamically, use/configure Autofs as per the instructions at the Debian Wiki for this service (link: https://wiki.debian.org/AutoFs).

You will note that you can set sync option for USB drives in a config file as per the following syntax and line:

 /var/autofs/removable   /etc/auto.removable --timeout=2,sync,nodev,nosuid

The above is just an example you can configure to your specific needs; go to wiki link for full instructions on how to do this.

I think the debian wiki is appropriate for ubumtu users as this distro derives from it.

Hope this helps.

Guest-urrr
  • 1