1

Dual boot machine with Ubuntu 20.04 and Windows 10 on seperate m.2 nvme storage devices. I have an external hard drive (14TB) set up as NTFS. On either operating system I can write to the disc. However, when I open files on the HD in Windows 10, if I generated those files using Ubuntu 20.04, they are often corrupted. For example:

D:\my\path> type myfile.mrc.tlt
The file or directory is corrupted and unreadable.

I have seen this behavior on two external hard drives (one Seagate and another WD). I had assumed the problem was with the Seagate drive. But I have now replicated it with a WD one.

Not sure where to start toubleshooting from here.

When I mount the drive while running journalctl -f I get the following:

Nov 05 17:12:21 axoneme udisksd[894]: Mounted /dev/sdd1 at /media/jared/Elements on behalf of uid 1000
Nov 05 17:12:21 axoneme dbus-daemon[1641]: [session uid=1000 pid=1641] Activating via systemd: service name='org.freedesktop.Tracker1' unit='tracker-store.service' requested by ':1.1' (uid=1000 pid=1637 comm="/usr/libexec/tracker-miner-fs " label="unconfined")
Nov 05 17:12:21 axoneme systemd[1629]: Starting Tracker metadata database store and lookup manager...
Nov 05 17:12:21 axoneme dbus-daemon[1641]: [session uid=1000 pid=1641] Activating service name='org.gnome.Shell.HotplugSniffer' requested by ':1.37' (uid=1000 pid=1860 comm="/usr/bin/gnome-shell " label="unconfined")
Nov 05 17:12:21 axoneme dbus-daemon[1641]: [session uid=1000 pid=1641] Successfully activated service 'org.gnome.Shell.HotplugSniffer'
Nov 05 17:12:21 axoneme dbus-daemon[1088]: [session uid=125 pid=1088] Successfully activated service 'org.freedesktop.Tracker1'
Nov 05 17:12:21 axoneme systemd[1072]: Started Tracker metadata database store and lookup manager.
Nov 05 17:12:21 axoneme dbus-daemon[1641]: [session uid=1000 pid=1641] Successfully activated service 'org.freedesktop.Tracker1'
Nov 05 17:12:21 axoneme systemd[1629]: Started Tracker metadata database store and lookup manager.
Nov 05 17:12:21 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10255 > 9984): Illegal seek
Nov 05 17:12:21 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10256 > 9984): Illegal seek
Nov 05 17:12:21 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10164 > 9984): Illegal seek
Nov 05 17:12:21 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10165 > 9984): Illegal seek
Nov 05 17:12:22 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10009 > 9984): Illegal seek
Nov 05 17:12:22 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10010 > 9984): Illegal seek
Nov 05 17:12:22 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10030 > 9984): Illegal seek
Nov 05 17:12:22 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10031 > 9984): Illegal seek

Similarly, if I run ls -lth in a directory on the NTFS HD with Ubuntu 20.04, I get the following in the corrupted directorys:

Nov 05 17:16:03 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10294 > 9984): Illegal seek
Nov 05 17:16:03 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10290 > 9984): Illegal seek
Nov 05 17:16:03 axoneme ntfs-3g[5491]: Trying to read non-allocated mft records (10360 > 9984): Illegal seek
Jacob Anderson
  • 121
  • Are these files really corrupted, or is it just Windows reads them as such? Can you remember e.g. md5sum /path/to/myfile while the myfile is on Ubuntu partition, then copy myfile to the NTFS disk, check md5sum of that copy, then reboot into Windows, get the "file is corrupted" error, reboot back to Ubuntu, and check md5sum of the same file again. Do they match? – Hi-Angel Nov 05 '22 at 20:57
  • I can check. In the meantime, if I go to the directory with the corrupted (or sometimes deleted) files in Ubuntu 20.04 and simply run ls -lth I see lines like this:

    ls: cannot access 'TS014.mrc.prexf': Input/output error

    And beneath it: ????????? ? ? ? ? ? TS014.mrc.prexf Where there is usually permissions, creator of file, size of file, date last modified, and file name.

     – Jacob Anderson Nov 05 '22 at 21:06
  • Huh, that's interesting. Try to unmount the NTFS disk completely in Ubuntu, then run a journalctl -f in a separate terminal tab, and then mount the disk back, and run the ls -lth command. Do you see anything interesting in the journalctl tab, like I/O errors while the disk is getting mounted or when running the ls? If you do, would be nice to add that information to your post. – Hi-Angel Nov 05 '22 at 21:10
  • Got it. Thanks for the suggestion. I have updated the question accordingly. – Jacob Anderson Nov 05 '22 at 21:17
  • Alright, thank you! Another couple of questions: does it happen if you reboot Ubuntu but not boot Windows? IOW, does the problem only start happening after booting to Windows? For example, try to format the disk (if possible) as a fresh new NTFS in Windows, then reboot to Ubuntu, and create new files, unmount/remount/reboot, see if the problem starts happening or not. Another question: what is your usecase for NTFS on the external drive? Is it to simply share files between the systems? – Hi-Angel Nov 05 '22 at 21:41
  • If I create and write to the NTFS HD from Ubuntu 20.04, if I reboot back to Ubuntu 20.04, the files are intact. No question marks, "cannot access" issues, or "Illegal Seek" messages when running journalctl -f. I suppose something about accessing the drive from Windows 10 is corrupting the files? – Jacob Anderson Nov 05 '22 at 22:13
  • Is Windows fast startup/hibernation off? Note that Windows updates may turn it back on. With hibernation, Windows restores the NTFS as it last saw it and any any other changes are lost. Not sure how the newer NTFS drivers are handling hibernation. It used to be that you could only normally mount read only. https://askubuntu.com/questions/843153/unable-to-mount-windows-10-partition-it-is-in-an-unsafe-state & https://askubuntu.com/questions/145902/unable-to-mount-windows-ntfs-filesystem-due-to-hibernation – oldfred Nov 06 '22 at 02:40

1 Answers1

1

So, discussion in comments showed that the problem starts happening once Windows gets to access the NTFS partition. So, is it a Windows problem? Looks likely, though there's a chance the ntfs-3g FUSE driver interprets something in an incorrect way compared to the Windows one, resulting in an incompatibility.

It is interesting that this problem seems to be extremely rare (I found just a few posts with the exact errors from journalctl, one from 2008 year, and another about some odd interaction with RAID). That is something to note, because that might imply you have some special configuration that causes these problems, and it would be very interesting to find out what that could be. But I'll leave that as an exercise to a reader.

In terms of a workaround what you can try is:

  1. Try the new ntfs3 kernel driver (as opposed to ntfs-3g you're using), contributed into the kernel by Paragon Software since Linux 5.15. Not to be confused with older read-only ntfs kernel driver, which still haven't been removed. You will need to update to 5.15 or higher Linux kernel version. The 5.15 seems to be used by default in 22.04 (and I recommend you to upgrade 20.04 → 22.04 because by having an older software on your 20.04 you're missing out on lots of optimizations and features).

    Offhand, I don't know how to make file managers to use ntfs3 by default, but you might, for example, add a /etc/fstab entry that makes use of ntfs3 driver.

    This may or may not help with your problem. But if it doesn't, then I am 97% sure this is purely a fault of specifically yours Windows system (see also my point about the rareness). The reason for my confidence is that Paragon Software is an old company that were selling their filesystem drivers for a long time, and I am pretty sure they had enough expertise and practical experience to work out possible incompatibilities with the original Windows driver.

  2. If you're using NTFS specifically to share the files, you might also consider:

    1. Using UDF filesystem instead. It is supported by both Windows and Linux.
    2. Using exfat. Since 5.7 SAMSUNG has added a driver for exfat, and they also released exfatprogs, so the proper support is in place.

P.S.: ideally, you'd also have try latest ntfs-3g, then if problem is still reproducible, report a bug. Though you might need to convince the developers that it's really a ntfs-3g problem. If the ntfs3 driver will work fine for you, then that might be an implicit evidence that the problem is in ntfs-3g driver.

Hi-Angel
  • 3,702
  • 1
  • 29
  • 36