0

Alternative way to earn the bounty: Show me how to store a file system like ext4 (a FS which can store large files, I don't really need all of the good stuff ext4 provides for this use case) in many small files (< 4 GiB) stored on a (working) FAT-formatted partition.


I have an SD card (adverties as 64 GB but other than usual storage media has less than that (it has 63'416'827'904 Byte)) and an SD slot on my laptop (an Acer Aspire VN7-591G-70CY). The SD card worked with my previous laptop without problems and is formatted with ext4.

I tried sudo modprobe -r r852; sudo modprobe -r sdhci_pci; sudo modprobe r852; sudo modprobe sdhci_pci which helped others but did nothing for me.

The SD card works with an SD card to USB adapter. When using this adapter the SD card is /dev/sdb and the single partition on it is /dev/sdb1. When I put it into the SD card slot it is /dev/mmcblk0 and its partition is /dev/mmcblk0p1.

When I enter the SD card into the slot, nautilus says:

Error mounting /dev/mmcblk0p1 at /media/christoph/e07d3be4-bd85-4eb2-9205-d6638ab37704: Command-line `mount -t "ext4" -o "uhelper=udisks2,nodev,nosuid" "/dev/mmcblk0p1" "/media/christoph/e07d3be4-bd85-4eb2-9205-d6638ab37704"' exited with non-zero exit status 32: mount: wrong fs type, bad option, bad superblock on /dev/mmcblk0p1, missing codepage or helper program, or other error

   In some cases useful info is found in syslog - try
   dmesg | tail or so.

Mounting it manually results in pretty much the same error message:

mount: wrong fs type, bad option, bad superblock on /dev/mmcblk0p1, missing codepage or helper program, or other error

   In some cases useful info is found in syslog - try
   dmesg | tail or so.

Now the weird part is that dd doesn't seem to work in either case. I copied the first 100 MiB of each /dev/sdb, /dev/sdb1, /dev/mmcblk0 and /dev/mmcblk0p1 and earlier this day even got matching sha256-sums.

They now don't match any more but are very similar in the way that except for a little bit in the beginning almost exclusively consist of zero bytes even though the SD card is almost entirely full (and filled with compressed and encrypted data, so there shouldn't me many zero bytes).

So I replaced every instance of 4 consecutive zero bytes in $mmcblk0 (the copies I made via dd are named exactly the same as the devices but I but a dollar sign before the name so they can be differentiated) with plain nothingness in a hex editor and ended up with files just being 129.4 kB for r$sdb (the 'r' is for 'reduced' / 'replaced') and 312.7 kB for r$mmcblk0 instead of being 104.9 MB each.

r$sdb has a mount point stated in the beginning (and I have no idea why). It really is the mount point the SD card was mounted on. Then comes randomly seeming data with a lot of repetitions, followed by a whole lot of FF bytes, followed by actual names of files saved on the SD card, followed by a little more randomly seeming data but not much.

About the fist 1600 Byte in r$mmcblk0 are exactly the same as in r$sdb (this includes the mount point even though the SD card couldn't be mounted while in the slot). Then comes randomly seeming data with a lot of repetitions but with big blocks made from printable characters including file paths, followed by FF bytes, followed by exactly the same as the ending of r$sdb (actual names of files saved on the SD card, followed by a little more randomly seeming data but not much).

Can you help me to resolve the problem and maybe tell me why dd outputs this weird data?


I figured out how to make it work with FAT myself. I ran apt-get install nfs-common, rebooted and now FAT is working. Unfortunately, FAT is a shitty file system which doesn't support loads of stuff and I need file many GB in size which aren't supported by FAT. So I cannot use the SD card with FAT at all.

If I format the partition with ext4, the same error as previously occurs. For FAT everything seems fine.

Is there a command to make it work for ext4 as well?


$ sudo fdisk -l

Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 2D8D2D1D-B17D-4F86-ACD1-743E6B376EFE

Device          Start        End    Sectors   Size Type
/dev/sda1        2048    1230847    1228800   600M Windows recovery environment
/dev/sda2     1230848    1845247     614400   300M EFI System
/dev/sda3     1845248    2107391     262144   128M Microsoft reserved
/dev/sda4  1716043776 1920843775  204800000  97.7G Microsoft basic data
/dev/sda5  1920843776 1953523711   32679936  15.6G Windows recovery environment
/dev/sda6     2107392   36923391   34816000  16.6G Linux swap
/dev/sda7    36923392 1716043775 1679120384 800.7G Linux filesystem

Partition table entries are not in disk order.
Disk /dev/mmcblk0: 59.1 GiB, 63416827904 bytes, 123860992 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd69abc36

Device         Boot Start       End   Sectors  Size Id Type
/dev/mmcblk0p1       2048 123860991 123858944 59.1G 83 Linux

Disk /dev/mapper/cryptswap1: 16.6 GiB, 17825267712 bytes, 34814976 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

 

$ lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation 4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation HM86 Express LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
01:00.0 3D controller: NVIDIA Corporation GM107M [GeForce GTX 860M] (rev a2)
07:00.0 Network controller: Intel Corporation Wireless 7265 (rev 48)
08:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)

I think that the last part of the output of dmesg is interesting:

[10106.153215] mmc0: new ultra high speed SDR50 SDXC card at address 59b4
[10106.153349] mmcblk0: mmc0:59b4 SD    59.0 GiB 
[10106.158564]  mmcblk0: unknown partition table
[10109.748018]  mmcblk0: p1
[10110.393758] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[10111.098030] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[10111.862596] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[10111.862610] EXT4-fs (mmcblk0p1): Magic mismatch, very weird!
[10201.953257] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[10202.553754] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[10202.555661] EXT4-fs (mmcblk0p1): ext4_check_descriptors: Checksum for group 48 failed (26727!=0)
[10202.555664] EXT4-fs (mmcblk0p1): group descriptors corrupted!
UTF-8
  • 5,710
  • 10
  • 31
  • 67
  • Why not ntfs then? – daltonfury42 Jul 04 '15 at 19:53
  • Also, what partition table do you use? Try changing it. – daltonfury42 Jul 04 '15 at 20:28
  • @daltonfury42 I tried ntfs but it also doesn't work. Right after formatting the partition to ntfs, GParted shows me a warning symbol on the partition and when I right-click the partition and select "Information", the error message says: Unable to read the contents of this file system! Because of this some operations may be unavailable. The cause might be a missing software package. The following list of software packages is required for ntfs file system support: ntfsprogs / ntfs-3g. Package ntfs-3g is installed, ntfsprogs has no installation candidate (is superseded by ntfs-3g?). – UTF-8 Jul 04 '15 at 20:33
  • http://askubuntu.com/questions/512461/how-do-i-install-ntfs-3g-ntfsprogs-manually-on-ubuntu Quote: "The ntfsprogs package was renamed to ntfs-3g in Ubuntu 13.04." – UTF-8 Jul 04 '15 at 20:40
  • yeah, I was wrong, from it's manpage, "The ntfsprogs are part of the ntfs-3g package" – daltonfury42 Jul 04 '15 at 20:42
  • I tried ntfs. It doesn't produce error messages outside of GParted. I copied the same files as before onto it and it took very long. Somehow accessing them takes even longer. Way, way longer. I waited several hours and only 2 of them got hashed in that time (remember: about 2.1 GB each). Even worse: The checksums don't match. So ntfs returned garbage while claiming no errors occurred which isn't exactly preservation of data integrity. – UTF-8 Jul 04 '15 at 23:54
  • So what happened, did anything work out for you? It's a difficult question, and the fact that windows can extract the data adds to the mistry. – daltonfury42 Jul 10 '15 at 08:47
  • @daltonfury42 No, unfortunately nothing worked. – UTF-8 Jul 10 '15 at 12:40
  • Have you got the oppertunity to try another 64 GB card on the reader? – daltonfury42 Jul 10 '15 at 14:12
  • @daltonfury42 No, I didn't. But I will later this day. – UTF-8 Jul 10 '15 at 15:31
  • Well, it should be the card, but we can confirm that only after you try another card. If it doesn't work, try a smaller card if you can get hold of one. Then we can say what is the issue. – daltonfury42 Jul 10 '15 at 15:35
  • Also you could try exfat. See my answer. – daltonfury42 Jul 11 '15 at 06:18
  • @daltonfury42 I have a new installation of Ubuntu 15.04 (I changed my SSHD to an SSD) and got my hands on a 250 MB SD card). I can read the correct information (a 200 MiB big file of random numbers) I put there from the card (ext4-formatted). Unfortunately, I don't have my 64 GB SD card available to try it with the new installation. So Ubuntu supports reading correct information via this card reader from the small SD card and Windows can read 64 GB of information without a single bit being wrong. But Ubuntu doesn't read correct information from the 64 GB SD card. – UTF-8 Jul 11 '15 at 22:19
  • But even though reading the file did work, the SD card behaves weird. For example I first took a dd from the SD card. After I did my experiments, I did the dd back and the error message "dd: error writing ‘/dev/mmcblk0p1’: No space left on device" appeared immediately. Not after a plausible period of time for dd to have written enough information but instantly when I executed the command. However, dd kept running and copied the information back onto the SD card. Now there are the correct files on the SD card, but when I ddagain, I get back a different image (volume wasn't mounted). – UTF-8 Jul 11 '15 at 22:24
  • But this time, the files are pretty much identical. I wasn't able to find any differences by hand by looking at it in a hex editor. – UTF-8 Jul 11 '15 at 22:49

3 Answers3

1

In my opinion, your SD Card Reader is failing. There is a very high probability of data loss. FAT is working probably because it has less error checking features, but that doesn't mean no errors! Use FAT with extreme caution. Take backups if you will be storing important data. Or rather, don't store important data, or simply use the USB Card Reader. That being said,

Try repairing your SD card which is a standard procedure:

sudo e2fsck -f -C 0 /dev/mmcblk0p1

e2fsck is the linux utility to repair ext based filesystems. Anyways, I don't expect it to work.

So, as another answer suggests, the card reader might not support such a large drive. But the fact that FAT works indicates that it is possible, with the added possibility of getting screwed up.

These are things you can try, but I don't know if they will work: Partition it into two, or even four drives if it will suit you, and then go forward with ext4. Or how about creating a smaller partition and leaving the rest as free space? You could give it a try.

Finally, since FAT works, (remember, probably because it does little error checking) exFAT could meet your requirements of storing file sizes greater than 4 GB. See how to do it here.

daltonfury42
  • 5,499
  • I'll try this command after I gave NTFS a try. How would making smaller partitions help? Why does the card reader even care about partitions in the first place? I'd expect a card reader to just read what's on the SD card, whether there are partitions according to some kind of partition table or whether there is something else on the card (like the whole thing being full of random data, not organized in partitions). – UTF-8 Jul 04 '15 at 20:38
  • Ok, I'll try to explain it for the last time. The card is not broken. Every cardreader has a built-in controller that is hardwired to do things. Pretty much like a regular hard drive. If you could insert another platter in it to extend the capacity it would not recognize it because it is not hardwired to do so. Similarly if you insert a card in the cardreader of a size that it is not hardwired for it will not recognize it. However the cardreader can read partition tables and sees that there are more sectors defined than it can find so it starts throwing I/O errors. – wie5Ooma Jul 04 '15 at 23:56
  • @wie5Ooma, if you are right, and it is doing as it is supposed to do, then why does it work with FAT? – daltonfury42 Jul 05 '15 at 04:55
0

I believe the problem is caused by the fact that this particular SD cardreader is unable to read SD cards that have a 64 GB capacity while the other one in your previous laptop apparently can read such cards. Not all card readers have the same specifications. When you buy a card reader it is stated in the documentation what the biggest capacity is that the reader can handle. When it sits already in a laptop you'll have to figure it out what it can handle. Currently the limit is usually 32 GB. Unfortunately you can't change this property.

wie5Ooma
  • 1,789
  • But then, why the partitions even be figured out? GParted can even tell me space is used and how much is unused. Btw.: I now figured out that useful data comes after the very nothing-saying beginning. It just so happened that the beginning is just that boring, probably because there all inodes are saved and the partition held very few (like very few) files. 3874816 inods with each taking 256 bye means ~967 MiB taken up by inodes. Given that I only read the fist 100 MiB it had to be expected that this would happen. But the question remains: Why can I read data in the first place? – UTF-8 Jul 01 '15 at 17:38
  • You 'lost' an amount of space because the metadata of the filesystem uses it so your observation regarding the inodes sounds valid. It works the same way for a regular hard drive. How did you test it with GParted? Was the card in the cardreader or in the SD to USB adapter? – wie5Ooma Jul 01 '15 at 21:05
  • I simply opened GParted and switched to the SD card while it was in the SD card slot of the laptop (no adapter). – UTF-8 Jul 02 '15 at 00:21
  • 1
    cards larger than 32GB are called SDXC (instead of SDHC or SD), for eXtended capacity. This calls for a different type of reader, for arbitrary and historic reasons. Apparently you have an SDHC reader installed. I can advise you to get a 32GB card if you want it to work with your laptop. Or use a USB card reader (you seem to have one lying around). – Maarten Klop Jul 06 '15 at 15:12
0

Considering the new information you've added to the question I've a different approach for you.

In GParted format the partition on the card as ext4. Then create a mountpoint. Let's assume this is sd. So you do:
mkdir -p /media/sd.
Then to mount it you do:
mount -t ext4 -O noatime,nodiratime /dev/mmcblk0p1 /media/sd.
The second command above is supposed to be on one line. Probably you have to prepend these commands with sudo. You can also use the flags in the second command in /etc/fstab.

wie5Ooma
  • 1,789
  • I tried mounting it manually from the beginning (see original question). Even when stating the file system type in the mount command, the error message remains the same: `$ sudo mount -t ext4 -O noatime,nodiratime /dev/mmcblk0p1 /media/sd mount: wrong fs type, bad option, bad superblock on /dev/mmcblk0p1, missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail or so.`
    
    – UTF-8 Jul 03 '15 at 23:02
  • This suggests the card is not formatted as ext4. Are you really sure it is ext4? The mount procedure described certainly works. – wie5Ooma Jul 03 '15 at 23:05
  • I know that the command is correct. I have used it many times to mount various media over several years. I created a new partition table and then a new partition on the SD card several times. (Using GPareted.) The whole thing that dd doesn't put out correct data doesn't make any sense at all. – UTF-8 Jul 03 '15 at 23:09
  • I added new information to the bottom of the answer. – UTF-8 Jul 03 '15 at 23:09
  • This means that there's something wrong with the partition table, which means the card is about to give up or it can mean that the cardreader thinks there's something wrong with the card which efectively means the cardreader is malfunctioning. Do both readers give this result? – wie5Ooma Jul 03 '15 at 23:23
  • No, in the converter (the external SD card to USB stick thingy) works like a charm. The SD card is just a few weeks old and worked in my old notebook without causing any trouble. – UTF-8 Jul 03 '15 at 23:31
  • Well, if I were you I would not trust that cardreader. Apparently there's something wrong with it or, and I refer to my previous answer, it can't read partitions this big. FAT formatted partitions which you've tried also are limited to 32 GB maximum which it apparently can read. – wie5Ooma Jul 03 '15 at 23:41
  • Yesterday I formatted the SD card with FAT32 using Ubuntu and copied 29 files onto it, each one containing 2.1 GB of random data (taken from /dev/urandom), so they took up almost the entire storage space. I hashed the files before I copied them and saved the sha256sums. Today I tried to hash them but it didn't work (I/O error). I tried copying them but it also resulted in an I/O error. Then I booted Windows on the same machine and copied the files onto the internal HDD. I switched back to Ubuntu and hashed the files. All checksums were correct. So it works with FAT. Why not with ext4? – UTF-8 Jul 04 '15 at 19:11