1

I had some problem. I need to copy about 10 million small files to an external hard drive. I was trying to do the job by several ways, but nothing worked.

First I tried cp command. But after about a million of files were copied, it slowed down badly.

Second I tried to make a tar. But after about 20 hours of work only 5 million files were archived.

Then I tried the rsync command. But the result was the same as with the copy command.

At the end my external drive stopped working at all...

I have a new external hard drive. The problem begin when I was trying to copy fileswith the help of rsync command

After an hour about 1 million files were copied, but then the process stopped I canceled the process and now I cannot open my hard drive The process was canceled by CTRL + C in the terminal. After that I couldn't open the dist and just restarted the PC

If I try to open it, it gives an Error:

Unable to access Transcend. An operation is already pending

If I open the Disks. I can see the Hard Drive there.

Assessment: Disk is Ok

But there is an active Job.

Mounting Filesystem.

Menu for the disc is unavaliable. I cannot format it.

I've already restarted the PC. No changes I've also tried to remove the hard drive and insert it again. But also no changes

How can I fix this problem?

I'm using Ubuntu 20.04. External Hard Drive format is ext4

Finally I have 2 problems.

  1. I still don't know how to copy such amount of files.
  2. My external hard drive stopped working (

Update lsblk:

sde           8:64   0   3,7T  0 disk 
└─sde1        8:65   0   3,7T  0 part

df -i command

djangodjames
  • 111
  • You've provided no specifics as to what OS/release you're using, what file-system is involved, and when you cancelled the rsync process, if you flushed buffers to the drives & cleanly unmounted the file-system. – guiverc Jun 01 '22 at 11:20
  • I'm using Ubuntu 20.04. External Hard Drive format is ext4 – djangodjames Jun 01 '22 at 11:32
  • The process was canceled by CTRL + C in the terminal. After that I couldn't open the dist and just restarted the PC – djangodjames Jun 01 '22 at 11:32
  • 1
    Restarted the PC in a safe way? or not?. If you told the system to restart (via command, even SysRq keyboard combination if the GUI/system was unresponsive) I'd expect it to be good after restart; however if you used an unclean shutdown (such as holding power button down) I'd expect you'd need to perform fsck or file-system checks before normal operation. – guiverc Jun 01 '22 at 11:36
  • I restarted the PC in a safe way. After the restart the Disks shows that it try to mount it. – djangodjames Jun 01 '22 at 11:38
  • Check the S.M.A.R.T. info of the file. Also [edit] your question and add the lsblk output after inserting your disk. – FedKad Jun 01 '22 at 11:43
  • 2
    You may have inode issue as well as file issue. Each file uses inode and block. New drives are 4K, so each file will be at minimum 4K. df -i to see inodes Do you really need all these as files? Sounds like the data should be in a database. – oldfred Jun 01 '22 at 12:44
  • I added the lsblk info – djangodjames Jun 01 '22 at 14:41
  • Check the S.M.A.R.T. - What is it and how to check it? – djangodjames Jun 01 '22 at 14:42
  • df -i
    Where should I run this command?     – djangodjames Jun 01 '22 at 14:42
  • anywhere, and then look for the line about the drive in question (sde1 apparently) – Esther Jun 01 '22 at 14:44
  • Added the screenshot for df -f command. – djangodjames Jun 01 '22 at 14:48
  • There is no info about sde or sde1 – djangodjames Jun 01 '22 at 14:50
  • is it mounted? otherwise mount it first. – Esther Jun 01 '22 at 14:52
  • It is not mounted. And that is the problem. If I insert it to the PC, it's trying to mount it, but cannot do it. I wrote about in the describtion of the problem in details – djangodjames Jun 01 '22 at 14:53
  • Try running sudo lshw -C disk and look for your external drive model on the list. If it is not there you might have either a bad USB port, bad USB cable or the USB drive drive or its casing went bad. Try it on another computer to see if it can see your drive. – Terrance Jun 01 '22 at 15:40
  • I don't think this the case here, but Chinese marketeers sell lots of disks that do not have the capacity that they have been advertised for. If that would have been the case here you ended up with the same results as you described. Just in case it is a new drive or you just bought it have a look at fight flash fraud https://askubuntu.com/questions/737473/check-real-size-of-usb-thumb-drive. Apart from that you should use sync after copying files to USB drives. If the inodes are the problem you should use tar to create one container with all files in it. Does gparted work on that disk? – AlexOnLinux Jun 27 '22 at 07:41

0 Answers0