2

Hi All: I'm using Ubuntu 22.04. I know there have been a lot of posts regarding dd but I'm still confused. I have a 1 Terabyte portable drive ( WD Passport ) with directories and files that add up to about 500 Gigabytes. Then I have another blank portable drive ( WD Passport ) that I want to copy the 500 Gigabytes to so I can have it as a backup. This command does not give the correct result.

dd if=/dev/sdb1 of=/dev/sdc1 status=progress

Based on what I've read on threads in askubuntu.com, it seems like maybe I have to mount something first possibly ? Thanks for wisdom or help or even a link. None of the links I have found don't explain about mounting anything before I start. Thanks a lot.

P.S: rsync and cp are not options because I tried them and it seems like it would take weeks for them to finish.

Raffa
  • 32,237
mark leeds
  • 131
  • 5
    What do you mean by "This command does not give the correct result"? Why do you think dd is faster than cp? – Pilot6 Aug 09 '22 at 14:51
  • 2
    "[S]eems like it would take weeks for them to finish" may be hyperbole, but how can we know? Be patient. It might take a few hours, but weeks suggests a hardware fault that the system is fighting. – user535733 Aug 09 '22 at 15:17
  • If there is a fair amount of free space on the partition cloned by dd, it is rather inefficient compared to copying only the used drive space, and you can do that either by rsync, which often is quite efficient, or by Clonezilla, which is a good alternative to dd, when the file system is recognized by Partclone (which is used by Clonezilla). – sudodus Aug 09 '22 at 16:00
  • Sorry Raffa. I had the slashes wrong in my example but I have my slashes correct in my attempts. Let me print out and read everything that everyone write and then get back. Thanks to all. – mark leeds Aug 09 '22 at 16:08
  • @Raffa: Thank you for your incredibly thorough and clear answer. When I get back late tonight, I am going to try "First way". I just want to make sure that all I have to do is take off the 1 digits of the /dev/sdb1 and /dev/sdc1 and that's it. It then knows that I just want the directories and files ? Thanks because I didn't see that anywhere in what I've read on the internet. – mark leeds Aug 09 '22 at 20:58
  • @user535733: I tried cp and rsync a long time ago ( months ) and the rate that they were copying the 500 Gig basically made them not possible to use. But definitely there could have been some option that I was leaving out that would have made them feasible ? Thanks for input. I will let everyone know how Raffa's "first way" goes assuming that I understand the usage. – mark leeds Aug 09 '22 at 21:00
  • One more thing: As far as it "not working", I get a file when I use my current command but it's not the files and directories of the source data on the hard drive. I'm not sure what it is. – mark leeds Aug 09 '22 at 21:01
  • You might know that, but you need to disconnect and reconnect the destination disk after the imaging process finishes so that the partitions get automatically mounted by the udisks utility as usual with external disks … or alternatively, you can mount them manually if you want. – Raffa Aug 09 '22 at 21:41
  • @Raffa: I let the current command ( where I used /sdb1 and /sdc1 ) finish since it took two days and I didn't want to stop it. Then I did what you said in terms of disconnecting the portable and re-connecting it. WALA !!!! Everything looks great. df -k showed that I have two identical portables and the files and directory structures are there. Maybe your approach without the 1's would be faster so I'll definitely try that first the next time. For this time, I'd rather just stay with what worked because it was very difficult to get to this point. Thanks so much for your great help. – mark leeds Aug 10 '22 at 10:40
  • @user535733: Just one more thing that I left out about rsync and cp taking too long. They take too long in the case where I am copying from portable to portable. In a case where I am copying from computer ( so hard drive ) to portable, they are fine. So, I should have mentioned that the problem only occurs with them when it's portable to portable where two portables are connected. Thanks for your help also. – mark leeds Aug 10 '22 at 10:42
  • @markleeds Copying without the 1's (ie. disk to disk) is for the case when the target disk does not contain any partitions. In your case, the target disk almost certainly did contain a partition, so copying partition to partition worked. There is no speed difference. However, there is a speed difference if you use a different block size in the dd command. The default block size is 512 bytes, it is far too small. Using eg. bs=1M (you need to test which size gives the best results) as a parameter to dd command will speed copying up a lot. – raj Aug 11 '22 at 13:21
  • @raj setting bs= other than the default might not be a good advice in all cases … e.g. using dd with block syncing option and it might result in errors otherwise … mind the word might ;-) – Raffa Aug 11 '22 at 13:43
  • @Raffa From my experience, dd with default block size is unacceptably slow on copying large disks/partitions... – raj Aug 11 '22 at 15:17
  • Thanks to all. This is my last question on this thread. If I have any more questions, I will create a new one. My question is: When you buy a portable like the one below, does it already have partitions ? Thanks. https://www.amazon.com/dp/B07VTWX8MN?psc=1&ref=ppx_yo2ov_dt_b_product_details – mark leeds Aug 12 '22 at 15:22
  • 1
    It may or may not … Looking up that specific model on WD website, they state it’s compatible with Windows 10 which suggests it might be partitioned with an NTFS filesystem … but no way to know for sure unless it’s clearly stated … this however shouldn’t be a big concern as it can be easily partitioned or repartitioned later on from your OS disk tools like gnome disks and gparted … looks like a good deal though :-) – Raffa Aug 12 '22 at 16:09
  • Ok. Thanks. It worked on ubuntu so I guess it's compatible with both windows and linux ? Next time I do copying from portable to portable, I will read this thread carefully again and add a block size argument. Maybe the latter will speed things up some. The 500 Gigs took about 2 days to copy which seemed a litttle slow. – mark leeds Aug 14 '22 at 00:09

1 Answers1

6

dd is indeed faster than cp and even rsync ... The reason is dd doesn't process directories and files, while cp and rsync do ... dd just reads and writes (although it can process data in other usage cases) ... But, cp and rsync are more manageable and more fixable and stopable in case of a mistake while dd is NOT. ... dd will DESTROY the destination disk as soon as you hit the Enter key.

Big Dangerous Ugly Warning (disk names do change)

Disk names(and partition names as a result) under /dev/ can and will change depending on their connection order i.e. the first connected disk gets the first available letter in the order /dev/sda then the next one will be named /dev/sdb then the third one /dev/sdc/ and so on … So if two disks get disconnected and then connected again in a different order, their previously assigned names might get swapped and your source disk might suddenly become the destination for dd and it will be DESTROYED … The same goes for all types of disks e.g. NVMEs, SSDs … etc. … So unless you are an experienced user, DO NOT use dd.

Notice: Clonezilla as discussed in this answer is a bit safer and should give the same performance as dd.

Warnings made clear? ... If true; then continue.

What you are doing with dd is called partition/disk imaging and you can do it in a few ways.

First way(disk to disk)

In this way you copy raw disk(overlooking partitions but including them) data as an image to another disk ... This will make the destination disk an exact image of the source disk ... Extra space however on the destination disk is not utilized as the image will only occupy a size identical to the total size of the readable area on the source disk ... Any data, partitions and images on the destination disk will be DESTROYED ... Filesystems i.e. partitions must not be mounted during the imaging process on both source and destination … This way can be done with something like this(where /dev/sdb is the source and /dev/sdc is the destination):

dd if=/dev/sdb of=/dev/sdc

Second way(partition to disk)

In this way you copy a partition as an image to another disk ... This will make the destination disk contain an identical partition(with filesystem and data) to the source partition ... Extra space however on the destination disk is not utilized as the image will only occupy a size identical to the total size of the source partition ... Any data, partitions and images on the destination disk will be DESTROYED ... Filesystems i.e. partitions must not be mounted during the imaging process on both source and destination … This way can be done with something like this(where /dev/sdb1 is the source partition and /dev/sdc is the destination disk):

dd if=/dev/sdb1 of=/dev/sdc

Third way(disk/partition to a file)

In this way you copy a disk or a partition as an image to a file on a mounted filesystem ... This will make the destination file contain an identical image of the source partition/disk ... The destination file is an image that can be mounted in-place, moved, copied and restored ... Filesystems i.e. partitions must not be mounted during the imaging process on the source … This way can be done with something like this(where /dev/sdb and /dev/sdb1 are the source disk/partition and /mnt/backup is the destination mount-point of a supported filesystem and file.img is a suggested filename for the image file):

dd if=/dev/sdb of=/mnt/backup/file.img

or(to image a partition instead):

dd if=/dev/sdb1 of=/mnt/backup/file.img

Remarks

If you are an advanced user, then you know your ways ... But if you are a normal user willing to save your data and time, then this part is for you.

dd is best suitable for copying/imaging physical disks in raw data mode i.e. raw disk data blocks … these are different from data manipulation/processing that happen on the LV/partition(logical devices/disks) and the filesystem(data structure implemented and used by your OS to control how data is stored and retrieved) … to copy on the LV/partition level and especially on the filesystem level, don’t use dd … use filesystem level tools like partclone or cp or rsync or even tar.

You might find on the internet instructions on using dd to "copy" disk partitions and they'll tell you to do it by copying the source partition to the destination partition(without enough explanation of what else you need to care for when using this method) e.g. just like this(not advised):

dd if=/dev/sdb1 of=/dev/sdc1

That might go wrong on many levels as the destination partition, if it exists, is merely a logical boundary i.e. a logical(not real) device/disk created and identified with software ... So you would be imaging the source partition /dev/sdb1 with its area being read by dd on the disk raw level into /dev/sdc1 which is a logical disk that depends on identifiers by software to be correctly utilized by the OS ... Those identifiers might not remain valid after that imaging process as e.g. the filesystem might change ... If, on the other hand, the destination partition e.g. /dev/sdc1 doesn't exist, then what's the point of attempting to create a partition on /dev/sdc the wrong way(dd is not a disk partitioning tool) as although(if the imaging operation succeeds by pure luck and a 50/50 chance) the destination /dev/sdc1 will contain the a filesystem(copied from the source i.e./dev/sdb1) it will not really be a partition and the destination desk /dev/sdc will not contain a partition table identifying /dev/sdc1 as a partition ... But, you most likely(50% of the time if we are being optimistic) will not see this not advised approach yield any expected result and you will end up in a loop not knowing what went wrong like these e.g. (To name just a few ... The list is long):

Furthermore, partitioning is a disk specific(for one unique geometry of a certain distinct physical disk) procedure ... Correctly created partitions are deeply rooted into that disk's partition table with their purposes/types, filesystems, flags and geometry set ... They shouldn't be selectively copied(using low level tools like dd) to another disk with a different partition table and be expected to simply magically work! ... And even worse things could happen, if dd is used to copy a partition from one disk into a partition in another disk destroying its filesystem, causing un-allocated space on the partition level and probably mangling it's logical boundaries ... If correctly cloning a partition is what you want then use other partition cloning tools that will do it right and not dd ... If you must do it with dd, then at least protect the integrity of your valuable data(assuming preserving data is your goal) and do it partition to physical disk and not to a logical device that might get mangled very badly during the process ... Yes, physical disks can hold a filesystem and data without partitioning(read and write speeds might be a bit slower though) and it should be safer than imaging partition to partition as far as data concerned.

Please go through this example of How to create swap partition in Linux as it illustrates how a partition is really created and set covering some of the underlying deep roots of a partition.

A good and easy reading resource on this matter is File System - Partition / Volume (Logical Disk Partition)

Raffa
  • 32,237