How can I get wget to preserve sub-directory structure while ignoring parent directories?

Question

Suppose the directory

http://www.example.com/content/media/images/vacation_photos

contains links to a number of sub-directories:

/sweden
/egypt
/canada

each of which contains a bunch of JPEGs, and I want to download them all to my local folder:

/home/jack/VacationPhotos

So that I end up with everything in directories

/home/jack/VacationPhotos/sweden
/home/jack/VacationPhotos/egypt
/home/jack/VacationPhotos/canada

the problem is, if I give wget that URL, apply the -r (recursive) option and the -P /home/jack/VacationPhotos option, it downloads everything to

/home/jack/VacationPhotos/content/media/images/vacation_photos/sweden

and so on, rather than the structure I wanted. Is it possible to get this kind of behavior in wget?

score 5 · Accepted Answer · answered Feb 28 '17 at 12:15

Look at the -nHand --cut-dirsoption in wget. From the manpage:

   --cut-dirs=number
       Ignore number directory components.  This is useful for getting a fine-grained control over the directory where recursive retrieval will be saved.

       Take, for example, the directory at ftp://ftp.xemacs.org/pub/xemacs/.  If you retrieve it with -r, it will be saved locally under ftp.xemacs.org/pub/xemacs/.  While the -nH option can
       remove the ftp.xemacs.org/ part, you are still stuck with pub/xemacs.  This is where --cut-dirs comes in handy; it makes Wget not "see" number remote directory components.  Here are
       several examples of how --cut-dirs option works.
           No options        -> ftp.xemacs.org/pub/xemacs/
           -nH               -> pub/xemacs/
           -nH --cut-dirs=1  -> xemacs/
           -nH --cut-dirs=2  -> .

           --cut-dirs=1      -> ftp.xemacs.org/xemacs/
           ...

   If you just want to get rid of the directory structure, this option is similar to a combination of -nd and -P.  However, unlike -nd, --cut-dirs does not lose with subdirectories---for
   instance, with -nH --cut-dirs=1, a beta/ subdirectory will be placed to xemacs/beta, as one would expect.

Not obvious, but one can specify something like --cut-dirs 100 to remove almost all parent dirs, and this will not cut any subdirs. https://stackoverflow.com/questions/5043239/how-do-i-mirror-a-directory-with-wget-without-creating-parent-directories#comment58693987_5045370 — CoderGuy123, May 02 '20 at 15:05

How can I get wget to preserve sub-directory structure while ignoring parent directories?

1 Answers1