15

Background

In Linux you can:

  • List a group of files with ls Filename*
  • Remove a group of files with rm Filename*
  • Move a group of files with mv Filename* /New/Directory
  • But you cannot copy a group of files with: cp Filename* *.bak

Change Linux cp command to copy group of files

I have a group of files I'd like to copy without entering the names one by one and using the cp command:

$ ls gmail-meta3*
gmail-meta3                          gmail-meta3-REC-1558392194-26467821
gmail-meta3-LAB-1558392194-26467821  gmail-meta3-YAD-1558392194-26467821

How can I use something like the old DOS command copy gmail-meta3* *.bak?

I don't want to type similar command four times:

cp gmail-meta3-LAB-1558392194-26467821 gmail-meta3-LAB-1558392194-26467821.bak

I'm looking for a script/function/app that accepts parameters for old and new filename group and not something with hard-coded filenames. For example, a user can type:

copy gmail-meta3* *.bak

or they might type:

copy gmail-meta3* save-*
  • 1
    The issue looks to me using glob operator twice which none of your other commands use. bash isn't smart enough to handle it. – qwr Jun 26 '19 at 13:32
  • 2
    @qwr, the fact that bash expands metacharacters and tokenizes the input prior to handing it off to a command to be exec'ed is part of the design of the UNIX shells. To somehow try and program an exception for the cp command would break the entire consistency of bash, which would not be smart at all. As an exercise, try and figure out what is happening here, and why it is the shell's metacharacter expansion that makes it so: touch aa ab ba; mkdir bb; cp a* b*; ls * – Mike S Jun 27 '19 at 02:11
  • @MikeS Thanks for the pointers. Yesterday someone else said you can use a wildcard * for the source filenames but not for the target filenames. As such as substitute wildcard (I think ## was suggested but I'm leaning towards %) will have to be used for the target. I think this is what you are reinforcing? I didn't expect to change the cp command at all. Just create a wrapper script called copy that emulated (within reason) the DOS copy command. – WinEunuuchs2Unix Jun 27 '19 at 02:27
  • @WinEunuuchs2Unix that person was correct. The shell metacharacters are independent of the commands. So all wildcards will try and match all files that match the pattern. If you're trying to make a general-purpose "match everything and copy them to whatever they were but add this suffix" program, then yes, putting an unescaped metacharacter as the target will probably not do what you want. Because all metacharacters in the shell command line are expanded. If you KNOW for certain that the target metacharacter will never form a match, you could use it- because the shell can't expand it. – Mike S Jun 27 '19 at 02:56
  • ...but that would be an ugly thing to do. Better to use a special character. % or underscore are good, they're generally not metacharacters (but be careful of using % in a crontab file; it's special there). – Mike S Jun 27 '19 at 02:57
  • If you are interested, I have a script that does this with several config options, as well as one that does the inverse (remove the .bak from *.bak files). – istrasci Jun 27 '19 at 19:21

8 Answers8

20

Here is an example of one atypical usage of sed, that is applicable in this case:

sed -i.bak '' file-prefix*

In this way, actually, sed will not change the files, because we do not provide any commands '', but due to the option -i[suffix] it will create a backup copy of each file. I found this approach when I was asking Is there any way to create backup copy of a file, without type its name twice?

pa4080
  • 29,831
  • 1
    It probably doesn't matter what storage device you have for these tests, if this is happening on a Linux machine. The kernel is extremely good at buffering files in memory. That's what the "buff/cache" value is when using the free command. Actual writes to the device take place at a moment chosen by an algorithm that takes into account age of the cache and memory pressure on the machine. If you're trying multiple tests, then the first file reads will come off the disk but subsequent reads will most likely come straight out of memory (see sync; echo 3 > /proc/sys/vm/drop_caches). – Mike S Jun 27 '19 at 11:52
  • 1
    Yesterday I made few tests with large files over 2GB and - yes, this approach is relatively slow than using of the cp command, but I can't say there is any significant performance difference. – pa4080 Jun 28 '19 at 14:08
  • 1
    Although I upvoted your answer and all the others in this thread I regret to say I've accepted my own answer tonight. I used cps last .bak in the command line and four files in my directory were copied. I didn't need to remember sed command syntax you posted here two years ago. It's rather brilliant though. – WinEunuuchs2Unix May 27 '21 at 00:58
16

You can use find:

find . -max-depth 1 -name 'gmail-meta3*' -exec cp "{}" "{}.bak" \;

That will find in the current directory . all files with a name matching the glob pattern (mind the single quotes around the pattern to prevent shell globbing). For each file found, it will exec cp from name to name.bak. The \; at the end ensures it will do each file individually instead of passing all of them at once. The max depth as 1 only searches the cuurent directory instead of recursing down.

cbojar
  • 261
  • 1
  • 6
  • 1
    Will it work with find . -max-depth 1 -name '"$1"'' -exec cp "{}" "{}$2" \; when $1 is source and $2 is extension? – WinEunuuchs2Unix Jun 26 '19 at 03:27
  • $2 should be ok to substitute so long as it is reasonable. $1 could be trickier as we can't do variable substitution inside single quotes. I'm not sure offhand but it might be possible to use $1 in double quotes since the pattern is stored in a string. – cbojar Jun 26 '19 at 03:36
11

You can use a for loop with bash. Normally, I would just type it as a one-liner because this isn't a task I perform often:

for f in test* ; do cp -a "$f" "prefix-${f}.ext" ; done

However, if you need it as a script:

cps() {
   [ $# -lt 2 ] && echo "Usage: cps REGEXP FILES..." && return 1

   PATTERN="$1" ; shift

   for file in "$@" ; do
      file_dirname=`dirname "$file"`
      file_name=`basename "$file"`
      file_newname=`echo "$file_name" | sed "$PATTERN"`

      if [[ -f "$file" ]] && [[ ! -e "${file_dirname}/${file_newname}" ]] ; then
         cp -a "$file" "${file_dirname}/${file_newname}"
      else
         echo "Error: $file -> ${file_dirname}/${file_newname}"
      fi
   done
}

Usage is similar to rename. To test:

pushd /tmp
mkdir tmp2
touch tmp2/test{001..100}     # create test files
ls tmp2
cps 's@^@prefix-@ ; s@$@.bak@' tmp2/test*    # create backups
cps 's@$@.bak@' tmp2/test*    # more backups ... will display errors
ls tmp2
\rm -r tmp2                   # cleanup
popd
xiota
  • 4,849
  • FYI: $ time for f in gmail-meta3* ; do cp -a "$f" "${f}.bak" ; done = real 0m0.046s – WinEunuuchs2Unix Jun 26 '19 at 23:18
  • Um no I don't want to optimize for time. It is 0.046 seconds which means 0 seconds for human perception. I was just trying to show how I was testing posted answers and passing along interesting tidbits to viewers who looked at the sed command above. Or at least I was interested in comparing sed to cp.... – WinEunuuchs2Unix Jun 26 '19 at 23:25
  • Your solution with cp is faster than solution with sed though. So it's a cause for celebration :) – WinEunuuchs2Unix Jun 26 '19 at 23:38
  • (1) -a is a non-standard test operator.  Why not use -e?  (2) “Unable to create temporary directory.” is a somewhat misleading error message.  (3) Why not just use mktemp -d?  (4) You should test exit statuses.  For example, you should say ! mkdir "$FOLDER" && echo "Unable to create temporary directory." && return 1 or mkdir "$FOLDER" || { echo "Unable to create temporary directory."; return 1;}.  Likewise for cp and rename (and maybe even pushd, if you want to be careful).  … (Cont’d) – G-Man Says 'Reinstate Monica' Jun 27 '19 at 15:43
  • (Cont’d) …  (5) Arrggghhhh!  Don’t say $@; say "$@". (5b) There’s no need to use { and } when you reference variables the way you are doing ("${FOLDER}", "${PATTERN}" and "${file}"); just do "$FOLDER", "$PATTERN" and "$file". (6) This assumes that files are in the current directory.  cps 's/$/.bak/' d/foo will copy d/foo to foo.bak in the current directory, not d/foo.bak. – G-Man Says 'Reinstate Monica' Jun 27 '19 at 15:43
  • @G-Man 1. Done. 2. N/A b/c scrapped the rename script. 3. N/A. 4. Tried. Testing exit statuses doesn't work. Don't know why. 5. Done. 6. Rewrote to work with files that may not be contained in current directory. – xiota Jun 27 '19 at 19:50
  • @WinEunuuchs2Unix Rewrote to work with files that aren't in the current directory and to use regular expressions via sed. May be slower than previous versions. – xiota Jun 27 '19 at 19:51
  • @xiota: Well, that’s a bit of a shame; I thought the idea of piggybacking on the existing rename program was brilliant, even if the implementation was flawed. But +1 for making the improvements. – G-Man Says 'Reinstate Monica' Jun 27 '19 at 21:47
  • @G-Man Couldn't figure out how to fix problems you pointed out while still using rename. – xiota Jun 27 '19 at 22:43
6

The closest you will likely get to the DOS paradigm is mcp (from the mmv package):

mcp 'gmail-meta3*' 'gmail-meta3#1.bak'

If zsh is available, its contributed zmv module is perhaps a little closer:

autoload -U zmv

zmv -C '(gmail-meta3*)' '$1.bak'

I'd avoid ls regardless - a variant on your own answer that's safe for whitespace (including newlines) would be

printf '%s\0' gmail-meta3* | while IFS= read -r -d '' f; do cp -a -- "$f" "$f.bak"; done

or perhaps

printf '%s\0' gmail-meta3* | xargs -0 -I{} cp -a -- {} {}.bak
steeldriver
  • 136,215
  • 21
  • 243
  • 336
  • I understand mmv is the package but in comments you say the command is mcp but then in the command you use mmv which is also a command in mmv package. I like the direction of the printf examples and in a polished script I'd ensure $1 and $2 were passed. +1 for getting ball rolling :) – WinEunuuchs2Unix Jun 26 '19 at 01:19
  • @WinEunuuchs2Unix apologies - the mcp/mmv was a brainfart. Actually mcp is just a synonym for mmv -c – steeldriver Jun 26 '19 at 01:21
  • Pfft no worries. If I had a dollar for every typo I've made I'd be a millionaire :) I'd like clarification on printf command I've never used really. Are you saying printf '%s\0' "$1"* would work if gmail-meta3 was passed as parameter 1? – WinEunuuchs2Unix Jun 26 '19 at 01:24
  • @WinEunuuchs2Unix I'd probably let the calling context do the globbing i.e. cps gmail-meta3* and then write printf '%s\0 "$@" | while ...in the function. Or simply usefor f; do cp -- "$f" "$f.bak"; done` (like xiota's answer, but as a function) – steeldriver Jun 26 '19 at 01:40
  • 1
    Note that with zmv you can use "wildcard replacement" mode, which I find a bit easier to grok: zmv -W -C 'gmail-meta3*' '*.bak' – 0x5453 Jun 27 '19 at 22:03
  • @0x5453 thanks - I wasn't aware of that option – steeldriver Jun 27 '19 at 23:33
5

rsync only solution

If you just want to backup your files, you can copy them to a new directory

rsync /path/to/dir/Filename* /path/to/backupdirectory

This will copy the Filename files from /path/to/dir/ to /path/to/backupdirectory.


rsync + filerename

If you want your backup files to have a suffix, things get hacky with rsync...

rsync -Iu /path/to/dir/Filename* /path/to/dir/Filename* -b --backup-dir=/path/to/backupdirectory --suffix=.bak

This would overwrite the existing files... with the existing files (-I) but only if they are (-u) newer (which they aren't) and creating a backup, with a suffix.

You can also do that in the same directory. But better exclude existing backups.

rsync -Iu /path/to/dir/Filename* /path/to/dir/Filename* -b --backup-dir=/path/to/backupdirectory --suffix=.bak --exclude '*.bak'

Robert Riedl
  • 4,351
  • I love rsycnc so I upvoted but, a simpler method would be cp Filename* /path/to/backup/dir because files wouldn't need *.bak uniquifier if they were in a separate directory. – WinEunuuchs2Unix Jun 26 '19 at 22:59
4

I wrote this one-liner into my ~/.bashrc. Much better answers using find can be posted I suppose. Even better answers could be written in in C. Hopefully this Q&A gets the ball rolling for better answers:

cps () {
    # cps "Copy Splat", copy group of files to backup, ie "cps Filename .bak"
    # Copies Filename1 to Filename1.bak, Filename2 to Filename2.bak, etc.
    # If Filename1.bak exists, don't copy it to Filename1.bak.bak
    for f in "$1"*; do [[ ! "$f" == *"$2" ]] && cp -a "$f" "$f$2"; done

    # OLD version comments suggested to remove 
    # ls "$1"* | while read varname; do cp -a "$varname" "$varname$2"; done
}
  • for f in "$1"*; do : $1 is the gmail-meta3 parameter and f is the list of files matching. Combined this means for gmail-meta3, gmail-meta3-LAB-9999, etc. do the following
  • [[ ! "$f" == *"$2" ]] && : $f is the same as f above. $2 is the .bak parameter passed. Combined this means if the filename doesn't end in .bak (because we don't want to copy .bak and create .bak.bak) then do the following
  • cp -a "$f" "$f$2"; copy gmail-meta3 to gmail-meta3.bak, etc.
  • done : loop back and grab next filename in gmail-meta3* list.

cps gmail-meta3 .bak Sample Output

Using the question as an example here is how it looks in action:

───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/gmail$ ll gmail-meta3*
-rw-rw-r-- 1 rick rick 26467821 May 20 16:43 gmail-meta3
-rw-rw-r-- 1 rick rick 26467821 May 20 16:43 gmail-meta3.bak
-rw-rw-r-- 1 rick rick      643 May 20 16:43 gmail-meta3-LAB-1558392194-26467821
-rw-rw-r-- 1 rick rick      643 May 20 16:43 gmail-meta3-LAB-1558392194-26467821.bak
-rw-rw-r-- 1 rick rick    49607 May 20 16:44 gmail-meta3-REC-1558392194-26467821
-rw-rw-r-- 1 rick rick    49607 May 20 16:44 gmail-meta3-REC-1558392194-26467821.bak
-rw-rw-r-- 1 rick rick   728954 Jun 27 17:04 gmail-meta3-YAD-1558392194-26467821
-rw-rw-r-- 1 rick rick   728954 Jun 27 05:46 gmail-meta3-YAD-1558392194-26467821.bak
───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/gmail$ cps gmail-meta3 .bak
───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/gmail$ ll gmail-meta3*
-rw-rw-r-- 1 rick rick 26467821 May 20 16:43 gmail-meta3
-rw-rw-r-- 1 rick rick 26467821 May 20 16:43 gmail-meta3.bak
-rw-rw-r-- 1 rick rick      643 May 20 16:43 gmail-meta3-LAB-1558392194-26467821
-rw-rw-r-- 1 rick rick      643 May 20 16:43 gmail-meta3-LAB-1558392194-26467821.bak
-rw-rw-r-- 1 rick rick    49607 May 20 16:44 gmail-meta3-REC-1558392194-26467821
-rw-rw-r-- 1 rick rick    49607 May 20 16:44 gmail-meta3-REC-1558392194-26467821.bak
-rw-rw-r-- 1 rick rick   728954 Jun 27 17:04 gmail-meta3-YAD-1558392194-26467821
-rw-rw-r-- 1 rick rick   728954 Jun 27 17:04 gmail-meta3-YAD-1558392194-26467821.bak
───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/gmail$ 

Note: This uses the -a flag with the cp command to preserve timestamps and give you better grasp of your file backups.

Notice how the file copies have the exact same date and time as the originals. If the -a parameter was omitted they would be given the current date and time and wouldn't look like a true backup except that the file size would be the same.

  • 7
    don't people always recommend against parsing ls – qwr Jun 26 '19 at 13:30
  • 3
    Since you mention find I assume you’re aware of the dangers of parsing ls? But in your case neither is necessary: just do for file in "$1"*; do copy -a "$file" "$file$2"; done — this is completely safe and much simpler than any kind of indirection via ls or find and a while loop. – Konrad Rudolph Jun 26 '19 at 13:40
  • @KonradRudolph Thanks for your suggestion. I implemented and tested your suggestion with a couple minor changes. – WinEunuuchs2Unix Jun 27 '19 at 23:21
4

This one should do as requested:

cps(){ p="${@: -1}"; for f in "${@:1:$#-1}"; do cp -ai "$f" "${p//\?/$f}"; done  }

Usage:

cps FILES... pattern
Example 1: cps gmail-meta3* ?.bak
Example 2: cps * save-?
Example 3: cps * bla-?-blubb

I chose ? because # must be quoted when it's the first character of pattern, otherwise it's recognized as the start of a comment.

Test:

$ touch 'test};{bla#?"blubb'
$ cps test* bla-?-blubb
$ ls
test};{bla#?"blubb  bla-test};{bla#?"blubb-blubb


Some earlier versions of the script for adding a suffix:

Similar to @WinEunuuchs2Unix answer, but I think more flexible and not parsing ls:

cps(){ S="$1"; shift; printf '%s\0' "$@" | xargs -0 -I{} cp -abfS "$S" {} {}; }

Put this in your .bashrc.

Usage:

cps SUFFIX FILES...
Example: cps .bak gmail-meta3*

Alternative, with the suffix as last argument (via and via):

cps(){ S="${@: -1}"; printf '%s\0' "${@:1:$#-1}" | xargs -0 -I{} cp -abfS "$S" {} {}; }

Usage:

cps FILES... SUFFIX
Example: cps gmail-meta3* .bak

pLumo
  • 26,947
  • Nice coding but it's kind of hard after decades of using Source then Target to switch copy command to Target then Source – WinEunuuchs2Unix Jun 26 '19 at 10:35
  • Added the function with the suffix in the back. – pLumo Jun 26 '19 at 11:14
  • Thanks that is more intuitive. Calling it a suffix is accurate the way my answer coded it but it's really a target or destination. Other users might want to use: copy gmail-meta3* old-meta3*. In my answer I couldn't figure out how to get * into the destination name like my question requested... – WinEunuuchs2Unix Jun 26 '19 at 11:22
  • problem is that * is interpreted by the shell, so the function will not know about it. You would need some other character or quote it, then replace it with the original filename within the function. – pLumo Jun 26 '19 at 11:26
  • I guess the # could be used as a replacement wildcard for *? So you could type copy filenames# save-#. I think you'd want the wildcard character to be the same for source and target. – WinEunuuchs2Unix Jun 26 '19 at 11:53
  • I added one with pattern, that should be pretty much what you want, but I let the filenames be chosen from the shell outside the function, so I use * for the files and # for the pattern. I think it's better. – pLumo Jun 26 '19 at 12:01
2

Another method of achieving your requirement is to copy the files to a temporary directory and use the rename command to rename them.

$ mkdir backup
$ cp filename* /tmp/rename-backup/
$ rename 's/(filename.*)/$1.bak/' /tmp/rename-backup/*
$ mv /tmp/rename-backup/* ./

If you need it as a script, you can use it like so

cps () {
    mkdir -p /tmp/rename-backup/
    cp "$1"* /tmp/rename-backup/
    rename "s/($1.*)/\$1.$2/" /tmp/rename-backup/*
    mv "/tmp/rename-backup/$1"*".$2" .
}

And you can use it like so:

cps file bak

This is an example

$ ls -l
total 0
-rw-r--r--  1 danny  danny  0 Jun 26 16:23 file a
-rw-r--r--  1 danny  danny  0 Jun 26 16:23 file ab
-rw-r--r--  1 danny  danny  0 Jun 26 16:23 file ac
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename1
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename2
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename3
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename4
$ cps file bak
$ ls -l
total 0
-rw-r--r--  1 danny  danny  0 Jun 26 16:23 file a
-rw-r--r--  1 danny  danny  0 Jun 26 16:41 file a.bak
-rw-r--r--  1 danny  danny  0 Jun 26 16:23 file ab
-rw-r--r--  1 danny  danny  0 Jun 26 16:41 file ab.bak
-rw-r--r--  1 danny  danny  0 Jun 26 16:23 file ac
-rw-r--r--  1 danny  danny  0 Jun 26 16:41 file ac.bak
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename1
-rw-r--r--  1 danny  danny  0 Jun 26 16:41 filename1.bak
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename2
-rw-r--r--  1 danny  danny  0 Jun 26 16:41 filename2.bak
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename3
-rw-r--r--  1 danny  wheel  0 Jun 26 16:41 filename3.bak
-rw-r--r--  1 danny  danny  0 Jun 26 16:05 filename4
-rw-r--r--  1 danny  danny  0 Jun 26 16:41 filename4.bak
Dan
  • 13,119