1

I'm trying to accomplish a simple task using the excellent post here: I would like to remove all files from a folder except those listed in a .txt file contained in the folder. The syntax works fine, but for some reason all files are removed - basically the script is saying that there are no correspondences between the filenames in the folder and those in the .txt file - but I can see that there are!

I thought maybe there might be some invisible white space after my file names that I cannot see (in the folder or the .txt file). But I don't think so...Is there another solution that I am not thinking of?

MeC
  • 111
  • 2
    Try looking with the od -c command. You might find things like \r (return) are hidden just before the \n(newline). – ubfan1 Oct 25 '19 at 22:09
  • You could also try this frome within the directory ** replace exclude.txt with your list file ** ls -p | grep -v / | sed 's/\<exclude.txt\>//g' | sort | comm -3 - <(sort exclude.txt) | xargs this will only output without removing. To remove run ls -p | grep -v / | sed 's/\<exclude.txt\>//g' | sort | comm -3 - <(sort exclude.txt) | xargs rm – Raffa Oct 25 '19 at 23:41
  • @ubfan1 - there are indeed \n and \r at the end of each line. I removed the \r but don't I need to keep the \n to differentiate between each filename entry in the txt file? – MeC Oct 25 '19 at 23:44
  • is there a syntax error in that @raffa ? – MeC Oct 25 '19 at 23:51
  • ls -p | grep -v / | sed 's/\<a.txt\>//g' | sort | comm -3 - <(sort a.txt) | xargs No it is just a one line command that is easier and will do the job. I replaced exclude.txt with a.txt assuming this is the name of your list. If you run this command, it will only show you the output. If that is what you want, append rm after adding one space at the end to remove the files. – Raffa Oct 25 '19 at 23:55
  • This leaves me with the same issue, however. It wasn't that the initial syntax was wrong, it's that there are \r and \n at the end of each line and I don't know how to structure my txt file so that each filename is read separately but also so that \n and \r are not present on each line – MeC Oct 26 '19 at 00:10
  • \n is okay and should be there if you run od -c a.txt but \r is not and shouldn't be there. – Raffa Oct 26 '19 at 00:14
  • I removed \r and still both approaches fail to recognize file names listed in the txt file and just remove everything. – MeC Oct 26 '19 at 00:16
  • Probably, recreate the file in nano editor. Copy and paste the lines one by one and press Enter once after each line except the last line no Enter after it. This should work. Good Luck – Raffa Oct 26 '19 at 00:20
  • I can't manually enter. There are 13000 lines in the txt file. – MeC Oct 26 '19 at 00:21

2 Answers2

1

It seems that your a.txt file contains hidden characters due to possibly being created or edited on other OS's like Windows.

You need to sanitize the file first in order to use it without syntax errors.

The easiest way is to use a tool called dos2unix. Please follow these steps to do so:

Firstly: Install dos2unix by opening a terminal and running the following command:

sudo apt install dos2unix

Secondly: Back up your a.txt file.

Thirdly: Use dos2unix on your file by running the following command in the terminal from within the directory containing your a.txt file:

dos2unix a.txt

Finally: Test your file. It should be fine now.

Raffa
  • 32,237
0

The script you are referring to will give false positives. for will expand filenames literally, this means special characters will be preserved and using it as a pattern will break the matching.

Example:

#!/bin/bash

# the script is adapted and slightly shortened.
shopt -s nullglob
for i in a* b*; do
   grep -Fx -e "$i" exclude.txt
done

Output:

$ ls
a   'a'$'\n''b'   b   exclude.txt

$ cat exclude.txt
a
b

$ script.sh
a
a
b
b

A better approach would be to preserve the shell-escape. 

#!/bin/bash

shopt -s extglob

eval "list=( \
    $(ls --quoting-style=shell-escape !(exclude.txt) | \
        grep -vFx -f exclude.txt))"

for i in "${list[@]}"; do ls "$i"; done

Example:

$ cat exclude.txt  
'a'$'\n''b'

$ script.sh
a
b

$ cat exclude.txt  
a
b

$ script.sh
'a'$'\n''b'