How to delete words from txt file, that exists on another txt file?

Question

File a.txt has about 100k words, each words is in new line

july.cpp
windows.exe
ttm.rar
document.zip

File b.txt has 150k words, one word by line - some words are from file a.txt, but some words are new:

july.cpp    
NOVEMBER.txt    
windows.exe    
ttm.rar    
document.zip    
diary.txt

How can I merge this files into one, delete all duplicate lines, and keep lines that are new (lines that exist in a.txt but don't exist in b.txt, and vice versa)?

Looks like a homework from laboratory classes on University, am I right? Anyway, Your question is not about Ubuntu, please ask it on StackOverflow site. — , Jul 25 '14 at 21:44
@MikołajBartnicki no, this is not for university homework, we have holiday now. I could done it in microsoft excel, but I don't want to use MS software. — Kate-Kasia, Jul 25 '14 at 21:51
@MikołajBartnicki Unix.SE would probably be a better place to ask — Glutanimate, Jul 25 '14 at 21:52
@MikołajBartnicki I saw your answer, it was in one line, but after refreshing page, answer disappeared. — Kate-Kasia, Jul 25 '14 at 22:01
Kasia, I have made a mistake in my answer, that is why I deleted it. I am working on a new one. — , Jul 25 '14 at 22:04
@Seth I agree. I should have phrased my comment differently. What I meant was: If we were to argue about scope and the right SE site, Unix.SE would probably be more apt than SO. — Glutanimate, Jul 26 '14 at 11:39
there is trailing spaces in the second list, very hard to detect, can be a reason answers got so overcomplicated. — , Dec 02 '19 at 21:09

score 13 · Accepted Answer · 2014-07-25T22:33:07.153

13

There is a command to do this: comm. As stated in man comm, it is plain simple:

   comm -3 file1 file2
          Print lines in file1 not in file2, and vice versa.

Note that comm expects files contents to be sorted, so You must sort them before calling comm on them, just like that:

sort unsorted-file.txt > sorted-file.txt

So to sum up:

sort a.txt > as.txt

sort b.txt > bs.txt

comm -3 as.txt bs.txt > result.txt

After above commands, You will have expected lines in the result.txt file.

edited Jul 25 '14 at 22:33

answered Jul 25 '14 at 22:06

thank you, it works like a charm. PS. to zdjęcie z tłuczkiem na Twoim profilu jest fajne ;-) – Kate-Kasia Jul 25 '14 at 22:18

score 2 · Answer 2 · edited Apr 13 '17 at 12:24

Here is a short python3 script, based on Germar's answer, which should accomplish this while retaining b.txt's unsorted order.

#!/usr/bin/python3

with open('a.txt', 'r') as afile:
    a = set(line.rstrip('\n') for line in afile)

with open('b.txt', 'r') as bfile:
    for line in bfile:
        line = line.rstrip('\n')
        if line not in a:
            print(line)
            # Uncomment the following if you also want to remove duplicates:
            # a.add(line)

score 1 · Answer 3 · answered Jul 25 '14 at 22:09

1

#!/usr/bin/env python3

with open('a.txt', 'r') as f:
    a_txt = f.read()
a = a_txt.split('\n')
del(a_txt)

with open('b.txt', 'r') as f:
    while True:
        b = f.readline().strip('\n ')
        if not len(b):
            break
        if not b in a:
            print(b)

answered Jul 25 '14 at 22:09

Germar

6,377
2
26
39

2

Man, you are shooting a mosquito with a naval cannon! – Jul 25 '14 at 22:11
:-) You're right. I missed the 'k' in 100k – Germar Jul 25 '14 at 22:14

score 1 · Answer 4 · answered Jul 25 '14 at 22:11

Have a look at the coreutils comm command - man comm

NAME
       comm - compare two sorted files line by line

SYNOPSIS
       comm [OPTION]... FILE1 FILE2

DESCRIPTION
       Compare sorted files FILE1 and FILE2 line by line.

       With  no  options,  produce  three-column  output.  Column one contains
       lines unique to FILE1, column two contains lines unique to  FILE2,  and
       column three contains lines common to both files.

       -1     suppress column 1 (lines unique to FILE1)

       -2     suppress column 2 (lines unique to FILE2)

       -3     suppress column 3 (lines that appear in both files)

So for example you can do

$ comm -13 <(sort a.txt) <(sort b.txt)
diary.txt
NOVEMBER.txt

(lines unique to b.txt)

How to delete words from txt file, that exists on another txt file?

4 Answers4