52

I want merge (union) output from two different commands, and pipe them to a single command.

A silly example:

Commands I want to merge the output:

cat wordlist.txt
ls ~/folder/*

into:

wc -l

In this example, if wordlist.txt contains 5 lines and 3 files, I want wc -l to return 8.

$cat wordlist.txt *[magical union thing]* ls ~/folder/* | wc -l
8

How do I do that?

David Oneill
  • 12,144

3 Answers3

78

Your magical union thing is a semicolon... and curly braces:

    { cat wordlist.txt ; ls ~/folder/* ; } | wc -l

The curly braces are only grouping the commands together, so that the pipe sign | affects the combined output.

You can also use parentheses () around a command group, which would execute the commands in a subshell. This has a subtle set of differences with curly braces, e.g. try the following out:

    cd $HOME/Desktop ; (cd $HOME ; pwd) ; pwd
    cd $HOME/Desktop ; { cd $HOME ; pwd ; } ; pwd

You'll see that all environment variables, including the current working directory, are reset after exiting the parenthesis group, but not after exiting the curly-brace group.

As for the semicolon, alternatives include the && and || signs, which will conditionally execute the second command only if the first is successful or if not, respectively, e.g.

    cd $HOME/project && make
    ls $HOME/project || echo "Directory not found."
BeastOfCaerbannog
  • 14,585
pablomme
  • 5,500
  • 2
    That does work! Help me learn - what exactly are the curly braces doing in here? What other magical powers do they have? – David Oneill May 07 '12 at 19:13
  • 1
    @DavidOneill { list; } is a compound command. From bash(1): list is simply executed in the current shell environment. list must be terminated with a newline or semicolon. [..] The return status is the exit status of list. – Lekensteyn May 07 '12 at 19:18
  • 1
    I've added a bit more info to the answer. The definitive guide to shell scripting with bash is the bash manpage, type man bash from the command line and browse it. – pablomme May 07 '12 at 19:24
  • should the ; in those commands not be && in case the file wordlist.txt does not exists? – Rinzwind May 07 '12 at 19:33
  • If wordlist.txt does not exist an error from cat will appear on standard error, but not on standard output, so wc -l will not count any lines from it. Same goes for ~/folder not existing. One could add 2> /dev/null between each of these commands and their semicolon to prevent noise on standard error, but other than being ugly, error messages are harmless for the purpose of counting lines. – pablomme May 07 '12 at 19:39
  • yes but now the user is confident and tries to parse a file and tries to remove the content of that with an rm... suddenly the ; looks very scary. – Rinzwind May 07 '12 at 19:51
  • Which is why I described the && and || operators. The semicolon is appropriate in the cat ; ls example though. – pablomme May 07 '12 at 20:26
  • Note the mandatory space: Works: { cat f.txt;ls -l;}|wc -l, Syntax Error: {cat f.txt;ls -l;}|wc -l – Rolf Mar 11 '18 at 11:25
  • If using the fish shell, replace { with begin; and replace } with end. So the example would be: begin; cat wordlist.txt ; ls ~/folder/* ; end | wc -l – Aaron Feldman Nov 30 '18 at 02:43
9

Since wc accepts a file path as input, you can also use process substitution:

wc -l <(cat wordlist.txt; ls ~/folder/*)

This roughly equivalent to:

echo wordlist.txt > temp
ls ~/folder/* >> temp
wc -l temp

Mind that ls ~/folder/* also returns the contents of subdirectories if any (due to glob expansion). If you just want to list the contents of ~/folder, just use ls ~/folder.

Lekensteyn
  • 174,277
  • 1
    The wc -l was just a made up example, i'm actually piping it into something more complicated that doesn't have this option. – David Oneill May 07 '12 at 19:12
  • 3
    @DavidOneill Well, as cat accepts a file argument, you can use cat <(cat wordlist.txt; ls wordlist.txt) | wc -l and even cat wordlist.txt <(ls wordlist.txt) | wc -l. This is of course very ugly, but it demonstrates the endless possibilities with command line tools. – Lekensteyn May 07 '12 at 19:14
  • 2
    @DavidOneill Correct, I've corrected it now, thanks. – Lekensteyn May 07 '12 at 19:25
  • 2
    You want ls ~/folder/ so that if ~/folder is a symbolic link ls lists the contents of its target instead of the link itself. By the way, all the ls commands should be followed by -1 so that a single file per line is printed and wc -l is a valid way of counting files. – pablomme May 07 '12 at 20:28
  • @pablomme You're true about the symlink thing, but -1 is implied as the output is not a terminal but a pipe. – Lekensteyn May 07 '12 at 20:44
  • You're right, sorry. – pablomme May 07 '12 at 21:45
3

I was asking myself the same question, and ended up writing a short script.

magicalUnionThing (I call it append):

#!/bin/sh
cat /dev/stdin
$*

Make that script executable

chmod +x ./magicalUnionThing

Now you do

cat wordlist.txt |./magicalUnionThing ls ~/folder/* | wc -l

What it does:

  • Send standard input to standard output
  • Execute argument. $* returns all arguments as a string. The output of that command goes to script standard output by default.

So the stdout of magicalUnionThing will be its stdin + stdout of the command that is passed as argument.

There are of course simpler ways, as per other answers.
Maybe this alternative can be useful in some cases.

Rolf
  • 2,031