How to insert a line break between stderr and stdout

Question

This question helped me understand how to combine stderr and stdout:

with this command:

gunzip -vc /opt/minecraft/wonders/logs/20* 2>&1

But how may one insert a line break between the two so that the output can appear on separate lines instead of clumped together?

The stderr comes first, which is always 1 line, something like this:

/opt/minecraft/wonders/logs/2017-08-28-2.log.gz:

Whereas the stdout is usually (but not always) multiple lines, like:

06:17:05: Starting minecraft server version 1.10.2
06:18:21: Loading properties
06:18:21: Default game type: SURVIVAL
06:18:21: Generating keypair
06:18:21: Starting Minecraft server on *:25565
06:18:22: Using default channel type

But because there is no newline between them the first line from each file to be unzipped is always something like this:

/opt/minecraft/wonders/logs/2018-08-18-2.log.gz:    06:17:05: Starting minecraft server version 1.10.2

I need these two parts on different lines so I can filter them with grep. But I also want the corresponding stdout directly following its stderr (its filename) so I know which file it comes from. Like this:

/opt/minecraft/wonders/logs/2018-08-18-2.log.gz:    
06:17:05: Starting minecraft server version 1.10.2
06:18:21: Loading properties
06:18:21: Default game type: SURVIVAL
06:18:21: Generating keypair
06:18:21: Starting Minecraft server on *:25565
06:18:22: Using default channel type

This is the command I am piping into using with grep:

egrep -iv "^\[.*(stopping|starting|saving|keep|spawn|joined|lost|left|: (<.*>|\/)|\/(help|forge))

Its purpose is to remove certain lines from the log files, leaving the pertinent ones.

Can you add more detail as to how exactly you're trying to filter with grep ? Especially, can you explain the end goal you're trying to achieve ? If you need filtering on both streams, you could probably do gunzip -vc /opt/minecraft/wonders/logs/20* > >(grep 'stdin pattern' ) 2> >(grep 'stderrpattern' ) — Sergiy Kolodyazhnyy, Aug 20 '18 at 22:34

score 3 · Answer 1 · answered Aug 20 '18 at 22:09

3

Here is one simple way to do this, assume this command:

ls -d Documents/ NotExist/ /dev/null

the output is:

ls: cannot access 'NotExist/': No such file or directory
/dev/null  Documents/

the first line is stderr and the second is stdout, if I pipe the output to xargs I can print a new line after stderr and then print the stdout:

ls -d Documents/ NotExist/ /dev/null | xargs echo -e '\n'

which gives me:

ls: cannot access 'NotExist/': No such file or directory

 /dev/null Documents/

answered Aug 20 '18 at 22:09

Ravexina

55,668
25
164
183

This didn't work. I added more detail to my question which could explain why. The error message I get is xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option. Using the -O option, double quotes or no quotes removed the error message but now only a small bit of output shows up, as far as I can tell the first line of each file from stdout. Thanks. – Jesse Yishai Aug 20 '18 at 22:30

Sergiy Kolodyazhnyy · Answer 2 · 2018-08-20T23:38:14.133

2

It's possible with duplicating file descriptors. Long story, short: pipe stderr, where it can be processed, and send stdin to the controlling terminal ( short hand for that is /dev/tty ) so that it still shows on the screen:

$ stat /etc/passwd noexist 2>&1 > /dev/tty | while IFS= read -r line; do echo "---"; echo "$line"; echo "---"; done
  File: /etc/passwd
  Size: 2380        Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d  Inode: 937653      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-08-20 13:05:01.171247259 +0800
Modify: 2018-07-24 04:51:36.799057737 +0800
Change: 2018-07-24 04:51:36.867020035 +0800
 Birth: -
---
stat: cannot stat 'noexist': No such file or directory
---

Here I'm separating stderr with --- line, but you get the idea.

Another way, if you want to visually separate them is to colorize the output. Based on Balazs Pozar's answer, you could do:

command 2> >(while read line; do echo -e "\e[01;31m$line\e[0m" >&2; done)

To address the edit on the question, if you need to filter both stdout and stderr with grep (for example, if you're looking for a pattern in stdout, but another pattern in stderr ), you use multiple process substitutions >():

gunzip -vc /opt/minecraft/wonders/logs/20* > >(grep 'stdin pattern' ) 2> >(grep 'stderrpattern' )

Addressing the comments, if you only need to separate the filename /opt/minecraft/wonders/logs/2017-08-28-2.log.gz: from further content, you could use sed:

gunzip -vc /opt/minecraft/wonders/logs/20* | 
sed -r 's/^(\/opt\/minecraft\/wonders\/logs\/2018-08-18-2.log.gz:) *(.*)$/\1\n\2/'

Test:

$ echo /opt/minecraft/wonders/logs/2018-08-18-2.log.gz:    06:17:05: Starting minecraft server version 1.10.2 | sed -r 's/^(\/opt\/minecraft\/wonders\/logs\/2018-08-18-2.log.gz:) *(.*)$/\1\n\2/'
/opt/minecraft/wonders/logs/2018-08-18-2.log.gz:
06:17:05: Starting minecraft server version 1.10.2

edited Aug 20 '18 at 23:38

answered Aug 20 '18 at 22:13

Sergiy Kolodyazhnyy

105,154
20
279
497

1

@JesseYishai I think there's no way to reorder them, or no simple method at least, without writing whole program that monitors both streams and maybe timestamps them. – Sergiy Kolodyazhnyy Aug 20 '18 at 22:44
All 3 methods did indeed separate the stdout and the stderr, but also produced the results without proper order, by which I mean the stdout of a file is not always lined up (following) its stderr. They are out of order. Also, I don’t need filtering on the stderr, except for the option to omit it entirely if it is followed by certain elements (or lack of them) in the stdout. Thanks. – Jesse Yishai Aug 20 '18 at 22:46
@JesseYishai So you're basically trying to do grep -v 'pattern' on stderr - hide "pattern" but show other errors ? – Sergiy Kolodyazhnyy Aug 20 '18 at 22:50
I am grep -v 'pattern' on stdout, hiding certain elements, with the heading from stderr (the filename the stdout is coming from) preceding it. – Jesse Yishai Aug 20 '18 at 22:52
@JesseYishai OK. So you just want to find that filename and separate it ? I'll edit my answer again, hopefully that can help, so please wait. Btw, if these lines are coming from your Minecraft server, and not gnzup, they're just regular stdin in the log file. – Sergiy Kolodyazhnyy Aug 20 '18 at 23:01
@JesseYishai OK. I've added my last edit. I think I'm lacking clarity as to what exactly should be done, but I did try my best to answer. Best regards – Sergiy Kolodyazhnyy Aug 20 '18 at 23:07
Thanks @Sergiy, this almost does it, but what I need is the filename directly above its content instead of concatenated with the first line of its content. I want to know which file the content comes from, but I want the filename and the content on separate lines so I can filter them with grep. I understand if you're done with this one, but thanks for trying. – Jesse Yishai Aug 20 '18 at 23:31
@JesseYishai I think I see what you mean, and I've changed the sed command. Please review – Sergiy Kolodyazhnyy Aug 20 '18 at 23:38
1

You might want to use a different sed delimiter, like |, so you don't have to escape the slashes: 's|^(/opt/minecraft/wonders/logs/2018-08-18-2.log.gz:) *(.*)$|\1\n\2|' – wjandrea Aug 21 '18 at 00:14
@wjandrea Yes, that'd be a better approach. – Sergiy Kolodyazhnyy Aug 21 '18 at 00:15
Still no luck. But shouldn't sed look like this sed -r 's|^(/opt/minecraft/wonders/logs/20.*) *(.*)$|\1\n\2|') anyway? – Jesse Yishai Aug 21 '18 at 00:29
@JesseYishai The .* in the first group of the pattern would probably be greedy match, capturing whole line, so I think that wouldn't work. Maybe place : at the end of .*. Also remove the last ) – Sergiy Kolodyazhnyy Aug 21 '18 at 00:33

score 1 · Answer 3 · answered Aug 21 '18 at 01:51

What's really happening I think is that stdout is getting mixed into stderr, because the former is line-buffered and the latter unbuffered by default. To illustrate:

$ gunzip -vc ../foo.txt.gz
../foo.txt.gz:  some text here
more text here
 26.7%

(note that -v is causing two pieces of information to be emitted on stderr: the filename, and the % compression - with standard output falling in between).

If you line-buffer stderr as well, the streams will appear on separate lines:

$ stdbuf -eL gunzip -vc ../foo.txt.gz
some text here
more text here
../foo.txt.gz:   26.7%

however not necessarily in the order that you require - you might try using the sponge utility to soak up all of stdout before outputing it, so that the stderr output (filename and %) appears in the terminal first:

gunzip -vc ../foo.txt.gz | sponge
../foo.txt.gz:   26.7%
some text here
more text here

How to insert a line break between stderr and stdout

3 Answers3