367

What can I do to configure SSH on both client and servers to prevent Write Failed: broken pipe errors? It often occurs if you sleep your client computer and resume later.

LiveWireBT
  • 28,763
sorin
  • 9,888
  • 8
    Nothing really. The session was interrupted, and the security of the session was compromised. If you don't put the comp to sleep you can set a keep alive time for the client to shoot a keep alive heart beat to the server, but if the system's going to sleep then there is nothing that can be done. – darkdragn Apr 28 '12 at 23:44
  • SHH sessions rotate encryption keys over time, to prevent long timescale brute force attacks. Sleep / resume will break this security and the connection. – david6 Apr 29 '12 at 01:58
  • 4
    In this case I'm looking for something that would allow me to re-initiate a broken ssh connection (based probably on the exit code) and restore using screen? – sorin Apr 30 '12 at 07:11
  • 7
    You people are wrong: I have TWO desktop client machines connecting to the SAME server. One is ubuntu 12.10, Quantal, whose SSH client works well, and keeps the connection for hours. The other is Ubuntu 14.10, Utopic, just aside the other and in a fresh install; after a couple of minutes, it blocks itself with this message. The rest of the network functions in the machine are not interrupted. So no, it's neither a network problem, nor a server problem, but a specific SSH CLIENT software problem, which CAN be solved, opposite to what "darkdragan" dares to say, that "nothing can be done". – David L Dec 18 '14 at 06:09
  • 3
    And indeed, as I said: people talk too much when they say "nothing can be done", just as @darkdragn dared. I read Aram Kocharyan 's answer, and I applied it: 20 minutes ago... I realized that in my old Quantal Ubuntu 12.10, I had applied that instruction in that file [I just checked], two years ago, and that was the reason of the stability there. I did it here, and in these last 20 minutes, the connection has been stable since then. So please, people: refrain yourselves when daring to think that "nothing can be done", and refrain even more when trying to leave that message to other people. – David L Dec 18 '14 at 06:34
  • Nothing is as pleasant as the following situation. A person saying: "it can't be done / Nothing can be done", is replied or interrupted by someone saying: "look... It's done, and it was easy" @darkdragn – David L Dec 18 '14 at 07:06
  • 1
    @DavidL Agree that naysaying can be annoying, esp from a position of ignorance.

    However, it also seems you took it a bit personally (aka a tad ranty). darkdragn prob meant well

    – cavalcade Mar 29 '15 at 02:06
  • 13
    @DavidL you should read the questions better before ranting. Your problem is not the same as the OP's, who clearly mentions putting the computer to sleep. Which by the way only one of the answers address ("mosh"), and it was posted 2 years after the question. However the other answers do the next best thing, which is proposing solutions to cases that can be solved more easily, like yours. Chill out, don't be so stressed, ranting doesn't do any good around here... – msb Apr 08 '15 at 21:40
  • The same problem in the network when two devices with the same address. Turn off one of them and the problem is solved. – it ultrasport Dec 22 '16 at 11:05
  • I too faced the same issue. (even though the NATs were configured for the ports 22, and it could hit the sshd). Had issues in the PAM module. # ssh -l @127.0.0.1 Password: Read from remote host 127.0.0.1: Connection reset by peer Connection to 127.0.0.1 closed. Was facing this error in the ssh (in /var/log/auth.log)- " sshd[26712]: fatal: PAM: pam_setcred(): Permission denied" and switching "PermitEmptyPassword yes" to "PermitEmptyPassword no" solves the problem as in this post https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=315040 – ravi.zombie Jan 26 '17 at 01:54
  • Are you using tmux? I have the same problem on OS X, but only while tmux is running. Without tmux, the session persists -- even across sleeps. ServerAliveInterval is set BTW. – Jason Stewart Apr 24 '21 at 15:14

10 Answers10

345

I have tried this in /etc/ssh/ssh_config for Linux and Mac:

Host *
ServerAliveInterval 120

This is how often, in seconds, it should send a keepalive message to the server. If that doesn't work then train a monkey to press enter every two minutes while you work.

You could set either ServerAliveInterval in /etc/ssh/ssh_config of the client machine or ClientAliveInterval in /etc/ssh/sshd_config of the server machine. Try reducing the interval if you are still getting the error.

Configuration for a single user can be set in file ~/.ssh/config both on the server and client side. Make sure the file has correct permissions chmod 644 ~/.ssh/config.

teekarna
  • 103
  • 1
    Yeah I do similar and it works quite well for most things. – Oli Aug 25 '12 at 00:25
  • 5
    I'm not on a Mac, but Ubuntu 12.04 and the file for this operating system also seems to be ~/.ssh/config. – H2ONaCl Dec 14 '12 at 13:26
  • 7
    OS X 10.8.4 gives an error Bad configuration option: ClientAliveInterval – ohho Jul 15 '13 at 04:46
  • Hmm I'm using ServerAliveInterval 60 and ServerAliveCountMax 100 for OS X 10.8.4. – Aram Kocharyan Jul 15 '13 at 11:52
  • 3
    I get that same Bad configuration option error on OSX 10.8.4. – Nick Heiner Jul 18 '13 at 19:57
  • OS X 10.8.5 gives the same error as @ohho listed. – program247365 Sep 25 '13 at 16:09
  • 12
    Generally, you put these two commands into different parts of the system. Only ServerAliveInterval on the OSX client side... and only ClientAliveInterval on the sshd config file... – ftrotter Oct 23 '13 at 22:56
  • On Linux you can modify either /etc/ssh/ssh_config (system wide config) or ~/.ssh/config (user wide config). But because both configs apply only to the ssh client (and not server) you want to use only the property ServerAliveInterval, the other one applies only for the configuration of sshd and won't change a thing here. Check the man pages. – Huygens Jan 27 '14 at 21:51
  • 2
    Please do man ssh_config (client) and man sshd_config (server) and you'll see that both options are not available on both sides. – Ludovic Kuty Jan 29 '14 at 08:23
  • 1
    On the client or the remote? – AGamePlayer Feb 18 '14 at 07:19
  • 1
    Please see this answer: http://unix.stackexchange.com/a/2013/16127 – lobner Jun 25 '14 at 11:52
  • 1
    I would leave the time interval to 30 seconds, and a 30 seconds interval has worked perfectly for 2 years in my case, with a Quantal Ubuntu 12.10 installation. In a fresh install of Ubuntu Utopic 14.10 the same Broken Pipe problem was happening, until you remembered me of that great solution. Now this other machine ALSO works great, keeping the connection. Thanks, bro. – David L Dec 18 '14 at 06:37
  • Does this prevent the either of the machines from actually sleeping? – cavalcade Mar 27 '15 at 06:59
  • 6
    My monkey said to me: "Why don't you type yourself "top [ENTER]" – Augusto Jul 09 '15 at 15:34
  • 3
    Hello, I'm the monkey, and I managed to make it work, getting free bananas without doing anything living the dream – Marware Aug 26 '16 at 20:49
  • ClientAliveInterval fixed it for all clients, thank you! – Stanimir Stoyanov Oct 12 '16 at 10:05
  • 1
    Do you have to restart sshd after adding this change? – ZN13 Jun 18 '17 at 11:47
  • What happens if this happens while you're using the connection? – Owl Mar 11 '19 at 09:49
  • I still had the issues after making this change to the file.

    What worked for me was to set .ssh/ssh_config AND etc/ssh/ssh_config to the following:

    Host * ServerAliveInterval 10 ServerAliveCountMax 60

    – MrOli3000 May 27 '19 at 09:33
114

SSH sessions may break due to numerous and possibly unavoidable reasons.

A useful utility which can be used to mitigate problems caused by this is called screen. Screen is a powerful utility that allows you to control multiple terminals which will stay alive independently of the ssh session. For example, if you run screen in an ssh session you will see a new terminal open and you can use that to run jobs. Lets say your ssh session dies in the process. Running screen -d then screen -r will reopen the last session and you will be able to continue from there. Make sure you read some of the documentation before using it.

eltommo
  • 1,916
  • 7
    This is probably the best answer, I'm not sure why it's not voted up higher. The other "fixes" are helpful in the special case where you would actually care about maintaining an SSH connection, but in most use cases I imagine the real concern is that the intended processes continue to run, regardless of any client/server connection issues whatsoever. – Paul 'Joey' McMurdie Jun 03 '13 at 17:33
  • 23
    I would also add Tmux as an alternative to screen. I find it more versatile and stable than screen. – fridaymeetssunday Jan 05 '15 at 15:28
  • 4
    just leaving this here for future reference – you can conveniently run screen -d -r to recover your last session. – doplumi Nov 21 '16 at 07:41
  • 3
    Or simply screen -dr. Or screen -x depending on what you're planning on doing. The point is, one ought to know what all those switches do, so that one can use the appropriate ones and not just blindly follow internet people's suggestions. There's a nice compact summary available here: https://ss64.com/bash/screen.html – flith May 28 '18 at 06:27
  • 3
    This is not an answer to the problem – user3728501 Apr 16 '19 at 09:26
  • Is the solution to run screen after logging into the server or start screen in local PC and do SSH inside screen? – vineeshvs Sep 05 '19 at 07:41
  • The link to the documentation is no longer valid, this is a more recent one https://www.gnu.org/software/screen/manual/screen.html – Duccio A Nov 21 '19 at 09:31
  • 1
    And here an article on how to use it https://www.digitalocean.com/community/tutorials/how-to-install-and-use-screen-on-an-ubuntu-cloud-server – Duccio A Nov 21 '19 at 09:42
  • 1
    One last comment for newbies like me: screen should run on the server, not on your local machine. Before you run the process that should be independent of the SSH connection, run screen and then ctrl+a c. This creates a new window where the process can be started. Once started, type ctrl+a d to detach the window and go back the the original one. This can now be closed together with the SSH connection. When you reconnect screen -ls will show you that the window where the process is running is still available and accessible with screen -r <pid>. – Duccio A Nov 21 '19 at 10:08
63

Client configuration

Try creating the file:

~/.ssh/config

Add the contents:

Host *
  ServerAliveInterval 30
  ServerAliveCountMax 5

Now ssh to your server and see if your problem is fixed. ClientAliveInterval option is only useful when configuring the ssh server (aka sshd), it does not change a thing on the ssh client side, so don't use it in the above configuration file.

This will send a hello-are-you-there signal to the server if no packets have been received in the preceding 30 seconds (as specified above). However, if the number of consecutive hello-are-you-there signals reach ServerAliveCountMax then ssh will disconnect from the server. This value is defaulting to 3 (so 3*30 = 90 seconds without server activity), increase it if it suits your needs. There are alot more config options to the .ssh/config file and you could read:

Using an SSH Config File

For more information on other options. You may not want to apply this to every server you connect to which this example will. Or restrain it to only a particular server by replacing the line Host * with Host <IP> (replace by an IP address, see ssh_config man page).

Server configuration

Similarly you can tell the server to be gentle with your clients. The configuration file is /etc/ssh/sshd_config.

ClientAliveInterval 20
ClientAliveCountMax 5

You can either deactivate it by setting ClientAliveInterval to 0 or tweak ClientAliveInterval and ClientAliveCountMax to set a maximum ssh client inactivity without responding to the probes. One advantage of this settings over TCPKeepAlive is that the signals are sent through the encrypted channels, so it is less likely to be spoofable.

David Foerster
  • 36,264
  • 56
  • 94
  • 147
Matt
  • 712
34

I'm remotely upgrading an Ubuntu server from lucid to precise and lost the ssh connection in the middle of the upgrade with the message "Write failed. Broken pipe". ClientAliveInterval and ServerAliveInterval did nothing. The solution is to turn on TCPKeepAlive options in client ssh:

TCPKeepAlive yes

in

/etc/ssh/ssh_config

Hope this will help

28

For the client, edit your ~/.ssh/config (or /etc/ssh/ssh_config) file as follow:

Host *
  TCPKeepAlive yes
  ServerAliveInterval 120

TCPKeepAlive - Specifies whether the system should send TCP keepalive messages to the other side. If they are sent, death of the connection or crash of one of the machines will be properly noticed. However, this means that connections will die if the route is down temporarily, and some people find it annoying (The default is 'yes').

ServerAliveInterval - Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message through the encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server.


For the server, edit your /etc/ssh/sshd_config as:

ClientAliveInterval 600
ClientAliveCountMax 0

If you want ssh client to exit (timeout) automatically after 10 minutes (600 seconds).

ClientAliveCountMax – This indicates the total number of checkalive message sent by the ssh server without getting any response from the ssh client. Default is 3.

ClientAliveInterval – This indicates the timeout in seconds. After x number of seconds, ssh server will send a message to the client asking for response. Deafult is 0 (server will not send message to client to check.).


See also: What do the options ServerAliveInterval and ClientAliveInterval in sshd_config do, precisely?

kenorb
  • 10,347
  • Setting a ServerAliveCountMax higher than the default on the client should also help keep the connection live for slow connections. – I'm Root James Oct 01 '18 at 20:29
20

I absolutely love Mosh. I frequently ssh into a server, close my laptop and go to a cafe, open it up and carry on as if nothing changed.

Mosh (mobile shell)

Remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes.

Mosh is a replacement for SSH. It's more robust and responsive, especially over Wi-Fi, cellular, and long-distance links.

Mosh is free software, available for GNU/Linux, FreeBSD, Solaris, Mac OS X, and Android.

Jake
  • 301
  • 2
  • 3
6

I posted my answer here, as it was not a Ubuntu VM.

https://unix.stackexchange.com/questions/259225/packet-write-wait-broken-pipe-even-leaving-top-running

ssh -o IPQoS=throughput user@host
rupert160
  • 171
  • 1
  • 2
6

For me, I was getting Write failed: Broken pipe even when I was actively typing in vim or at the shell prompt. I couldn't browse the internet locally either for awhile either. (I was connecting remotely to Ubuntu using Terminal.)

Others in my network stream a lot of video from Netflix and other places. I can't prove it, but I suspect its an ISP or router issue. For example, Verizon and Netflix are pointing fingers at each other for their customer's network problems.

If you've got a dial-up connection and are streaming video or music with a simultaneous SSH or telnet connection, it's inevitable at some point you'll get a broken pipe message. Upgrading my ISPs broadband package seemed to make my broken connection less frequent.

pmagunia
  • 1,522
3

I have a script on the remote server that never seems to fails, regardless of the SSH configuration client or server.

#!/bin/bash
while true; do date; sleep 10; done;

Save it to some dummy.sh file and quickly run it before you minimize the window or move away from it. It will keep printing the current time stamp on the server and keeps your connection alive as long as the connection is not dropped by any other reason. When you get back to that terminal, just hit CTRL+C and keep working.

JulioHM
  • 634
1

You can add these args every time you invoke ssh: -o ServerAliveInterval=15 -o ServerAliveCountMax=3

You don't have to edit /etc/ssh/*config files if you do this.

You can create an bash alias or function or script to make this easy.

E.g. these bash functions, you can add into your .bashrc, do_ssh is used manually to turn on keepalives. do_ssh_pty is used within scripts to set pty and avoid prompts.

do_ssh() {
    ssh -o ServerAliveInterval=15 -o ServerAliveCountMax=3 $*
}

do_ssh_pty() {
    ssh -tt -o "BatchMode=yes" -o "StrictHostKeyChecking=no" -o ServerAliveInterval=15 -o ServerAliveCountMax=3 $*
}

Now do_ssh user@host can be used or do_ssh user@host <args> <command> and keepalives will be active.

gaoithe
  • 541
  • 4
  • 10
  • 2
    So for me the client side QoS setting didn't work but I compared sshd_config to a raspberry pi that did not have this and I found IPQoS 0x00 at the end of the config. I added this to my Ubuntu config, restarted sshd and now I'm good. – rjb101 Apr 15 '22 at 16:33