5

I'm running KDE Neon which is Ubuntu 18.04 Bionic (LTS) with the latest KDE desktop packages on top of it.

Since upgrading to this from Xenial I have lost the ability for processes to survive SSH session disconnects... specifically nohup, tmux, screen, byobu and all their children are killed when disconnecting.

Here's a simple test to demonstrate the problem

# connect over ssh
cd /tmp
nohup watch date &
ps -ef | grep watch
# disconnect and reconnect
ps -ef | grep watch # process is gone

I tried reverting tmux and byobu to older versions before discovering nohup was also effected. This leads me to believe that it's either SSHd config or systemd.

Apparently systemd decided to change it's default behaviour to nuke all users processes when a user disconnects some time ago. I tried to revert the behaviour by editing /etc/systemd/logind.conf and setting KillUserProcesses=no and then rebooting (service systemd-logind restart didn't work). It didn't work... I'm at my wits end.

Help!

ironstorm
  • 51
  • Hi Ironstorm, I see is not a single command you can send in background with a "&". Try to set it as a .service following this tread, link: https://askubuntu.com/questions/919054/how-do-i-run-a-single-command-at-startup-using-systemd . Let me know if was useful. – NwalmeThule Aug 28 '19 at 20:39
  • @NwalmeThule The & operator sends a process to the background as a job, but it's still owned by the shell (bash) which is in turn owned by SSH, which default configuration says to kill all the child processes on disconnect. – Kristopher Ives Aug 28 '19 at 21:03
  • Does this help https://unix.stackexchange.com/q/504606/323121 ? – Rusi Sep 05 '19 at 12:46

2 Answers2

4

I just had exactly the same issue, coincidentally also with Ubuntu 18.04 with the KDE Neon packages installed after the fact.

It turns out that systemd is terminating all of your user processes when your session ends, for example when you terminate your SSH login because you think that you're just going to re-attach later with tmux. ;)

To fix, do the following steps:

  1. Make the following changes to /etc/systemd/logind.conf:

    KillUserProcesses=no
KillExcludeUsers=root YOUR_USER_NAME

    and restart systemd-logind with:

    sudo systemctl restart systemd-logind

  2. Enable lingering for your account with:

    loginctl enable-linger YOU_USER_NAME

I initially only had the lingering enabled and the KillUserProcesses=no, but my tmux still got killed. Only after I modified KillExcludeUsers (thanks to https://askubuntu.com/a/1097134/59971 ) could I continue my neural network training in peace.

BeastOfCaerbannog
  • 14,585
Charl Botha
  • 2,826
1

You want to use the setsid command to make it a session leader.

nohup means "no hang up" which for the most part just disconnects the various standard input/output streams, but that process is still "owned" by the bash process which is in turn owned by the SSH process. When you disconnect from SSH, all those child processes go with it.

By using setsid you re-parent the process outside of the scope of SSH.

Kristopher Ives
  • 5,509
  • Nope, that's not how it works on RHEL or any other normal Unix-like system... When nohup is run and someone hangs up (disconnects), the process is supposed to continue running with init (pid 1) as the parent. – ironstorm Aug 28 '19 at 21:13
  • 1
    There are many implementations of nohup because it's both a command and a bash intrinsic, and ultimately any program can (and some do) go out of their way to reconnect the signals. In either case, setsid will solve these issues by explicitly re-parenting the process to pid 1 instead of relying on it to happen at a later time. – Kristopher Ives Aug 28 '19 at 23:57