25

I have a VPS running Ubuntu 16.04.5 that's been going for a number of years now with little issue. Today, however, I found I was unable to access the server using SSH, receiving 'connection refused' errors. I accessed the server using my VPS host's serial console service, and traced the issue down to openssh server failing to start. Here's an output of service status, service start, and sshd -t following a fresh reboot:

root@167:/# service ssh status
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: failed (Result: start-limit-hit) since Fri 2019-01-18 04:56:42 EST; 24min ago
  Process: 983 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=255)

Jan 18 04:56:42 167 systemd[1]: Failed to start OpenBSD Secure Shell server.
Jan 18 04:56:42 167 systemd[1]: ssh.service: Unit entered failed state.
Jan 18 04:56:42 167 systemd[1]: ssh.service: Failed with result 'exit-code'.
Jan 18 04:56:42 167 systemd[1]: ssh.service: Service hold-off time over, scheduling restart.
Jan 18 04:56:42 167 systemd[1]: Stopped OpenBSD Secure Shell server.
Jan 18 04:56:42 167 systemd[1]: ssh.service: Start request repeated too quickly.
Jan 18 04:56:42 167 systemd[1]: Failed to start OpenBSD Secure Shell server.
Jan 18 04:56:42 167 systemd[1]: ssh.service: Unit entered failed state.
Jan 18 04:56:42 167 systemd[1]: ssh.service: Failed with result 'start-limit-hit'.
root@167:/# service ssh start
Job for ssh.service failed because the control process exited with error code. See "systemctl status ssh.service" and "journalctl -xe" for details.
root@167:/# sshd -t
Missing privilege separation directory: /var/run/sshd

I've attempted some research into this, but nothing that's come up seems to have an actual solution - Just endless cycles of 'I have this problem' with no answers, answers that are outdated, or just generally unhelpful information.

Does anybody have any ideas on what to do next to troubleshoot/resolve this issue? SSH was last working about 12 hours ago when I logged in to run updates and rebooted the server.

UPDATE

Issue was resolved using workaround 1 as provided here: https://askubuntu.com/a/1110843/531533

SierraKomodo
  • 351
  • On my system, /var/run is a link to /run and /run/sshd exists, is owned by root and has permissions 755. – Jos Jan 18 '19 at 10:44
  • My understanding of that directory is it's a temporary folder that starts empty every time the OS boots - My research into this issue indicated the ssh service is not creating the /var/run/sshd directory, though I've yet to find anything to give a precise reason as to why or how to correct it.

    Supposedly, this can be 'temporarily' fixed by creating the folder manually, but I'd have to do this every time the server boots.

     – SierraKomodo Jan 18 '19 at 10:49
  • 3
    I'm experiencing the same problem! And had post the same question (I should post an answer there.). Aparantly this is as bug of the current version of systemd with old kernels. The workaround that I doun is to modify /usr/lib/tmpfiles.d/sshd.conf in this way d /run/sshd 0755 root root. – pa4080 Jan 18 '19 at 10:53

1 Answers1

10

According to the bug report here, the problem can be worked around by adding the directory creation code to /etc/rc.local.

I've done some research and I still can't figure out what is supposed to create that directory on a normal server startup - there're a couple of scripts under /etc that do this, but they appear to be deprecated as they relate to the old SysV startup and I can't find anything in the SystemD init setup to create that directory.

[Update]: Just noticed @pa4080 comment, and this seems to be it - the file /usr/lib/tmpfiles.d/sshd.conf exists and contains the line d /var/run/sshd 0755 root root, in openssh-server version 1:7.2p2-4ubuntu2.6.

The tmpfiles.d directory is part of the systemd-tmpfiles service and is documented here .

The could be an update anomaly as I've seen relevant bug reports that discuss this as a failed update, and updating the openssh-server to the latest version might fix it.

Guss
  • 3,535
  • The version for openssh matches what you're showing - It doesn't look like updating is an option right now, as apt states this is the newest version. I'll see if a patch comes out later today. If not, I'll give @pa4080's suggestion a try - I'm not too keen on making changes to a system file that was suggested back in '08, so I probably won't be trying the update to rc.local. – SierraKomodo Jan 18 '19 at 11:00
  • Check the file specified by pa4080 - does it have the a line like in my updated answer? – Guss Jan 18 '19 at 11:03
  • It does - Exactly as described: 
    `d /var/run/sshd 0755 root root`
    – SierraKomodo Jan 18 '19 at 11:08
  • 2
    On a 16.04 test system I got, running systemd-tmpfiles --create causes the directory to be created. This command should have been run by the SystemD init process on boot, but if there was no restart after an update that changed the behavior for how openssh-server uses that directory, that could explain the problem you had. – Guss Jan 18 '19 at 11:22
  • 2
    SierraKomodo, I just post an answer where I've described also a third method by using crontab. – pa4080 Jan 18 '19 at 11:38
  • Modifying /usr/lib/tmpfiles.d/sshd.conf resolved the issue. – SierraKomodo Jan 18 '19 at 20:31