16

Yesterday I created an Ubuntu 18.04 droplet, with a MongoDB v4.0.2 image at DigitalOcean and today I checked the /var/log/auth.log file... What I saw is this:

Oct  1 16:16:25 mongodb-server-1 sshd[9171]: Failed password for root from 116.31.116.16 port 61535 ssh2
Oct  1 16:16:30 mongodb-server-1 sshd[9171]: message repeated 2 times: [ Failed password for root from 116.31.116.16 port 61535 ssh2]
Oct  1 16:16:30 mongodb-server-1 sshd[9171]: Received disconnect from 116.31.116.16 port 61535:11:  [preauth]
Oct  1 16:16:30 mongodb-server-1 sshd[9171]: Disconnected from authenticating user root 116.31.116.16 port 61535 [preauth]
Oct  1 16:16:30 mongodb-server-1 sshd[9171]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:17:01 mongodb-server-1 CRON[9173]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct  1 16:17:01 mongodb-server-1 CRON[9173]: pam_unix(cron:session): session closed for user root
Oct  1 16:17:34 mongodb-server-1 sshd[9176]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:17:36 mongodb-server-1 sshd[9176]: Failed password for root from 116.31.116.16 port 60613 ssh2
Oct  1 16:17:40 mongodb-server-1 sshd[9176]: message repeated 2 times: [ Failed password for root from 116.31.116.16 port 60613 ssh2]
Oct  1 16:17:40 mongodb-server-1 sshd[9176]: Received disconnect from 116.31.116.16 port 60613:11:  [preauth]
Oct  1 16:17:40 mongodb-server-1 sshd[9176]: Disconnected from authenticating user root 116.31.116.16 port 60613 [preauth]
Oct  1 16:17:40 mongodb-server-1 sshd[9176]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:18:43 mongodb-server-1 sshd[9178]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:18:45 mongodb-server-1 sshd[9178]: Failed password for root from 116.31.116.16 port 30163 ssh2
Oct  1 16:18:49 mongodb-server-1 sshd[9178]: message repeated 2 times: [ Failed password for root from 116.31.116.16 port 30163 ssh2]
Oct  1 16:18:49 mongodb-server-1 sshd[9178]: Received disconnect from 116.31.116.16 port 30163:11:  [preauth]
Oct  1 16:18:49 mongodb-server-1 sshd[9178]: Disconnected from authenticating user root 116.31.116.16 port 30163 [preauth]
Oct  1 16:18:49 mongodb-server-1 sshd[9178]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:19:50 mongodb-server-1 sshd[9183]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:19:53 mongodb-server-1 sshd[9183]: Failed password for root from 116.31.116.16 port 55398 ssh2
Oct  1 16:19:57 mongodb-server-1 sshd[9183]: message repeated 2 times: [ Failed password for root from 116.31.116.16 port 55398 ssh2]
Oct  1 16:19:57 mongodb-server-1 sshd[9183]: Received disconnect from 116.31.116.16 port 55398:11:  [preauth]
Oct  1 16:19:57 mongodb-server-1 sshd[9183]: Disconnected from authenticating user root 116.31.116.16 port 55398 [preauth]
Oct  1 16:19:57 mongodb-server-1 sshd[9183]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:20:57 mongodb-server-1 sshd[9186]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:20:59 mongodb-server-1 sshd[9186]: Failed password for root from 116.31.116.16 port 24942 ssh2
Oct  1 16:21:04 mongodb-server-1 sshd[9186]: message repeated 2 times: [ Failed password for root from 116.31.116.16 port 24942 ssh2]
Oct  1 16:21:05 mongodb-server-1 sshd[9186]: Received disconnect from 116.31.116.16 port 24942:11:  [preauth]
Oct  1 16:21:05 mongodb-server-1 sshd[9186]: Disconnected from authenticating user root 116.31.116.16 port 24942 [preauth]
Oct  1 16:21:05 mongodb-server-1 sshd[9186]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:22:15 mongodb-server-1 sshd[9188]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:22:18 mongodb-server-1 sshd[9188]: Failed password for root from 116.31.116.16 port 17758 ssh2
Oct  1 16:22:22 mongodb-server-1 sshd[9188]: message repeated 2 times: [ Failed password for root from 116.31.116.16 port 17758 ssh2]
Oct  1 16:22:23 mongodb-server-1 sshd[9188]: Received disconnect from 116.31.116.16 port 17758:11:  [preauth]
Oct  1 16:22:23 mongodb-server-1 sshd[9188]: Disconnected from authenticating user root 116.31.116.16 port 17758 [preauth]
Oct  1 16:22:23 mongodb-server-1 sshd[9188]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:23:15 mongodb-server-1 sshd[9190]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:23:17 mongodb-server-1 sshd[9190]: Failed password for root from 116.31.116.16 port 17471 ssh2
Oct  1 16:23:21 mongodb-server-1 sshd[9190]: message repeated 2 times: [ Failed password for root from 116.31.116.16 port 17471 ssh2]
Oct  1 16:23:22 mongodb-server-1 sshd[9190]: Received disconnect from 116.31.116.16 port 17471:11:  [preauth]
Oct  1 16:23:22 mongodb-server-1 sshd[9190]: Disconnected from authenticating user root 116.31.116.16 port 17471 [preauth]
Oct  1 16:23:22 mongodb-server-1 sshd[9190]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:24:19 mongodb-server-1 sshd[9209]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:24:20 mongodb-server-1 sshd[9209]: Failed password for root from 116.31.116.16 port 37695 ssh2
Oct  1 16:24:25 mongodb-server-1 sshd[9209]: message repeated 2 times: [ Failed password for root from 116.31.116.16 port 37695 ssh2]
Oct  1 16:24:26 mongodb-server-1 sshd[9209]: Received disconnect from 116.31.116.16 port 37695:11:  [preauth]
Oct  1 16:24:26 mongodb-server-1 sshd[9209]: Disconnected from authenticating user root 116.31.116.16 port 37695 [preauth]
Oct  1 16:24:26 mongodb-server-1 sshd[9209]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:25:26 mongodb-server-1 sshd[9214]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:25:27 mongodb-server-1 sshd[9214]: Failed password for root from 116.31.116.16 port 17403 ssh2
Oct  1 16:25:31 mongodb-server-1 sshd[9214]: message repeated 2 times: [ Failed password for root from 116.31.116.16 port 17403 ssh2]
Oct  1 16:25:32 mongodb-server-1 sshd[9214]: Received disconnect from 116.31.116.16 port 17403:11:  [preauth]
Oct  1 16:25:32 mongodb-server-1 sshd[9214]: Disconnected from authenticating user root 116.31.116.16 port 17403 [preauth]
Oct  1 16:25:32 mongodb-server-1 sshd[9214]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root

Oct  1 16:26:25 mongodb-server-1 sshd[9367]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root
Oct  1 16:26:27 mongodb-server-1 sshd[9367]: Failed password for root from 116.31.116.16 port 42236 ssh2
Oct  1 16:26:31 mongodb-server-1 sshd[9367]: message repeated 2 times: [ Failed password for root from 116.31.116.16 port 42236 ssh2]
Oct  1 16:26:32 mongodb-server-1 sshd[9367]: Received disconnect from 116.31.116.16 port 42236:11:  [preauth]
Oct  1 16:26:32 mongodb-server-1 sshd[9367]: Disconnected from authenticating user root 116.31.116.16 port 42236 [preauth]
Oct  1 16:26:32 mongodb-server-1 sshd[9367]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=116.31.116.16  user=root

Thousands of connection attempts logs! And it's still going!

I'm the only one with access to the server and the only port I've left open, is 22!

What's happening?

Sotiris Kaniras
  • 271
  • This may be a question better suited for the IT security stack exchange. Firstly though, I'd investigate that 116.36.116.16 IP and see what you can find on it, see if anyone else has complained about it or if it's from somewhere you'd absolutely never expect a connection from, hope this helps. – tommy61157 Oct 01 '18 at 17:04
  • I have almost no knowledge of security! How can I search for it? – Sotiris Kaniras Oct 01 '18 at 17:17
  • 3
    https://security.stackexchange.com/q/180321 –  Oct 01 '18 at 17:18
  • @jdv So it's something the system does? – Sotiris Kaniras Oct 01 '18 at 17:20
  • 1
    @SotirisKaniras I'm not sure where you'd get that idea. It is a common brute-force automated attack looking for weak passwords. But that link discusses things pretty thoroughly. –  Oct 01 '18 at 17:22
  • @jdv I misread it the first time! How on earth both of my servers that I created yesterday have already been targeted? Randomly? And will it stop? – Sotiris Kaniras Oct 01 '18 at 17:33
  • AskUbuntu uses a Question/Answer format so future readers can benefit. Comments are not intended for conversation, but to help you refine your question. If you are more comfortable with conversation-based support or need one-on-one training, then you might find our sibling site http://www.ubuntuforums.org a more satisfying venue. – user535733 Oct 01 '18 at 17:41
  • @SotirisKaniras your servers specifically weren't targeted, these services target all public IP ranges and when they find a server that is alive and responds to connection attempts, they tend to brute-force attempt. It doesn't mean you yourself are personally under attack, it's just something that is a standard of the Internet that there are service scanners and then service attacking bots in China that attempt to breach servers to hijack them. – Thomas Ward Oct 01 '18 at 18:22
  • Welcome to the Internet :) – marcelm Oct 01 '18 at 20:00
  • 1
    Do you have a server? If so, it's under attack: being under attack is the natural state of servers. – Mark Oct 01 '18 at 21:45
  • Change the default service port. – Pedro Lobito Oct 02 '18 at 14:36
  • 1
    @PedroLobito That's just 'security through obscurity' - it's not a long-term effective protection practice. – Thomas Ward Oct 02 '18 at 16:21
  • @PedroLobito: If you have advice to give please write an answer so it can go through the regular voting and review process. – David Foerster Oct 02 '18 at 17:43

1 Answers1

47

This specific traffic is from a Chinese-sourced IP address (basic info of the IP address on dnslytics.com) and it is attempting to login with password authentication to your root user over SSH.

There are major concerns when running any Internet-facing service:

  1. ALL IP addresses everywhere when they get online are probed.
  2. When some probes find open ports such as SSH ports, malicious threat actors will attempt to continue probes to see if they can get into your system with password attacks.

Both of these are a defacto standard of Internet-facing services. And as such, many of these threats are ongoing. However, this happens to many services - not just SSH.

These types of probes are unlikely to cease. This is why you should be careful when exposing services to the Internet.

Based on what I've seen in the past, and my knowledge of IT Security, as well as the first-hand knowledge I've gained thanks to running multiple Internet-facing services myself, this activity looks like typical service scanning and probing activity that happens to most systems that are directly facing the Internet. It does not mean your server is directly under attack. Merely, what has happened is a service scanner found your server responded on port 22, and is repeatedly coming back and attempting to authenticate with weak passwords in an attempt to breach the server. This is not uncommon to see on Internet-facing connections.

There are a few things you can do, however, to mitigate this a little bit more:

  1. Disable SSH login access for the root user directly.

    Edit /etc/ssh/sshd_config, find the line that says PermitRootLogin and make sure it's set to prohibit-password or no.

    Note that you will need to have a non-root user that you can login to if you do this; this way you protect the root user, and you have a non-root user who can have sudo access configured for them so they can still execute superuser commands as needed. (NEVER SSH as root for your admin functions and actions!)

  2. Disable password authentication, and set up SSH Key Authentication as the only viable SSH login mechanism. There are a lot of guides on how to do this, such as this one from Digital Ocean.

  3. Set up something like fail2ban to help block the brute force attempts. This is a complicated process in and of itself, but you can get basic setup done by doing sudo apt install fail2ban. This will set itself up by default to be enabled to protect SSH connectivity.

  4. Set up a firewall before you continue adding additional services. This way you can only receive connections you trust to services you want to offer to the Internet rather than leave everything exposed.

Rich
  • 753
Thomas Ward
  • 74,764
  • 2
    This is a great summary, and there are some good tutorials on the digital ocean documentation site to follow to harden a server. One thing to add is setting up a firewall, UFW, and getting it configured and running before adding any new services. – Ian McGowan Oct 01 '18 at 19:22
  • I already had ufw rules and now I disabled root access... I think I'm a little more secure now... – Sotiris Kaniras Oct 01 '18 at 20:12
  • 1
    @SotirisKaniras I would still move towards SSH key auth only for login, password auth can mean your standard user could be bruteforced still, while key auth helps to avoid that risk. Otherwise, enjoy! :) – Thomas Ward Oct 01 '18 at 20:13
  • @ThomasWard I already had enabled ssh keys on my application servers, but unfortunately I can't do the same on my db servers, because if I do, I'm having a connectivity issue between them! – Sotiris Kaniras Oct 01 '18 at 20:18
  • 1
    @ThomasWard 1 last question... If I install fail2ban, will it interfere with my existing ufw rules? – Sotiris Kaniras Oct 01 '18 at 20:20
  • I wonder if you could create some sort of honeypot "shell" and start feeding them all sorts of garbage. Being from China, maybe give them a bunch of Falun Gong materials. Would need to somehow negotiate with the remote to use a null cipher so the authorities could get a good look at it. – Nick T Oct 01 '18 at 20:38
  • 1
    @SotirisKaniras why would your db servers be open for connections from internet? shouldn't they be on same internal network as with services using them? – eis Oct 02 '18 at 06:16
  • 1
    @eis I didn't really thought about that... (I'm still new to this) Do you know if I can change that, now that my servers are deployed? – Sotiris Kaniras Oct 02 '18 at 06:19
  • @SotirisKaniras Configure ufw to only allow communication between the two servers for the DB component on the DB server. Block all other IPs reaching to the port. That way you shield the DB parts from probes. (Too tricky to explain how on my phone but the ufw man page might help explain how to do IP-source-based allows for certain ports and traffic) – Thomas Ward Oct 02 '18 at 11:40
  • @ThomasWard You could also consider adding: 5. Change the port that sshd uses to something other than 22. – The Vermilion Wizard Oct 02 '18 at 17:29
  • @ThomasWard That was part of my initial setup, so I've done it already! Do you believe I'm somewhat protected now? – Sotiris Kaniras Oct 03 '18 at 10:15
  • @Erik Security through Obscurity - that's not really going to 'fix' the problem at all. Changing the default port is only security through obscurity. Relevant: https://security.stackexchange.com/questions/32308/should-i-change-the-default-ssh-port-on-linux-servers – Thomas Ward Oct 03 '18 at 14:41