My server potentially hacked . How to clean a server from rogue processes

Question

I am a newbie and my first server seems to have been hacked. I am not sure how to clean it. There are processes hogging the CPU that am not the owner of. All the 12 CPUs are almost all running at 100%. Kindly see htop screen shot htop on the server

I have run a netstat and below is a summary of the output

sudo netstat -anolp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    Timer
tcp        0      0 127.0.0.1:9001          0.0.0.0:*               LISTEN      4869/java           off (0.00/0/0)
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      15327/mongod        off (0.00/0/0)
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      4124/mysqld         off (0.00/0/0)
tcp        0      0 127.0.0.1:9100          0.0.0.0:*               LISTEN      24415/node_exporter off (0.00/0/0)
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      16039/nginx -g daem off (0.00/0/0)
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      5824/config.ru      off (0.00/0/0)
tcp        0      0 127.0.0.1:9168          0.0.0.0:*               LISTEN      24370/ruby          off (0.00/0/0)
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      13595/sshd          off (0.00/0/0)
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      2370/master         off (0.00/0/0)
tcp        0      0 127.0.0.1:9121          0.0.0.0:*               LISTEN      24450/redis_exporte off (0.00/0/0)
tcp        0      0 127.0.0.1:9090          0.0.0.0:*               LISTEN      24429/prometheus    off (0.00/0/0)
tcp        0      0 127.0.0.1:9187          0.0.0.0:*               LISTEN      24421/postgres_expo off (0.00/0/0)
tcp        0      1 my.ser.ver.ip:41037    172.247.116.47:2017     SYN_SENT    26521/mbb           on (57.32/6/0)
tcp        0      0 my.ser.ver.ip:45599    124.112.1.160:29135     ESTABLISHED 32265/DDosClient    off (0.00/0/0)
tcp        0      0 127.0.0.1:9001          127.0.0.1:54729         ESTABLISHED 4869/java           keepalive (2412.64/0/0)
tcp        0      0 my.ser.ver.ip:49366    31.222.161.239:8080     ESTABLISHED 14106/hald-daemon   off (0.00/0/0)
tcp        0      0 my.ser.ver.ip:57275    163.172.204.219:443     ESTABLISHED 20238/wDHnu         keepalive (45.40/0/0)
tcp        0      0 127.0.0.1:9001          127.0.0.1:54909         ESTABLISHED 4869/java           keepalive (2445.40/0/0)
tcp        0      0 my.ser.ver.ip:56654    45.125.34.159:6969      ESTABLISHED 27006/XDTQK         off (0.00/0/0)

I tried killing them using kill with their PID one by one but after some time they came up again. So I checked my cron, and there were some entries there that I didn't do so I clear them as well. I compared the crons with a new box I just launched.

I disabled the root login, created another sudoer with pem file instead. Now I need to find a way to clean this once and for all and I lack experience. Can anyone please help me out?

Thanks in advance

Nuke it from the orbit and rebuild. That's the only sure way — Sergiy Kolodyazhnyy, Nov 29 '17 at 02:14

Videonauth · Answer 1 · 2017-11-29T01:29:07.160

If you're 100% sure you got hacked, the best way of action is to set up from scratch.

Sure you could maybe fix it and get your machine back, but would you really be able to trust in that installation afterwards?
Do you know you fixed all backdoors they might have installed?

And most importantly take that server offline so it cant be used for the attackers purposes.

While the machine is offline for public you might want to diagnose the breach and learn from it security wise, and then set it up from scratch. I see from your screen-shot that you have GitLab running on that machine. This should be the only data you should backup and take over, but not before scanning it thoroughly and checking the database for any rogue entries which would allow new breaches.

And forget about playing back a backup, it might contain still the hole which allowed the attacker to get in.

So back up your data and then install this machine from scratch, and foremost re-roll all passwords which might have been compromised as well in the process.

Sorry to tell you this but thats the best course of action you can take.

Very well noted. Will wait for couple more answers and will take a decision — highjo, Nov 28 '17 at 18:47
Fine with me :) I wont be able to get any reputation todays anyways hehehe, hit the cap already for today. — Videonauth, Nov 28 '17 at 18:48
If you are a new user it is safe to assume any potential cracker knows more than you do, thus a fresh install is the way to go. I would not transfer ANYTHING from the cracked server PERIOD. Read up on security, there is a reason the root account is locked by default, and I guess you found out why. — Panther, Nov 28 '17 at 18:51

My server potentially hacked . How to clean a server from rogue processes

1 Answers1