Help request - company server potentially compromised with xmrig

[szelenak]

Hello. Yesterday, my boss asked me to take a look at our Unraid server as the CPU was running quite hot. I logged in over SSH to look at htop and all our 24 of our CPU cores were absolutely pinned at 100%. The process using the most CPU turns out to be "xmrig" which I understand is a Monero cryptominer. Since I killed the process, I've been having a hard time locating the source. My understanding is that either our server is compromised, or one of our Docker plugins is infected. The process has not started again since I killed it yesterday.

 

I've looked through the syslog and didn't see anything suspicious, although it only goes back 84 days which is the last time the server was restarted. I'm the only Linux guy here at the office but I'm not very familiar with the s6 init system or Unraid in general, so looking around as root has yielded no findings.

 

I've noticed in other threads that the diagnostic .zip file is often requested to look through. This is our main company server that runs our DC, file server, and WDS, so I'd prefer to only upload specific log files if I possibly can.

 

I'm thinking it's one of our Docker containers but can't be sure. Maybe someone can point me in the right direction. Thanks a lot!

syslog.txt

[trurl]

Is your server accessible from the internet? If so fix that before anything else.

 

Possibly getting loaded from your go file. Shutdown, put flash in PC, post contents of config/go

[szelenak]

3 minutes ago, trurl said:

Is your server accessible from the internet? If so fix that before anything else.

Yes, I've identified that as a massive security risk to my boss. Ports 80 and 443 are open to the internet. We don't host the website locally but we host some services like GitLab.

 

7 minutes ago, trurl said:

Possibly getting loaded from your go file. Shutdown, put flash in PC, post contents of config/go

This was one of the first recommendations I saw when I was searching the forums. I don't believe there's anything weird going on here.

go

[trurl]

go file is normal.

 

27 minutes ago, szelenak said:

Ports 80 and 443 are open to the internet.

You absolutely must take your server off the internet immediately

 

Then disable Docker and VM Manager in Settings and reboot. See if xmrig is still running after that.

 

[trurl]

Only connection attempts I see in syslog is 12.1.1.x. This is not a private IP but maybe your company isn't doing this right. Seems to belong to AT&T.

[trurl]

10 minutes ago, trurl said:

Only connection attempts I see in syslog is 12.1.1.x

These do seem to be failed attempts using accounts that might be tried by hackers though.

[szelenak]

3 hours ago, trurl said:

go file is normal.

 

You absolutely must take your server off the internet immediately

 

Then disable Docker and VM Manager in Settings and reboot. See if xmrig is still running after that.

 

Great, thanks. I'll wait and see if xmrig starts again. Unfortunately, I killed the process before taking a closer look at it. All I saw in htop is "xmrig" but I noticed in this screenshot from this thread that there is a lot more information on how the process was started:

 

47 minutes ago, trurl said:

These do seem to be failed attempts using accounts that might be tried by hackers though.

Yep, I've confirmed those login attempts are from our PCs, unless of course one of our computers are compromised.

 

I had no luck finding any related .sh files or crontab entries, so I'm thinking the process was possibly executed from a tmp location and deleted itself, or maybe even one of our docker containers. I'm kicking myself for not taking a close look at the process when I had it in front of me...

[trurl]

4 minutes ago, szelenak said:

those login attempts are from our PCs

Why aren't they private IPs?

 

https://en.wikipedia.org/wiki/Private_network

[szelenak]

19 hours ago, trurl said:

Why aren't they private IPs?

 

https://en.wikipedia.org/wiki/Private_network

No idea. That's just how the network is set up here. I know they should be 10.1.1.X and I'll bring that up to the network guy.

 

In other news, xmrig is back and I've managed to capture it this time. Since I logged off yesterday, there's nothing new in the syslog besides this line:
Jul  7 00:40:16 SL-HOST crond[2156]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null
I've looked at the crontab and haven't seen anything suspicious. Anyways, screenshots and syslog will be attached.

 

edit: this doesn't look like a Docker thing to me. It must be on the system because it's able to use all of my cores, no?

syslog.txt

Edited July 7, 2022 by szelenak