Dreaded error: Out Of Memory errors detected on your server

[MissMagdalene]

 

I haven't logged into my Unraid server in a month or two but I went in today and found this error above ^^. Clicking "More Information" led me to this forum post which essentially says to capture diagnostics BEFORE REBOOTING, which I did download and have attached here, and make a post under General Support.

 

My first instinct is to review my RAM memory usage, screenshot below.

 

And my overall storage array still have 18T of space remaining.

 

Any help would be greatly appreciated as I have not read through any diagnostics before and am not sure if I would have noticed any warning signs leading up to this. I have not rebooted my server yet in case anyone who can help needs more information.

 

Thanks in advance internet friends.

bespin-diagnostics-20240201-1446.zip

[trurl]

Feb  1 14:38:48 Bespin root: Fix Common Problems: Warning: Possible mining software running

Is this expected?

[trurl]

Have you been hacked?

Jan 28 04:30:08 Bespin root: FCP Debug Log: nobody   25491 69.9 14.6 3231888 2401512 ?     Sl   Jan18 9934:36      \_ ./xmrig -o randomxmonero.eu.nicehash.com:3380 -u NHbW7fJpRFQxQU8nFHP72FC6zCoDA714Xgg6.15 -k --nicehash --coin monero -a rx/0 -cpu-max-threads-hint=100 --cpu-freq=4000
Jan 28 04:30:08 Bespin root: FCP Debug Log: nobody   14347 63.3 14.6 3231888 2399984 ?     Sl   Jan21 6200:16      \_ ./xmrig -o randomxmonero.eu.nicehash.com:3380 -u NHbW7fJpRFQxQU8nFHP72FC6zCoDA714Xgg6.15 -k --nicehash --coin monero -a rx/0 -cpu-max-threads-hint=100 --cpu-freq=4000
Jan 28 04:30:08 Bespin root: FCP Debug Log: nobody   16019 62.6 14.6 3231892 2399976 ?     Sl   Jan22 5157:40      \_ ./xmrig -o randomxmonero.eu.nicehash.com:3380 -u NHbW7fJpRFQxQU8nFHP72FC6zCoDA714Xgg6.15 -k --nicehash --coin monero -a rx/0 -cpu-max-threads-hint=100 --cpu-freq=4000
Jan 28 04:30:08 Bespin root: FCP Debug Log: nobody    6083 53.6 14.6 3231888 2399940 ?     Sl   Jan23 3868:27      \_ ./xmrig -o randomxmonero.eu.nicehash.com:3380 -u NHbW7fJpRFQxQU8nFHP72FC6zCoDA714Xgg6.15 -k --nicehash --coin monero -a rx/0 -cpu-max-threads-hint=100 --cpu-freq=4000
Jan 28 04:30:08 Bespin root: FCP Debug Log: nobody   10950 61.6 14.6 3231888 2399628 ?     Sl   Jan25 2809:17      \_ ./xmrig -o randomxmonero.eu.nicehash.com:3380 -u NHbW7fJpRFQxQU8nFHP72FC6zCoDA714Xgg6.15 -k --nicehash --coin monero -a rx/0 -cpu-max-threads-hint=100 --cpu-freq=4000
Jan 28 04:30:08 Bespin root: Fix Common Problems: Warning: Possible mining software running

 

[MissMagdalene]

Quote

Have you been hacked?

I saw that error too and was planning to make a separate post about it. 
 

but any direction/suggestions on how to remove a hacker/unknown would be greatly appreciated as well. 

[trurl]

https://docs.unraid.net/unraid-os/manual/security/

[trurl]

26 minutes ago, MissMagdalene said:

remove a hacker/unknown

Should be your first priority.

[MissMagdalene]

So I search in the forums for the "Possible mining software running" but nothing came up but my post here. Broadening my search to just "mining software" I ran into this forum post. Which basically directed me to a few files to review if they were compromised or had stuff I didn't put in them, which did not seem to be the case for me.

 

I'm still unsure how or if I was indeed hacked, no other users were visibly present via the GUI or looking via console/terminal.

 

I have restarted my server and will monitor if either of these errors come up again. I have also strengthened my Unraid server user password and reviewed my port forwards on my router and those are secure to be sure, they align with the suggestions referenced by @trurl here https://docs.unraid.net/unraid-os/manual/security/.

[trurl]

23 minutes ago, MissMagdalene said:

still unsure how or if I was indeed hacked

If you didn't intend to have mining software on your server, then someone else put it there.

 

Do you have any docker containers you didn't install?

[MissMagdalene]

 

28 minutes ago, trurl said:

Do you have any docker containers you didn't install?

No, there were no docker containers that I didn’t install myself. Unless they could be hidden from the GUI view?

 

same with VMs, none that were not installed by me. 

[trurl]

2 minutes ago, MissMagdalene said:

no docker containers that I didn’t install myself. Unless they could be hidden from the GUI view?

On Dockers page, move slider at upper right to Advanced View.

[T0rqueWr3nch]

Run the following command and give us the output:

 

ps -auxf | grep -v grep | grep -i xmrig

 

This is what Fix Common Problems is looking for. Kudos @Squid for thinking to include this.

 

We need to go into damage control mode and figure out if they've established persistence and how.

 

Did you ever expose your Unraid server to the internet? Ever port forwarded to SSH?

[MissMagdalene]

36 minutes ago, trurl said:

On Dockers page, move slider at upper right to Advanced View.

No additional containers showing under Advanced View.

 

10 minutes ago, T0rqueWr3nch said:

Run the following command and give us the output:

 

ps -auxf | grep -v grep | grep -i xmrig

 

Output reveals nothing:

root@Bespin:~# ps -auxf | grep -v grep | grep -i xmrig
root@Bespin:~# 

 

11 minutes ago, T0rqueWr3nch said:

Did you ever expose your Unraid server to the internet? Ever port forwarded to SSH?

I have never exposed SSH port forward. I have a Plex server port forwarded with the Plex default (32400) and 3 other port forwards in the 8xxx range.

 

I have a static IP if that makes any difference.

[T0rqueWr3nch]

On one hand, good that the output reveals nothing, which is probably to be expected since you currently aren't running out of memory...on the other hand, now we're in this ambiguous state on if we're still compromised since persistence is always a concern.

 

Good that you've never exposed SSH. And you've never exposed your Unraid Web GUI to the internet correct? What are the other forwards to?

 

The logs show this happened on the 28th- did you have anything (Docker containers, plugins, etc.) then that you don't have now?

[MissMagdalene]

12 minutes ago, T0rqueWr3nch said:

And you've never exposed your Unraid Web GUI to the internet correct? What are the other forwards to?

No I have my GUI exposed to the internet (my static IP) but I have beefed up my password at the start of this. I like to check things from anywhere is the reason.

 

The other port forwards are to Tautulli and Deluge dockers, the GUIs. EDIT: And Krusader, I forgot.

 

12 minutes ago, T0rqueWr3nch said:

The logs show this happened on the 28th- did you have anything (Docker containers, plugins, etc.) then that you don't have now?

None that were added or removed by me, again I hadn't logged into my Unraid until authoring this post, so Feb 1st. Not sure if the logs would provide entries for anything added or removed in that window of time?

 

There was a lot of scanning happening on my Plex server coming from Deluge downloads recently (and from me moving download locations via Deluge GUI), but I can't remember how far back that was, it feels like well before 01/28 but maybe I'm wrong. Reviewing my Deluge activity I'm seeing now that the 'Date Added' in my Deluge GUI is incorrect. Some entries date added shows 12/01/2024...So I'm not sure if I can identify what was being downloaded/seeded/moved on that date.

Edited February 2 by MissMagdalene
Forgot about Krusader port forward. That totals the 3 I mentioned

[trurl]

4 minutes ago, MissMagdalene said:

I have my GUI exposed to the internet (my static IP)

Looks like your server has a LAN IP address so not clear what you mean.

 

In any case, you should use Wireguard (builtin) or other VPN, or Unraid Connect, to access your server remotely.

[MissMagdalene]

Woke up to the mining software error again this morning. Ran the grep for xmrig and found something this time.

 

root@Bespin:~# ps -auxf | grep -v grep | grep -i xmrig
nobody   18199  292 14.7 3231768 2404120 ?     Sl   Feb03 4047:15      \_ ./xmrig -o 167.88.169.6:3333 -u NHbW7fJpRFQxQU8nFHP72FC6zCoDA714Xgg6.15 -k --nicehash --coin monero -a rx/0 --cpu-max-threads-hint=100 --cpu-freq=4000

 

No unexpected dockers running in the advanced view via GUI.

Edited February 4 by MissMagdalene
added details about docker

[trurl]

Diagnostics shows stock go file, and no /boot/extra folder. So either the OS has been hacked, or one of your plugins, dockers, or VMs had been hacked.

 

Disable Docker and VM Manager in Settings, then reboot in SAFE mode. This will insure no dockers, VMs, or plugins are running. Then check for xmrig.

 

Let it run like that for a while and keep checking for xmrig.

 

If it doesn't come back, then one of your plugins, dockers, or VMs has been hacked and is causing it.