I don't post much but I feel this is important to those using NGINX on their servers.

 

Technical details are here:

https://www.bleepingcomputer.com/news/security/new-malware-hides-as-legit-nginx-process-on-e-commerce-servers/

or, more directly:

https://sansec.io/research/cronrat

https://sansec.io/research/nginrat

 

Summary:

A vulnerability in NGINX allows a threat actor to install a RAT running virtually undetectable on your server.  One of the options is for it to also hide in CRON with a date of Feb 31.

 

I mention this because I believe my server got hit and it's very likely others could be vulnerable as well.  In the past I have noticed what I thought was a corrupt cron.d/root file and I've manually cleaned that file in the past.  Where I'm stumped is on how to clean the NGINX infestation.  I can identify the malicious processes and their solution is to just terminate them, however, every time I check, the process ID is different.

 

If anyone else has detected this activity on their server, I would really like to find a way to permanently eradicate NginRAT from my server.  All I've done so far is block the payload IP address on my router.  I only discovered this issue today.

How do you think you got hit?  And did it hit something like the nginx container or the WebUI for Unraid?

I don't know.  I've been running jlesage/nginx-proxy-manager for years and haven't touched it since the initial configuration.  It may not even be related to that.  I've only noticed issues with the CRON files but dismissed them as being potentially file corruption and didn't think anything of it at that time.

 

I run the sudo grep -al LD_L1BRARY_PATH /proc/*/environ | grep -v self/ command on the host directly and not from within a container.  Every time I run it I receive a new process ID.  I don't know if the process is restarting itself or if every instance is a new attack on my server.  There is never more than one process returned by this command.

 

There's supposed to exist /dev/shm/php-shared but my /dev/shm folder is empty.  Not sure if this is normal for Unraid or not.

 

 

On 12/2/2021 at 9:16 PM, earthworm said:

I don't know.  I've been running jlesage/nginx-proxy-manager for years and haven't touched it since the initial configuration.  It may not even be related to that.  I've only noticed issues with the CRON files but dismissed them as being potentially file corruption and didn't think anything of it at that time.

 

I run the sudo grep -al LD_L1BRARY_PATH /proc/*/environ | grep -v self/ command on the host directly and not from within a container.  Every time I run it I receive a new process ID.  I don't know if the process is restarting itself or if every instance is a new attack on my server.  There is never more than one process returned by this command.

 

There's supposed to exist /dev/shm/php-shared but my /dev/shm folder is empty.  Not sure if this is normal for Unraid or not.

 

 

 

Hmm i have not had anything like this, but the post makes me think of security in the dockers images we download. Payload can be downloaded with images and then attack the host from within maybe. Dont know how the security in that regard works... 

 

Which is why docker apps are awesome.  IF you've allowed a container to be reached from the outside world one way or another and it gets compromised, then that container only has access to certain folders etc that you have decided to give it, with the permissions (ro / rw) that you also gave it access to.

 

And, if you are allowing outside access to any given app then you also should make sure that the app is constantly up to date.  Either by checking yourself say monthly, or just simply having it auto update via the plugin so that you are getting the latest versions of the app / container, security updates etc.

 

Note that updates aren't just limited to the containers.  The major plugins (CA family, Dynamix, UD and more) are all constantly being reviewed for security issues and having updates issued to them not to mention the OS itself.

3 minutes ago, Squid said:

Which is why docker apps are awesome.  IF you've allowed a container to be reached from the outside world one way or another and it gets compromised, then that container only has access to certain folders etc that you have decided to give it, with the permissions (ro / rw) that you also gave it access to.

 

And, if you are allowing outside access to any given app then you also should make sure that the app is constantly up to date.  Either by checking yourself say monthly, or just simply having it auto update via the plugin so that you are getting the latest versions of the app / container, security updates etc.

 

Note that updates aren't just limited to the containers.  The major plugins (CA family, Dynamix, UD and more) are all constantly being reviewed for security issues and having updates issued to them not to mention the OS itself.

yeah i know about the mappings. but i was more thinking of the attack vector to attack at ports from inside the network to other dockers or the host it self. As there really is no docker firewall as far as i know

 

I'm really appreciating all the replies here.  My main goal was to bring attention to this issue and have people check their servers to ensure they haven't been compromised.  I doubt I'm the only one who has this problem.  It's possible it started with the CRON file and when that event fired it infected NGINX.

32 minutes ago, earthworm said:

It's possible it started with the CRON file

What did you see in the cron file?

I also see different process ID every time I run sudo grep -al LD_L1BRARY_PATH /proc/*/environ | grep -v self/

 

But I blocked the payload ip, what can I do more?

 

root@Thebox:~# sudo grep -al LD_L1BRARY_PATH /proc/*/environ | grep -v self/
/proc/10704/environ
root@Thebox:~# sudo grep -al LD_L1BRARY_PATH /proc/*/environ | grep -v self/
/proc/10759/environ
root@Thebox:~# sudo grep -al LD_L1BRARY_PATH /proc/*/environ | grep -v self/
/proc/10895/environ
root@Thebox:~# sudo grep -al LD_L1BRARY_PATH /proc/*/environ | grep -v self/
/proc/10907/environ
root@Thebox:~# ps
  PID TTY          TIME CMD
10685 pts/0    00:00:00 bash
11253 pts/0    00:00:00 ps
root@Thebox:~# sudo grep -al LD_L1BRARY_PATH /proc/*/environ | grep -v self/
/proc/11451/environ
root@Thebox:~# crontab -l
# If you don't want the output of a cron job mailed to you, you have to direct
# any output to /dev/null.  We'll do this here since these jobs should run
# properly on a newly installed system.  If a script fails, run-parts will
# mail a notice to root.
#
# Run the hourly, daily, weekly, and monthly cron jobs.
# Jobs that need different timing may be entered into the crontab as before,
# but most really don't need greater granularity than this.  If the exact
# times of the hourly, daily, weekly, and monthly cron jobs do not suit your
# needs, feel free to adjust them.
#
# Run hourly cron jobs at 47 minutes after the hour:
47 * * * * /usr/bin/run-parts /etc/cron.hourly 1> /dev/null
#
# Run daily cron jobs at 4:40 every day:
40 4 * * * /usr/bin/run-parts /etc/cron.daily 1> /dev/null
#
# Run weekly cron jobs at 4:30 on the first day of the week:
30 4 * * 0 /usr/bin/run-parts /etc/cron.weekly 1> /dev/null
#
# Run monthly cron jobs at 4:20 on the first day of the month:
20 4 1 * * /usr/bin/run-parts /etc/cron.monthly 1> /dev/null

I cant kill the process since its a new one every time and don't even show up with ps.

I was thinking of making a new Unraid usb and transferring my key but that sounds like a last resort, I tried scanning with antivirus but nothing.

 

any feedback would be appreciated.

Disable autostart on all containers in Docker, reboot.  Process will odds on be gone

 

Then start up the containers one at a time and see when it starts showing back up again.

same thing without dockers.

 

root@Thebox:~# sudo grep -al LD_L1BRARY_PATH /proc/*/environ | grep -v self/
/proc/14386/environ
root@Thebox:~# sudo grep -al LD_L1BRARY_PATH /proc/*/environ | grep -v self/
/proc/14396/environ
root@Thebox:~# ps
  PID TTY          TIME CMD
14035 pts/0    00:00:00 bash
14425 pts/0    00:00:00 ps
root@Thebox:~# 

 

If i start over with a new usb, is there anything i can/should save from my previous usb?

Post your diagnostics before we go down that road.

17 hours ago, Squid said:

Post your diagnostics before we go down that road.

 

thebox-diagnostics-20220126-0856.zip

Nothing particularly wrong in configuration.  Is this server directly accessible from the internet?  What ports are forwarded to it itself?

 

To create a "fresh" boot, simply delete everything from /config/ on the flash drive except for super.dat and your .key file and reboot

I have plex (32400 ), wireguard (51820) and swag, nextcloud, organizr and ombi. I used to have a rust game server (28015-28016).

I think the problem started the one time i forgot that i left some ports open to test things, server cpu was at 100% and things stopped working, pulled the power on server and fixed my router settings. It was the same time ubiquiti had a log4j problem so I did a fw reset, update and change pw to router. now only Cloudflare is allowed to connect to 443 that I mapped to 1443 for swag but that was not the case back then. 

 

everything works but I'm worried since I can't get rid off the  LD_L1BRARY_PATH thingy.

 

 

 

 

 

Edited January 26, 20224 yr by [email protected]
spelling

I just tested this myself and if you only get one process ID returned, it's the pid of the grep command you just ran since it also contains the string "LD_L1BRARY_PATH". Same happens if you type any string in case you want to test it for yourself

That's why it's exiting before you can see what it is and gives you a new pid every time, nothing to worry about if it's just one

 

root@TOWER:~# sudo grep -al LD_L1BRARY_PATH /proc/*/environ | grep -v self/
/proc/5098/environ
root@TOWER:~# sudo grep -al LD_L1BRARY_PATH_TEST /proc/*/environ | grep -v self/
/proc/5314/environ
root@TOWER:~# sudo grep -al HelloUnraid /proc/*/environ | grep -v self/
/proc/13914/environ