Warning: Unraid Servers exposed to the Internet are being hacked


jonp

Recommended Posts

On 3/25/2021 at 3:20 AM, jonp said:

Hello Unraid Community!

 

It has come to our attention that in recent days, we've seen a significant uptick in the amount of Unraid server's being compromised due to poor security practices.  The purpose of this post is to help our community verify their server's are secure and provide helpful best-practices recommendations to ensuring your system doesn't become another statistic.  Please review the below recommendations on your server(s) to ensure they are safe.

 

 

 

I do wonder if its just not a lot easier to have a VPN system running on your router or another device (openvpn, wireguard)..  Sure you need to set up the access before you can lopg on to the webpage but that does not seem like a big step to take ?  Its the way I have been doing this for years...

 

Is exposing your server to the internet (even with SSL) actually a good idea ?

Link to comment
4 minutes ago, Helmonder said:

 

Is exposing your server to the internet (even with SSL) actually a good idea ?

Depends on your skill and knowledge level. If you are asking the question because you don't know how to do it safely, the answer is no.

 

Each bit of exposure increases your risk. If you follow best practices, the risk is quite low. Threats are constantly changing, so exposing any device to the internet requires ongoing learning and adapting.

Link to comment
Depends on your skill and knowledge level. If you are asking the question because you don't know how to do it safely, the answer is no.
 
Each bit of exposure increases your risk. If you follow best practices, the risk is quite low. Threats are constantly changing, so exposing any device to the internet requires ongoing learning and adapting.

Its not that difficult.. I was just wondering why people prefer this to the von solution…. Also wondering my idea that a vpn is preferable is actually -true- or just me beiing an old guy..


Verzonden vanaf mijn iPhone met Tapatalk
  • Like 1
Link to comment
4 minutes ago, Helmonder said:


Its not that difficult.. I was just wondering why people prefer this to the von solution…. Also wondering my idea that a vpn is preferable is actually -true- or just me beiing an old guy..


Verzonden vanaf mijn iPhone met Tapatalk

Not sure what portion you are saying is not difficult. Proper security while exposing services IS difficult, because of all the interconnected moving pieces that are involved.

 

VPN simplifies it greatly, because you only have 1 service and port to audit and keep up to date.

  • Like 1
Link to comment
20 minutes ago, jonathanm said:

VPN simplifies it greatly, because you only have 1 service and port to audit and keep up to date.

 

Plus a VPN solution, like WireGuard has build-in protection to keep intruders out. Though nothing is perfect, at least it is a whole lot harder than a password only defense.

  • Thanks 1
Link to comment

A few suggestions if I may, from my experiences in the Cloud Infrastructure World;

 

First, Reviewing Docker Folder Mappings (and to some extent VM Shares).

 

Do all you Docker Containers need read and write access to non appdata folders? If it does, is the scope of the directories restricted to what is needed, or have you given it full read/write to /mnt/user or /mnt/user0 ?

 

For example I need Sonnarr and Radarr to have write access to my TV and Movie Share, so they are restricted to just that, they don't need access to my Personal Photos, or Documents etc. Whereas for Plex, since I don't use the Media Deletion Feature, I dont need Plex, to do anything to those Folders, just read the content. So it has Read Only Permissions in the Docker Config.

 

Additionally, I only have a few containers that need read/write access to the whole server (/mnt/user) and so these are configured to do so, but since they are more "Administration" containers, I keep them off until I need them, most start up in less than 30 seconds.

 

That way, if for whatever reason a container was compromised, the risk is reduced in most cases.

 

Shares on my VM's are kept to only the required directories and mounted as Read Only in the VM.

 

For Docker Containers that use VNC or VMs, set a secure password for the VNC component too, to prevent something on the Network from using it without access (great if you don't have VLAN's etc).

 

This may be "overkill" for some users, but have a look at the Nessus or OpenVAS Containers, and run regular Vulnerability Scans against your Devices / Local Network. I use the Nessus one and (IMO) its the easier of the two to setup, the Essentials (Free) version is limited to 15 IPs, so I scan my unRAID Server, VMs, and a couple of other physical devices and it has SMTP configured so once a week sends me an email with a summary of any issues found, they are categorized by importance as well. 

 

I don't think many people do this, but don't use the GUI mode of unRAID as a day to day browser, outside of Setup and Troubleshooting (IMO) it should not be used. Firefox, release updates quite frequently and sometimes they are for CVE's that depending on what sites you visit *could* leave you unprotected.

 

On the "Keeping your Server Up-to-Date" part, while updating the unRAID OS is important, don't forget to update your Docker Containers and Plugins, I use the CA Auto Update for them, and set them to update daily, overnight. Some of the Apps, could be patched for Security Issues, and so keeping the up-to-date is quite useful. Also, one that I often find myself forgetting is the NerdPack Components, I have a few bits installed (Python3, iotop, etc), AFAIK these need to be updated manually. Keeping these Up-to-Date as well is important, as these are more likely to have Security Issues that could be exploited, depending on what you run.

 

Also on the Updates, note, if you have VM's and they are running 24/7 keep these up-to-date too and try and get them as Hardened as possible, these can often be used as a way into your server/network. For Linux Debian/Ubuntu Servers, you can look at Unattended Upgrades, similar alternatives are available for other Distros. For Windows you can configure Updates to Install Automatically and Reboot as needed. Hardening the OS as well, is something I would also recommend, for most common Linux Distros and Windows, there are lots of guides useful online, DigitalOcean is a great source for Linux stuff I have found.

 

If something is not available as a Docker Container or Plugin, don't try and run it directly on the unRAID Server OS itself (unless, its for something physical, e.g. Drivers, or Sensors etc), use a VM (with a Hardened Configuration), keeping only the bare minimum running directly on unRAID, helps to reduce your attack surface.

 

Also, while strictly not part of Security, but it goes Hand in Hand, make sure you have a good Backup Strategy and that all your (important/essential) Data is backed up, sometimes stuff happens and no matter how much you try, new exploits come out, or things get missed and the worst can happen. Having a good backup strategy can help you recover from that, the 321 Backup method is the most common one I see used.

 

If something does happen and you need to restore, where possible, before you start the restore, try and identify what happened, once you have identified the issue, if needed you can restore from Backups to a point in time, where there was no (known) issue, and start from there, making sure you fix whatever the issue was first in your restored server. I have seen a few cases (at work) where peoples Servers have been compromised (typically with Ransomware), they restore from backups, but don't fix the issue (typically a Weak Password for an Admin account, and RDP exposed to the Internet) and within a few hours of restoring, they are compromised again.

 

Other ideas about using SSH Keys, Disabling Telnet/FTP etc, are all good ones, and definitely something to do, and something I would love to see done by default in future releases.

 

EDIT: One other thing I forgot to mention was, setup Notifications for your unRAID server, not all of them will be for Security, but some of the apps like the Fix Common Problems, can alert you for security related issues and you can get notified of potential issues quicker than it may take you to find/discover them yourselves.

 

Edited by timethrow
Mention Setup of Notifications
  • Thanks 9
Link to comment
2 hours ago, jaybee said:

What has changed then? Why are hacks occurring more now?

 

Answered earlier in the thread.

 

In the past people may not have been exposing their server to the internet but now they may want the "My Servers" functionality so they tried to do something. They forwarded too many ports or DMZed their server and simply didn't secure it properly to start with.

 

Nothing wrong with "My Servers", but it may have been the push for some to try out things they really shouldn't have. Humans are flawed. They tend to skip instructions or pick parts randomly to follow and ignore the rest.

  • Like 1
Link to comment
2 hours ago, jaybee said:

What has changed then? Why are hacks occurring more now?

 

Another theory...  When the announcement was made that LimeTech was providing tools to allow users to easily access their servers directly from the Internet, hackers suddenly decided that this might well be a low hanging piece of fruit.  These hackers (while they might be classed as 'loners') do 'network' with each other to exchange information.   And I suspect that there will be a lot of Unraid users who have going to trying to see what can be done without having sufficient knowledge about the required security, monitoring techniques, and tools to keep the bad guys at bay.    

 

I have spend enough time attempting to help people on this forum that I know folks don't like to read, understand and follow instructions.  When their Unraid server is securely behind a router and its firewall, this usually only results in something not working.  When you move beyond that firewall and expose the server to the Internet, you can't afford to make a mistake!  There will be somebody out waiting there to take advantage of you.  And often they will find you within a couple of hours.

  • Like 1
Link to comment

 

8 hours ago, timethrow said:

Do all you Docker Containers need read and write access to non appdata folders?

 

Perhaps a simple change would makes it easier to users to find out what may happen:

 

On "Docker" page, add the permission (r/w/rw) behind the path in the column "VOLUME MAPPINGS (APP TO HOST)". E.g:

 

/config --> /mnt/pool_nvme/system/appdata/plex (r/w)
/mnt --> /mnt (r Slave)
/transcode --> /tmp/plex/ (r/w)

 

 

Edited by hawihoney
Link to comment
On 3/26/2021 at 9:43 AM, Helmonder said:

 

I do wonder if its just not a lot easier to have a VPN system running on your router or another device (openvpn, wireguard)..  Sure you need to set up the access before you can lopg on to the webpage but that does not seem like a big step to take ?  Its the way I have been doing this for years...

 

Is exposing your server to the internet (even with SSL) actually a good idea ?

 

Remote access is a tricky thing to get perfect.  Ultimately there is a sliding scale between convenience, security, and complexity in terms of how you can do it and the risks/costs that come with each method can vary.  For example, let's say you go with a Wireguard solution on Unraid.  By default, anyone connecting would have a full connection to the entire server, not just a single application.  So for management access, sure, that is a more overall secure way of approaching this, but it comes at the added cost of needing to download and configure a VPN client for the devices you wish to use to connect.  Might not be a big deal to you, but if you want the flexibility of being able to use ANY device with a browser to connect to your server, a VPN isn't going to serve that purpose if you don't have the rights to install apps on the device you want to use.

 

Another consideration is the application you want to serve remotely.  Plex can work with a VPN, but its far easier to use without one.  Do you really want to walk your non-techie friends/family through how to connect to your VPN and use Plex?

 

So for just the management side of things, I definitely think a full VPN tunnel is great, but if you need more flexibility or don't want to have to configure that, HTTPS is very solid.

Link to comment
3 hours ago, Maor said:

This just sreams, how Unraid has poor security. How long till we can setup more users and control permissions like on any other linux distro?
Will we ever get SFTP of NFSv4 with password protection...

 

I used to respond to messages like this with the "Unraid is meant to be used within your network and should not be exposed to the internet" paradigm..

Link to comment
On 3/25/2021 at 10:32 PM, SpencerJ said:

It’s found under Users in each password section. Good question

Thanks, didn't realize the script and gui was only triggered when the password was typed into the users change password interface.  Makes sense though. 

  • Like 1
Link to comment
On 3/27/2021 at 3:09 AM, gnollo said:

The only way to reach content remotely on my server is via the emby docker. I set it up using cloudflare as a go to as in the guide below

https://blog.awelswynol.co.uk/2018-01-setting-up-cloudflare-with-emby/

Is this a good setup security wise? I have a password for my emby admin login, will update it to make it stronger as well.

 

Using Cloudflare is a great idea since they offer restriction rules such as GeoIP filtering and some exploit protection.

 

It's still not a good idea to expose services directly to the Internet, pointing your port forwards at a reverse proxy container like traefik will also require attackers to know the DNS before they can do anything and effectively hides services from appearing in port scan directories (someone searching specifically for emby servers to exploit will have a harder time finding it) .

 

For anything not web-based VPN with private key authentication is the only solution.

Link to comment
3 hours ago, zee said:

 

Using Cloudflare is a great idea since they offer restriction rules such as GeoIP filtering and some exploit protection.

 

It's still not a good idea to expose services directly to the Internet, pointing your port forwards at a reverse proxy container like traefik will also require attackers to know the DNS before they can do anything and effectively hides services from appearing in port scan directories (someone searching specifically for emby servers to exploit will have a harder time finding it) .

 

For anything not web-based VPN with private key authentication is the only solution.

 

Some emby users suggested to lock down the firewall to only accept tcp/443 connections from the Cloudflare ip ranges and also use the fail2ban cloudflare integration with monitoring of the emby logs to ban anyone who tries to brute force a login. I have also hidden all accounts from the login screen. More info in the thread attached.

https://emby.media/community/index.php?/topic/69779-remote-security-i-just-dont-get-it/

  • Thanks 1
Link to comment
On 4/2/2021 at 2:39 PM, trurl said:

You can have any other linux distro in a VM.

I can just stop using Unraid... Not a solution. The security issues are known for ages, hopefully more hacks will happen and the devs are forced to fix them.

Link to comment
3 minutes ago, Maor said:

I can just stop using Unraid... Not a solution

You seem to believe that unRAID is a full Linux distribution.  It is not.  It may never implement users, permissions and security in the way you would expect of a full Linux distro or any other OS.

 

As a special use appliance OS, unRAID does what it is expected to do in the way it was designed to do it.  Are there issues and problems that need to be fixed?  Yes, absolutely.  Does unRAID need to be completely redesigned to fix these issue?  No.

 

Obviously, you can quit using unRAID at any time if it does not meet your needs.  It is not for everyone and perhaps it does not do what you want or expect it to do.

 

The hacks are a result of unRAID being used improperly rather than unRAID failing to function in the manner in which it was designed.  I would expect to see more emphasis on educating and helping users to implement unRAID and its supporting infrastructure properly rather than a product redesign to address problems it was never intended to address.

  • Like 10
  • Thanks 1
Link to comment
On 4/6/2021 at 1:02 AM, Hoopster said:

The hacks are a result of unRAID being used improperly rather than unRAID failing to function in the manner in which it was designed.  I would expect to see more emphasis on educating and helping users to implement unRAID and its supporting infrastructure properly rather than a product redesign to address problems it was never intended to address.

I disagree, letting users to open a root user access from internet is a basic security sin. Just changing the username from root to "John" would make it so much harder. 
Up until now, Unraid was not supposed to be accesible from internet and the official narrative says "You can control Unraid from internet" with only a password, blaming users for improper use is wrong. Fortunately, the devs are making progress in 6.9.2 introducing blocking of IPs for failed logins! I hope more security features to come in the future

Edited by Maor
Link to comment
5 minutes ago, Maor said:

I disagree, letting users to open a root user access from internet is a basic security sin. Just changing the username from root to "John" would make it so much harder. 
Up until now, Unraid was not supposed to be accesible from internet and the official narrative says "You can control Unraid from internet" with only a password, blaming users for improper use is wrong. There has to be more security features enabled by default, ie. disable access after failed login attemt.

I can imagine, that Unraid would promote "make your shares available on internet" without providing enough security... blaming leaked data to improper behaviour of users.

But I don't want to only blame Unraid, I appreciate the announcement of the hack and providing some security guidelines!

See the 6.9.2 release announcement for more info on what's to come.

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.