[UPDATED] Lost Data Due To Hacking... (orig: Terrified I lost Data after USB Drive Died somehow)

[MooTheKow]

So - yesterday I found myself unable to access my unraid server.  Hooked up monitor to machine and booted up and saw a message about it being unable to find some .sys or .dll file (can't recall the exact error message).  Searched for the error and saw suggestion 'plug into a windows PC and look at the USB drive to see if the file is there).  USB drive plugged into computer was empty.  I assume bad USB drive.  Download some data recovery software -- it finds all (most?) of the files.  I recover the config folder -- though it has a couple filenames with a _ in place of the original start letter for some reason . Fix file names, run the 'create USB' tool from the Unraid website -- then copy config files back over to drive.    Try booting up server.. everything seems to be going ok.. hurray.  Try to connect to the share i created.. can't connect.  Load up the web interface and to my horror it shows both my drives (3 total - 2 8TB data drives and 1 8TB parity drive) as having darned near 100% free space.  I had nearly 13 TB of data on them. Immediately shut down server.

Was thinking maybe USB was still corrupt and that impacted something.. so created a new USB drive, replaced key, etc -- but I'm still seeing the same thing. 

Am I completely hosed here? I don't understand how/why the data would be gone ... do I have any hope that it's still there and I can get to it somehow?  I have no idea what step to take next year...

 

 

UPDATE: looking through forums see other similar things.  Looks like I had a port forwarding to my unraid server so I could access the dashboard from work.. set up a long time ago and forgot.  This mean I was probably hacked and am just hosed? 😞

Edited March 23, 2021 by MooTheKow
More accurate title

[trurl]

Have you allowed access to your server from outside your LAN? 

[JorgeB]

There have been multiple hacked users in the last week or so, flash and all array data was deleted, did you have any forwarded ports from the router to your server? Or using dmz?

[itimpi]

The USB drive getting corrupt will never touch data on the main array of server - this only happens if active action is taken by something/someone.

 

It sounds like there is a chance your system has been hacked as there seems to be a flurry of such incidents recently with data being deleted.   What ports do you have forwarded to the server (if any)or is the server in the router's DMZ zone as there can leave the server open to attack.   You may need to use some recovery software such as UFS Explorer to try and recover the data if this is the case.

 

It is possible that posting ' your systems diagnostics zip file (obtained via Tools->Diagnostics) to your NEXT post might give soke clue as to what happened.

 

Do you have backups of any critical data?  You should have as parity is never a substitute for a backup.

 

[MooTheKow]

Looks like I had a port forwarding to my unraid server so I could access the dashboard from work.. set up a long time ago and forgot.  This mean I was probably hacked and am just hosed? 😞.  Nothing "critical" per se ... fortunately all my family photos and videos are on a separate hardware raid array ... mostly just a _lot_ of media -- TV shows and Movies and a bunch of other random types of backups...

kowunraid-diagnostics-20210323-0807.zip

Edited March 23, 2021 by MooTheKow

[itimpi]

27 minutes ago, MooTheKow said:

Looks like I had a port forwarding to my unraid server so I could access the dashboard from work.. set up a long time ago and forgot.  This mean I was probably hacked and am just hosed?

Unfortunately yes.  

 

If you want remote access it can be done securely in the current unRaid releases as you can use the built in WireGuard VPN software.

 

We have not had any feedback on whether UFS Explorer (or a similar tool) has successfully recovered  an array disk that has been wiped in one of these apparent hack attacks  

 

 

[MooTheKow]

Yeah.. thanks .  I don't even need remote access ... was just something I set up when I first started using unRaid just to see how it worked and if I could do it -- then never remembered to turn it off.  Lesson learn I suppose...

 

_early_ results using USF Explorer trial (well, actually the cheaper 'Recovery Explorer Standard') shows I may be able to recover some data.. scanning is taking a _lot_ time ... I canceled after it was like .1% done just to see if it had found anything - and it appeared to indicate it had.

 

Now the problem I have is that I don't have any storage available to recover the data on to .. may be time to buy some additional hard drives I suppose...

[MooTheKow]

Question - what do the files look like on the drive normally?  I'm trying the USF Explorer and it's finding files - but I'm unclear if the folder structure is just lost/hosed, or if it is how the files were actually stored by the filesystem ...

Edited March 23, 2021 by MooTheKow

[xxxliqu1dxxx]

3 hours ago, JorgeB said:

There have been multiple hacked users in the last week or so, flash and all array data was deleted, did you have any forwarded ports from the router to your server? Or using dmz?

Hi JorgeB, I see you posted comments like these in other threads - is this a security vulnerability with the new release 6.9.1? I find it somewhat concerning that "in recent weeks" there have been several hacking attempts and when 6.8.3 was stable there were not as many (or none)? I understand if the server was exposed to external connections, but still concerned with the raise in frequency coinciding with the new release. Any info would be appreciated.

[JorgeB]

1 minute ago, xxxliqu1dxxx said:

is this a security vulnerability with the new release 6.9.1?

AFAIK there's nothing on v6.9.x that makes the server more vulnerable, here's one example of a user who got hacked still on v6.8.x, what I suspect is that there is currently one or more hackers actively looking for open Unraid servers.

[xxxliqu1dxxx]

3 minutes ago, JorgeB said:

AFAIK there's nothing on v6.9.x that makes the server more vulnerable, here's one example of a user who got hacked still on v6.8.x, what I suspect is that there is currently one or more hackers actively looking for open Unraid servers.

Thanks. If that's indeed the case, I believe limetech should put out a PSA/blog post/announcement and/or a banner on the unraid server itself... or something to prompt users to review their configuration and mitigate these issues... I mean I get suggestions to run BOINC on my server, but would love to know if there's a security misconfiguration which could avoid issues like the ones mentioned here...

[Hoopster]

4 minutes ago, xxxliqu1dxxx said:

I believe limetech should put out a PSA/blog post/announcement and/or a banner on the unraid server itself... or something to prompt users to review their configuration and mitigate these issues

I made the same suggestion over the weekend to SpencerJ.  He indicated the whole team is very concerned about the recent spate of attacks against unRAID servers.  They are looking at several ways to get the word out about proper security measures and external access; including in the GUI itself. 

 

As JorgeB indicated, there is nothing inherently less or more secure about the most recent version of unRAID.  Almost all these cases come down to new users not understanding how to properly secure their servers or more experienced users forgetting they had left the back door open.

 

It certainly does appear that there are one or more hacker actively looking for exposed unRAID servers.

[PeteAron]

I appreciate JorgeB, Hoopster, and others trying to raise this to a higher priority level.  

 

I am going to review everything in my setups, and even though i have a pretty good idea of what to do, some sort of primer on server security would be helpful, no doubt to many others.  I am wondering, after i have opened a port for my server to accept connections, what is necessary to prevent any unintended access to that port or ports.

 

Any guidance, from general to specific, from our most experienced users, should go a long way to helping the Luddites among us (me included).

[Hoopster]

31 minutes ago, PeteAron said:

Any guidance, from general to specific

Port forwarding for specific ports needed by applications such as WireGuard, OpenVPN, Plex, etc. for remote access is not a security risk and is needed for proper LAN/WAN networking.  Unnecessarily opening up ports 22, 80, 443, etc. without very strict use cases and rock-solid protections is asking for trouble.

 

I have several ports forwarded on my router for the applications I mentioned above.  However, from the Internet, they appear to be in stealth mode and do not respond to external probes except from those applications that are specifically designed to use those ports.

 

I also prefer to disable Universal Plug n Play (UPnP) on my router.  It was designed for local networking discovery only but has been exposed and exploited over the WAN by past problems.

 

Although some question the methodology and conclusions of the author, I use the tools at Gibson Research to get an idea how secure my server is from access from the outside world.  Specifically, I run the Shields Up! UPnP Exposure, Common Ports and All Service Ports tests.  You can also test specific ports like those commonly used for external access by trusted applications such as those mentioned above.

 

Results like this indicate your server is not responding to random connection requests from the Internet:

 

 

 

Edited March 23, 2021 by Hoopster

[MooTheKow]

Anyone catch my question about the file system and layout of the files?

 

Would I expect to see a folder structure on the drive that exactly mimics what I see when browsing my file share.. (like.. a \media\, \media\tv shows\show name', etc etc ) -- or are things like the $Folder06411E68 that this recovery utility finding the actual folders that would have existed on the drive?

[Hoopster]

1 minute ago, MooTheKow said:

Anyone catch my question about the file system and layout of the files?

 

Would I expect to see a folder structure on the drive that exactly mimics what I see when browsing my file share.. (like.. a \media\, \media\tv shows\show name', etc etc ) -- or are things like the $Folder06411E68 that this recovery utility finding the actual folders that would have existed on the drive?

What you are seeing is normal for most recovery tools.  When a drive is "wiped" the filesystem which tracks the folder structure is overwritten and no longer exists and the pointers to files in that structure are also gone.  However, most of the data is still on the drive (unless it was overwritten) and the recovery tools just do a sector by sector scan to see which contain data.  The recovery software can no longer tell you how/where it was stored in a folder structure.

[MooTheKow]

4 minutes ago, Hoopster said:

What you are seeing is normal for most recovery tools.  When a drive is "wiped" the filesystem which tracks the folder structure is overwritten and no longer exists and the pointers to files in that structure are also gone.  However, most of the data is still on the drive (unless it was overwritten) and the recovery tools just do a sector by sector scan to see which contain data.  The recovery software can no longer tell you how/where it was stored in a folder structure.

 

Thanks, I largely figured as much, but I am not familiar with the XSF file system and never looked at how it was actually storing the files on the actual drives - so wasn't sure if there was something goofy going under the hood that somehow got translated to the folder structure I was seeing somewhere.

 

Better than nothing I suppose ... for media shouldn't be horrible, everything named pretty consistently so should b eable to manually sort it if I put some time into it.... for random backups of hard drives i think I'm just going to have to chuck it up as lost...

Edited March 23, 2021 by MooTheKow

[MooTheKow]

So - aside from the 'Don't expose the server to the public internet' thing, is there any known issue that can result in someone being able to wipe all the data like this?  Are they exploiting some sort of unknown loophole that the devs haven't figured out yet?  Is there some default username/password they're able to use that I must have forgot to change?  

[PeteAron]

tyvm Hoopster - i will try this tonight.

[trurl]

1 hour ago, JorgeB said:

AFAIK there's nothing on v6.9.x that makes the server more vulnerable, here's one example of a user who got hacked still on v6.8.x, what I suspect is that there is currently one or more hackers actively looking for open Unraid servers.

One thing many of these seem to have in common is these threads are started by new forum users.

[John_M]

6 minutes ago, MooTheKow said:

So - aside from the 'Don't expose the server to the public internet' thing, is there any known issue that can result in someone being able to wipe all the data like this?  Are they exploiting some sort of unknown loophole that the devs haven't figured out yet?  Is there some default username/password they're able to use that I must have forgot to change?  

 

The public Internet is a known dangerous place but don't discount the possibility of malicious action by a spiteful member of the family or so-called friend who has access via your LAN. The user is root and the default password is no password so, yes, it is important to change that.

[Hoopster]

7 minutes ago, MooTheKow said:

Are they exploiting some sort of unknown loophole that the devs haven't figured out yet?

If such a thing exists and Limetech has not figured it out, I doubt anyone here knows any more.  However, this is highly unlikely.  UnRAID relies on router level firewall and other security protections as it does not have this built-in due to it NOT being a full-fledged soup-to-nuts secure OS.  It is stripped-down Slackware providing very specific services and is more of an appliance than an OS

 

11 minutes ago, MooTheKow said:

Is there some default username/password they're able to use that I must have forgot to change?  

The only user that can allow access to the GUI is the 'root' user.  Other users are for share level access controls via SMB/NFS only and have no GUI/admin rights.  If you have a secure password on the root user, you have done what you can from that standpoint. 

 

The default root password is blank (no password) to allow initial login to the system to get it setup.  Problems arise when unRAID users fail to set a password on root and then expose ports 22, 80, etc. thinking that is what they need to do for remote access.  Even if root has a password set, brute-force attacks can break that password if it is not sufficiently secure.

[kizer]

Always Always Always set a secure password. How secure should the password be? Well honestly I'd make it good enough knowing that some day it might be put to the test by somebody trying to get in and erase your data. 

[S80_UK]

1 hour ago, kizer said:

Always Always Always set a secure password. How secure should the password be? Well honestly I'd make it good enough knowing that some day it might be put to the test by somebody trying to get in and erase your data. 

My suggestions - take it or leave it...

 

I use at least 16 characters, a mixture of numbers, letters (upper and lower case) and punctuation symbols.  Try to avoid names, real words, dates of personal significance.  I don't use numbers as substitutes for letters in words (too easy to have bots making smart guesses).  Also, be absolutely certain that the password is unique and never used for anything else, so I have completely different passwords for any other devices such as PCs, or for account logins, etc.  To help my memory I do use some patterns, sometimes based around old car registration numbers, model numbers, etc.  I go back over 40 years with some of those.        

 

There are a number of pretty decent password strength checkers on the web, but I would then worry about whether any of them retained a tested password against my IP address (call me paranoid...), so I only use them with passwords of similar make up.  I then see estimates of "time to crack" between 2000 and 2000000 years.  At that point I have to assume that the password is good enough.  

[MooTheKow]

2 minutes ago, S80_UK said:

My suggestions - take it or leave it...

 

I use at least 16 characters, a mixture of numbers, letters (upper and lower case) and punctuation symbols.  Try to avoid names, real words, dates of personal significance.  I don't use numbers as substitutes for letters in words (too easy to have bots making smart guesses).  Also, be absolutely certain that the password is unique and never used for anything else, so I have completely different passwords for any other devices such as PCs, or for account logins, etc.  To help my memory I do use some patterns, sometimes based around old car registration numbers, model numbers, etc.  I go back over 40 years with some of those.        

 

There are a number of pretty decent password strength checkers on the web, but I would then worry about whether any of them retained a tested password against my IP address (call me paranoid...), so I only use them with passwords of similar make up.  I then see estimates of "time to crack" between 2000 and 2000000 years.  At that point I have to assume that the password is good enough.  

This has always been my gold standard for explanations of password strength: (xkcd  -- gotta love it):