MooTheKow Posted March 23, 2021 Share Posted March 23, 2021 (edited) So - yesterday I found myself unable to access my unraid server. Hooked up monitor to machine and booted up and saw a message about it being unable to find some .sys or .dll file (can't recall the exact error message). Searched for the error and saw suggestion 'plug into a windows PC and look at the USB drive to see if the file is there). USB drive plugged into computer was empty. I assume bad USB drive. Download some data recovery software -- it finds all (most?) of the files. I recover the config folder -- though it has a couple filenames with a _ in place of the original start letter for some reason . Fix file names, run the 'create USB' tool from the Unraid website -- then copy config files back over to drive. Try booting up server.. everything seems to be going ok.. hurray. Try to connect to the share i created.. can't connect. Load up the web interface and to my horror it shows both my drives (3 total - 2 8TB data drives and 1 8TB parity drive) as having darned near 100% free space. I had nearly 13 TB of data on them. Immediately shut down server. Was thinking maybe USB was still corrupt and that impacted something.. so created a new USB drive, replaced key, etc -- but I'm still seeing the same thing. Am I completely hosed here? I don't understand how/why the data would be gone ... do I have any hope that it's still there and I can get to it somehow? I have no idea what step to take next year... UPDATE: looking through forums see other similar things. Looks like I had a port forwarding to my unraid server so I could access the dashboard from work.. set up a long time ago and forgot. This mean I was probably hacked and am just hosed? 😞 Edited March 23, 2021 by MooTheKow More accurate title Quote Link to comment
trurl Posted March 23, 2021 Share Posted March 23, 2021 Have you allowed access to your server from outside your LAN? Quote Link to comment
JorgeB Posted March 23, 2021 Share Posted March 23, 2021 There have been multiple hacked users in the last week or so, flash and all array data was deleted, did you have any forwarded ports from the router to your server? Or using dmz? Quote Link to comment
itimpi Posted March 23, 2021 Share Posted March 23, 2021 The USB drive getting corrupt will never touch data on the main array of server - this only happens if active action is taken by something/someone. It sounds like there is a chance your system has been hacked as there seems to be a flurry of such incidents recently with data being deleted. What ports do you have forwarded to the server (if any)or is the server in the router's DMZ zone as there can leave the server open to attack. You may need to use some recovery software such as UFS Explorer to try and recover the data if this is the case. It is possible that posting ' your systems diagnostics zip file (obtained via Tools->Diagnostics) to your NEXT post might give soke clue as to what happened. Do you have backups of any critical data? You should have as parity is never a substitute for a backup. Quote Link to comment
MooTheKow Posted March 23, 2021 Author Share Posted March 23, 2021 (edited) Looks like I had a port forwarding to my unraid server so I could access the dashboard from work.. set up a long time ago and forgot. This mean I was probably hacked and am just hosed? 😞. Nothing "critical" per se ... fortunately all my family photos and videos are on a separate hardware raid array ... mostly just a _lot_ of media -- TV shows and Movies and a bunch of other random types of backups... kowunraid-diagnostics-20210323-0807.zip Edited March 23, 2021 by MooTheKow Quote Link to comment
itimpi Posted March 23, 2021 Share Posted March 23, 2021 27 minutes ago, MooTheKow said: Looks like I had a port forwarding to my unraid server so I could access the dashboard from work.. set up a long time ago and forgot. This mean I was probably hacked and am just hosed? Unfortunately yes. If you want remote access it can be done securely in the current unRaid releases as you can use the built in WireGuard VPN software. We have not had any feedback on whether UFS Explorer (or a similar tool) has successfully recovered an array disk that has been wiped in one of these apparent hack attacks Quote Link to comment
MooTheKow Posted March 23, 2021 Author Share Posted March 23, 2021 Yeah.. thanks . I don't even need remote access ... was just something I set up when I first started using unRaid just to see how it worked and if I could do it -- then never remembered to turn it off. Lesson learn I suppose... _early_ results using USF Explorer trial (well, actually the cheaper 'Recovery Explorer Standard') shows I may be able to recover some data.. scanning is taking a _lot_ time ... I canceled after it was like .1% done just to see if it had found anything - and it appeared to indicate it had. Now the problem I have is that I don't have any storage available to recover the data on to .. may be time to buy some additional hard drives I suppose... Quote Link to comment
MooTheKow Posted March 23, 2021 Author Share Posted March 23, 2021 (edited) Question - what do the files look like on the drive normally? I'm trying the USF Explorer and it's finding files - but I'm unclear if the folder structure is just lost/hosed, or if it is how the files were actually stored by the filesystem ... Edited March 23, 2021 by MooTheKow Quote Link to comment
xxxliqu1dxxx Posted March 23, 2021 Share Posted March 23, 2021 3 hours ago, JorgeB said: There have been multiple hacked users in the last week or so, flash and all array data was deleted, did you have any forwarded ports from the router to your server? Or using dmz? Hi JorgeB, I see you posted comments like these in other threads - is this a security vulnerability with the new release 6.9.1? I find it somewhat concerning that "in recent weeks" there have been several hacking attempts and when 6.8.3 was stable there were not as many (or none)? I understand if the server was exposed to external connections, but still concerned with the raise in frequency coinciding with the new release. Any info would be appreciated. Quote Link to comment
JorgeB Posted March 23, 2021 Share Posted March 23, 2021 1 minute ago, xxxliqu1dxxx said: is this a security vulnerability with the new release 6.9.1? AFAIK there's nothing on v6.9.x that makes the server more vulnerable, here's one example of a user who got hacked still on v6.8.x, what I suspect is that there is currently one or more hackers actively looking for open Unraid servers. Quote Link to comment
xxxliqu1dxxx Posted March 23, 2021 Share Posted March 23, 2021 3 minutes ago, JorgeB said: AFAIK there's nothing on v6.9.x that makes the server more vulnerable, here's one example of a user who got hacked still on v6.8.x, what I suspect is that there is currently one or more hackers actively looking for open Unraid servers. Thanks. If that's indeed the case, I believe limetech should put out a PSA/blog post/announcement and/or a banner on the unraid server itself... or something to prompt users to review their configuration and mitigate these issues... I mean I get suggestions to run BOINC on my server, but would love to know if there's a security misconfiguration which could avoid issues like the ones mentioned here... Quote Link to comment
Hoopster Posted March 23, 2021 Share Posted March 23, 2021 4 minutes ago, xxxliqu1dxxx said: I believe limetech should put out a PSA/blog post/announcement and/or a banner on the unraid server itself... or something to prompt users to review their configuration and mitigate these issues I made the same suggestion over the weekend to SpencerJ. He indicated the whole team is very concerned about the recent spate of attacks against unRAID servers. They are looking at several ways to get the word out about proper security measures and external access; including in the GUI itself. As JorgeB indicated, there is nothing inherently less or more secure about the most recent version of unRAID. Almost all these cases come down to new users not understanding how to properly secure their servers or more experienced users forgetting they had left the back door open. It certainly does appear that there are one or more hacker actively looking for exposed unRAID servers. 1 Quote Link to comment
PeteAron Posted March 23, 2021 Share Posted March 23, 2021 I appreciate JorgeB, Hoopster, and others trying to raise this to a higher priority level. I am going to review everything in my setups, and even though i have a pretty good idea of what to do, some sort of primer on server security would be helpful, no doubt to many others. I am wondering, after i have opened a port for my server to accept connections, what is necessary to prevent any unintended access to that port or ports. Any guidance, from general to specific, from our most experienced users, should go a long way to helping the Luddites among us (me included). 1 Quote Link to comment
Hoopster Posted March 23, 2021 Share Posted March 23, 2021 (edited) 31 minutes ago, PeteAron said: Any guidance, from general to specific Port forwarding for specific ports needed by applications such as WireGuard, OpenVPN, Plex, etc. for remote access is not a security risk and is needed for proper LAN/WAN networking. Unnecessarily opening up ports 22, 80, 443, etc. without very strict use cases and rock-solid protections is asking for trouble. I have several ports forwarded on my router for the applications I mentioned above. However, from the Internet, they appear to be in stealth mode and do not respond to external probes except from those applications that are specifically designed to use those ports. I also prefer to disable Universal Plug n Play (UPnP) on my router. It was designed for local networking discovery only but has been exposed and exploited over the WAN by past problems. Although some question the methodology and conclusions of the author, I use the tools at Gibson Research to get an idea how secure my server is from access from the outside world. Specifically, I run the Shields Up! UPnP Exposure, Common Ports and All Service Ports tests. You can also test specific ports like those commonly used for external access by trusted applications such as those mentioned above. Results like this indicate your server is not responding to random connection requests from the Internet: Edited March 23, 2021 by Hoopster 4 Quote Link to comment
MooTheKow Posted March 23, 2021 Author Share Posted March 23, 2021 Anyone catch my question about the file system and layout of the files? Would I expect to see a folder structure on the drive that exactly mimics what I see when browsing my file share.. (like.. a \media\, \media\tv shows\show name', etc etc ) -- or are things like the $Folder06411E68 that this recovery utility finding the actual folders that would have existed on the drive? Quote Link to comment
Hoopster Posted March 23, 2021 Share Posted March 23, 2021 1 minute ago, MooTheKow said: Anyone catch my question about the file system and layout of the files? Would I expect to see a folder structure on the drive that exactly mimics what I see when browsing my file share.. (like.. a \media\, \media\tv shows\show name', etc etc ) -- or are things like the $Folder06411E68 that this recovery utility finding the actual folders that would have existed on the drive? What you are seeing is normal for most recovery tools. When a drive is "wiped" the filesystem which tracks the folder structure is overwritten and no longer exists and the pointers to files in that structure are also gone. However, most of the data is still on the drive (unless it was overwritten) and the recovery tools just do a sector by sector scan to see which contain data. The recovery software can no longer tell you how/where it was stored in a folder structure. Quote Link to comment
MooTheKow Posted March 23, 2021 Author Share Posted March 23, 2021 (edited) 4 minutes ago, Hoopster said: What you are seeing is normal for most recovery tools. When a drive is "wiped" the filesystem which tracks the folder structure is overwritten and no longer exists and the pointers to files in that structure are also gone. However, most of the data is still on the drive (unless it was overwritten) and the recovery tools just do a sector by sector scan to see which contain data. The recovery software can no longer tell you how/where it was stored in a folder structure. Thanks, I largely figured as much, but I am not familiar with the XSF file system and never looked at how it was actually storing the files on the actual drives - so wasn't sure if there was something goofy going under the hood that somehow got translated to the folder structure I was seeing somewhere. Better than nothing I suppose ... for media shouldn't be horrible, everything named pretty consistently so should b eable to manually sort it if I put some time into it.... for random backups of hard drives i think I'm just going to have to chuck it up as lost... Edited March 23, 2021 by MooTheKow Quote Link to comment
MooTheKow Posted March 23, 2021 Author Share Posted March 23, 2021 So - aside from the 'Don't expose the server to the public internet' thing, is there any known issue that can result in someone being able to wipe all the data like this? Are they exploiting some sort of unknown loophole that the devs haven't figured out yet? Is there some default username/password they're able to use that I must have forgot to change? Quote Link to comment
PeteAron Posted March 23, 2021 Share Posted March 23, 2021 tyvm Hoopster - i will try this tonight. Quote Link to comment
trurl Posted March 23, 2021 Share Posted March 23, 2021 1 hour ago, JorgeB said: AFAIK there's nothing on v6.9.x that makes the server more vulnerable, here's one example of a user who got hacked still on v6.8.x, what I suspect is that there is currently one or more hackers actively looking for open Unraid servers. One thing many of these seem to have in common is these threads are started by new forum users. Quote Link to comment
John_M Posted March 23, 2021 Share Posted March 23, 2021 6 minutes ago, MooTheKow said: So - aside from the 'Don't expose the server to the public internet' thing, is there any known issue that can result in someone being able to wipe all the data like this? Are they exploiting some sort of unknown loophole that the devs haven't figured out yet? Is there some default username/password they're able to use that I must have forgot to change? The public Internet is a known dangerous place but don't discount the possibility of malicious action by a spiteful member of the family or so-called friend who has access via your LAN. The user is root and the default password is no password so, yes, it is important to change that. Quote Link to comment
Hoopster Posted March 23, 2021 Share Posted March 23, 2021 7 minutes ago, MooTheKow said: Are they exploiting some sort of unknown loophole that the devs haven't figured out yet? If such a thing exists and Limetech has not figured it out, I doubt anyone here knows any more. However, this is highly unlikely. UnRAID relies on router level firewall and other security protections as it does not have this built-in due to it NOT being a full-fledged soup-to-nuts secure OS. It is stripped-down Slackware providing very specific services and is more of an appliance than an OS 11 minutes ago, MooTheKow said: Is there some default username/password they're able to use that I must have forgot to change? The only user that can allow access to the GUI is the 'root' user. Other users are for share level access controls via SMB/NFS only and have no GUI/admin rights. If you have a secure password on the root user, you have done what you can from that standpoint. The default root password is blank (no password) to allow initial login to the system to get it setup. Problems arise when unRAID users fail to set a password on root and then expose ports 22, 80, etc. thinking that is what they need to do for remote access. Even if root has a password set, brute-force attacks can break that password if it is not sufficiently secure. Quote Link to comment
kizer Posted March 24, 2021 Share Posted March 24, 2021 Always Always Always set a secure password. How secure should the password be? Well honestly I'd make it good enough knowing that some day it might be put to the test by somebody trying to get in and erase your data. Quote Link to comment
S80_UK Posted March 24, 2021 Share Posted March 24, 2021 1 hour ago, kizer said: Always Always Always set a secure password. How secure should the password be? Well honestly I'd make it good enough knowing that some day it might be put to the test by somebody trying to get in and erase your data. My suggestions - take it or leave it... I use at least 16 characters, a mixture of numbers, letters (upper and lower case) and punctuation symbols. Try to avoid names, real words, dates of personal significance. I don't use numbers as substitutes for letters in words (too easy to have bots making smart guesses). Also, be absolutely certain that the password is unique and never used for anything else, so I have completely different passwords for any other devices such as PCs, or for account logins, etc. To help my memory I do use some patterns, sometimes based around old car registration numbers, model numbers, etc. I go back over 40 years with some of those. There are a number of pretty decent password strength checkers on the web, but I would then worry about whether any of them retained a tested password against my IP address (call me paranoid...), so I only use them with passwords of similar make up. I then see estimates of "time to crack" between 2000 and 2000000 years. At that point I have to assume that the password is good enough. 1 Quote Link to comment
MooTheKow Posted March 24, 2021 Author Share Posted March 24, 2021 2 minutes ago, S80_UK said: My suggestions - take it or leave it... I use at least 16 characters, a mixture of numbers, letters (upper and lower case) and punctuation symbols. Try to avoid names, real words, dates of personal significance. I don't use numbers as substitutes for letters in words (too easy to have bots making smart guesses). Also, be absolutely certain that the password is unique and never used for anything else, so I have completely different passwords for any other devices such as PCs, or for account logins, etc. To help my memory I do use some patterns, sometimes based around old car registration numbers, model numbers, etc. I go back over 40 years with some of those. There are a number of pretty decent password strength checkers on the web, but I would then worry about whether any of them retained a tested password against my IP address (call me paranoid...), so I only use them with passwords of similar make up. I then see estimates of "time to crack" between 2000 and 2000000 years. At that point I have to assume that the password is good enough. This has always been my gold standard for explanations of password strength: (xkcd -- gotta love it): 2 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.