Just lost data from 5 out of 6 drives!

[Corvus]

Hey guys,
 

My Unraid server has been functioning flawlessly for the past 2 years. However just today I had a look at the Main tab and noticed that 5 of the 6 drives are showing as empty! Previously, my array was half full.
 

I've tried rebooting, but this didn't fix the problem. A parity check reveals no errors. Drives are showing normal operation. I've had a search, and previous threads suggest it could be Sonarr or Radarr, except I've lost more than just movies/TV shows. I've also lost entire shares that contained photos and documents!

EDIT: Looks like it would've had to have happened overnight (last night) because I can see from my Tautulli history that a Plex user streamed some content that is now missing.
 

My diagnostic is attached.

Please tell me there's some way I can recover this data because I'm freaking out!

nas-diagnostics-20221206-1047.zip

Edited December 6, 2022 by Corvus

[Corvus]

Hello?

[JorgeB]

Do you have the server exposed to the internet? Other option would be some container, I don't use it but I believe Plex has an option to delete watched movies or something similar.

[Corvus]

28 minutes ago, JorgeB said:

Do you have the server exposed to the internet? Other option would be some container, I don't use it but I believe Plex has an option to delete watched movies or something similar.

I do, but let's say they somehow got access to my password how the hell would they even delete specific HDDs? I wouldn't even know how to target specific HDDs because the shares are configured to be evenly distributed across the array. Also, I'm the only one with write access to the array, and only from my laptop, which stays in my room at all times.
It wasn't just movies and TV series that were deleted, it was also a bunch of documents and photos stored in other shares across the array.

Anyway, I'm going to try following the advice contained in this thread: 
How to Recover Deleted Files [SOLVED] - General Support - Unraid

My game plan is to:
- shut down the Unraid server,
- take out one of the affected HDDs,
- plug it into my Windows desktop PC (which has a brand new HDD connected to it to copy over recovered data),
- run the recovery program on the affected drive,
- copy the data over to the spare drive,
- connect the affected drive back into exactly the same SATA port on Unraid,
- boot Unraid, 
- Start array,
- Copy data from spare drive over the network to the original drive 

And rinse and repeat until all drives are done.

I still don't know how I'm going to copy the recovered data from the spare drive (connected to my Windows PC) *specifically* to the original drive once it's plugged into Unraid, since the shares are designed to distribute data evenly across the disks. Any ideas?

My aim is to preserve the file directory structure exactly as it was, because I have a bunch of Dockers that work together and rely on files located in specific directories, and it took me forever to set it up so it works correctly. I also would like to restore my VMs (which have also mysteriously been deleted) exactly as they were.

Does this sound possible?

Edited December 6, 2022 by Corvus

[JorgeB]

9 minutes ago, Corvus said:

I still don't know how I'm going to copy the recovered data from the spare drive (connected to my Windows PC) *specifically* to the original drive once it's plugged into Unraid, since the shares are designed to distribute data evenly across the disks. Any ideas?

Since you are recovering disk by disk you also restore to the same disk.

 

9 minutes ago, Corvus said:

I do,

And fix this:

 

[Corvus]

2 minutes ago, JorgeB said:

Since you are recovering disk by disk you also restore to the same disk.

 

But how?
I know how to copy files over the LAN using Samba file sharing on Windows, but how do I target specific disks? Since the files are distributed equally across the array, Unraid has a unified file structure for all disks. There is nowhere in the file structure that separates by disk.

[JorgeB]

Enable disk share

[Corvus]

2 minutes ago, JorgeB said:

Enable disk share

Thanks, I'll try that when I boot the server again.
Turning this option on won't affect my existing shares, right?

[Kilrah]

29 minutes ago, Corvus said:

I do, but let's say they somehow got access to my password how the hell would they even delete specific HDDs? I wouldn't even know how to target specific HDDs because the shares are configured to be evenly distributed across the array.

 

  

14 minutes ago, Corvus said:

There is nowhere in the file structure that separates by disk.

 

Of course there is, on unraid itself /mnt/diskX are all the individual disks. 

 

Edited December 6, 2022 by Kilrah

[Corvus]

2 minutes ago, Kilrah said:

 

  

 

Of course there is, on unraid itself /mnt/diskX are all the individual disks. 

 

Right, but I can't see any option to just delete an entire disk from the webUI, assuming someone got access to it.

 

[Kilrah]

If the Dynamix file manager plugin is installed, one checkbox and Delete...

 

 

But someone who hacked into your box wouldn't bother using the UI, a single terminal command to rm -rf /mnt/diskX/* and the job is done. Or they could format it, or do anything really...

Edited December 6, 2022 by Kilrah

[Corvus]

Just now, Kilrah said:

If the Dynamix file manager plugin is installed, one checkbox and Delete...

 

 

But someone who hacked into your box wouldn't bother using the UI, a single terminal command to rm -rf /mnt/diskX/* and the job is done. 

So then why wouldn't they finish the job? Why would they leave one drive untouched? They had all night to do it. 
Also why? Just for shits and kicks? What does someone have to gain from doing this?

[Kilrah]

8 minutes ago, Corvus said:

So then why wouldn't they finish the job? Why would they leave one drive untouched? They had all night to do it. 
Also why? Just for shits and kicks? What does someone have to gain from doing this?

 

If there's no ransomware or such to gain from it's pretty much the computer equivalent of street vandalism/bored kids smashing random stuff for shits...

[Corvus]

Ok so I've recovered data from the first drive.
I've placed the original drive back into the Unraid array and booted. So far so good.
Now I'm trying to copy the data from the spare drive (which is connected to my windows PC on the same LAN) to this specific original drive. It's important that I do this because if I allow Unraid to decide which disk to put the data, it will overwrite recoverable data on the other (now empty) drives which I haven't recovered data from.
I've enabled disk share, and I can see the disk in samba shares in Windows.

Two problems:
1. Since the original drive is now empty, how do I know what the original file structure was? Disk 6 (which was untouched and still has files intact) has folders that correspond to the shares that I set up in Unraid, however since I set up the array to distribute data in the 'high water' setting, it doesn't contain all the files, naturally. Should I just recreate this folder structure in the original drive which is now empty and start copying my files to the relevant folders?
2. When I copy anything to Unraid, it first gets copied to my cache drive. Since the amount of data I'm copying far exceeds the capacity of my cache drive, what will happen once the capacity is reached? Will it just automatically start writing to the disk itself? Or will it stop and tell me it's out of space until I invoke the Mover script? Where can I go to check this setting?

Edited December 7, 2022 by Corvus

[JorgeB]

9 hours ago, Corvus said:

Since the original drive is now empty, how do I know what the original file structure was?

Don't you have the original structure in the recovered drive?

 

9 hours ago, Corvus said:

2. When I copy anything to Unraid, it first gets copied to my cache drive.

If you use the disk share it won't use cache.

[Corvus]

14 minutes ago, JorgeB said:

Don't you have the original structure in the recovered drive?

 

If you use the disk share it won't use cache.

Yeah I do. I tried copying the recovered files back to the affected drive, and the share now successfully reports those files mixed in with the other files on the unaffected drive in the array. Now I just need to do this whole process 4 more times

I forgot that copying to a disk directly bypasses the cache. Although since I've sparked my own curiosity, what's the answer to this question, and where are the settings that govern whether or not it starts copying to the array directly when the cache is full?

[JorgeB]

1 minute ago, Corvus said:

and where are the settings that govern whether or not it starts copying to the array directly when the cache is full?

If the share is set to cache=yes it will start copying to the array after it hits the share's minimum free space or cache floor, which ever is higher.

[Corvus]

Ok cool thanks. I'll report back after this is all done. Thanks for your help!