So i screwed up guys... help with some data recovery please

[xexitenglishx]

First off, everyone on this forum is amazing.  I was able to ALMOST move from my main windows server to Unraid.  I have the server set up, I created a windows server VM, and various other plug in etc.  I have slowly been adding more hard drives to the array while I transfer all my files off of my old server.  When i first started unraid up I began with a 500gb hard drive just to get familiar to the system and get everything going.  Since then ive added about 6 more hard drives.  Today i got the bright idea that I wanted to remove the 500gb hd from the system.  I do not have a parity disk yet because I am still working on transferring over about 40tb of media.  See below for the (mis)steps i took and I would like anyone's advice to help fix the situation.

 

I used midnight commander to move all of the files off of the 500gb hd to the cache drive.

I set all of the shares to only use disks 2 - 6 and not to write any other data onto disk 1 which is the 500gb hd

I then deleted everything on the 500gb HD

I *THOUGHT* the cache drive would move everything over to the other hard drives as I set the shares to only allow disks 2 -6 while excluding disk 1

The cache disk did not do what I thought it would do so I decided to try to use MC to then move ALL the files from the cache over to the /mnt/usr folder thinking it would just repopulate the files from the 500gb hd. 

This didnt work and while it said it moved all the files over from the cache drive it didnt. it appears to have just put place holders but didnt actually write the data.

Now the cache drive is empty, the 500gb drive has all the files deleted from it, and my VMs and plugins arent working (obviously because I basically deleted everything from the beginning of the unraid install when I was only using the 500gb hd.)

 

I understand i went in way over my head without doing the proper research first...I guess thats just the learning curve I took.

 

My question is.... Can I pull the 500gb hd from unraid, install it in my windows PC, use "recovery explorer standard" to recover all of the files (https://www.sysdevlabs.com/product.php?id=rxs6#specification ), put the 500gb hd back into the unraid cluster, install the temp hard drive with the recovered files as an unassigned device, and recopy the recovered files back to the 500gb hard drive?  I should be able to recover the files from the 500gb as i only did a delete and before i deleted anything I set unraid to not write any files to the 500gb hd.

 

If I can do this, how would I correctly move the recovered files from the temp hard drive to the original 500gb hd without further screwing everything up.

[xexitenglishx]

This is also my log from yesterday and today.  This is when i began messing around with the file system.

syslog 6.7 amd 6.8.txt

[JonathanM]

You experienced first hand the results of mixing disk shares (/mnt/diskX,/mnt/cache/,/mnt/<poolname>) and user shares (everything under /mnt/user)

 

/mnt/user normally merges all the identically named root folders on all the disks into those named user shares. Since they are the same files and but appear in different paths, linux doesn't understand what you want to do when you copy between them, so it ends up overwriting the files with zero byte files.

 

So, to move files from disk to disk, use the full disk paths. To move files between user shares, only use /mnt/user/* paths. Don't mix the two.

 

You can try doing a recovery, hopefully you will be able to get your deleted files back.

 

This is the reason disk shares are disabled by default, with a warning in the help text for disk shares. Using mc allows you to access things not normally exposed.

[xexitenglishx]

2 minutes ago, jonathanm said:

You experienced first hand the results of mixing disk shares (/mnt/diskX,/mnt/cache/,/mnt/<poolname>) and user shares (everything under /mnt/user)

 

/mnt/user normally merges all the identically named root folders on all the disks into those named user shares. Since they are the same files and but appear in different paths, linux doesn't understand what you want to do when you copy between them, so it ends up overwriting the files with zero byte files.

 

So, to move files from disk to disk, use the full disk paths. To move files between user shares, only use /mnt/user/* paths. Don't mix the two.

 

You can try doing a recovery, hopefully you will be able to get your deleted files back.

 

This is the reason disk shares are disabled by default, with a warning in the help text for disk shares. Using mc allows you to access things not normally exposed.

Understood.... So now *if* i am able to recover the files how would I go about repopulating those files back on to the array?  part of what I lost was my VM which is what I was using to move files into the array/shares.  I cant do that since the whole VM is now lost.  I need to know how to manually move the files I recover back into the array like nothing ever happened. i.e. i need to get my VM back to where it has to be...the docker files where they need to be to start the VM etc..

[JonathanM]

Tools, diagnostics, download the zip file. That should contain clues to the paths where the files are expected to be.

 

Currently are the VM and Docker tabs available in the GUI?

 

 

[xexitenglishx]

10 hours ago, jonathanm said:

Tools, diagnostics, download the zip file. That should contain clues to the paths where the files are expected to be.

 

Currently are the VM and Docker tabs available in the GUI?

 

 

The docker tab is available but the VM tab give me an error since I have rebooted the machine.  Please see the attached diagnostics zip.  any assistance would be awesome.  Thanks for your help!

 

Update: I am currently recovering all of the data.  It was all recoverable it seems.  I can worry about restoring the media later on but I just want to get the core files back into unraid so I can run my VM etc...

diagnostics-20210609-0513.zip

Edited June 9, 2021 by xexitenglishx
adding info

[xexitenglishx]

Update.  I recovered all the files.  I need to figure out how to get the follwing files in the correct places:

docker.img

libvirt.img

virtio-win-0.1.190-1.iso

 

Can anyone assist? 

 

Once i put disk on back in the array and start the array, where and how to i put those files back?

[JorgeB]

Docker image can easily be recreated, and probably best than re-using existing one, but if you it it should be put in:

DOCKER_IMAGE_FILE="/mnt/user/system/docker/docker.img"

 

libvirt:

IMAGE_FILE="/mnt/user/system/libvirt/libvirt.img"

 

virtio iso:

VIRTIOISO="/mnt/user/isos/virtio-win-0.1.190-1.iso"

 

Make sure docker and VM services are stopped before doing it.

[xexitenglishx]

Thank you very much JorgeB!  One last question.  What should I use to transfer the files?  I was transferring all of my media using my windows VM and the share folders.  Im not very proficient in linux/command line but if you point me in the right direction I can definitely figure it out.

[JorgeB]

You can still use Windows, can also use for example midnight commander (mc on the console), important part it to only transfer from user share to user share, or disk share to disk share.

[xexitenglishx]

37 minutes ago, JorgeB said:

You can still use Windows, can also use for example midnight commander (mc on the console), important part it to only transfer from user share to user share, or disk share to disk share.

Copy!  Thank you very much for your assistance!  I really appreciate it!  I will follow up with the final result once I get home. 

[xexitenglishx]

following up.  I was able to get my VM services and docker services to load.  I had to redo all the dockers.  I am now having an issue were i cant unmount the arrey and my cache disk is filling up and not offloading.  As of right now I am thinking about starting from scratch with the unraid setup.  Is there a way to start from scratch with unraid but still keep all of my data intact on the array?  i.e. the media i already transferred but as a new unraid setup so i can reconfigure everything?

[JonathanM]

2 minutes ago, xexitenglishx said:

my cache disk is filling up and not offloading

Which shares are staying on the cache? Read the help descriptions for the cache usage on one of the shares configuration page, you probably have the wrong settings for one or more of your shares.

[JonathanM]

5 minutes ago, xexitenglishx said:

i cant unmount the arrey

You can use the "open files" plugin to see which files are still in use when you try to stop the array.

[xexitenglishx]

nothing running in open files...docker wont start again and cache drive is filled.  I think starting from scratch will be much easier as i didnt fully has the system set up anyways.  I was just focusing on transfering all my media.  Is there a way to just start from scratch but keep my media and shares?