Used rsync incorrectly with --remove-source-files, can I recover files?

[ogi]

I'm temporarily without internet so I'm writing this out on my phone, apologise for typos out missing information in advance.

 

I was shifting around data between disks to minimize the number of shares per disk. On a 6TB drive I had 3.X TB of data I was originally trying to move to a 3TB drive. After a while, realizing my issue (but before an error occurred) I cut rsync off and then specified rsync to move data to /mnt/user0/. Rsync was not looking like it was responding, so I attempted to ctrl-c but didn't have luck either.

 

Stupid me, had --remove-source-files in the command, and the next day I noticed I had a bunch of empty folders in my share that I was trying to remove.

 

My disk is formatted in btrfs, I haven't had any other activity to this disk since this occurred.

 

Any guidance with how to recover the data would be great.

 

Thanks!

 

Sent from my Pixel XL using Tapatalk

 

 

[JorgeB]

Try this:

 

http://ram.kossboss.com/?p=2017

 

Let us know if it worked so it can be recommended in the future to users in a similar situation.

[ogi]

Try this:
 
http://ram.kossboss.com/?p=2017
 
Let us know if it worked so it can be recommended in the future to users in a similar situation.

Thanks for the link, I'm running the first command

Btrfs-find-root /dev/md18 &> /root/bfr.txt

And it's taking quite a while, then again it is a 8TB disk... Will postv follow up findings.

Sent from my Pixel XL using Tapatalk

[ogi]

Try this:
 
http://ram.kossboss.com/?p=2017
 
Let us know if it worked so it can be recommended in the future to users in a similar situation.

Hmm, so far not good, while running that first command (I was transferring data between other disks, array was up) all my ssh sessions disconnected, could briefly ssh in to see the server was still up, but shares were not online, web page was inaccessible

Waited a while, decided to issue the reboot command, and the system appears to be hung there in reboot.

Would attach a screen shot but as I said I'm without internet at home so please pardon the picture of the screen:

I have to step out for a while so I suppose I'll let this run for a while and see what happens when I return.

Sent from my Pixel XL using Tapatalk

[JorgeB]

You should run that with the array unmounted, ie, in maintenance mode.

[ogi]

Figured that now, but should I manually power off in the mean time?

Sent from my Pixel XL using Tapatalk

[JorgeB]

If it doesn't reboot after 15 minutes you'll need to force it.

[ogi]

If it doesn't reboot after 15 minutes you'll need to force it.

Forced the reboot, system came back with a notice about unclean shutdown detected. Started in maintenance mode, running the btrfs-find-root command again, hopefully to different results.

 

Sent from my Pixel XL using Tapatalk

 

 

 

[ogi]

So that didn't work, command was running well in excess of 24 hours, didn't generate any output, no memory usage, just 1 thread at 100% cpu utilization.

 

Good news is I think I can recover my data with just running btrfs restore /dev/md18

 

Problem is I need to restore the data to another folder, which means I need to mount a different unraid disk to move the restored files to.

 

I haven't been able to mount another of my drives while in maintenance mode, the command I've run is

 

mkdir /mnt/disk20

mount /dev/sdo1 /mnt/disk20

 

I get a 'filesystem mounted, but mount ( failed: No space left on device' error.

 

Suggestions as to how to proceed with be appreciated.

 

Sent from my Pixel XL using Tapatalk

 

 

 

 

[JorgeB]

Don't know if btrfs restore is going restore deleted data without using a previous root, but to restore to another disk this would be the easiest way, and parity is maintained:

 

-stop the array

-click on disk18 and change filesytem to xfs

-start the array, disk18 will be unmountable, but the rest of the array will be accessible, then restore to another disk, e.g.:

 

btrfs restore -v /dev/md18 /mnt/disk1/restore

Don't forget that if you restore to a new folder it needs to exist before starting the restore.

 

[ogi]

11 hours ago, johnnie.black said:

Don't know if btrfs restore is going restore deleted data without using a previous root, but to restore to another disk this would be the easiest way, and parity is maintained:

 

-stop the array

-click on disk18 and change filesytem to xfs

-start the array, disk18 will be unmountable, but the rest of the array will be accessible, then restore to another disk, e.g.:

 

btrfs restore -v /dev/md18 /mnt/disk1/restore

Don't forget that if you restore to a new folder it needs to exist before starting the restore.

 

 

That's clever to change the file-system to prevent mounting.... server is in a parity check right now (from the unexpected crash when I was doing btrfs-find-root with the drive mounted (yeah I was an idiot for that).  Will attempt this after it finishes.  

 

I did take a catalog of the data-loss, I have a backup available for this data, so I'm not too concerned should btrfs restore not work (mostly want to get some experience with the restore procedure, so should I not have a backup in the future, I'm not freaking out).

 

I did have the same issue this poster did regarding duration of btrfs-find-root command: https://www.spinics.net/lists/linux-btrfs/msg61340.html . in my case, I let it run for ~26 hours before I killed it.  

 

Strangely enough, when I ran 'brtfs-find-root /dev/md18

 

I immediately got the following printed out (but the program did not terminate and went on to run for a while longer).

root@Tower:~# btrfs-find-root /dev/md18
Superblock thinks the generation is 12237
Superblock thinks the level is 1

 

 

Ogi

[JorgeB]

7 hours ago, ogi said:

I did have the same issue this poster did regarding duration of btrfs-find-root command: https://www.spinics.net/lists/linux-btrfs/msg61340.html . in my case, I let it run for ~26 hours before I killed it.  

 

Strangely enough, when I ran 'brtfs-find-root /dev/md18

 

I immediately got the following printed out (but the program did not terminate and went on to run for a while longer).

 

I did some googling and it appears that btrfs-find-root hasn't been working correctly for a while, with the exact same symptom you describe.

[ogi]

After going through more steps in the mailing list above, I was unable to recover the files that were removed, so in the interest of time, I've decided to restore from backup.

 

Sorry I can't provide a better test-bed for testing the recovery of files, only thing of value I can suggest now is for unraid to include an older version of btrfs-find-root utility for now :-/