rm -rf

[Kees Fluitman]

I know im the biggest idiot on the planet. Who uses this right? Why did i?
But I did, even though I never thought I would. And since i was just changing my backup solution, all backups were still on the server itself.

 

I had a script to build and copy a gohugo website to my swag/nginx/www folder. In order to have it properly copied and not leave any old residues, i would ssh rm -rf the $dir/* and then ssh cp the files there.

Then i saw stuff was still dangling about, sth was off. So I quickly manually rm -rf the command $dir/* and bam. It was /rm -rf my /* dir.
So now i need to use a file recovery system, since booting from my recovered flash drive doesnt work properly. I hope anything is left. But i fear the worst.

 

Any tips? I will avoid any writing to the disks right now. Set up a separate ubuntu boot usb and just try to extract whatever i can...it's sooo stupid i know. Especially since I wasnt copying the appdata folders and pictures to my PC and backup server anymore, since I just reinstalled it...

[Michael_P]

Big oof - but if your FS is xfs you can give this a shot if you haven't written to the drives 

 

https://github.com/ianka/xfs_undelete

 

 

[Kees Fluitman]

27 minutes ago, Michael_P said:

Big oof - but if your FS is xfs you can give this a shot if you haven't written to the drives 

 

https://github.com/ianka/xfs_undelete

 

 

My cachepool is btrfs. But my nvme drive (for all database and application data) is xfs.
My array also xfs.
But it seems it needs to copy all data to a different drive. So thats gonna be a bit of a pain.

But even then, it says ill need to rename all the files myself. as it cannot save that.

 

Edited August 14 by Kees Fluitman

[Michael_P]

9 minutes ago, Kees Fluitman said:

But it seems it needs to copy all data to a different drive. So thats gonna be a bit of a pain.

You deleted all your files dude lol - Getting them back was never going to be easy. For btrfs try the equivalent script.

[Kees Fluitman]

19 minutes ago, Michael_P said:

You deleted all your files dude lol - Getting them back was never going to be easy. For btrfs try the equivalent script.

haha i know. i did stop the command though. but it just didnt show anythnig in /mnt anymore.
Now on the ubuntu filesystem, i actually saw all my data was still present on all my ssds. the nvme and the 2 ssd cachepool.
So now ill check the parity drives.
But then the question is, how shall I boot back into my unraid. As it did delete some important data apparently, since booting unraid from my recovered USB didnt work, it just kept looping a reboot.

Edited August 14 by Kees Fluitman

[Michael_P]

9 minutes ago, Kees Fluitman said:

So now ill check the parity drives.

There's nothing on the parity drives, no actual data is stored on them

[Kees Fluitman]

14 minutes ago, Michael_P said:

There's nothing on the parity drives, no actual data is stored on them

srry i meant th array drives. miswrote that.
The nvme comes out clean, nothing was deleted. The ssd cache pool i believe that suffer a little. But ill check them afterwards. Now im running the array, which still has part of the data. but I guess im lucky it was busy deleting the filesystem and the movies dirs first. As it's recovering all movies now.

next step would be to try to get my unraid to boot?

[Michael_P]

14 minutes ago, Kees Fluitman said:

would be to try to get my unraid to boot?

Probably best to start fresh there, and recover any apps/docker containers you had installed.

[Kees Fluitman]

5 minutes ago, Michael_P said:

Probably best to start fresh there, and recover any apps/docker containers you had installed.

ill see. as the nvme had no deleted files at all. all appdata, vms, docker file, etc. are still in tact. the cache pool i havent done yet, since it's running on one of the 10tb drives now and it takes a while. But Ill probably backup all my personal data first and see if i can restart the array then. Ive seen some hints on how to boot again, so shouldnt be impossible to boot from a recovery. The recovery always takes place at 04:00 am so that wasnt too long ago.

[Kees Fluitman]

3 hours ago, Michael_P said:

Probably best to start fresh there, and recover any apps/docker containers you had installed.

btw. i noticed my flash drive doesnt have a efi folder. the usb creation tool didnt add it, although it is in my .zip file. Can i just add it?
It seems i can recover all important data easily. Luckily, my nvme xfs drive was not affected by any deletions, and my cache pool was btrfs and seems recoverable. So now it's just a matter of seeing what of my array drives i really wish to recover and what to run without.

[Kees Fluitman]

so i got it booted. got my data from my cache pools. only thing left is see if i can see what i lost on the array.

[itimpi]

4 minutes ago, Kees Fluitman said:

so i got it booted. got my data from my cache pools. only thing left is see if i can see what i lost on the array.

If it really matters a tool such as UFS Explorer on Windows should be able to retrieve data from the array drives.

[Kees Fluitman]

3 hours ago, itimpi said:

If it really matters a tool such as UFS Explorer on Windows should be able to retrieve data from the array drives.

hey. ye i saw that. its so hard sometimes to guess what's missing or if i got everything since data is on all disks. And Im using the undelete_xfs now with options such to ignore matroska files or files bigger than 2G, to only really get relevant data. But then it just doesn't find anything. So Im thinking i might just bite the bullet here now and go startup that array again.

Only thing im wondering is, should I get one ssd full of the data before i restart the array? Or better to start it, and then get the data back on there? How does unraid handle missing files? or two ssds in the cache pool with different sets of data?
 

[Kilrah]

You didn't start the array? That would have been the first step, THEN based on what you saw there you'd decide on the best course of action. Most likely most of the stuff would have still been there if you cancelled quick enough.

/bin comes first and that being deleted will already crash unraid pretty quick, next is /boot which is the usb, and there are a few more system folders in alphabetical order before reaching /mnt which holds the actual data.

Usually a reboot, or restoring a USB backup gets most people going with their stuff still there as long as it was caught in time.

Edited August 14 by Kilrah

[Kees Fluitman]

8 minutes ago, Kilrah said:

You didn't start the array? That would have been the first step, THEN based on what you saw there you'd decide on the best course of action. Most likely most of the stuff would have still been there if you cancelled quick enough.

/bin comes first and that being deleted will already crash unraid pretty quick, next is /boot which is the usb, and there are a few more system folders in alphabetical order before reaching /mnt which holds the actual data.

Usually a reboot, or restoring a USB backup gets most people going with their stuff still there as long as it was caught in time.

Thanks for the headsup. I basically instantly tried to use my backup usb, which failed because the usb creation tool is bugged. So only after my confusion and thinking I should attempt sth else, did i read up and saw that the boot device was bugged. But i'd already tried booting in ubuntu and see what was still on the drives. Seemingly not finding part of the files...but i guess i was wrong then, and if i start the array, everything will be fine?

So /mnt does not entail any disks once /* has started running? since nthing is mounted anymore?

[Kilrah]

1 minute ago, Kees Fluitman said:

So /mnt does not entail any disks once /* has started running? since nthing is mounted anymore?

It does, stuff is still mounted, rm just goes in alphabetical order so everything depends on how quick you are at seeing the mistake and killing it, if you use default names then cache will be first in the mnt folder, then disk1, 2... Starting things normally you should rather easily see where it stopped and which disks are left intact and which need intervention.

[Kees Fluitman]

4 minutes ago, Kilrah said:

It does, stuff is still mounted, rm just goes in alphabetical order so everything depends on how quick you are at seeing the mistake and killing it, if you use default names then cache will be first in the mnt folder, then disk1, 2... Starting things normally you should rather easily see where it stopped and which disks are left intact and which need intervention.

Ye i did not start the array cause i didnt want to have any reads corrupt my data. i did now, and yes, it did go through my 3 disks first, cutting them back from 15TB to around 2-3TB. The cache pool seems mostly oke, but it could be that some stuff is missing there too.

Im gonna set up a windows to go usb now with some heavier gear to see if i can recover any data with it's filenames and folder structure.

[Kees Fluitman]

So as a conclusion. I've skipped 2 hdds that only had bulk stuff that wasnt so important. But most importantly, it never got to my cache pools or nextcloud and pictures shares, fortunately...
So it took me a couple of days, but got evrthg back in order. Just letting the parity check do it's fixing now.

Only issue. is im getting kernel panics after a couple of hours.
So ill let the parity check run without docker, and if that works, afterwards recreate the diocker image.

Edited August 19 by Kees Fluitman

[Kees Fluitman]

Even after docker image deletion. It still gets kernel panic every 10 hours or so. Back to the drawing board I guess....I guess it's possible to do a clean install, save xml files of used apps and all docker compose files and use existing databases?
Ill first check the syslogs and then shut down some containers one by one to see if that helps.
I noticed 2 more things:
1. Since network settings were deleted and somehow the Create Flash Drive programm overwritten my backuped one. It was set to bonding and bridging, so a macvlan and bridge were both active. Which can cause instability. I reset it and changed back to my old settings (although my router refused to accept the old static ip i had set).
2. Plugins and NERD packages were not properly installed anymore (since they had been deleted and not recovered properly from flash backup), i reinstalled all of them.
Let's see if that solves it.

Was probably the network settings. Ive had that before. Has been on for more than 15 hours now.

Edited August 22 by Kees Fluitman