March 25, 201511 yr This is one of those horror stories every NAS user is afraid of. I was writing a script, and a malformed variable let a "rm -rf *" command got executed inside my "Movies" folder. The result: 15TB of video files got deleted, and no backup of them. I've just put the array on "Maintance mode", and now I'm trying to recover using UFS Explorer Recovery from a VM. Will keep you posted about my success.
March 26, 201511 yr Which file system? I did something similar with 3TB of TV on a reiserfs, and scan whole partition got back 95%. In the words of a former US president "I feel your pain"
March 26, 201511 yr Author Thank you all guys for your best wishes! Which file system? I did something similar with 3TB of TV on a reiserfs, and scan whole partition got back 95%. In the words of a former US president "I feel your pain" All drivers formatted with XFS, that's why I'm using UFS Recovery. I think that wold be my success rate too. 5% is 45 movies, and I can redownload that in one day.
March 26, 201511 yr ... and no backup of them. Ouch !! http://lime-technology.com/forum/index.php?topic=31020.0 It'll be interesting to see if UFS Recovery can come close to the stellar success that reiserfsck is known for. I continue to be surprised at how much data folks collect without bothering to back any of it up.
March 26, 201511 yr Good luck to you! This is why i have the recycle bin enabled. is there a plugin or another method to enable the recycle bin? Ive had a few heart attacks while deleting from a windows machine. Ill delete 1 file in movies, but the damn progress window says "Deleting Tower/Movies/" even though its not deleting the directory. Id love the peace of mind
March 26, 201511 yr Good luck to you! This is why i have the recycle bin enabled. is there a plugin or another method to enable the recycle bin? Ive had a few heart attacks while deleting from a windows machine. Ill delete 1 file in movies, but the damn progress window says "Deleting Tower/Movies/" even though its not deleting the directory. Id love the peace of mind I think there might be a plugin somewhere, but I just created a file called "smb-extra.conf" and put it in the config foler on your flash drive, then reboot or restart SMB. Contents of smb-extra.conf: # Documentation http://www.samba.org/samba/docs/man/manpages-3/smb.conf.5.html # Forum topic http://lime-technology.com/forum/index.php?topic=5446.0 # %S Samba share name (which in unRaid equals the user share name), eg. Storage # %P User share full path, eg /mnt/user/Storage # %u Username of the current service, if any # To apply changes execute: /etc/rc.d/rc.samba restart # If the single custom bin solution is chosen, create the dir manually, and grant it g+rwx permissions [global] vfs objects = recycle recycle:repository = /mnt/user/.Recycle.Bin/%P recycle:keeptree = Yes recycle:touch = Yes recycle:touch_mtime = Yes recycle:versions = Yes recycle:exclude = *.tmp recycle:exclude_dir = .Recycle.Bin acl allow execute always = yes **** NOTE: I'm not sure if this would help with what OP did, but I know it works for stuff you delete via samba.
March 26, 201511 yr Author ... and no backup of them. Ouch !! http://lime-technology.com/forum/index.php?topic=31020.0 It'll be interesting to see if UFS Recovery can come close to the stellar success that reiserfsck is known for. I continue to be surprised at how much data folks collect without bothering to back any of it up. First disk done. So far, so good. UFS had done a very good job maintaining the directory structure. These are most scene files, so I do not bother backing them up. Family pictures, music, documents are all backed up on two different backup sets. Hard drives are extremely expansive in Brazil; it's uneconomical to backup everything.
March 26, 201511 yr **** NOTE: I'm not sure if this would help with what OP did, but I know it works for stuff you delete via samba. I'm pretty sure it wouldn't help in the OPs case since his error happend when running a script... but this is really cool, so thanks for sharing. So to empty the trash you have to go to /.Recycle.Bin and delete it from there as well? Can this be done from windows or does it have to be done from a terminal? Is there a problem if you don't have the leading period before the trash directory? Anyone know if there is a way to implement this on AFP or NFS shares?
March 30, 201511 yr Author Ok, guys, this is the result: From a total of 834 movies, I was able to successfully recover 543. Not bad but not stellar too. Again, thanks a lot for your kind words, guys.
March 30, 201511 yr Contrats on your getting around a third of the movies back. I am hoping to get syncthing setup on my machine and my brothers machine so if one melts we will have alittle more peace of mind.
March 30, 201511 yr gfjardim - Sorry for your trouble, and happy for the degree of recovery you achieved. You are the first user to have such kind of problem with XFS formatted disk. I was hoping you might provide some details of your process and results, including - what version of the tool you used - installation and setup of the tool - length of time it took for different disk sizes - play by play (steps you performed) - How did you confirm the validity of the recovered movies (md5, spot checks, other?) - thoughts on results (did it perfectly recover from some disks and others not at all, did you have a bunch of oddly named files to go through, other thoughts?) - lessons learned (would you do anything different if it happened again?, was it a good tool?) - anything else to share?
March 30, 201511 yr Ok, guys, this is the result: From a total of 834 movies, I was able to successfully recover 543. Not bad but not stellar too. Again, thanks a lot for your kind words, guys. Thinking about the factors that affect file recoveries, and what we can learn here... Number one factor of course is the file system and how well a recovery tool can accurately recognize pieces of the damaged file system structures, and use them to reconstruct a new file system. Next factor would be the quality of the recovery tool. Judging by your example, it doesn't look very good, for XFS, but then this is only one data point. It would be good to find out how mature the UFS Recovery tool is, and how much ongoing effort is going into it now. Perhaps we can apply some friendly pressure? The next factor I think, is how contiguous the file clusters are. Files that are one contiguous piece will always have the best odds of recovery. Start splitting the chains of clusters and the odds start dropping, ESPECIALLY if they are not in physical order. Which brings us to some questions - gfjardim, can you guess at a percentage of how fragmented the files might have been. This has to do with how they were saved, always one at a time or sometimes multiple saved at the same time, and were they ever edited or modified in any way. Files copied new to the drive *should* be completely contiguous, but the file system *may* write file system structure pieces that break the perfect cluster stream. As to the 'fragments out of order' problem, how confident are you that while there are 543 videos apparently recovered, are there possibly some sections out of order? Obviously you cannot listen to all of them from start to finish, but perhaps some spot checking might be good. Perhaps a VideoReDo Quick Stream Fix test would discover trouble, I don't know. Or better yet, did you have any MD5 or SHA checksums on the files you could check? With ReiserFS, file fragmentation was not a known issue, partly because file recovery was amazingly good without worrying about it (and partly because there weren't any defragging tools any way)! I'm wondering if defragging might be at all useful for XFS, to improve the odds of file recovery (if there is such a defragging tool). This is one more reason to have CRC checking systems in place. Are there any clues at all as to why almost 300 videos could not be recovered? That seems worrying, especially on a fresh XFS system. Edit: looks like Brian beat me, with some similar thoughts.
March 30, 201511 yr Actually 2/3rds recovery is pretty good ... not sure how much different this might have been if the file system was Reiser (my gut feel is it would have been better, but no way to know). I remain convinced, however, that the #1 way to ensure good disaster recovery is to have backups
March 30, 201511 yr Community Expert Actually 2/3rds recovery is pretty good ... not sure how much different this might have been if the file system was Reiser (my gut feel is it would have been better, but no way to know). I remain convinced, however, that the #1 way to ensure good disaster recovery is to have backups My guess is that you have not written new files to the array then ReoserFS may have archived close to 100% recovery. I have been quite amazed at how good ReiserFS has been at recovering from very severe corruption (e.g. Accidentally assigning a data drive as parity and starting parity sync so that the sectors at the start of Fisk have their contents completely destroyed - and not realising until GB of corruption have occurred). Hopefully, XFS is more stable as a file system so that we will not need to recover too often n the first place!
March 30, 201511 yr Author There are command line tools that do XFS recover, but since the XFS wiki has referenced UFS, I did used it. I cannot judge it because all I did was recover MKV files within the Movies folder, so many MKV's remained in directories that the program was unable to determine their names. In fact, it was a lazy recover. The program itself is easy to use and it's not very expensive, and it was fast, it reads the disk and then let you select what you want to recover. gfjardim - Sorry for your trouble, and happy for the degree of recovery you achieved. You are the first user to have such kind of problem with XFS formatted disk. I was hoping you might provide some details of your process and results, including - what version of the tool you used - installation and setup of the tool - length of time it took for different disk sizes - play by play (steps you performed) - How did you confirm the validity of the recovered movies (md5, spot checks, other?) - thoughts on results (did it perfectly recover from some disks and others not at all, did you have a bunch of oddly named files to go through, other thoughts?) - lessons learned (would you do anything different if it happened again?, was it a good tool?) - anything else to share? 1) UFS Explorer Standard Recovery ( http://www.ufsexplorer.com/download_stdr.php); 2) I used a Windows 8 VM running from a SSD. 3) It took 3 hours to read a 2TB disk and 4,5 hours to read a 3TB disk. I don't know the time lapse for the recover phase, but the speeds were similar to those I saw while reading. It's very fast. 3) I disabled all writing services of unRAID(SAMBA, NFS, plugins, Docker etc.). While recovering a disk, the array remained stopped. The recover source and target were RDM to the VM; the target disk was a spare 3TB disk. After recovering, I shutdown the VM, started the array in Maintenance mode, mounted the target and source disks and then I moved the files back. 4) I did use bitrot to verify the files and, for those it had the hash, it all checked fine. 5-6-7) Lesson learned: do not play with your production machine! NEVAH! :
March 30, 201511 yr 5-6-7) Lesson learned: do not play with your production machine! NEVAH! : Hopefully you also learned another important lesson: Keep current backups !!
March 31, 201511 yr Author 5-6-7) Lesson learned: do not play with your production machine! NEVAH! : Hopefully you also learned another important lesson: Keep current backups !! I do, just not from scene movies and tv; everything else is backed up twice.
April 24, 201511 yr Good luck to you! This is why i have the recycle bin enabled. How do you enable the recycling bin on the server. I don't see the setting.
April 24, 201511 yr Good luck to you! This is why i have the recycle bin enabled. How do you enable the recycling bin on the server. I don't see the setting. It's a plugin as far as I know.
April 24, 201511 yr Good luck to you! This is why i have the recycle bin enabled. How do you enable the recycling bin on the server. I don't see the setting. It's a plugin as far as I know. Unless someone has turned this into a plugin or we are talking about another implementation of recyle bin, It's not even plugin... http://lime-technology.com/forum/index.php?topic=5446.0
April 24, 201511 yr Community Expert Good luck to you! This is why i have the recycle bin enabled. How do you enable the recycling bin on the server. I don't see the setting. It's a plugin as far as I know. Unless someone has turned this into a plugin or we are talking about another implementation of recyle bin, It's not even plugin... http://lime-technology.com/forum/index.php?topic=5446.0 See page 9 of that thread for plugin. Influencer also had one I think for v5 but he's not been around for a while.
April 24, 201511 yr Good luck to you! This is why i have the recycle bin enabled. How do you enable the recycling bin on the server. I don't see the setting. It's a plugin as far as I know. Unless someone has turned this into a plugin or we are talking about another implementation of recyle bin, It's not even plugin... http://lime-technology.com/forum/index.php?topic=5446.0 See page 9 of that thread for plugin. Influencer also had one I think for v5 but he's not been around for a while. Woo-hoo I'm not going crazy
Archived
This topic is now archived and is closed to further replies.