Restoring appdata results in apparent total array data loss


SnowLprd

Recommended Posts

My goal was to upgrade my cache drive from an older 60 GB SSD to a larger 240 GB SSD via these docs. Following unRAID 6.2.4's Community Applications > Appdata Backup/Restore process as carefully as possible, I completed the appdata restore and immediately received notifications that multiple disks had returned to "normal" utilization levels. Puzzled, I looked through the disks in the array and noticed that they are 99% empty. Other than a few scattered files, it appears as though nearly everything in the array has been lost.

 

I tried to re-trace my steps and see whether I did anything that could cause this. I performed the following before restoring the data:

 

* 6.1.9 --> 6.2.4 upgrade didn't seem to create a "system" share, so I created it manually with "Use cache disk" set to: "Prefer"

* I created a "docker" subfolder within "system" and enabled Docker to (re-)create a 20 GB file at: /mnt/user/system/docker/docker.img

* I created an "appdata" share with "Use cache disk" set to: "Prefer"

* I set "Default appdata storage location" to: /mnt/user/appdata/

* I used the default restore settings to restore:

 

Source Directory:	/mnt/disk10/CommunityApplicationsAppdataBackup
Destination Directory:	/mnt/cache

 

My main question is... Is all my data truly lost? Is there nothing that I can do to recover it?

 

Any and all assistance would be hugely appreciated. I feel sick. I thought I was being so cautious.

 

Sincere thanks in advance for any help.

Link to comment

Post up a Diagnostics file    ('Tools'  >>>  'Diagnostics')  to your next post.  Hopefully, you have not rebooted the machine.  If you have, look in the  logs  folder on your flash drive  and post up the last syslog and diagnostics  files that you find.  These files may provide a starting point to the Gurus to figure out what happened. 

Link to comment

It is very unlikely that your data is lost.  Your data is written onto independent file systems on your data disks and nothing you've done so far sounds like it would have over-written your data.  Unfortunately, making panicked decisions at a time like this can and has resulted in data loss though, so you're doing the right thing - post your diagnostics and let the experienced folks here help you through this. 

 

Oh, and the obligatory warning - any form of RAID including unRAID is not a backup strategy.  If you don't have a good backup strategy then this should be all the motivation you need to put one in place (once things have settled down).

Link to comment

Thank you both for the advice and encouragement. I am sincerely grateful for any and all guidance. I was indeed in a panic and rebooted the machine, so I'm not sure logs were preserved. I don't see anything that looks like log files/directories on the flash drive. Beyond the reboot and running the diagnostics, I have not performed any other actions, in hopes of soliciting advice from those more experienced with these situations.

 

As suggested, I have attached the resulting output of Tools > Diagnostics > Download. Is there any other information I can provide to help determine what I should do next?

tower-diagnostics-20161226-1817.zip

Link to comment

Source Directory:	/mnt/disk10/CommunityApplicationsAppdataBackup
Destination Directory:	/mnt/cache

 

I tried to duplicate what you did and all data on the array was deleted, looking at the above looks like where you went wrong was selecting /mnt/cache as source instead of /mnt/cache/appdata.

 

Still was not expecting that restoring to /mnt/cache would delete everything in /mnt/user, but it does.

 

@Squid, maybe restrict the app from using /mnt/cache as a valid backup/restore path.

 

For the OP, your only option, if you don't have backups, is to use a xfs compatible file recovery tool, most/all data should be recoverable, I remember there were some mentioned on the forum but can't find them.

Link to comment

Source Directory:	/mnt/disk10/CommunityApplicationsAppdataBackup
Destination Directory:	/mnt/cache

 

I tried to duplicate what you did and all data on the array was deleted, looking at the above looks like where you went wrong was selecting /mnt/cache as source instead of /mnt/cache/appdata.

 

Still was not expecting that restoring to /mnt/cache would delete everything in /mnt/user, but it does.

 

@Squid, maybe restrict the app from using /mnt/cache as a valid backup/restore path.

 

For the OP, your only option, if you don't have backups, is to use a xfs compatible file recovery tool, most/all data should be recoverable, I remember there were some mentioned on the forum but can't find them.

 

johnnie.black, did you PM Squid and point him to this post? 

Link to comment

CA Appdata Backup is only supposed to backup appdata, right?  If you're backing up your entire cache drive then it's going to backup TV, or Movies or whatever temporary shares exist on the cache drive at that point, waiting to be written next time Mover runs.

 

So when you restore using CA Backup it will overwrite everything in the folders it is restoring. 

 

 

Link to comment

Sounds like the --delete option was used in rsync, so it deleted everything in the destination folder that didn't exist in the source folder

 

But even then, I would have expected just the cache drive content to be deleted not the array

The module works perfectly as is.  However, under certain circumstances when overriding the source browser by manually typing in a path that is only the root of a drive then part of the cleanup process after the restore is completed winds up deleting the files in /mnt/user0.  Net result is to not override the folder browser setting by typing in a path.  An update will get issued to prevent this behavior in the future.
Link to comment

I have written up a data recovery plan and would deeply appreciate feedback so as to increase the odds of salvaging as much as I can.

 

Done so far:

 

1. Confirmed that AppData was restored to new cache drive

2. Shut down unRAID Tower

3. Set up another machine with fresh installation of macOS 10.12.2 ("Sierra")

 

Planned next steps:

 

1. Remove 4TB disk10 from Tower (that disk was empty until used as backup target for AppData; should still contain that backup)

2. Connect disk10 to Mac via SATA. Boot Mac.

3. Test UFS Explorer on disk10. Can it copy AppData backup to Mac's internal SSD?

4. Remove disk3 from Tower (that disk has least important/amount of data) and connect to Mac.

5. Use UFS Explorer to recover deleted files on disk3 and copy them to disk10. Do best to verify integrity of recovered files.

6. Copy recovered files from disk10 back to disk3 (assuming UFS Explorer supports copying un-deleted files)

7. Put disk3 back in Tower. Remove files from disk10 in order to free up space for recovering the the next disk's files.

8. Repeat steps 4 through 7 until files from all disks have been recovered and copied back to original locations.

9. Remove disk10 from Mac and put back in Tower.

10. Boot Tower, start array, and cross fingers.

 

Does this sound like the most sensible plan? Anything seem unwise? Anyone have suggestions for improvement?

Link to comment
  • 1 month later...

I'm not sure OSX can write to XFS, you may need to use one of the disks as a temp disk and then transfer over lan to unRAID, rest of the plan sounds good, as you already probably know the main thing you can't do is write to any of disks before all recoverable data is recovered and checked.

 

This seems like sensible advice. I am preparing to transfer the first drive's recovered data over the network to the unRAID tower. In an effort to ensure that no writes (or filesystem/directory modifications of any kind) are made to any of the other drives, I disconnected power to all drives except the parity drive and the drive to which I intend to copy the recovered data. Upon powering up the Tower, I see that I cannot start the array, presumably due to the "Too many wrong and/or missing disks!" message. I assumed unRAID would allow me to temporarily ignore the missing drives and allow me to copy the recovered data to the single data drive, but clearly that is not the case.

 

What should I do here? Is my worry about having data drives (from which data has not been recovered yet) powered up and connected an unfounded concern? Is it safe to connect everything back the way it was and start the array, as long as I don't actively make any writes? I suppose my main concern was any low-level garbage collection or other behind-the-scenes filesystem manipulation, but I have no idea if that concern is valid.

 

Or is there another way I should be handling this? Any and all suggestions would be most welcome.

Link to comment

I think that disconnecting them is not a bad idea, especially since you'll need to remove one by one to recover the data.

 

To start the array with only parity plus a data disk you'll need to do a new config (tools -> new config), a parity sync will begin on array start, only after that it finishes will it be protected, you then can add one disk at the time to the array after data is recovered (this will result in the disk being cleared and all data deleted so only do this after all data is recovered and verified).

Link to comment

In that scenario, how will the cache drive be treated? Will its data also be purged when doing Tools > New Config?

 

If so, what should I do? I want to preserve the appdata on the cache drive, given that's the only part of the system that seems to have fully survived the data loss event.

Link to comment

I see. Thank you for clarifying. I think the following is what led me to wonder:

 

[...] you then can add one disk at the time to the array after data is recovered (this will result in the disk being cleared and all data deleted so only do this after all data is recovered and verified).

Link to comment

Clearing is not related to a new config, it's needed every time a disk is added to a parity protected array, so parity is maintained.

 

You could do a new config and not assign parity, this way you can add disks without clearing, so the process would be much faster and only assign parity in the end, obviously the disadvantage is that the array would be unprotected in case of a disk failure.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.