I had a power loss on my server (UPS not configured correctly in Unraid) and the array did not come back correctly.

disk1 was disabled and showed drive xfs unmountable, parity2 was disabled as well.

The array did not work correctly as my dockers and VMs where only partially working.

1. I rebooted the system and went into maintenance mode to do a xfs check.
   1. disk1 and disk3 showed that they would need to be mounted to replay xfs log.
2. restarted array in normal mode to get the replay fixed
   1. disk3 got fixed
   2. disk1 and parity2 were still disabled
   3. array came up, but dockers and VMs still were not working correctly
3. rebooted system
4. restarted array in maintenance because dockers and VMs still were not working correctly
5. stopped array
6. removed parity2
7. started array in normal mode
8. stopped array
9. added same disk as parity2
10. started array in normal mode
11. rebuild started automatically
    1. still some dockers and VMs were only partially working
12. accidentaly formated disk1 during the parity check (assumed that it would not affect the disk itself as it was emulated)
    1. realized that disk1 shows empty and stopped the parity check
13. restarted array in maintenance to realize that disk1 was empty
14. Stopped array to start this thread

 

Setup:

• 7 Disks total in array (2 parity)
• disks were added roughly in this order:
  • disk1 + parity1
  • disk2
  • disk3
  • disk4
  • disk5 + parity2
• Cache pool with 2 x 2TB NVME (mirror)
• Folders appdata, system, domains and iso have cache as primary and array as secondary storage.

 

First up:

• I should have posted before point 1.
• I can not upload the diagnostics file (error code: -200)
• The data on this system is mostly backup and monitoring or recoverable by other means. Only one Folder with lots of videos is unique to this server (would be nice to get them back but I can suffer this loss).

 

main questions:

1. Is there any way to get the data of disk1 back? (files, folder structure, file names)
2. what are the next steps?

 

Edited August 18, 20241 yr by Plak

Try Diagnostics again. 

 

Not clear whether you rebuilt disk1, or only parity. If disk1 was not rebuilt then the disk might still contain its data. If you did rebuild disk1 after formatting emulated disk1, then you rebuilt the formatting on it.

 

 

I have attached the diag with the array stopped: unraid-01-diagnostics-20240818-0652.zip

and with the array started in maintenance mode with disk1 and parity2 removed from it: unraid-01-diagnostics-20240819-0753.zip

The server is in a remote location and I am in the process of bringing it home for further steps.

 

I do recall that I was not able to mount disk1 individually in maintenance mode due to xfs superblock missing. now I can mount disk1 individually and it is empty.

 

mkdir /mnt/temp
mount /dev/md1p1 /mnt/temp
# or
mount /dev/sdc1 /mnt/temp
ls -la /mnt/temp
# no files or folders present

 

4 hours ago, Plak said:

now I can mount disk1 individually and it is empty.

That's expect if it was formatted, see if the old disk mounts with UD

Since you mounted the md device, it's possible you have mounted the emulated disk and not the physical disk, but we don't have enough information to know what your current condition is.

 

On 8/18/2024 at 5:44 AM, trurl said:

Try Diagnostics again

 

Disk1 mounts with UD and is empty. I think that I did mount the whole array with all disks after formatting to get rid of a message about the XFS log needing replaying that suggested to mount so the log could get applied.

unraid-01-diagnostics-20240819-1333.zip

If you also formatted the actual disk, not just the emulated one, the data will be gone, only option is using a file recovery app like UFS Explorer.

16 hours ago, Plak said:

a message about the XFS log needing replaying that suggested to mount so the log could get applied.

XFS tool doesn't know that Unraid had already determined the disk was unmountable. That same message also would have told you to use the -L switch to proceed without the log. That is what you should have done.

 

https://docs.unraid.net/unraid-os/manual/storage-management/#drive-shows-as-unmountable

 

Emulated disk1 is empty.

 

You have 2 unassigned. Logs indicate you mounted sdc. Is that the disk that was assigned as disk1?

Yes

sdc1 is disk1

Will try to get back my data with UFS Explorer.

 

Next steps:

* Remove disk1 and recover data with UFS Explorer (fingers crossed for this one)

* Order new disk for use as new disk1

* Put parity2 back in (not sure on when to do this.

 

I am not clear on how to get the data from the recovery into the new disk 1.

Do I keep the array in maintenance mode and mount the new disk1 and put the recovered data on it and run a parity check afterwards or do I need another route?

You can rebuild the array and then copy the recovered data there.

So I added parity2 to the array and it is rebuilding. Disk1 is emulated and I will pull it and check if any data can be recovered.

From what I gather it would be ok to recover to the array directly and not to the disk1 in particular, right?

42 minutes ago, Plak said:

From what I gather it would be ok to recover to the array directly and not to the disk1 in particular, right?

Yes, that fine, but better if you already have a new disk1 in place when you restore the data, or any writers to the emulated disk can be slower.

Currently UFS Explorer is restoring data from the formated disk to a new drive.

Part of that data already has helped in getting my backups running again (restic config and data was recovered and moved back to unraid).

Parity2 is working again.

Once I am sure I can not get back any more data from disk1, it will be added back to the server and then the recovered data will be moved from the new drive to the array

Due to the size of the disks and the remote location of the server, this process will take a couple weeks.

Thanks for the help and the information.

 

What I take with me for the future is: format is not a solution to any problems with unraid.

Quick Update: I still did not have the opportunity to get the disks back to the server. Will update once I put all disks in.

UFS Explorer did work out. I got an additional drive to recover the data to from the formated one. Most of the data was recovered with correct folder and file names. only around 250GB out of 17TB were "lost-files". I got both disks back in the system. Disk1 was precleared and added to the array. I was able to restore the data from the new disk to the array at the same time.

Data copy to the array with the emulated disk1 had no speed issues. (single disk -> cache at 140MB/s, cache -> array at 120 MB/s, at 95% fill rate of the array the speed went down to around 70MB/s which is to be expected)

 

Everything is back up and running. The new disk is in preclearing process to add it to the array.

Edited November 3, 20241 yr by Plak