Hello,

Unraid failed to mount one of my disks in the array after a reboot. This was caused by an invalid superblock UUID of the BTRFS. Before I try to repair the fs, I replaced the disk and started the array, and everything was fine.

But wait – I just realized that the mover was running after the reboot and the moment I got aware of the failed disk and it had moved ~300M from cache to disk 2.

 

Did this affect the recovered data or did unraid calculate the parity data correctly regardless of the missing hard drive not showing up as emulated?

 

I have backups (resp. sources) of the important data, but I have no idea how to check and identify the affected files and how to restore them from the backup in reasonable time (spread over Nextcloud, Paperless etc.)

 

Some more background information:
One SSD in the cache pool failed several times and I could not identify the root cause (failed on different ports and cables). So I removed it for replacement and also replaced the PCIe controller just in case.

After this change I restarted the server and the data disk failed as well. This disk is connected to an onboard SATA port, not to the PCIe controller on which the failed cache SSD is plugged in. I have not touched it physically yet.

I did an extended SMART test on the failed data disk and it resulted with no errors.

 

Any suggestions on how to proceed?

 

tl;dr
Will an unmountable data disk, which is not emulated, still be recreated correctly from parity if data will be modified on another data disk in the pool, before the failed disk has been replaced?

 

Thanks for any help

 

 

rumo-diagnostics-20250414-1922.zip rumo-smart-20250414-1537.zip

Rebuilding an unmountable emulated disk will result in an unmountable disk, rebuilding cannot repair filesystem issues.

Well, apparently not.

Like I said, everything looks fine. Disk 1 has been rebuilt on the newly added hard drive and looks as expected. I made some random checks on the files and as far as I can tell, they are correct.

 

I just wonder if all data is binary correct or has the parity disk been altered by the mover without considering the data of the missing disk?

If so, how should I proceede with the failed disk? I would like to try to repair the fs and backup the data if possible.

1 hour ago, cheatinger said:

has the parity disk been altered by the mover without considering the data of the missing disk?

Mover has nothing to do with parity. All writes to the array update parity at the same time, whether mover or something else does the write. Parity knows nothing about filesystems. It is all just parity bits.

https://docs.unraid.net/unraid-os/overview/nas/#parity-protected-array

 

Looks like those diagnostics showed disk1 missing (and btrfs?), but since the array was not started, no way to know anything about any filesystems and whether any were unmountable.

 

Start the array in normal mode and post new diagnostics.

 

1 minute ago, trurl said:

no way to know anything about any filesystems and whether any were unmountable.

Actually, syslog did say disk1 was unmountable at one point. I wonder though if it was actually supposed to be xfs instead of btrfs.

2 minutes ago, trurl said:

Start the array in normal mode and post new diagnostics.

 

3 hours ago, trurl said:

Mover has nothing to do with parity. All writes to the array update parity at the same time, whether mover or something else does the write. Parity knows nothing about filesystems. It is all just parity bits.

https://docs.unraid.net/unraid-os/overview/nas/#parity-protected-array

I understand the concept of parity. I only mentioned the mover to emphasise that this was the only write access since I stopped the array as soon as I noticed the problem. However, the mover had already moved ~300M from the cache to disk 2.

Sorry for the confusion.

 

Let me rephrase my question:
Is the recreated disk 1 binary correctly rebuilt from parity even though there was a write access to disk 2 while the failed disk was unmounted?

How is the parity calculation implemented? Will it check the affected areas on each disk bit by bit on each write access and calculate those parity bits or will it check if a bit is toggled by the write access and if so, toggle the respective parity bit?
Will it emulate the unmounted disk and include it in the parity calculation?

 

4 hours ago, trurl said:

Actually, syslog did say disk1 was unmountable at one point. I wonder though if it was actually supposed to be xfs instead of btrfs.

The fs was btrfs on the failed disk (sdf) and is on the replacement disk (sdd).

 

rumo-diagnostics-20250415-2016.zip

41 minutes ago, cheatinger said:

Is the recreated disk 1 binary correctly rebuilt from parity even though there was a write access to disk 2 while the failed disk was unmounted?

• An unmounted disk can't be written, since its filesystem isn't loaded.
• If the emulated disk was mounted, then any writes to the emulated disk updates parity just as if the physical disk had been written, so any emulated writes can be recovered.

Maybe those 2 points answer your other questions.

Maybe the confusion is in your usage of the word "mounted".

 

That word doesn't mean a disk is installed or assigned.

 

It means the disk's filesystem has been loaded by the OS to make it available for access.

I understand the meaning of the term "mounted"

This is what it looked like when the problem arose:

 

I understand that unraid emulates a missing disk by providing the virtual data calculated from the existing disks, right?

 

This did not seem to be the case with the unmounted disk. At least no shares or files on this disk were shown under Shares or in the file manager, whereas they showed up in the file manager as soon as I started the array with the replacement disk.

 

In this state there was write access to Disk 2 and I still do not understand if the virtual data of Disk 1 was corrupted and thus the corrupted data was written to the replacement disk.

 

Thus I still don't know if I can safely start the Docker containers or access the shares.

 

I hope I have made my concerns clear now

 

If the data might be corrupted, I suppose my next step would be to try to repair the fs on the failed disk and compare it with the replacement.

 

Thank you for your support so far!

23 minutes ago, cheatinger said:

unraid emulates a missing disk by providing the virtual data calculated from the existing disks

You didn't have a missing disk. You had an unmountable disk. So nothing was being emulated.

 

If you had a missing disk, and the emulated disk were unmountable, then rebuild would have resulted in an unmountable disk..

 

Looks like rebuilt disk1 is mounted now though. Perhaps the unmountable condition was just due to not being able to accurately read the original disk.

 

If you had started the array without the original disk, probably it would have shown us that the emulated disk was mountable, since that is the result of the rebuild.

3 minutes ago, trurl said:

You didn't have a missing disk. You had an unmountable disk. So nothing was being emulated.

Yes, as you can see in the screenshot, there is no file manager link.

 

7 minutes ago, trurl said:

If you had a missing disk, and the emulated disk were unmountable, then rebuild would have resulted in an unmountable disk..

 

If you had started the array without the original disk, probably it would have shown us that the emulated disk was mountable, since that is the result of the rebuild.

Yes, but hat is hypothetical and in fact I ran in that state with the array in auto start mode and the mover caused write access on the array.
Did this corrupt the virtual data?

Accessing the array for reading/writing while a disk is being emulated works just as if the disk were there. That is really the whole point.

 

All other disks can be read and written normally, since the emulated disk isn't involved.

 

The emulated disk is read by reading all other disks and getting the data from the parity calculation. The emulated disk is written by updating parity, so all emulated writes can be recovered.

 

Doesn't matter at all what is doing the reading/writing, it all works the same.

Yes, I understand all that, but this is not my question. Forget about the mover.

 

I ran into the state where Disk 1 was neither missing nor emulated and write access happened on Disk 2!
Did this corrupt the virtual data (not the emulated) of Disk 1 by not taking Disk 1 in account to calculate the parity?

 

 

4 minutes ago, cheatinger said:

virtual data (not the emulated)

Not clear what you think the distinction is.

 

4 minutes ago, cheatinger said:

not taking Disk 1 in account to calculate the parity?

Disk 1 isn't changed when writing to disk2, so it isn't involved in updating parity.

 

https://docs.unraid.net/unraid-os/manual/storage-management/#array-write-modes

9 hours ago, trurl said:

Not clear what you think the distinction is.

Virtual: exists conceptually or as an abstraction, not physically.

Emulated: is recreated to mimic the behaviour of an original, although technically it is something else.

 

So virtual data is data that is not explicitly written on any disk but can be computed whereas emulated is how unraid makes the virtual data available as an accessible disk.

 

Not that important, just for the sake of subtlety 😁

 

9 hours ago, trurl said:

Disk 1 isn't changed when writing to disk2, so it isn't involved in updating parity.

https://docs.unraid.net/unraid-os/manual/storage-management/#array-write-modes

This is what I was trying to ask for above

13 hours ago, cheatinger said:

How is the parity calculation implemented? Will it check the affected areas on each disk bit by bit on each write access and calculate those parity bits or will it check if a bit is toggled by the write access and if so, toggle the respective parity bit?
Will it emulate the unmounted disk and include it in the parity calculation?

rumo-diagnostics-20250415-2016.zip 131.65 kB · 1 download

Luckily, I left md_write_method on auto, and I can safely assume that the re-created disk is all valid

Thank you so much for your help! 🤩

Edited April 16, 20251 yr by cheatinger