I had an issue in my dual parity array. I had two drives drop out for some reason. I was rebuilding them and they hit about 50% and the server appeared to hang for about 12 hours. The gui worked but nothing really was updating. I could see errors in syslog indicating errors connecting to the unraid API. I was able to ssh and issue a reboot but again everything was hung so I had little else I could do but powercycle the server.... now the array shows a 3rd drive that dropped out but the other two never finished rebuilding.... what is the best way to recover from this state. I've attached diags but I don't thing they will help much due to the need to power cycle.

nasvm-diagnostics-20240707-0706.zip

You can try this:

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disks 4 and 8
-Start array (in normal mode now) and post new diags.

16 minutes ago, JorgeB said:

You can try this:

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disks 4 and 8
-Start array (in normal mode now) and post new diags.

Set new config as suggested…however I don’t see “parity is already valid” as an option.

That suggests the new config wasn't correctly done, post a screenshot from the main array devices page.

1 hour ago, JorgeB said:

That suggests the new config wasn't correctly done, post a screenshot from the main array devices page.

I tried again and the new config worked.... I completed the recommended steps and attached is the latest diagnostic.

nasvm-diagnostics-20240707-1021.zip

Emulated disks are mounting, you can try rebuilding them again, it may be worth replacing disk9 cables/swapping slots before doing it, just in case that was the problem.

 

I'm also seeing a lot of these:

 

Jul  7 08:00:01 nasvm emhttpd: device /dev/sdh problem getting id
Jul  7 08:00:01 nasvm emhttpd: device /dev/sdg problem getting id

 

They mean Unraid is seeing some disks twice, likely because there are multiple cables between the HBA and the enclosure, Unraid doesn't support SAS multipath, there should be a single cable there.

Thank you for the guidance. I removed the second cable in the event that is causing an issue. After rebooting everything I am now rebuilding the drives.

 

I see a lot of these in the logs:

 

Jul  7 14:29:39 nasvm inotifywait[25822]: Failed to watch /mnt/disk13; upper limit on inotify watches reached!
Jul  7 14:29:39 nasvm inotifywait[25822]: Please increase the amount of inotify watches allowed per user via `/proc/sys/fs/inotify/max_user_watches'.

 

I have increased that value in the past when I've seen these... it's now at 80000.... I don;t want to keep increasing it if something else is causing the errors.

14 hours ago, ridewithjoe said:

I have increased that value in the past when I've seen these... it's now at 80000....

Should be perfectly OK to increase to more if needed, I've seen users with 10x that.