unRAID stuck starting array

[MechMack]

unRAID loads and gets stuck on Array Starting, I have left this running overnight and it doesn't change. I can't shut it down or reboot it using the web GUI buttons or via CLI. The only way to reboot it is to just hit the physical reset button. Array auto start is disabled, I start it by accessing the web GUI. I have no access to logging when this happens, clicking log brings up a page that stays blank the entire time.

 

What does work. I can access the web GUI and SSH into the server. I can start unRAID in safe mode, but that exhibits the exact same issue. I can start the array in maintenance mode and perform various functions, like get SMART data, diagnostics, settings, etc.. 

 

I did notice that when unRAID is stuck starting arrays, my cache drive was also stuck on "loading". Running check on the cache drive I get a whole list of “leaf parent key incorrect 1796378525696 Error looking up extent record -1“ errors. Since the drive is BTRFS I cannot perform xfs_repair, and the GUI info strongly warns against using any other repair functions on the drive.

 

I haven't done any unRAID updates recently, and only did some plugin and docker updates a few days ago. The biggest change recently, I added the I350-4 NIC, but that seems purely coincidental to this issue.

 

I have perused many forums and posts with similar problems and have had no progress, with many hours wasted. I am at a dead end as what to do next.

 

unRAID Plus, version 6.9.2,

Community Applications Plugin,

4930k, Asus Rampage IV BE, 32GB DDR3 1600MHz RAM,

LSI 9207-8i, I350-4 NIC, EVGA GTX970,

Intel P4500 2TB U.2 PCIE drive (cache),

4 WD Red 2TB SAS HDDs (data),

HGST 4TB SATA HDD (parity),

Chenbro 4-bay SAS backplane and 4u server case          

[trurl]

attach diagnostics to your NEXT post in this thread.

[MechMack]

Hey, thanks!

Diagnostics from my server attached.

 

I did read though the readme for help posts. As far as getting the syslog, I was not expecting to have to make a post, and most times when I had problems with unRAID a reboot or two would get it going again. So I don't have that pre-reboot data logged, but I did attach the syslog just in case it may be useful.

I attached the SMART report for the Intel P4500, it comes up as healthy, but since the check summary gave me errors it may have other issues.

 

BTW, when creating a new post, I didn't see an option to attach files. A very helpful feature would be the ability to attach diagnostics at the same time as posting a support question.

fortress-diagnostics-20220221-1026.zip fortress-syslog-20220221-1629.zip fortress-smart-20220221-1131.zip

[MechMack]

Hey, thanks!

Diagnostics from my server attached.

 

I did read though the readme for help posts. As far as getting the syslog, I was not expecting to have to make a post, and most times when I had problems with unRAID a reboot or two would get it going again. So I don't have that pre-reboot data logged, but I did attach the syslog just in case it may be useful.

I attached the SMART report for the Intel P4500, it comes up as healthy, but since the check summary gave me errors it may have other issues.

 

BTW, when creating a new post, I didn't see an option to attach files. A very helpful feature would be the ability to attach diagnostics at the same time as posting a support question.

 

----

 

Since I've still come no further than this post from last week, and gotten no feedback on what I can try to get unRAID working, I've been looking around online for possible solutions. It seems like this person had an issue that seems almost similar to mine. There was no mention which methods or commands they used, and unRAID warns against using any repair commands on btrfs, so I'll have to read around some more.

Maybe I'm just being kind of hopeful that's what it is, because I feel like I'm at a dead end otherwise. Once I have some time I'll try booting my server from a Linux pendrive and looking at the drives too.

[JorgeB]

There's nothing visible in the diags, if you suspect issues with the cache filesystem you can see if the array starts without it, stop the array, if cache is used for docker/VMs disable those services, unassign cache device, start array.