Kev600

Update - I replaced my SAS > SATA Cabling and updated to Unraid 7.2.4 > 7.2.6 and subsequently 7.3 and the server remains stable, with a massively reduced number of I/O related read errors on the SSD during the first Parity Check .... Subsequent Parity checks have shown ZERO read errors..

I still cannot really understand how Unraid got itself all messed up and becoming non-responsive - but I reseated my RAID card, and GPU when I changed the cabling; and rebooted cleanly after the first boot. (Still using USB to boot).

I'm not sure whether using Duplicati had any impact on the Unraid server becoming non-responsive - It still seems to heavily load the CPU cores, despite me pinning two cores to that Container. (That seems to be ignored) - I've read how the CPU Scheduling could be conflicting, and it's better to leave all cores available for it. But - the error was 'Duplicate Default Server' - Which appeared to correlate with a new router being fitted - But I see there are fixes around dhcp in Unraid 7.3.1... (Although I'm not using dhcp!)

Anyway - all is fixed and my server is running and responding well, remotely, from Thailand! ;)
>>Different topic entirely but - Has CA gotten worse for failed updates/dangling images? 75% of my containers fail to update successfully, unless I manually pull an image from Terminal.

And - That new Disk Talkers plugin, is AMAZING! :)

With over 6 years of usage and increasing errors, it's probably time to replace that drive.
It might be possible to run a bad-blocks check and reformat, to limp along another year or so - depending on what data/usage that drive has.
Treat yourself to a bigger drive mate.. (Matching or less than your Parity capacity, if using it in the array).

Progress made  - I didn't think this would do anything, since I had tried rebooting my whole UR Server..
..but I logged into the AIO Console, then manually stopped all Nextcloud containers in UR/Docker... After refreshing the AIO Console, once again, it told me that updates were ready to install. So I clicked the button to update and start containers. This has created a new nextcloud-aio-nextcloud container.
At the moment, some auto backup process is pausing the database>nextcloud>apache>notify push containers, in that order; from starting.
 
Essentially, I refreshed the AIO Console page a few minutes into the updates, the first time around - this is what I believe, caused this problem.

@JorgeB Update - I wanted to run 'zpool clear' but it didn't show the 'Clear' Button, only 'Scrub' So I ran that and it it finished quite fast.
I rebooted and saw a lot more Disk errors in the logs. I ran the scrub again, then it took the pool into DEGRADED mode and showed a errors for one of the disks..
It hung at about 68%. (3+ hrs)
I forced a hard shutdown/restart and ran a Scrub again - This time it showed loads of errors for both disks and went to SUSPENDED > DISABLED > SUSPENDED > DEGRADED status, and it seemed to hang again, although the UNRAID GUI was still responsive; I could not stop/start the Array. With more Device & I/O errors filling up the logs..
I rebooted again and the Array hung at 'mounting drives' - So I switched it off and went to bed.


This afternoon, I pulled the SAS/SATA Controller card and re-placed it into a different PCI Slot. 
I reseated the cable at the card, and to each of the 4-bays at the SSD caddy.
I also disconnected power and data cables from the DVD drive as it was unused.

I started it, and now, unbelievably, everything seems back to normal, and all my Containers are available - with the zfs cache pool showing;
pool: cache state: ONLINE scan: resilvered 72K in 00:00:00 with 0 errors on Sat Feb 3 15:56:05 2024 config: NAME STATE READ WRITE CKSUM cache ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 /dev/sdb1 ONLINE 0 0 0 /dev/sdc1 ONLINE 0 0 0 errors: No known data errors

How can I test the integrity of my cache pool & disks?

Note: The 250gb (old disk 4) Disk, has been added under UD for now, showing no SMART Errors.
I could potentially add 2x 4tb SSD drives to the Array at a later time. But need to know this 'half' of the storage is sound..

Thanks for your quick responses last night, and any other comments you might have.
Fresh logs attached.

Thanks,
Kev
kbnas-diagnostics-20240203-1623.zip kbnas-syslog-20240203-1622.zip

If this relates to the array failing to stop fully before shut-down, then you could try changing the Settings>Disk Settings>'Default spin down delay' - To - 'Never'.
I see you have shutdownTimeout set to '90', which should be ok. 
 
OR - Is your shutdown unexpected/unwanted?

This should have all the answers..