Sorry for the long post, I wanted to include as much info as possible off the bat.

 

I've recently setup my first Unraid server and everything was going well for the past couple of weeks as I set it all up and migrated data over to the array. I'm very new to Unraid so I've mostly been following a lot of the guides Spaceinvader One has put out.

 

Due to space issues I am running Unraid on a Beelink GTi12 mini PC and have two DAS setup with my drives in them attached through USB-C 10gbs ports. A TERRAMASTER D5-300C with 2x14TB and 3x8TB drives included in the main array and the TERRAMASTER D8 Hybrid with 1x20TB HDD in the array and 4x2TB NVME drives as the cache pool. The 4x2TB drives are setup in raidz1. There is no parity setup on the array yet, I was going to do that sometime next week after getting all of my data transferred over.

 

Yesterday while doing a backup of my main Windows desktop to the cache pool, the mini PC randomly shut off. There was no issue with power as I have it connected to a UPS and the UPS didn't register any events. I checked the zfs pool status and there were no errors listed, but I decided to run a scrub of the pool. It completed and didn't show any issues. I also tried to check the logs, but they had been cleared due to the shutdown. I set the syslog server to localhost to a share on the cache pool to see if I could capture any data if it happened again. It did end up happening again while doing another write to the cache pool and the log didn't show anything as far as I could tell, there were no entries at the time of the power off. I tried writing some more to the cache and it unfortunately shut down again. At this point after restarting the computer, I was unable to start the array. There was a yellow error in the logs to the effect of unable to mount cache due to uncorrectable I/O failure. I stupidly didn't save the error as I expected it to be saved to the syslog I was recoding locally, but I now realize that was not the case since the syslog was trying to write to the share on the cache, which was unmountable. I now have the syslog mirroring to the flash as I probably should have done from the beginning.

 

I gave up trying to troubleshoot the issue as it was late and wanted to look at it fresh in the morning. This morning I was able to get the array to start up without issue, and started copying an appdata backup off of the cache array since I (again, stupidly) hadn't saved the appdata backups anywhere else but the cache pool. I noticed that the copy was going unusually slow, so I checked the logs and didn't see anything initially so I just let it continue copying. At one point I noticed the transfer halted so I checked the dashbaord/syslog and I noticed I was getting errors such as the following:

 

Jan 19 11:10:08 MiniPC kernel: usb 3-2.3.1: reset high-speed USB device number 12 using xhci_hcd
Jan 19 11:10:08 MiniPC kernel: sd 6:0:0:0: [sdh] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=30s
Jan 19 11:10:08 MiniPC kernel: sd 6:0:0:0: [sdh] tag#0 CDB: opcode=0x28 28 00 ac 8e bf 60 00 00 f0 00
Jan 19 11:10:08 MiniPC kernel: I/O error, dev sdh, sector 2895036256 op 0x0:(READ) flags 0x4700 phys_seg 4 prio class 2
Jan 19 11:10:09 MiniPC kernel: zio pool=cache vdev=/dev/sdh1 error=5 type=1 offset=1482258407424 size=270336 flags=40080c80

 

 

This would always be proceeded by some of the cores on my processor getting pinned at 100%.

 

 

 

I checked the zpool status and it is showing unrecoverable errors as well as that note about the resilver. The resilver note is from first thing this morning when I started up after getting the "unable to mount cache due to uncorrectable I/O failure" from the night before:

 

  pool: cache
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: resilvered 7.69M in 00:00:00 with 0 errors on Sun Jan 19 09:10:05 2025
config:

	NAME           STATE     READ WRITE CKSUM
	cache          ONLINE       0     0     0
	  raidz1-0     ONLINE       0     0     0
	    /dev/sdh1  ONLINE      21     0     0
	    /dev/sdi1  ONLINE      62     0     0
	    /dev/sdf1  ONLINE       0     0     0
	    /dev/sdg1  ONLINE       0     0     0

 

 

At this point I'm pretty out of my depth and don't know what I should do to correct/troubleshoot. My current thought is that it's something to do with the cache pool being attached over USB even though this had been working without issue for the past couple of weeks with TBs of data being written to and read from the cache and the array. I've attached my diagnostics. Any insight or help would be much appreciated.

minipc-diagnostics-20250119-1205.zip

46 minutes ago, nicduv said:

something to do with the cache pool being attached over USB

It does look like USB problems, and note that USB is not recommended for array or pools, but try using a different port or USB cable.

I had run into some issue starting up Unraid when I had both DAS plugged in at once before this recent issue started. My solution was to just turn one of them off and then start it once Unraid had booted and this had seemed to work previously.

 

Interestingly, I now just restarted the computer and instead of having the TERRAMASTER D5-300C turned off during boot, I had the D8 Hybrid (where the cache pool is located) turned off. Now when I am copying the same files I had been before, where it was running very slow (60MB/s) and giving the I/O errors, it is copying from/to the same locations at full speed (260MB/s on a spinning disk on the array) and there are no errors.

I'm able to get the computer to consistently crash by trying to restore a backup of my Plex appdata. Every time I try to use the appdata backup lougin to restore the plex.tar.zst, the computer immediately crashes. Likewise if I try to use Unbalance plugin to move the Plex appdata folder to a different drive it crashes. It seems to be crashing when trying to move or restore files in the Database directory for Plex.

 

Any idea why this might be happening?

Enable the syslog server and post that after a crash.

Nothing showing in the syslog after logging to a remote server and experiencing a crash. I think there's more than one issue going on here. The first is the USB connectivity issue that was causing the USB devices to reset. This seems to be fixed by simply starting Unraid with both DAS powered off, then starting one and letting Unraid register the drives, then starting the other and again letting Unraid register the drives, then starting the array.

 

The other issue is with restoring the Plex appdata. I'm not sure why the Beelink PC is crashing when extracting this, but I switched to a Minisforum MS-01 and it is able to successfully extract the appdata folder without issue. I think there must be some issue with the USB controller on the Beelink so I'm sending that one back and am going to keep the MS-01 rather than try and chase ghosts with the Beelink.

 

I'll mark this as solved for now.