[SOLVED] Array Read Errors, Disks in Error State, Disks Duplicating in Unassigned Devices

[garrettw27]

I am running version 6.9.2 with 4 10TB sata drives (3 data, 1 parity) connected with a LSI 9211-8I SAS HBA in IT mode. I also have a 1 TB nvme drive for a cache along with a 500 GB nvme and 4 TB sata for a virtual machine. Within the past month, I replaced my cache drive from a 1 TB usb ssd to the nvme drive. Everything worked fine except my iommu groups changed on my virtual machine. I finally spent some time 2 days ago to get the VM back up and running. Intel VF-D ended up turned off in BIOS somehow. And the update in 6.9 with the binding to VFIO threw me off on trying to add the drives in the VM the old way. In the troubleshooting stages, I ended up doing multiple reboots to mess with the BIOS.

 

After getting the drives figured out, I finally got a new error when starting the VM where it couldn't find the Graphics ROM BIOS file. When I went back to the Main page, my disk 1 was in error state. After doing some research, I was pretty sure the drive hadn't failed but most likely failed a write in all of my reboots. I stopped the array, unassigned disk 1, started the array, stopped the array, and reassigned disk 1 to do a parity rebuild. After almost 19 hours, it successfully rebuilt everything. All of my docker images appeared to be up and running when I checked about 8 hours after the rebuild completed. I went back to work on my VM and browsed for the ROM BIOS in the dropdown. Within 5 seconds of selecting the ROM BIOS file, I got a "Array has 4 disks with read errors" notification and 2 seconds after that, my disk 2 was in error state. Also, the 4 disks are now showing up in unassigned devices as sdg-sdj (previously sdb-sde). I haven't messed with any hardware since installing the nvme cache drive almost a month ago and everything was working up until I started messing with the VM. Any suggestions?

tower-diagnostics-20211205-2107.zip

Edited December 6, 2021 by garrettw27
Update to solved

[JorgeB]

-device vfio-pci,host=0000:05:00.0,id=hostdev1,bus=pci.0,addr=0x9 \

 

05:00.0 RAID bus controller [0104]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
    Subsystem: Fujitsu Technology Solutions HBA Ctrl SAS 6G 0/1 [D2607] [1734:1177]
    Kernel driver in use: mpt3sas
    Kernel modules: mpt3sas

 

The LSI is being passed though to the VM, so when you start it Unraid loses access to it and the disks.

[garrettw27]

6 hours ago, JorgeB said:

-device vfio-pci,host=0000:05:00.0,id=hostdev1,bus=pci.0,addr=0x9 \

 

05:00.0 RAID bus controller [0104]: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
    Subsystem: Fujitsu Technology Solutions HBA Ctrl SAS 6G 0/1 [D2607] [1734:1177]
    Kernel driver in use: mpt3sas
    Kernel modules: mpt3sas

 

The LSI is being passed though to the VM, so when you start it Unraid loses access to it and the disks.

 

Yep. That would do it. I was editing in form view and don't recall checking that box but at least its an easy fix. Thanks for the help!