Disk 2 Read Errors: Any Disk, Any Cable, Different Controller(s)!

[Eric Smith]

I'm having a very difficult time chasing down the problem I'm seeing.

 

I've got an x570 Asrock Creator board with Ryzen 3950 and 64GB of RAM. The Creator has 8 SATA ports on board. I've got the board populated with 6x 16TB Segate Exos 7200RPM drives. 2x in Parity, 4x in Data, XFS formatted.  Also have an NVME 2TB cache.

 

About a month ago, disk 2 dropped out of the array with Read errors.  No SMART errors. Since I had several extra 16TB drives laying around, I followed the Wiki procedures for replacing a potentially bad drive. I verified all of the cabling, power connectors, etc.

 

Replacement disk started rebuilding, and after 5-6 hours, Unraid errored out on the disk saying that there were read errors.  Again, no SMART errors.

 

Repeated the process with another 16TB spare drive, and again, Unraid errored out with read errors on re-build, no SMART errors on the disk.

 

Thinking there MUST be something hinky with the cable(s), I swapped known working cables, and tried again.  Same error.

 

Re-shuffling the cables between drives results in any physical drive that Unraid thinks is Disk 2, regardless of what actual disk is in that slot, or where/how it's connected, all Disk 2's on this machine error out with read errors.  I've swapped 5 different disks in, under different cables, with four different controllers (MB, 2x different LSI boards and a Marvel PCI-e Sata controller)... no matter where I plug in the replacement disk, with whatever cable to whatever port, they always error out on the re-build.

 

Thinking it might be a configuration issue, I've reset my configuration and copied data from the original Disk 2 I pulled back manually onto a replacement Disk 2 and skipped the rebuild.  That worked for a few days of use, but then this "fresh" Disk 2 just threw the exact same Unraid read error.  I tried to do a re-build, and it just now failed again.

 

I'm at my wits end here... I have done everything but swap the motherboard.  None of these Disk 2's are the same in ANY WAY!

 

Has anyone seen anything like this?  Any ideas?

 

 

 

 

tower-diagnostics-20201130-1556.zip

[JorgeB]

This is a SATA controller problem, quite common with Ryzen boards:

Nov 30 13:48:22 Tower kernel: ahci 0000:03:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x001a address=0x7fffffc0000 flags=0x0000]
Nov 30 13:48:22 Tower kernel: ahci 0000:03:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x001a address=0x7fffffc0f80 flags=0x0000]

 

Using the latest beta usually helps, but you're already using the current one, so look for a BIOS update or disable IOMMU if not needed.

[Eric Smith]

Turning off IOMMU didn't resolve. Any other suggestions?  Updated HW Diagnostic attached. 

tower-diagnostics-20201202-0900.zip

[JorgeB]

53 minutes ago, Eric Smith said:

Any other suggestions? 

IOMMU is still enable in the latest diags.

[Eric Smith]

I disabled IOMMU and unsafe in the VM Settings.  Is there another switch for IOMMU in unraid GUI, or is this something I need to do in the BIOS as well?

 

Nevermind.  Googled, found it.  While doing that, saw that there was a new BIOS update for the Asrock X570 Creator, so I applied that.  Followed some Unraid bios guides for known-good settings (which included turning on IOMMU in the bios)... which I went ahead and turned on.  I'm going to try and see if the BIOS update solves the issue.  If it doesn't, I'll go in and disable IOMMU in the BIOS and try from there, now that I know where to enable/disable in the BIOS.

 

Edited December 2, 2020 by Eric Smith

[Eric Smith]

Update:  Wanted to close the loop on this topic.

 

After updating the BIOS on this Asrock X570 Creator to version 3.13 (which includes the AMD AGESA ComboAM4v2 1.1.0.0 Patch C, I was able to successfully rebuild "Disk 2" off the onboard SATA ports.  If it remains stable for the next few days, I guess it's been resolved without needing to turn off IOMMU and disabling HW passthrough!