Disk errors multiple disks after hardware change

[__nonSENSE]

Hi,

 

I recently switched out my motherboard and CPU (from x570 Aorus Elite with 3900X to TUF Gaming Plus Wifi 2700X); the CPU is a bit older but the mainboard is brand new.

 

After switching the hardware and booting unraid, I tried starting the array. I was greeted with millions of disk errors across all disks. I rebooted, and upgraded my BIOS (unrelated). When I started up again, the array was fine when running through the parity check (0 errors).

I noticed that AMD-V was disabled (again), so I enabled it and rebooted. Upon starting the array, 2 of the 4 disks were reporting errors again during the parity check (millions).

 

Any advice would be appreciated. Thanks!

 

 

tower-diagnostics-20221221-1417.zip

[trurl]

15 minutes ago, __nonSENSE said:

When I started up again, the array was fine when running through the parity check (0 errors).

I noticed that AMD-V was disabled (again), so I enabled it and rebooted. Upon starting the array, 2 of the 4 disks were reporting errors again during the parity check (millions).

Seems like disk controller doesn't like AMD-V

[__nonSENSE]

I was able to confirm that the AMD-V flag is indeed causing the issues. Could this be an issue due to a defective unit? I have read up a bit and saw that other users have used this same motherboard with Unraid.

Is there anything else I could try?

[JorgeB]

One of the SATA controllers is being passed-through to VFIO-PCI

 

0a:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
    Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901]
    Kernel driver in use: vfio-pci
    Kernel modules: ahci

 

That's why Unraid loses contact with all those disks.

[trurl]

Tools - System Devices

 

Post screenshot

[__nonSENSE]

I am currently in maintenance mode, rebuilding a drive, and I have disabled the AMD-V flag. Should I reenable it before I post the screenshot of System Devices or is it OK to post it now with the flag disabled?

 

In case it does not matter, here is the current list:

 

 

[__nonSENSE]

Some progress; I enabled the AMD-V flag and was able to start the array without any errors as long as the VMs are not enabled. I assume this is expected, as turning on VMs and starting the array will also auto-start some of the VMs (at least one of which have a SATA controller passed through, if I understood correctly?).

 

The next step would be to keep the VMs from auto-starting, but as far as I understood, this is not a feature that is available anymore post 6.9?

 

What do you suggest I should do? Can I somehow get rid of the VM definitions without having the VM manager enabled?

 

[JorgeB]

There was feature where if you disabled array auto-start the VMs would not start after first array start, allowing you to do the changes needed, but it's not working with v6.11, you could quickly downgrade to v6.10.3 to fix that then upgrade back, probably still the easiest way.

[__nonSENSE]

7 minutes ago, JorgeB said:

There was feature where if you disabled array auto-start the VMs would not start after first array start, allowing you to do the changes needed, but it's not working with v6.11, you could quickly downgrade to v6.10.3 to fix that then upgrade back, probably still the easiest way.

 

OK, I will try that. Is there any dangers / things to be aware of when downgrading to an earlier Unraid version?

[JorgeB]

Should be pretty safe.

[__nonSENSE]

4 hours ago, JorgeB said:

There was feature where if you disabled array auto-start the VMs would not start after first array start, allowing you to do the changes needed, but it's not working with v6.11, you could quickly downgrade to v6.10.3 to fix that then upgrade back, probably still the easiest way.

 

6.10.3 also seems to auto-start the VMs, maybe I need to go back further?

[JorgeB]

9 minutes ago, __nonSENSE said:

6.10.3 also seems to auto-start the VMs, maybe I need to go back further?

Disable array auto-start (Settings -> Disk settings), then reboot and start the array manually, VMs won't start.

[__nonSENSE]

6 minutes ago, JorgeB said:

Disable array auto-start (Settings -> Disk settings), then reboot and start the array manually, VMs won't start.

 

I had auto-start of disks disabled since the beginning. I always manually start the array

 

 

Could this point to something other than the VMs starting being the issue (if the VMs aren't started after all)? I would love to post more diagnostics, but the process just errors.

 

[__nonSENSE]

I am also reluctant to keep starting the array in this state, because it puts my disks into error state which I need to rebuild.

 

[JorgeB]

Maybe it's not the VMs, go to Settings -> VM Manager -> Enable VMs -> No and see if you still have issues after array start.

[__nonSENSE]

1 minute ago, JorgeB said:

Maybe it's not the VMs, go to Settings -> VM Manager -> Enable VMs -> No and see if you still have issues after array start.

 

With "Enable VMs: No", there are no issues / read errors after array start.

[JorgeB]

That points to a VM being the problem, backup/rename current libvirt.img, create a new one, then just recreate the VMs.

[__nonSENSE]

OK, I will try that. I will only lose the VMs configs, correct?

 

On another note: My Disk2 is showing as "Disabled (contents emulated)". Should I follow the guide to rebuild the drive onto itself, or is there a quicker way to restore it to the array? (I don't like to take shortcuts, but this is the 2nd rebuild within a day I would have to do)

[JorgeB]

19 minutes ago, __nonSENSE said:

OK, I will try that. I will only lose the VMs configs, correct?

Correct.

 

19 minutes ago, __nonSENSE said:

On another note: My Disk2 is showing as "Disabled (contents emulated)". Should I follow the guide to rebuild the drive onto itself, or is there a quicker way to restore it to the array? (I don't like to take shortcuts, but this is the 2nd rebuild within a day I would have to do)

As long as the emulated disk is mounting and contents look correct we usually recommend rebuilding, other option would be a new config but you'd need to run a correcting parity check, so same time and stress for disks.

[__nonSENSE]

7 minutes ago, JorgeB said:

As long as the emulated disk is mounting and contents look correct we usually recommend rebuilding, other option would be a new config but you'd need to run a correcting parity check, so same time and stress for disks.

 

Alright, I am rebuilding now. Won't be able to test the libvirt rename fix until Sunday. I'll report back if it worked.

 

Thanks in any case for all your help so far!

[__nonSENSE]

Update: Renaming the libvirt.img file so that it gets recreated did the trick!

 

I was able to Enable VMs and start the array without having any disk read errors on the array's disks. I even managed to create a new VM and start it and no issues have been recorded so far.

 

Thanks a lot @JorgeB and @trurl