Binding NVMe drive to vfio-pci causes Unraid to not be able to boot

[MustardTiger]

Hi everyone. I'm trying to passthrough a second NVMe drive to be able to use with VMs. I was previously able to bind the device to vfio-pci, reboot, but then I would get this error when I try to pass it through to a VM:

 

internal error: qemu unexpectedly closed the monitor: 2022-11-28T18:10:09.668599Z qemu-system-x86_64: -device {"driver":"vfio- 
pci","host":"0000:81:00.0","id":"hostdev2","bus":"pci.6","addr":"0x0"}: vfio 0000:81:00.0: failed to add PCI capability 
0x11[0x50]@0xb0: table & pba overlap, or they don't fit in BARs, or don't align

 

I did some searching online and found out that I might need to update the firmware on the drive as Unraid might have issues with the controller. The NVMe drive I'm trying to passthrough is exactly the same model as my NVMe drive I use for my cache, so I updated both to the latest firmware. The one I use for my cache still works perfectly fine for Unraid. 

 

However, the one that was bound to vfio stops Unraid from booting, it just keeps bootcycling. I ended up having to edit vfio-pci.cfg on another pc to remove the problem device, which allowed me to boot into Unraid. I tried binding the device to vfio, but the same thing happened. 

 

I tried booting into legacy mode, but the same thing happens. The only difference is that I'm able to see this error just before it reboots:

 

I also get a critical error in my Integrated Management Log in iLO (for HPE Proliant DL80 Gen9) with the description "PCI Bus Error (Slot 6, Bus 128, Device 2, Function 0)", which is the NVMe drive in question, but it shows no more information.

 

For reference, the NVMe drive is an Intel SSDPEKKF256G8L, and both the drive in question and the one I use for my cache drive both use separate NVMe to PCI-E adapters. 

 

If anyone has any advice, it would be most appreciated, thank you.

Diagnostics: tower-diagnostics-20221129-1323.zip

 

Edited November 29, 2022 by MustardTiger

[MustardTiger]

Update: I found another NVMe drive to test, and I seemed to be able to pass it through to the VM with no issues. Only problem is it's a 16GB Optane drive.

When I was trying to boot with the NVMe drive with the problem, I was seeing something in the POST screen about 'dirty bit', and I did not see that when I tried it with the Optane drive. So, I think I'll start investigating into that to see if it can be fixed, I'll try running fsck on it.

Edited November 29, 2022 by MustardTiger