NVMe Passthrough Problem (Samsung EVO 980 Pro)

[Arbadacarba]

Running Windows 10 on a passed through 512GB Intel 660P NVME drive on an EVGA Classified K Z170 board

 

I have the opportunity to swap the 660P for a 1TB Samsung EVO 980 Pro. I know my existing board won't take full advantage of this drive but the extra size and a little future proofing never hurt… (except when it does)

 

Unraid lets me reconfigure the VM but when I start it I get:

internal error: Unknown PCI header type '127' for device '0000:03:00.0'

 

 

And after that happens I don't see the drive anywhere accessible to unraid.

 

If I reboot the machine I can see the drive in Unassigned devices and even access it… and when I look at the System Devices Page I see:

 

But as soon as I try to launch the VM  I get the error and the detail page changes to:

!!! Unknown header type 7f

…

 

I spent a bitt looking around and I see lots of references to similar problems with passing GPU's through, but very little about NVMe drives.

 

I did find this page and it seems to be about the same thing but it goes a bit over my head:

 

1542494 – VMs with NVMe devices passed through sometimes fail to be launched (redhat.com)

 

Any help would be appreciated… Should I cut my losses and reinstall the 660P?

 

Thank you

 

I'll have to pull this drive and reinstall the 660p tonight, but I'll check back here in case anyone has any ideas or needs different info.

 

jupiter-syslog-20210308-2108.zip syslog-10.40.70.11.zip

[abc123]

For me (unraid 6.9.0) on System Devices page I check the NVMe drive i want to passthrough:

Then on my VM:

 

My VM is running so it is Greyed out but normally blue.  I also have this set for Primary vDisk location

 

This method keeps unraid from mounting the drive at all

Edited March 8, 2021 by abc123

[Arbadacarba]

that's what I got when the 660P was working correctly... I've tried blocking it out by binding it to vfio... but that gives the same error followed by the difference in System Devices after I attempt to launch a VM.

 

I've tried creating a new VM and attaching it to a VM running Acronis that I use to clone drives and which is the one that I attached the 660P to in order to clone it before I started the swap.

 

For clarity the only VM running from the start is my pfsense router which is not configured to use the NVMe.

 

Thanks for the input though

[Arbadacarba]

Gave up and went back to the 660P

 

Defeated

[Arbadacarba]

Could this be related to the "VFIO allow unsafe interrupts:" setting?

 

I was listening to the Podcast today and it struck me... I was not restarting the VM... But just trying to launch it after boot... Could it have something to do with the host OS mounting the drive before restarting it for the VM?

[tjb_altf4]

5 hours ago, Arbadacarba said:

Could this be related to the "VFIO allow unsafe interrupts:" setting?

 

I was listening to the Podcast today and it struck me... I was not restarting the VM... But just trying to launch it after boot... Could it have something to do with the host OS mounting the drive before restarting it for the VM?

Are you using unassigned device plugin?
There is a setting (per disk) that you need to enable, "passed through" to prevent the system from mounting the disk.

[Arbadacarba]

Yes, I am and that could be contributing to the problem... I'll have another 980 Pro this weekend and will try it again.

[Arbadacarba]

So I've tried with "VFIO allow unsafe interrupts:" set to yes but am getting the same results.

 

I swapped out the 660p for the 980 Pro - There is a Server 2019 Install on the 980 Pro... Checked that it was set to passthrough in the Unassigned Devices config Page and bound to VFIO in System Devices.

 

Same Error on startup of VM

 

If I edit the Windows VM it displays the drive as available and checked.

 

Here's the link to the Article about the same issue on a RedHat forum:

 

1542494 – VMs with NVMe devices passed through sometimes fail to be launched (redhat.com)

https://bugzilla.redhat.com/show_bug.cgi?id=1542494 

 

Any thoughts? I'm kind of stuck with this drive now.

 

Arbadacarba

[Arbadacarba]

Libvert Log:

 

2021-07-01 02:08:42.220+0000: 6700: info : libvirt version: 6.5.0
2021-07-01 02:08:42.220+0000: 6700: info : hostname: Jupiter
2021-07-01 02:08:42.220+0000: 6700: warning : networkNetworkObjTaint:5292 : Network name='default' uuid=03992cfe-2f56-45e7-a079-c12c04bf2bfe is tainted: hook-script
2021-07-01 02:08:42.440+0000: 6790: error : virPCIGetHeaderType:2818 : internal error: Unknown PCI header type '127' for device '0000:03:00.0'
2021-07-01 02:08:43.010+0000: 6700: warning : qemuDomainObjTaint:6075 : Domain id=1 name='pfSense - Cerberus' uuid=021a25c9-9a65-6a21-4bf6-dddf2baf2a4f is tainted: high-privileges
2021-07-01 02:08:43.010+0000: 6700: warning : qemuDomainObjTaint:6075 : Domain id=1 name='pfSense - Cerberus' uuid=021a25c9-9a65-6a21-4bf6-dddf2baf2a4f is tainted: host-cpu
2021-07-01 02:10:30.473+0000: 6688: error : virPCIGetHeaderType:2818 : internal error: Unknown PCI header type '127' for device '0000:03:00.0'
2021-07-01 02:10:30.485+0000: 6688: warning : virHostdevReAttachUSBDevices:1815 : Unable to find device 000.000 in list of active USB devices
2021-07-01 02:10:30.485+0000: 6688: warning : virHostdevReAttachUSBDevices:1815 : Unable to find device 000.000 in list of active USB devices
2021-07-01 02:15:16.456+0000: 6687: error : virFileIsSharedFixFUSE:3384 : unable to canonicalize /mnt/user/Software/virtio-win-0.1.160-1.iso: No such file or directory
2021-07-01 02:15:16.456+0000: 6687: error : qemuOpenFileAs:3175 : Failed to open file '/mnt/user/Software/virtio-win-0.1.160-1.iso': No such file or directory
2021-07-01 02:15:16.485+0000: 6688: error : virFileIsSharedFixFUSE:3384 : unable to canonicalize /mnt/user/Software/- ISOs/virtio-win-0.1.160-1.iso: No such file or directory
2021-07-01 02:15:16.485+0000: 6688: error : qemuOpenFileAs:3175 : Failed to open file '/mnt/user/Software/- ISOs/virtio-win-0.1.160-1.iso': No such file or directory
 

[ghost82]

Your device is not attached to vfio; I don't know if you changed something from your first post to your last one, but in the first screenshots there's no check next to the nvme controller and the output of lspci clearly shows that the nvme is using the "nvme" driver: you must read vfio, not "nvme".

The last 4 errors clearly show that files are missing on your box, or the paths are wrong.

Edited July 1, 2021 by ghost82

[Arbadacarba]

It's absolutely true that the file is not there... the trouble is I can't find any reference to virtio-win-0.1.160-1.iso.. Nothing should be asking for it. The VM's are all using 0.1.190.

 

I'm a little confused... You're right I had not bound it to the VFIO before... I believe that was prior to the newest unraid version, with its easier binding, and I was manually attaching it to the VM in the xml file... I notice the system reports unknown header type as soon Is I log into the gui as opposed to waiting until after I try to run the VM...

 

 

 

Not sure what else to try... 

 

Not sure what to do next to troubleshoot it.

 

Arbadacarba

[ghost82]

12 hours ago, Arbadacarba said:

Not sure what to do next to troubleshoot it.

 

It may be more difficult to fix, it doesn't sound as a configuration problem.

I found some interesting comments here:

https://bugzilla.kernel.org/show_bug.cgi?id=202055

 

If you look at it, especially the latest comments (2019), you will see that limetech also replied with some comments: maybe here @limetech is the only one that can help.