NVME passthrough works with issues

[xemiterx]

I added an NVME drive using a PCIE adapter and it showed up as a drive in unRaid. I then created a new Windows 10 VM and edited the XML to pass through the NVME controller to the VM and installed Windows 10 directly to the NVME drive.

 

The problem comes when I start the VM after it has been shut down. I have to do it twice, the first time always fails. 

 

There are no errors in the log, but there is a warning on both the failed and successful attempts to start the VM.

Tower kernel: nvme nvme0: failed to set APST feature (-19)

[Start VM Try #1]
Apr 26 16:22:55 Tower kernel: vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Apr 26 16:22:55 Tower kernel: nvme nvme0: failed to set APST feature (-19)
Apr 26 16:22:55 Tower kernel: br0: port 4(vnet2) entered blocking state
Apr 26 16:22:55 Tower kernel: br0: port 4(vnet2) entered disabled state
Apr 26 16:22:55 Tower kernel: device vnet2 entered promiscuous mode
Apr 26 16:22:55 Tower kernel: br0: port 4(vnet2) entered blocking state
Apr 26 16:22:55 Tower kernel: br0: port 4(vnet2) entered forwarding state
Apr 26 16:22:56 Tower avahi-daemon[2852]: Joining mDNS multicast group on interface vnet2.IPv6 with address fe80::fc54:ff:fede:f67d.
Apr 26 16:22:56 Tower avahi-daemon[2852]: New relevant interface vnet2.IPv6 for mDNS.
Apr 26 16:22:56 Tower avahi-daemon[2852]: Registering new address record for fe80::fc54:ff:fede:f67d on vnet2.*.
Apr 26 16:22:56 Tower kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0x1e@0x258
Apr 26 16:22:56 Tower kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x900
Apr 26 16:22:56 Tower kernel: vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x168
Apr 26 16:22:56 Tower kernel: vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1e@0x190
Apr 26 16:23:03 Tower avahi-daemon[2852]: Interface vnet2.IPv6 no longer relevant for mDNS.
Apr 26 16:23:03 Tower avahi-daemon[2852]: Leaving mDNS multicast group on interface vnet2.IPv6 with address fe80::fc54:ff:fede:f67d.
Apr 26 16:23:03 Tower kernel: br0: port 4(vnet2) entered disabled state
Apr 26 16:23:03 Tower kernel: device vnet2 left promiscuous mode
Apr 26 16:23:03 Tower kernel: br0: port 4(vnet2) entered disabled state
Apr 26 16:23:03 Tower avahi-daemon[2852]: Withdrawing address record for fe80::fc54:ff:fede:f67d on vnet2.
Apr 26 16:23:04 Tower kernel: vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Apr 26 16:23:04 Tower kernel: nvme nvme0: pci function 0000:04:00.0
Apr 26 16:23:04 Tower kernel: nvme 0000:04:00.0: enabling device (0400 -> 0402)
Apr 26 16:23:05 Tower kernel: nvme0n1: p1 p2 p3 p4
[Start VM Try #2]
Apr 26 16:25:07 Tower kernel: vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Apr 26 16:25:07 Tower kernel: nvme nvme0: failed to set APST feature (-19)
Apr 26 16:25:07 Tower kernel: br0: port 4(vnet2) entered blocking state
Apr 26 16:25:07 Tower kernel: br0: port 4(vnet2) entered disabled state
Apr 26 16:25:07 Tower kernel: device vnet2 entered promiscuous mode
Apr 26 16:25:07 Tower kernel: br0: port 4(vnet2) entered blocking state
Apr 26 16:25:07 Tower kernel: br0: port 4(vnet2) entered forwarding state
Apr 26 16:25:08 Tower kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0x1e@0x258
Apr 26 16:25:08 Tower kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x900
Apr 26 16:25:08 Tower kernel: vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x168
Apr 26 16:25:08 Tower kernel: vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1e@0x190
Apr 26 16:25:09 Tower avahi-daemon[2852]: Joining mDNS multicast group on interface vnet2.IPv6 with address fe80::fc54:ff:fede:f67d.
Apr 26 16:25:09 Tower avahi-daemon[2852]: New relevant interface vnet2.IPv6 for mDNS.
Apr 26 16:25:09 Tower avahi-daemon[2852]: Registering new address record for fe80::fc54:ff:fede:f67d on vnet2.*.

 

[xemiterx]

While trying to figure this out I looked at the XML for my VM and noticed there was an extra line in the NVME passthru bit. I removed that line and it's working correctly now. The line I removed was:

<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

I originally copy/pasted the XML from the first part of this post and just changed the bus. The only thing I can think of was that after I installed Windows and the virtio drivers I used the form view to remove the ISO's and that line got inserted when I saved.

[unrateable]

On 4/27/2018 at 5:28 PM, xemiterx said:

While trying to figure this out I looked at the XML for my VM and noticed there was an extra line in the NVME passthru bit. I removed that line and it's working correctly now. The line I removed was:

<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

I originally copy/pasted the XML from the first part of this post and just changed the bus. The only thing I can think of was that after I installed Windows and the virtio drivers I used the form view to remove the ISO's and that line got inserted when I saved.

 

I also have APST errors. when I remove that line unraid add its again. was that really the reason ? are your APST errors gone and what drive are you using ?

mine is Samsung 970 EVO.. 

 

[alfredo_2020]

Im getting the same error, i started another post about this. I do see that extra address tag in my pcie passthrough, however when i try to delete it then save, and come back into xml edit its there again. Some reason it keeps getting added again. But i noticed all my PCI passthrough devices have a source address tag and an address tag. So maybe that is normal...?

 

   <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>

 

Edited February 1, 2019 by alfredo_2020

[alfredo_2020]

Also, when i shut down the VM from UNRAID VM And select STOP, it always lets me re-start the VM without issues. Problem is reproduceable every time i shut down from within windows VM. The suqsequent time it doesnt start, crashes or something, then i start again and it comes up...

 

 

EDIT: I was able to determine that i had to change a windows setting in Windows to prevent fast boot. Then the VM can be shut down and restarted no problem repeatedly. It now works ok.

Edited February 5, 2019 by alfredo_2020

[alfredo_2020]

Problems again out of no where my Win10 VM and OSX mojave VM dont boot anymore. i tried to recreate the XMl file thinking it was a bad nvram, but no luck im stuck at the tianno screen again. Im about to give up on unraid, this doesnt seem reliable enough, to have 3K invested in hardware and have it work here and there is not acceptable. I'm going to go back to a chepo build just for plex and nas and thats it. So frustrated with this. Everytime i think i got it to be stable i reboot, shut down the vms couple times in a day im happy. Then the next day i go to use it and crap again... I would expect this type of results from open source, but for this being a paid solution, its very disappointing.