Z800 Gpu passthrough issue

[T8Tom]

Hi all,

 

Today i have been trying to set up an old HP Z800 (2x5660) unraid server with a Win 10 VM including GTX1050ti + Nvidia audio passthrough.

 

Using VNC mode i can get the VM running however when i add the GPU i get the following message....

 

Execution error

internal error: qemu unexpectedly closed the monitor: 2019-01-26T20:34:52.746016Z qemu-system-x86_64: -device vfio-pci,host=0f:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio error: 0000:0f:00.0: failed to setup container for group 14: failed to set iommu for container: Operation not permitted

 

Here are my PCI Devices and IOMMU Groups:

IOMMU group 0:[8086:3406] 00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 13)

IOMMU group 1:[8086:3408] 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)

IOMMU group 2:[8086:340a] 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)

IOMMU group 3:[8086:340e] 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)

IOMMU group 4:[8086:3425] 00:10.0 PIC: Intel Corporation 7500/5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 13)

[8086:3426] 00:10.1 PIC: Intel Corporation 7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 13)

IOMMU group 5:[8086:3427] 00:11.0 PIC: Intel Corporation 7500/5520/5500 Physical and Link Layer Registers Port 1 (rev 13)

[8086:3428] 00:11.1 PIC: Intel Corporation 7500/5520/5500 Routing & Protocol Layer Register Port 1 (rev 13)

IOMMU group 6:[8086:342e] 00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13)

[8086:3422] 00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)

[8086:3423] 00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)

IOMMU group 7:[8086:342f] 00:15.0 PIC: Intel Corporation 7500/5520/5500/X58 Trusted Execution Technology Registers (rev 13)

IOMMU group 8:[8086:3a37] 00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4

[8086:3a38] 00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5

[8086:3a39] 00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6

[8086:3a3c] 00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2

IOMMU group 9:[8086:3a3e] 00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller

IOMMU group 10:[8086:3a40] 00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1

[8086:3a48] 00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5

[8086:3a4a] 00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6

[14e4:1684] 01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5764M Gigabit Ethernet PCIe (rev 10)

[14e4:1684] 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5764M Gigabit Ethernet PCIe (rev 10)

IOMMU group 11:[8086:3a34] 00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1

[8086:3a35] 00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2

[8086:3a36] 00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3

[8086:3a3a] 00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1

IOMMU group 12:[8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)

[1106:3044] 37:04.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev 46)

[11c1:5811] 37:09.0 FireWire (IEEE 1394): LSI Corporation FW322/323 [TrueFire] 1394a Controller (rev 70)

IOMMU group 13:[8086:3a16] 00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller

[8086:2822] 00:1f.2 RAID bus controller: Intel Corporation SATA Controller [RAID mode]

IOMMU group 14:[10de:1c82] 0f:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] (rev a1)

[10de:0fb9] 0f:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)

IOMMU group 15:[8086:340a] 40:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)

IOMMU group 16:[8086:340e] 40:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)

IOMMU group 17:[8086:3410] 40:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)

IOMMU group 18:[8086:3425] 40:10.0 PIC: Intel Corporation 7500/5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 13)

[8086:3426] 40:10.1 PIC: Intel Corporation 7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 13)

IOMMU group 19:[8086:3427] 40:11.0 PIC: Intel Corporation 7500/5520/5500 Physical and Link Layer Registers Port 1 (rev 13)

[8086:3428] 40:11.1 PIC: Intel Corporation 7500/5520/5500 Routing & Protocol Layer Register Port 1 (rev 13)

IOMMU group 20:[8086:342e] 40:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13)

[8086:3422] 40:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)

[8086:3423] 40:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)

IOMMU group 21:[8086:342f] 40:15.0 PIC: Intel Corporation 7500/5520/5500/X58 Trusted Execution Technology Registers (rev 13)

IOMMU group 22:[1000:0058] 41:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)

IOMMU group 23:[8086:2c70] 7f:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02)

[8086:2d81] 7f:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02)

IOMMU group 24:[8086:2d90] 7f:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02)

[8086:2d91] 7f:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02)

[8086:2d92] 7f:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02)

[8086:2d93] 7f:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02)

[8086:2d94] 7f:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02)

[8086:2d95] 7f:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02)

IOMMU group 25:[8086:2d98] 7f:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02)

[8086:2d99] 7f:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02)

[8086:2d9a] 7f:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02)

[8086:2d9c] 7f:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02)

IOMMU group 26:[8086:2da0] 7f:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02)

[8086:2da1] 7f:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02)

[8086:2da2] 7f:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02)

[8086:2da3] 7f:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02)

IOMMU group 27:[8086:2da8] 7f:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02)

[8086:2da9] 7f:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02)

[8086:2daa] 7f:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02)

[8086:2dab] 7f:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02)

IOMMU group 28:[8086:2db0] 7f:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02)

[8086:2db1] 7f:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02)

[8086:2db2] 7f:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02)

[8086:2db3] 7f:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02)

IOMMU group 29:[8086:2c70] 7e:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02)

[8086:2d81] 7e:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02)

IOMMU group 30:[8086:2d90] 7e:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02)

[8086:2d91] 7e:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02)

[8086:2d92] 7e:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02)

[8086:2d93] 7e:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02)

[8086:2d94] 7e:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02)

[8086:2d95] 7e:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02)

IOMMU group 31:[8086:2d98] 7e:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02)

[8086:2d99] 7e:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02)

[8086:2d9a] 7e:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02)

[8086:2d9c] 7e:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02)

IOMMU group 32:[8086:2da0] 7e:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02)

[8086:2da1] 7e:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02)

[8086:2da2] 7e:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02)

[8086:2da3] 7e:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02)

IOMMU group 33:[8086:2da8] 7e:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02)

[8086:2da9] 7e:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02)

[8086:2daa] 7e:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02)

[8086:2dab] 7e:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02)

IOMMU group 34:[8086:2db0] 7e:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02)

[8086:2db1] 7e:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02)

[8086:2db2] 7e:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02)

[8086:2db3] 7e:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02)

 

 

That seems fine to me... Both in the same group.

 

So i followwed these steps:

step 1- make sure you're passing the audio and video in the vm manager.

 

step 2- verify that the audio and video of the card are the only 2 devices in iommu group 15, identified with their corresponding id (tools tab>system devices)

 

if not, then you will have to try enabling acs override (settings>vm manager, click basic-advanced toggle, enable acs override, reboot.)

 

if the audio/video are still not alone, you will need to post diagnostics (tools>diagnostics) and we'll help you with the next step)

 

 

if they are alone in group 15, then step 3:

 

modify your syslinux.cfg  (via main tab>flash) to the following;

 

append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot

then reboot.

 

if that does not work, then post diagnostics.

 

Seen in this thread...

https://forums.unraid.net/topic/57772-hp-z600-graphics-pass-thru/?tab=comments#comment-566252

 

Im still getting the same error message...

 

Any help would be appreciated, cheers

[1812]

but you didn't post diagnostics after trying to start the vm.... a bit harder to troubleshoot. you should post them.

 

I run three vm's on a z400 all with dedicated gpus. I would assume the setup would be similar on the z800, but it may also need the RMRR fix... but couldn't tell you without seeing the diagnostics after trying to start the vm and seeing the failure.

[T8Tom]

That wasnt me posting in that thread i referenced, i just did those steps regardless.

 

How do i post the diagnostics?

 

Cheers

[T8Tom]

https://we.tl/t-ajeIPAUpP5

 

I think thats the diag file

[JonathanM]

40 minutes ago, T8Tom said:

https://we.tl/t-ajeIPAUpP5

 

I think thats the diag file

Go to tools, diagnostics, and download the zip file. Attach the entire zip file with no modifications to your post here. Third party hosting sites are unnecessary and undesirable.

[T8Tom]

thanks

tower-diagnostics-20190127-1200.zip

[1812]

4 hours ago, T8Tom said:

That wasnt me posting in that thread i referenced, i just did those steps regardless.

 

Yes I know, it was me you quoted. And you didn't do those steps (or did them incorrectly, or undid those steps) because your syslinux.cfg shows

 

Command line: BOOT_IMAGE=/bzimage initrd=/bzroot

when it should show

 

Command line: BOOT_IMAGE=/bzimage vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot

 

 

So, at this point you should follow those directions, reboot, then try to run the vm again. if it fails to load, then post diagnostics from immediately afterward.

 

You can modify the syslinux.cfg via the main tab, under boot device, clicking Flash. 

 

I have a high percentage feeling this will get you going once done correctly.

Edited January 28, 2019 by 1812

[T8Tom]

Sorry, i follow you now.

 

I will do that now and post results shortly

 

Thank you for your help :)

[T8Tom]

Returns this error:

 

VM creation error

internal error: qemu unexpectedly closed the monitor: 2019-01-28T18:22:31.575662Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,id=hostdev0,bus=pci.0,addr=0x5,romfile=/mnt/cache/isos/MSI.GTX1050Ti.4096.160929.rom: vfio error: 0000:42:00.0: failed to setup container for group 22: failed to set iommu for container: Operation not permitted

 

I hope im doing the steps correctly, i just added in that code to the end of the existing syslinux info in the box.

 

Cheers

 

 

tower-diagnostics-20190128-1023.zip

[T8Tom]

I notice the GPU ROM file is mentioned in the error message so i tryed creating a VM without the GPU ROM and get this error:

 

VM creation error

internal error: process exited while connecting to monitor: 2019-01-28T18:25:43.492286Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio error: 0000:42:00.0: failed to setup container for group 22: failed to set iommu for container: Operation not permitted

 

Diagnostics attached following this step

tower-diagnostics-20190128-1026.zip

[1812]

1 hour ago, T8Tom said:

i just added in that code to the end of the existing syslinux info in the box.

you are not doing it correctly and/or not rebooting because your log still shows what I put at the top.

 

after clicking flash (on the main tap page) and scrolling down, make your syslinux.cfg look like one of the two sections below (depends on what view you have enabled), then reboot, then retest vm start. if it doesn't work, post new diagnostics.

 

 

[T8Tom]

I was putting it at the end of the global config! Doh!

 

Awesome, its working however i have no video output. Any ideas?

 

Cheers

[1812]

18 minutes ago, T8Tom said:

I was putting it at the end of the global config! Doh!

 

Awesome, its working however i have no video output. Any ideas?

 

Cheers

 

if its a black screen you will need to search around. you may have an Nvidia code 43 or something goofy. You may also need to isolate the card from unraid, provided you have another that could be used for boot processes. Searching "unraid windows 10 black screen" may yield some results.