billington.mark Posted September 23, 2015 Share Posted September 23, 2015 Hi All, I have an OpenELEC Vm with an AMD 5450 passed through working 100%, but when attempting to pass through another 5450 to a windows VM, it refuses to start with an error. Error when starting windows 10 box: 2015-09-23 15:36:56.220+0000: starting up libvirt version: 1.2.18, qemu version: 2.3.0 LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name Windows 10 -S -machine pc-q35-2.3,accel=kvm,usb=off,mem-merge=off -cpu qemu64,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -m 4096 -realtime mlock=on -smp 4,sockets=1,cores=4,threads=1 -uuid 879ec15a-1074-4d4d-17cf-e9026ada07f5 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows 10.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-hpet -no-shutdown -boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -device ich9-usb-ehci1,id=usb,bus=pci.2,addr=0x2.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.2,multifunction=on,addr=0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x3 -drive file=/mnt/cache/VMs/Windows 10/vdisk1.img,if=none,id=drive-virtio-disk2,format=raw,cache=writebacction=on,port=2,chassis=1,id=root.1 -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on -msg timestamp=on Domain id=2 is tainted: high-privileges Domain id=2 is tainted: custom-argv char device redirected to /dev/pts/1 (label charserial0) 2015-09-23T15:36:57.053958Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/37: Operation not permitted 2015-09-23T15:36:57.053991Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: vfio: failed to get group 37 2015-09-23T15:36:57.054002Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: Device initialization failed 2015-09-23T15:36:57.054013Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized 2015-09-23 15:36:57.101+0000: shutting down Windows10 XML: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Windows 10</name> <uuid>879ec15a-1074-4d4d-17cf-e9026ada07f5</uuid> <metadata> <vmtemplate name="Custom" icon="windows.png" os="windows"/> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='12'/> <vcpupin vcpu='1' cpuset='13'/> <vcpupin vcpu='2' cpuset='14'/> <vcpupin vcpu='3' cpuset='15'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-2.3'>hvm</type> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> </hyperv> </features> <cpu> <topology sockets='1' cores='4' threads='1'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/cache/VMs/Windows 10/vdisk1.img'/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/Software/Microsoft Software/en-gb_windows_10_pro_10074_x64_dvd.iso'/> <target dev='hda' bus='sata'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/VMs/virtio-win-0.1-100.iso'/> <target dev='hdb' bus='sata'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:93:db:17'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/Windows 10.org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on'/> </qemu:commandline> </domain> OpenELEC XML: <domain type='kvm' id='5' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>OpenELEC</name> <uuid>65f1ec1f-8929-873b-f419-d8cda09f28d3</uuid> <metadata> <vmtemplate name="OpenELEC" icon="openelec.png" os="openelec" openelec="5.95.3_1"/> </metadata> <memory unit='KiB'>2097152</memory> <currentMemory unit='KiB'>2097152</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='12'/> <vcpupin vcpu='1' cpuset='13'/> <vcpupin vcpu='2' cpuset='14'/> <vcpupin vcpu='3' cpuset='15'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-2.3'>hvm</type> </os> <features> <acpi/> <apic/> </features> <cpu> <topology sockets='1' cores='4' threads='1'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/VMs/OpenELEC-unRAID.x86_64-5.95.3_1.img'/> <backingStore/> <target dev='hda' bus='virtio'/> <readonly/> <boot order='1'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0' multifunction='on'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </controller> <filesystem type='mount' accessmode='passthrough'> <source dir='/mnt/user/VMs/OpenELEC/'/> <target dir='appconfig'/> <alias name='fs0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </filesystem> <interface type='bridge'> <mac address='52:54:00:dd:76:df'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/OpenELEC.org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=28:00.0,bus=pcie.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=28:00.1,bus=pcie.0'/> </qemu:commandline> </domain> Link to comment
billington.mark Posted September 23, 2015 Author Share Posted September 23, 2015 PCI Devices: 00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 13) 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13) 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13) 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13) 00:10.0 PIC: Intel Corporation 7500/5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 13) 00:10.1 PIC: Intel Corporation 7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 13) 00:11.0 PIC: Intel Corporation 7500/5520/5500 Physical and Link Layer Registers Port 1 (rev 13) 00:11.1 PIC: Intel Corporation 7500/5520/5500 Routing & Protocol Layer Register Port 1 (rev 13) 00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13) 00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13) 00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13) 00:15.0 PIC: Intel Corporation 7500/5520/5500/X58 Trusted Execution Technology Registers (rev 13) 00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5 00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5 00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6 00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90) 00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller 00:1f.2 RAID bus controller: Intel Corporation SATA Controller [RAID mode] 01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5764M Gigabit Ethernet PCIe (rev 10) 02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5764M Gigabit Ethernet PCIe (rev 10) 28:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430/5450/5470] 28:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] 40:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13) 40:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13) 40:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13) 40:10.0 PIC: Intel Corporation 7500/5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 13) 40:10.1 PIC: Intel Corporation 7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 13) 40:11.0 PIC: Intel Corporation 7500/5520/5500 Physical and Link Layer Registers Port 1 (rev 13) 40:11.1 PIC: Intel Corporation 7500/5520/5500 Routing & Protocol Layer Register Port 1 (rev 13) 40:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13) 40:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13) 40:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13) 40:15.0 PIC: Intel Corporation 7500/5520/5500/X58 Trusted Execution Technology Registers (rev 13) 41:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08) 42:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430/5450/5470] 42:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] 7e:00.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers (rev 05) 7e:00.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder (rev 05) 7e:02.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 0 (rev 05) 7e:02.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 (rev 05) 7e:02.4 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 1 (rev 05) 7e:02.5 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 1 (rev 05) 7e:03.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller (rev 05) 7e:03.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder (rev 05) 7e:03.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller RAS Registers (rev 05) 7e:03.4 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers (rev 05) 7e:04.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers (rev 05) 7e:04.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers (rev 05) 7e:04.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers (rev 05) 7e:04.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers (rev 05) 7e:05.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers (rev 05) 7e:05.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers (rev 05) 7e:05.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers (rev 05) 7e:05.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers (rev 05) 7e:06.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers (rev 05) 7e:06.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers (rev 05) 7e:06.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers (rev 05) 7e:06.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers (rev 05) 7f:00.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers (rev 05) 7f:00.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder (rev 05) 7f:02.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 0 (rev 05) 7f:02.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 (rev 05) 7f:02.4 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 1 (rev 05) 7f:02.5 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 1 (rev 05) 7f:03.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller (rev 05) 7f:03.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder (rev 05) 7f:03.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller RAS Registers (rev 05) 7f:03.4 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers (rev 05) 7f:04.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers (rev 05) 7f:04.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers (rev 05) 7f:04.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers (rev 05) 7f:04.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers (rev 05) 7f:05.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers (rev 05) 7f:05.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers (rev 05) 7f:05.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers (rev 05) 7f:05.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers (rev 05) 7f:06.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers (rev 05) 7f:06.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers (rev 05) 7f:06.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers (rev 05) 7f:06.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers (rev 05) IOMMU: /sys/kernel/iommu_groups/0/devices/0000:7f:00.0 /sys/kernel/iommu_groups/0/devices/0000:7f:00.1 /sys/kernel/iommu_groups/1/devices/0000:7f:02.0 /sys/kernel/iommu_groups/1/devices/0000:7f:02.1 /sys/kernel/iommu_groups/1/devices/0000:7f:02.4 /sys/kernel/iommu_groups/1/devices/0000:7f:02.5 /sys/kernel/iommu_groups/2/devices/0000:7f:03.0 /sys/kernel/iommu_groups/2/devices/0000:7f:03.1 /sys/kernel/iommu_groups/2/devices/0000:7f:03.2 /sys/kernel/iommu_groups/2/devices/0000:7f:03.4 /sys/kernel/iommu_groups/3/devices/0000:7f:04.0 /sys/kernel/iommu_groups/3/devices/0000:7f:04.1 /sys/kernel/iommu_groups/3/devices/0000:7f:04.2 /sys/kernel/iommu_groups/3/devices/0000:7f:04.3 /sys/kernel/iommu_groups/4/devices/0000:7f:05.0 /sys/kernel/iommu_groups/4/devices/0000:7f:05.1 /sys/kernel/iommu_groups/4/devices/0000:7f:05.2 /sys/kernel/iommu_groups/4/devices/0000:7f:05.3 /sys/kernel/iommu_groups/5/devices/0000:7f:06.0 /sys/kernel/iommu_groups/5/devices/0000:7f:06.1 /sys/kernel/iommu_groups/5/devices/0000:7f:06.2 /sys/kernel/iommu_groups/5/devices/0000:7f:06.3 /sys/kernel/iommu_groups/6/devices/0000:7e:00.0 /sys/kernel/iommu_groups/6/devices/0000:7e:00.1 /sys/kernel/iommu_groups/7/devices/0000:7e:02.0 /sys/kernel/iommu_groups/7/devices/0000:7e:02.1 /sys/kernel/iommu_groups/7/devices/0000:7e:02.4 /sys/kernel/iommu_groups/7/devices/0000:7e:02.5 /sys/kernel/iommu_groups/8/devices/0000:7e:03.0 /sys/kernel/iommu_groups/8/devices/0000:7e:03.1 /sys/kernel/iommu_groups/8/devices/0000:7e:03.2 /sys/kernel/iommu_groups/8/devices/0000:7e:03.4 /sys/kernel/iommu_groups/9/devices/0000:7e:04.0 /sys/kernel/iommu_groups/9/devices/0000:7e:04.1 /sys/kernel/iommu_groups/9/devices/0000:7e:04.2 /sys/kernel/iommu_groups/9/devices/0000:7e:04.3 /sys/kernel/iommu_groups/10/devices/0000:7e:05.0 /sys/kernel/iommu_groups/10/devices/0000:7e:05.1 /sys/kernel/iommu_groups/10/devices/0000:7e:05.2 /sys/kernel/iommu_groups/10/devices/0000:7e:05.3 /sys/kernel/iommu_groups/11/devices/0000:7e:06.0 /sys/kernel/iommu_groups/11/devices/0000:7e:06.1 /sys/kernel/iommu_groups/11/devices/0000:7e:06.2 /sys/kernel/iommu_groups/11/devices/0000:7e:06.3 /sys/kernel/iommu_groups/12/devices/0000:00:00.0 /sys/kernel/iommu_groups/13/devices/0000:00:01.0 /sys/kernel/iommu_groups/14/devices/0000:00:03.0 /sys/kernel/iommu_groups/15/devices/0000:00:07.0 /sys/kernel/iommu_groups/16/devices/0000:00:10.0 /sys/kernel/iommu_groups/16/devices/0000:00:10.1 /sys/kernel/iommu_groups/17/devices/0000:00:11.0 /sys/kernel/iommu_groups/17/devices/0000:00:11.1 /sys/kernel/iommu_groups/18/devices/0000:00:14.0 /sys/kernel/iommu_groups/18/devices/0000:00:14.1 /sys/kernel/iommu_groups/18/devices/0000:00:14.2 /sys/kernel/iommu_groups/19/devices/0000:00:15.0 /sys/kernel/iommu_groups/20/devices/0000:00:1a.0 /sys/kernel/iommu_groups/20/devices/0000:00:1a.1 /sys/kernel/iommu_groups/20/devices/0000:00:1a.2 /sys/kernel/iommu_groups/20/devices/0000:00:1a.7 /sys/kernel/iommu_groups/21/devices/0000:00:1c.0 /sys/kernel/iommu_groups/22/devices/0000:00:1c.4 /sys/kernel/iommu_groups/23/devices/0000:00:1c.5 /sys/kernel/iommu_groups/24/devices/0000:00:1d.0 /sys/kernel/iommu_groups/24/devices/0000:00:1d.1 /sys/kernel/iommu_groups/24/devices/0000:00:1d.2 /sys/kernel/iommu_groups/24/devices/0000:00:1d.7 /sys/kernel/iommu_groups/25/devices/0000:00:1e.0 /sys/kernel/iommu_groups/26/devices/0000:00:1f.0 /sys/kernel/iommu_groups/26/devices/0000:00:1f.2 /sys/kernel/iommu_groups/27/devices/0000:28:00.0 /sys/kernel/iommu_groups/27/devices/0000:28:00.1 /sys/kernel/iommu_groups/28/devices/0000:02:00.0 /sys/kernel/iommu_groups/29/devices/0000:01:00.0 /sys/kernel/iommu_groups/30/devices/0000:40:03.0 /sys/kernel/iommu_groups/31/devices/0000:40:07.0 /sys/kernel/iommu_groups/32/devices/0000:40:09.0 /sys/kernel/iommu_groups/33/devices/0000:40:10.0 /sys/kernel/iommu_groups/33/devices/0000:40:10.1 /sys/kernel/iommu_groups/34/devices/0000:40:11.0 /sys/kernel/iommu_groups/34/devices/0000:40:11.1 /sys/kernel/iommu_groups/35/devices/0000:40:14.0 /sys/kernel/iommu_groups/35/devices/0000:40:14.1 /sys/kernel/iommu_groups/35/devices/0000:40:14.2 /sys/kernel/iommu_groups/36/devices/0000:40:15.0 /sys/kernel/iommu_groups/37/devices/0000:42:00.0 /sys/kernel/iommu_groups/37/devices/0000:42:00.1 /sys/kernel/iommu_groups/38/devices/0000:41:00.0 HVM and IOMMU both show as enabled in the unraid GUI. A few things ive tried so far: Adding Rom files to the XML for VMs makes no difference Moving the cards to different slots makes no difference Adding a 3rd card dedicated to the unraid console session makes no difference Would appreciate any further diags\tests to get to the bottom of the issue. unraid-diagnostics-20150923-1649.zip Link to comment
archedraft Posted September 23, 2015 Share Posted September 23, 2015 What is the output if you SSH into unRAID and type the following: virsh start Windows 10 Link to comment
billington.mark Posted September 23, 2015 Author Share Posted September 23, 2015 Command ran from terminal: Link to comment
archedraft Posted September 23, 2015 Share Posted September 23, 2015 Try modifying your Windows 10 XML to one of the following and see if it will start <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=42:00.1,bus=pcie.0'/> </qemu:commandline> <qemu:commandline> <qemu:arg value=’-device’/> <qemu:arg value=’ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1’/> <qemu:arg value=’-device’/> <qemu:arg value=’vfio-pci,host=42:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on’/> <qemu:arg value=’-device’/> <qemu:arg value=’vfio-pci,host=42:00.1,bus=root.1,addr=00.1’/> </qemu:commandline> Link to comment
bungee91 Posted September 23, 2015 Share Posted September 23, 2015 Relevant error info Error when starting windows 10 box: 2015-09-23T15:36:57.053991Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: vfio: failed to get group 37 2015-09-23T15:36:57.054002Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: Device initialization failed 2015-09-23T15:36:57.054013Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized 2015-09-23 15:36:57.101+0000: shutting down[/code] Section of XML to modify (preferably by using the VM Manager Edit or Add VM routine (not editing manually)) Windows10 XML: <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on'/> </qemu:commandline> </domain> The suggestion archedraft gave you should fix this right up. Basically you can't get group 37 as there are 2 devices in that group (GPU and Audio) and you currently are only attempting to pass 1 of the 2 devices in that group. Once you add in the 42:00.1 device, you should be all set! Link to comment
billington.mark Posted September 23, 2015 Author Share Posted September 23, 2015 Same error as before on each. Should have mentioned before that ive also tried only assigning only the GPU, which would have been why the audio was missing in the above messages. Is there any way to see if the device is being 'tied up' by anything else? I cant even assign the 2nd graphics card to the OpenELEC VM with no other VMs powered on. Could it be possible its sharing some resource that's preventing it being passed through? W10 XML: <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=42:00.1,bus=pcie.0'/> </qemu:commandline> Error: 2015-09-23 18:33:26.995+0000: starting up libvirt version: 1.2.18, qemu version: 2.3.0 LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name Windows 10 -S -machine pc-q35-2.3,accel=kvm,usb=off,mem-merge=off -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -m 4096 -realtime mlock=on -smp 4,sockets=1,cores=4,threads=1 -uuid 879ec15a-1074-4d4d-17cf-e9026ada07f5 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows 10.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-hpet -no-shutdown -boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -device ich9-usb-ehci1,id=usb,bus=pci.2,addr=0x2.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.2,multifunction=on,addr=0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x3 -drive file=/mnt/cache/VMs/Windows 10/vdisk1.img,if=none,id=drive-virtio-disk2,format=raw,cache=writeback ion=on,port=2,chassis=1,id=root.1 -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on -device vfio-pci,host=42:00.1,bus=pcie.0 -msg timestamp=on Domain id=11 is tainted: high-privileges Domain id=11 is tainted: custom-argv Domain id=11 is tainted: host-cpu char device redirected to /dev/pts/2 (label charserial0) 2015-09-23T18:33:27.801027Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/33: Operation not permitted 2015-09-23T18:33:27.801065Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: vfio: failed to get group 33 2015-09-23T18:33:27.801083Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: Device initialization failed 2015-09-23T18:33:27.801095Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized 2015-09-23 18:33:27.850+0000: shutting down Link to comment
bungee91 Posted September 23, 2015 Share Posted September 23, 2015 Two things. Why Q35 for Win 10? Likely not the issue, but i440FX is the recommended. Also, prior to this was group 37, now it's IOMMU group 33. Did you change or move anything? General rule for testing and to know is if the device is in an IOMMU group, it all has to be passed, or the other devices "stubbed" (not able to be used) in order to work. Link to comment
billington.mark Posted September 23, 2015 Author Share Posted September 23, 2015 Yes, ive been testing the different PCie ports to see if it made any difference. (I wont move anything from this point on!) Q35 was the default back when i created this VM a few versions of unraid ago. no reason in particular. updated to i440FX, same error. another thing, will having 2 identical AMD 5450 cards be causing any issue? Link to comment
bungee91 Posted September 23, 2015 Share Posted September 23, 2015 Yes, ive been testing the different PCie ports to see if it made any difference. (I wont move anything from this point on!) Q35 was the default back when i created this VM a few versions of unraid ago. no reason in particular. updated to i440FX, same error. The change to i440fx may require reactivating, so if already all setup you may want to change it back. Does this VM work if you assign the working 5450 to this VM and try it (shut down OE, bring this one up)? Does this 5450 work in OE if you swap the XML to it? Link to comment
bungee91 Posted September 23, 2015 Share Posted September 23, 2015 another thing, will having 2 identical AMD 5450 cards be causing any issue? Others have without issue, so I'd doubt that. For USB devices, yes.. For physical PCIe devices with a proper address, I wouldn't think so. I have 2 GT 720's (however slightly different) that are presented as identical devices as far as the naming in the VM manager. This has never presented me any issues. Link to comment
billington.mark Posted September 23, 2015 Author Share Posted September 23, 2015 Reactivation isnt an issue, I have a technet subscription so access to multiple keys. Yes, if i swap the GPU assignment around, Windows 10 boots, and openELEC throws the same error: 2015-09-23 18:48:44.173+0000: starting up libvirt version: 1.2.18, qemu version: 2.3.0 Domain id=14 is tainted: high-privileges Domain id=14 is tainted: custom-argv Domain id=14 is tainted: host-cpu char device redirected to /dev/pts/2 (label charserial0) 2015-09-23T18:48:44.726194Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/33: Operation not permitted 2015-09-23T18:48:44.726225Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: vfio: failed to get group 33 2015-09-23T18:48:44.726237Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: Device initialization failed 2015-09-23T18:48:44.726249Z qemu-system-x86_64: -device vfio-pci,host=42:00.0,bus=pcie.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized 2015-09-23 18:48:44.775+0000: shutting down Link to comment
bungee91 Posted September 23, 2015 Share Posted September 23, 2015 Interesting... Let me make sure I've got this right. So whatever you do that particular card refuses to work, regardless of the location that it is installed to? If you move it into the location of the working one, and swap the XML (to use this GPU), you get the same error? Do you only have 2 GPU's total in this machine (no integrated GPU)? Is UnRAID displaying on any of them? If so, users have reported issues with outputting a VM to a display that unRAID is using, or that is the only display that is available. You can throw in a cheap PCI graphics card (since you have a slot, they're like ~10 on eBay) and this should sort out the issue. Link to comment
snocrash Posted September 24, 2015 Share Posted September 24, 2015 Interesting... Let me make sure I've got this right. So whatever you do that particular card refuses to work, regardless of the location that it is installed to? If you move it into the location of the working one, and swap the XML (to use this GPU), you get the same error? Do you only have 2 GPU's total in this machine (no integrated GPU)? Is UnRAID displaying on any of them? If so, users have reported issues with outputting a VM to a display that unRAID is using, or that is the only display that is available. You can throw in a cheap PCI graphics card (since you have a slot, they're like ~10 on eBay) and this should sort out the issue. I'm having similar issues as mark, actually. I've been through a billion reconfigs and variations of my setup, but if I enable the PCIe ACS Override setting, I can get one of my VMs to run, and passthrough a video card (doesnt matter which - any one will work). If I attempt to launch a second or a 3rd, I get errors just like his. My IOMMU groups are listed totally wrong, with the video and audio portion of the card getting different groups. When I disable the ACS Override, all my groups return to normal again and I don't get the error. Unfortunately, I still can't get more than one VM to work with a video card passthrough. Where my experience starts to differ from the expected is that when I get my one VM to work with a video card passthrough (using SeaBios), I STILL retain my VGA console. Works just fine, side by side with my passthrough VM. Link to comment
billington.mark Posted September 24, 2015 Author Share Posted September 24, 2015 So I slept on this and made a list of things to double check while i was out on my 5am Dog walk... First thing in the list... BIOS Update... Turns out my 'new' motherboard was a million or so BIOS revisions behind. :-[ After the BIOS update and reboot, everything is working 100%. Sorry for wasting everyone's time with this.. feel a bit stupid! So... moral of the story... update EVERYTHING no matter how small before faffing like i did! Only makes your head hurt and everything more complicated than it needs to be! Link to comment
archedraft Posted September 24, 2015 Share Posted September 24, 2015 So I slept on this and made a list of things to double check while i was out on my 5am Dog walk... First thing in the list... BIOS Update... Turns out my 'new' motherboard was a million or so BIOS revisions behind. :-[ After the BIOS update and reboot, everything is working 100%. Sorry for wasting everyone's time with this.. feel a bit stupid! So... moral of the story... update EVERYTHING no matter how small before faffing like i did! Only makes your head hurt and everything more complicated than it needs to be! Wow, I wouldn't have guessed bios due to you already being able to pass through one graphics card. Thanks for the update though, something to keep in mind. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.