October 21, 201411 yr I was trying to set up my Windows 8 VM (on my test machine) by using dmacias excellent Virtual Machine Management and the Web Virtual Manager support. The objective is to passthrough the NVIDIA GF106GL [Quadro 2000] video card along with the NVIDIA GF106 HD Audio Controller and making the DVD ROM available for that particular VM. vfio-bind 0000:01:00.0 0000:01:00.1 is working: Kernel driver in use: vfio-pci (please see below): root@Tower2:/mnt/cache/.VMs/win8# lspci -knn 00:00.0 Host bridge [0600]: Intel Corporation Core Processor DRAM Controller [8086:0040] (rev 12) Subsystem: ASUSTeK Computer Inc. Device [1043:8383] 00:01.0 PCI bridge [0604]: Intel Corporation Core Processor PCI Express x16 Root Port [8086:0041] (rev 12) Kernel driver in use: pcieport 00:16.0 Communication controller [0780]: Intel Corporation 5 Series/3400 Series Chipset HECI Controller [8086:3b64] (rev 06) Subsystem: ASUSTeK Computer Inc. Device [1043:8383] 00:1a.0 USB controller [0c03]: Intel Corporation 5 Series/3400 Series Chipset USB2 Enhanced Host Controller [8086:3b3c] (rev 06) Subsystem: ASUSTeK Computer Inc. Device [1043:8383] Kernel driver in use: ehci-pci 00:1b.0 Audio device [0403]: Intel Corporation 5 Series/3400 Series Chipset High Definition Audio [8086:3b56] (rev 06) Subsystem: ASUSTeK Computer Inc. Device [1043:8435] 00:1c.0 PCI bridge [0604]: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 1 [8086:3b42] (rev 06) Kernel driver in use: pcieport 00:1c.2 PCI bridge [0604]: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 3 [8086:3b46] (rev 06) Kernel driver in use: pcieport 00:1c.3 PCI bridge [0604]: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 4 [8086:3b48] (rev 06) Kernel driver in use: pcieport 00:1c.4 PCI bridge [0604]: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 5 [8086:3b4a] (rev 06) Kernel driver in use: pcieport 00:1c.5 PCI bridge [0604]: Intel Corporation 5 Series/3400 Series Chipset PCI Express Root Port 6 [8086:3b4c] (rev 06) Kernel driver in use: pcieport 00:1d.0 USB controller [0c03]: Intel Corporation 5 Series/3400 Series Chipset USB2 Enhanced Host Controller [8086:3b34] (rev 06) Subsystem: ASUSTeK Computer Inc. Device [1043:8383] Kernel driver in use: ehci-pci 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev a6) 00:1f.0 ISA bridge [0601]: Intel Corporation 5 Series Chipset LPC Interface Controller [8086:3b06] (rev 06) Subsystem: ASUSTeK Computer Inc. Device [1043:8383] 00:1f.2 SATA controller [0106]: Intel Corporation 5 Series/3400 Series Chipset 6 port SATA AHCI Controller [8086:3b22] (rev 06) Subsystem: ASUSTeK Computer Inc. Device [1043:8383] Kernel driver in use: ahci Kernel modules: ahci 00:1f.3 SMBus [0c05]: Intel Corporation 5 Series/3400 Series Chipset SMBus Controller [8086:3b30] (rev 06) Subsystem: ASUSTeK Computer Inc. Device [1043:8383] Kernel modules: i2c_i801 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF106GL [Quadro 2000] [10de:0dd8] (rev a1) Subsystem: Hewlett-Packard Company Device [103c:084a] Kernel driver in use: vfio-pci 01:00.1 Audio device [0403]: NVIDIA Corporation GF106 High Definition Audio Controller [10de:0be9] (rev a1) Subsystem: Hewlett-Packard Company Device [103c:084a] Kernel driver in use: vfio-pci 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller [10ec:8168] (rev 03) Subsystem: ASUSTeK Computer Inc. M4A785TD Motherboard [1043:83a3] Kernel driver in use: r8169 Kernel modules: r8169 05:00.0 Multimedia video controller [0400]: Micronas Semiconductor Holding AG nGene PCI-Express Multimedia Controller [18c3:0720] (rev 01) Subsystem: Micronas Semiconductor Holding AG Device [18c3:db02] 06:00.0 IDE interface [0101]: JMicron Technology Corp. JMB368 IDE controller [197b:2368] Subsystem: ASUSTeK Computer Inc. Device [1043:824f] Kernel driver in use: pata_jmicron Kernel modules: pata_jmicron 3f:00.0 Host bridge [0600]: Intel Corporation Core Processor QuickPath Architecture Generic Non-core Registers [8086:2c61] (rev 02) Subsystem: Intel Corporation Device [8086:8086] 3f:00.1 Host bridge [0600]: Intel Corporation Core Processor QuickPath Architecture System Address Decoder [8086:2d01] (rev 02) Subsystem: Intel Corporation Device [8086:8086] 3f:02.0 Host bridge [0600]: Intel Corporation Core Processor QPI Link 0 [8086:2d10] (rev 02) Subsystem: Intel Corporation Device [8086:8086] 3f:02.1 Host bridge [0600]: Intel Corporation Core Processor QPI Physical 0 [8086:2d11] (rev 02) Subsystem: Intel Corporation Device [8086:8086] 3f:02.2 Host bridge [0600]: Intel Corporation Core Processor Reserved [8086:2d12] (rev 02) Subsystem: Intel Corporation Device [8086:8086] 3f:02.3 Host bridge [0600]: Intel Corporation Core Processor Reserved [8086:2d13] (rev 02) Subsystem: Intel Corporation Device [8086:8086] Here's my xml file (I have no clue where I got the controller information from - this VM was running already a while ago): <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win8</name> <uuid>cc411d70-4463-4db7-bf36-d364c0cdaa9d</uuid> <description>None</description> <memory unit='KiB'>2097152</memory> <currentMemory unit='KiB'>2097152</currentMemory> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-2.1'>hvm</type> <loader>/usr/share/qemu/bios-256k.bin</loader> <boot dev='hd'/> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/mnt/cache/.VMs/win8/win8.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </disk> <device> <name>block_sr0_HL_DT_ST_DVDRAM_GH24NS95_K3XD1JC0659</name> <path>/sys/devices/pci0000:00/0000:00:1c.0/0000:06:00.0/ata1/host1/target1:0:1/1:0:1:0/block/sr0</path> <parent>scsi_1_0_1_0</parent> <capability type='storage'> <block>/dev/sr0</block> <bus>ata</bus> <drive_type>cdrom</drive_type> <model>DVDRAM GH24NS95</model> <vendor>HL-DT-ST</vendor> <serial>HL-DT-ST_DVDRAM_GH24NS95_K3XD1JC0659</serial> <capability type='removable'> <media_available>0</media_available> <media_size>0</media_size> </capability> </capability> </device> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <interface type='bridge'> <mac address='02:16:3e:1a:b3:4a'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </memballoon> </devices> </domain> Error code: XML error: PCI bus is not compatible with the device at 0000:00:03.0. Device requires a standard PCI slot, which is not provided by bus 0000:00 I was using the Node information for the DVD ROM <device> but I might need to completely start over again by using the existing win8.qcow2 file. So what is best practice?
October 24, 201411 yr Author I used copy & past from an existing VM: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win8</name> <uuid>cc411d70-4463-4db7-bf36-d364c0cdaa9d</uuid> <memory unit='KiB'>2097152</memory> <currentMemory unit='KiB'>2097152</currentMemory> <vcpu placement='static'>2</vcpu> <os> <type arch='x86_64' machine='pc-q35-2.1'>hvm</type> <boot dev='hd'/> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='host-passthrough'> <topology sockets='2' cores='2' threads='1'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/mnt/cache/.VMs/win8/win8.qcow2'/> <target dev='vda' bus='virtio'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <device> <name>block_sr0_HL_DT_ST_DVDRAM_GH24NS95_K3XD1JC0659</name> <path>/sys/devices/pci0000:00/0000:00:1c.0/0000:06:00.0/ata1/host1/target1:0:1/1:0:1:0/block/sr0</path> <parent>scsi_1_0_1_0</parent> <capability type='storage'> <block>/dev/sr0</block> <bus>ata</bus> <drive_type>cdrom</drive_type> <model>DVDRAM GH24NS95</model> <vendor>HL-DT-ST</vendor> <serial>HL-DT-ST_DVDRAM_GH24NS95_K3XD1JC0659</serial> <capability type='removable'> <media_available>0</media_available> <media_size>0</media_size> </capability> </capability> </device> <interface type='bridge'> <mac address='02:16:3e:1a:b3:4a'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> </interface> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <hub type='usb'> <address type='usb' bus='0' port='1'/> </hub> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>/> </qemu:commandline> </domain> The VM is starting fine when I exclude the following: <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>/> </qemu:commandline> If I'm using this the error message is: root@Tower2:/mnt/cache/.VMs/win8# virsh create win8.xml error: Failed to create domain from win8.xml error: internal error: early end of file from monitor: possible problem: 2014-10-24T11:24:42.013243Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to set iommu for container: Operation not permitted 2014-10-24T11:24:42.013293Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 13 2014-10-24T11:24:42.013307Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13 2014-10-24T11:24:42.013329Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. 2014-10-24T11:24:42.013348Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized lspci -knn is showing, that the devices are using the vfio driver: 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF106GL [Quadro 2000] [10de:0dd8] (rev a1) Subsystem: Hewlett-Packard Company Device [103c:084a] Kernel driver in use: vfio-pci 01:00.1 Audio device [0403]: NVIDIA Corporation GF106 High Definition Audio Controller [10de:0be9] (rev a1) Subsystem: Hewlett-Packard Company Device [103c:084a] Kernel driver in use: vfio-pci Any idea?
October 24, 201411 yr Author Just found something in syslog: Tower2 kernel: vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform I'm afraid that I have to change back the syslinux.cfg. This is what I'm having now: label KVM/unRAID OS menu default kernel /bzimage append pcie_acs_override=downstream initrd=/bzroot Let's see what the change will bring....
October 24, 201411 yr Author Yes, this was doing the trick. I can see the NVIDIA Quadro 2000 and fight with the device driver but it is looking promising. Will report later. @jonp, maybe you would consider to change your advise here: http://lime-technology.com/forum/index.php?topic=35732.0. This was related to the syslinux.cfg {EDIT}Issues with the device driver. The driver installation went well but the VM doesn't boot anymore.
October 24, 201411 yr @EdgarWallace Give this a try: http://lime-technology.com/forum/index.php?topic=35732.msg334596#msg334596 Worked on my NVIDIA based Evga GTX650. Pass-through worked right from boot menu. I have run Windows 7, 8.1 and 10 with the above template. Reed
October 29, 201411 yr Author Thanks rd_blair. I was able to install Windows 8.1 from scratch and installed the NVIDIA device driver. Unfortunately the VM isn't starting anymore. Here is my win8.xml: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win8</name> <uuid>cc411d70-4463-4db7-bf36-d364c0cdaa9d</uuid> <memory unit='KiB'>2097152</memory> <currentMemory unit='KiB'>2097152</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='1'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='3'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-2.1'>hvm</type> <boot dev='cdrom'/> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='host-passthrough'> <topology sockets='2' cores='2' threads='1'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/mnt/cache/.VMs/win8/win8.qcow2'/> <target dev='vda' bus='virtio'/> </disk> <disk type='file' device='cdrom'> <source file='/mnt/user/backup/Oliver/Archiv/IT/unRaid/ISOs/win81enterp64.iso'/> <target dev='hdb' bus='sata'/> <readonly/> </disk> <disk type='file' device='cdrom'> <source file='/mnt/user/backup/Oliver/Archiv/IT/unRaid/ISOs/virtio-win-0.1-81.iso'/> <target dev='hdc' bus='sata'/> <readonly/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb0'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb0'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb0'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x2'/> </controller> <interface type='bridge'> <mac address='02:16:3e:1a:b3:4a'/> <source bridge='br0'/> <model type='virtio'/> </interface> <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>/> </qemu:commandline> </domain> Any additional ideas?
October 29, 201411 yr Did you change the disk type? I know I had to set mine to SATA instead of Virtio, but that was because I created my VM with VirtualBox.
November 11, 201411 yr Author Here is how the issues have been resolved: http://lime-technology.com/forum/index.php?topic=35732.msg337070#msg337070
Archived
This topic is now archived and is closed to further replies.