November 12, 201411 yr So I have a successful windows 8 install without passthrough. I want to change that. Here is my current working XML that was created using Dmacias's plugin: <domain type='kvm' id='8'> <name>win8pro</name> <uuid>abd14b60-59d0-db2a-9f7a-5da4be5bd0b0</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <vcpu placement='static'>8</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type> <boot dev='cdrom'/> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/mnt/cache/images/win8/win8.qcow2'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/cache/images/win8/windows8pro(1).iso'/> <backingStore/> <target dev='hdc' bus='ide'/> <readonly/> <alias name='ide0-1-0'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/cache/images/win7/virtio-win.iso'/> <backingStore/> <target dev='hdd' bus='ide'/> <readonly/> <alias name='ide0-1-1'/> <address type='drive' controller='0' bus='1' target='0' unit='1'/> </disk> <controller type='usb' index='0'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='bridge'> <mac address='52:54:00:99:f5:fc'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='tablet' bus='usb'> <alias name='input0'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='5900' autoport='yes' websocket='5700' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc52b'/> <address bus='1' device='3'/> </source> <alias name='hostdev0'/> </hostdev> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </memballoon> </devices> </domain> I typed the following in command line vfio-bind 0000:01:00.0 0000:01:00.1 My go file currently looks like this: #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & cd /boot/packages && find . -name '*.auto_install' -type f -print | sort | xargs -n1 sh -c /usr/local/sbin/vfio-bind 0000:01:00.0 0000:01:00.1 And this is my syslinux default /syslinux/menu.c32 menu title Lime Technology prompt 0 timeout 50 label unRAID OS menu default kernel /bzimage append pcie_acs_override=downstream initrd=/bzroot label unRAID OS Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest label Xen/unRAID OS kernel /syslinux/mboot.c32 append /xen dom0_mem=2097152 --- /bzimage --- /bzroot label Xen/unRAID OS Safe Mode (no plugins) kernel /syslinux/mboot.c32 append /xen dom0_mem=2097152 --- /bzimage --- /bzroot unraidsafemode This is my GPU pci_0000_01_00_0 0x1002:0x9442 None AMD/ATI [Advanced Micro Devices, Inc.] RV770 [Radeon HD 4850] pci_0000_01_00_1 0x1002:0xaa30 None AMD/ATI [Advanced Micro Devices, Inc.] RV770 HDMI Audio [Radeon HD 4850/4870] This is the XML of the USB device I want to passthrough. Its a logitech keyboard/mouse combo <device> <name>usb_1_1_5</name> <path>/sys/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.5</path> <parent>usb_1_1</parent> <driver> <name>usb</name> </driver> <capability type='usb_device'> <bus>1</bus> <device>3</device> <product id='0xc52b'>USB Receiver</product> <vendor id='0x046d'>Logitech</vendor> </capability> </device> I have edited my xml to being close to Jonp's as possible, as a base to wok off: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win81pro</name> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='1'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='3'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-2.1'>hvm</type> <boot dev='cdrom'/> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough'> <topology sockets='2' cores='2' threads='1'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/mnt/cache/images/win8/win8pass.qcow2'/> <target dev='vdc' bus='virtio'/> </disk> <disk type='file' device='cdrom'> <source file='/mnt/cache/images/win8/windows8pro(1).iso'/> <target dev='hda' bus='sata'/> <readonly/> </disk> <disk type='file' device='cdrom'> <source file='/mnt/cache/images/win7/virtio-win.iso'/> <target dev='hdd' bus='sata'/> <readonly/> </disk> <interface type='bridge'> <mac address='52:54:00:99:f5:fc'/> <source bridge='br0'/> <model type='virtio'/> </interface> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb0'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb0'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb0'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x2'/> </controller> <hostdev mode='subsystem' type='usb'> <source> <vendor id='0x046d'/> <product id='0xc52b'/> </source> </hostdev> <memballoon model='virtio'> <alias name='balloon0'/> </memballoon> </devices> <seclabel type='none' model='none'/> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>/> </qemu:commandline> </domain> I get the following errors when trying to start the VM: Error while starting domain: internal error: early end of file from monitor: possible problem: 2014-11-12T06:02:47.585829Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device 2014-11-12T06:02:47.585857Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. 2014-11-12T06:02:47.585868Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized This is not my first attempt either. I must have tried about a dozen different ways of getting it to work but now its time to ask for help. This is my syslog for when I tried to start the VM. I dont know if its any use for this but here it is: Nov 12 06:02:46 Tower kernel: device vnet0 entered promiscuous mode Nov 12 06:02:46 Tower kernel: br0: topology change detected, sending tcn bpdu Nov 12 06:02:46 Tower kernel: br0: port 2(vnet0) entered forwarding state Nov 12 06:02:46 Tower kernel: br0: port 2(vnet0) entered forwarding state Nov 12 06:02:47 Tower avahi-daemon[5818]: Withdrawing workstation service for vnet0. Nov 12 06:02:47 Tower kernel: br0: port 2(vnet0) entered disabled state Nov 12 06:02:47 Tower kernel: device vnet0 left promiscuous mode Nov 12 06:02:47 Tower kernel: br0: port 2(vnet0) entered disabled state If anyone can tell me where to begin it would be much appreciated or what further info I should post because at this stage I don't think I can work it out myself (even after reading the many pages in the KVM section)
November 12, 201411 yr Are you certain you have vt-d enabled in the BIOS?.. There is a command to run to see the IOMMU grouping for each device also (can't think of it off hand) I'd also look there.. Maybe lspci - k?
November 12, 201411 yr Process of elimination...remove either the USB passthrough section or the PCI (VGA) passthrough section and see if the VM will start. Also, see if this helps: http://lime-technology.com/forum/index.php?topic=36110.0 PCI passthrough would not work for me without the changes shown below: default /syslinux/menu.c32 menu title Lime Technology prompt 0 timeout 50 label unRAID OS kernel /bzimage append initrd=/bzroot label KVM/unRAID OS menu default kernel /bzimage append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot label unRAID OS Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest label Xen/unRAID OS kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot label Xen/unRAID OS Safe Mode (no plugins) kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot unraidsafemode Be aware that if you have an AMD CPU, the first part should read append amd_iommu=on instead of append intel_iommu=on.
November 12, 201411 yr Author Are you certain you have vt-d enabled in the BIOS?.. There is a command to run to see the IOMMU grouping for each device also (can't think of it off hand) I'd also look there.. Maybe lspci - k? Yep, its enabled alright. I tried lspci - k and I cant see much from it. Then again, I'm not too sure what I'm seeing, Here is the top part from running that cmd root@Tower:~# lspci -k 00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07) Subsystem: ASUSTeK Computer Inc. Device 84ef 00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07) Kernel driver in use: pcieport 00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07) Kernel driver in use: pcieport 00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07) Kernel driver in use: pcieport 00:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Mi sc, System Management (rev 07) Subsystem: ASUSTeK Computer Inc. Device 84ef 00:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07) Subsystem: ASUSTeK Computer Inc. Device 84ef 00:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07) Subsystem: ASUSTeK Computer Inc. Device 84ef 00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtua l Root Port (rev 05) Kernel driver in use: pcieport 00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05) Subsystem: ASUSTeK Computer Inc. Device 84ef 00:19.0 Ethernet controller: Intel Corporation 82579V Gigabit Network Connection (rev 05) Subsystem: Intel Corporation Device 0000 Kernel driver in use: e1000e Kernel modules: e1000e 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 05) Subsystem: ASUSTeK Computer Inc. Device 84ef Kernel driver in use: ehci-pci 00:1b.0 Audio device: Intel Corporation C600/X79 series chipset High Definition Audio Controller (rev 05) Subsystem: ASUSTeK Computer Inc. Device 8436 00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root P ort 1 (rev b5) Kernel driver in use: pcieport 00:1c.1 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root P ort 2 (rev b5) Kernel driver in use: pcieport 00:1c.2 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root P ort 3 (rev b5) Kernel driver in use: pcieport 00:1c.3 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root P ort 4 (rev b5) Kernel driver in use: pcieport 00:1c.4 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root P ort 5 (rev b5) Kernel driver in use: pcieport 00:1c.5 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root P ort 6 (rev b5) Kernel driver in use: pcieport 00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root P ort 8 (rev b5) Kernel driver in use: pcieport 00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 05) Subsystem: ASUSTeK Computer Inc. Device 84ef Kernel driver in use: ehci-pci 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) 00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (re v 05) Subsystem: ASUSTeK Computer Inc. Device 84ef 00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA A HCI Controller (rev 05) Subsystem: ASUSTeK Computer Inc. Device 84ef Kernel driver in use: ahci Kernel modules: ahci 00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller ( rev 05) Subsystem: ASUSTeK Computer Inc. Device 84ef Kernel driver in use: i801_smbus Kernel modules: i2c_i801 01:00.0 VGA compatible controller: AMD/ATI [Advanced Micro Devices, Inc.] RV770 [Radeon HD 4850] Subsystem: ASUSTeK Computer Inc. Device 025e 01:00.1 Audio device: AMD/ATI [Advanced Micro Devices, Inc.] RV770 HDMI Audio [R adeon HD 4850/4870] Subsystem: ASUSTeK Computer Inc. Device aa30 06:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Cont roller Subsystem: ASUSTeK Computer Inc. Device 8488 Kernel driver in use: xhci_hcd 07:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Cont roller Subsystem: ASUSTeK Computer Inc. Device 8488 Kernel driver in use: xhci_hcd 08:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Cont roller Subsystem: ASUSTeK Computer Inc. Device 8488 Kernel driver in use: xhci_hcd 09:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller ( rev 01) Subsystem: ASUSTeK Computer Inc. Device 84b7 Kernel driver in use: ahci Kernel modules: ahci 0a:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6315 Series Firewire Cont roller (rev 01) Subsystem: VIA Technologies, Inc. VT6315 Series Firewire Controller 0b:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9128 PCIe SATA 6 Gb/s RAID controller with HyperDuo (rev 11) Subsystem: ASUSTeK Computer Inc. P8P67 Deluxe Motherboard Kernel driver in use: ahci Kernel modules: ahci 0c:01.0 VGA compatible controller: NVIDIA Corporation NV18 [GeForce4 MX 4000] (r ev c1) ff:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07) Subsystem: ASUSTeK Computer Inc. Device 84ef ff:08.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (re v 07) Subsystem: ASUSTeK Computer Inc. Device 84ef ff:08.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (re v 07) Subsystem: ASUSTeK Computer Inc. Device 84ef ff:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07) Subsystem: ASUSTeK Computer Inc. Device 84ef ff:09.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (re v 07) Subsystem: ASUSTeK Computer Inc. Device 84ef ff:09.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (re v 07) Subsystem: ASUSTeK Computer Inc. Device 84ef ff:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 07) Subsystem: ASUSTeK Computer Inc. Device 84ef ff:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 07) Kernel driver in use: snbep_uncore Process of elimination...remove either the USB passthrough section or the PCI (VGA) passthrough section and see if the VM will start. Also, see if this helps: http://lime-technology.com/forum/index.php?topic=36110.0 PCI passthrough would not work for me without the changes shown below: default /syslinux/menu.c32 menu title Lime Technology prompt 0 timeout 50 label unRAID OS kernel /bzimage append initrd=/bzroot label KVM/unRAID OS menu default kernel /bzimage append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot label unRAID OS Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest label Xen/unRAID OS kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot label Xen/unRAID OS Safe Mode (no plugins) kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot unraidsafemode Be aware that if you have an AMD CPU, the first part should read append amd_iommu=on instead of append intel_iommu=on. I have edited my syslinux as you suggested and tried a restart but still get the following errors: This is my xml unedited: Error while starting domain: internal error: early end of file from monitor: possible problem: 2014-11-12T17:10:15.417723Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device 2014-11-12T17:10:15.417754Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. 2014-11-12T17:10:15.417767Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized I get the same error when I remove the USB Error while starting domain: internal error: early end of file from monitor: possible problem: 2014-11-12T17:12:06.292464Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device 2014-11-12T17:12:06.292491Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. 2014-11-12T17:12:06.292499Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized When I go to Tower and look for iommu_group I cannot see it. I'm looking here root@Tower:/sys/bus/pci/devices/0000:01:00.0# ls boot_vga config device enabled broken_parity_status consistent_dma_mask_bits dma_mask_bits irq class d3cold_allowed driver_override local_cpulist Should it be in there?
November 12, 201411 yr Author root@Tower:~# dmesg | grep IOMMU Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA Intel-IOMMU: enabled So, its enabled alright
November 12, 201411 yr Check your qemu commandline. Looks like you have an extra /> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>/> </qemu:commandline>
November 12, 201411 yr The - k is not the right flag, and I am away from my notes. Your error clearly states the lack of IOMMU grouping. There is an output for lspci that will show what is grouped to what. However you also have an Asus board & those are known to be flakey with actually supporting and working with vt-d (regardless of the manual/specs saying it's supported). I'm not saying this is your issue, however you may want to make sure someone else with the same board has completed this successfully.
November 12, 201411 yr So I have a successful windows 8 install without passthrough. I want to change that. If anyone can tell me where to begin it would be much appreciated or what further info I should post because at this stage I don't think I can work it out myself (even after reading the many pages in the KVM section) Okay, my suggestion is to go in stages. Figure out what USB devices you want to passthru, and add them to the XML, where it's easy to test in a VNC window. Update all drivers, and get that all working fine before moving to the GPU passthru. This will confirm that everything but the GPU works, so when you move onto the GPU, you know that's the only thing you're tryig to fix, and that you're not having other issues. Here is the section I have for my USB passthru stuff... <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x050d'/> <product id='0x0201'/> <address bus='3' device='3'/> </source> <alias name='hostdev0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x045e'/> <product id='0x00db'/> <address bus='3' device='5'/> </source> <alias name='hostdev1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x045e'/> <product id='0x0745'/> <address bus='3' device='6'/> </source> <alias name='hostdev2'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0a12'/> <product id='0x0001'/> <address bus='3' device='7'/> </source> <alias name='hostdev3'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x147a'/> <product id='0xe03e'/> <address bus='3' device='4'/> </source> <alias name='hostdev4'/> </hostdev> This comes from the "Devices" section of the vnMan plugin. If you select devices, then select the "usb_devices" tab, it will show only the stuff you can pass. Mine looks like this the screenshot attached. You can see the device names and the codes I used in the XML section above. I would suggest that once you get this all working with all the USB devices passed thru, you copy the entire XML file, and then create a new VM from XML, and copy that working code to the new VM, then give it a new name and make changes to the new one only. This lets you keep the known working VNC only VM in tact, and any trouble you have with the passthru VM can remain separate. The other screenshot I have shows how I do this. When you create the new XML, which you will use for passthru, change the name, the description, the id and the uuid (I just changed the last digit from a to b). <domain type='kvm' id='2' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Windows8_Passthru</name> <uuid>2e151e44-f801-4773-a6df-bfb01b0812fb</uuid> <description>Windows8 with Passthru</description> Then add the bit at the end to do the GPU passthru. Mine looks like this... <seclabel type='none' model='none'/> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=07:00.0,bus=pcie.0'/> </qemu:commandline> The last 2 lines are to passthru my TV tuner card; you probably don't need them. the last bits are the /boot/syslinux/syslinux.cfg file. Mine has this... append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot and the go file, which has this... /usr/local/sbin/vfio-bind 0000:01:00.0 0000:01:00.1 0000:07:00.0 That should get you there, but if you have issues, write back.
November 12, 201411 yr Author Check your qemu commandline. Looks like you have an extra /> Good spot. The same errors occurred after deleting it The - k is not the right flag, and I am away from my notes. Your error clearly states the lack of IOMMU grouping. There is an output for lspci that will show what is grouped to what. However you also have an Asus board & those are known to be flakey with actually supporting and working with vt-d (regardless of the manual/specs saying it's supported). I'm not saying this is your issue, however you may want to make sure someone else with the same board has completed this successfully. Ive seen a few posts where people have successful thorough with the same cpu and mobo as me. But who knows what the story is now. New Bios updates seems to break/fix/break VT-D. I hope the issue is to do with me and not my hardware Okay, my suggestion is to go in stages. Figure out what USB devices you want to passthru, and add them to the XML, where it's easy to test in a VNC window. Update all drivers, and get that all working fine before moving to the GPU passthru. This will confirm that everything but the GPU works, so when you move onto the GPU, you know that's the only thing you're tryig to fix, and that you're not having other issues. Here is the section I have for my USB passthru stuff... <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x050d'/> <product id='0x0201'/> <address bus='3' device='3'/> </source> <alias name='hostdev0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x045e'/> <product id='0x00db'/> <address bus='3' device='5'/> </source> <alias name='hostdev1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x045e'/> <product id='0x0745'/> <address bus='3' device='6'/> </source> <alias name='hostdev2'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0a12'/> <product id='0x0001'/> <address bus='3' device='7'/> </source> <alias name='hostdev3'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x147a'/> <product id='0xe03e'/> <address bus='3' device='4'/> </source> <alias name='hostdev4'/> </hostdev> This comes from the "Devices" section of the vnMan plugin. If you select devices, then select the "usb_devices" tab, it will show only the stuff you can pass. Mine looks like this the screenshot attached. You can see the device names and the codes I used in the XML section above. I would suggest that once you get this all working with all the USB devices passed thru, you copy the entire XML file, and then create a new VM from XML, and copy that working code to the new VM, then give it a new name and make changes to the new one only. This lets you keep the known working VNC only VM in tact, and any trouble you have with the passthru VM can remain separate. The other screenshot I have shows how I do this. When you create the new XML, which you will use for passthru, change the name, the description, the id and the uuid (I just changed the last digit from a to b). <domain type='kvm' id='2' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Windows8_Passthru</name> <uuid>2e151e44-f801-4773-a6df-bfb01b0812fb</uuid> <description>Windows8 with Passthru</description> Then add the bit at the end to do the GPU passthru. Mine looks like this... <seclabel type='none' model='none'/> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=07:00.0,bus=pcie.0'/> </qemu:commandline> The last 2 lines are to passthru my TV tuner card; you probably don't need them. the last bits are the /boot/syslinux/syslinux.cfg file. Mine has this... append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot and the go file, which has this... /usr/local/sbin/vfio-bind 0000:01:00.0 0000:01:00.1 0000:07:00.0 That should get you there, but if you have issues, write back. Thank Justin. I can confirm passthrough for my USB keyboard/mouse works fine when I check it in VNC. My Go file and syslinux.cfg are the same as yours and I have entered /usr/local/sbin/vfio-bind 0000:01:00.0 0000:01:00.1 Passthrough looks to be the problem. As johnodon suggested I have tried the same xml after removing <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/> </qemu:commandline> and removing anything to do with VNC It looks to be running ok. Passthrough looks to be the culprit
November 12, 201411 yr okay, what do you have in your syslinux.cfg and in your go file? Also, what exactly do you have in your xml specific to pasthru? finally, you have to remove the <video> and <graphics> sections from the xml when you have the passthru entered. you can't have them both in the same xml, they don't work together.
November 12, 201411 yr Author okay, what do you have in your syslinux.cfg and in your go file? default /syslinux/menu.c32 menu title Lime Technology prompt 0 timeout 50 label unRAID OS menu default kernel /bzimage append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot label unRAID OS Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest label Xen/unRAID OS kernel /syslinux/mboot.c32 append /xen dom0_mem=2097152 --- /bzimage --- /bzroot label Xen/unRAID OS Safe Mode (no plugins) kernel /syslinux/mboot.c32 append /xen dom0_mem=2097152 --- /bzimage --- /bzroot unraidsafemode #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & cd /boot/packages && find . -name '*.auto_install' -type f -print | sort | xargs -n1 sh -c /usr/local/sbin/vfio-bind 0000:01:00.0 0000:01:00.1 Also, what exactly do you have in your xml specific to pasthru? <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc52b'/> </source> </hostdev> finally, you have to remove the <video> and <graphics> sections from the xml when you have the passthru entered. you can't have them both in the same xml, they don't work together. Yep, I have them both removed
November 12, 201411 yr Also, what exactly do you have in your xml specific to pasthru? <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc52b'/> </source> </hostdev> Yep, I have them both removed That's for USB passthru, not GPU passthru. That will look more like this... <seclabel type='none' model='none'/> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=07:00.0,bus=pcie.0'/> </qemu:commandline> In fact, post your full xml. Also, what exactly isnt' working? the VM wont' start, you get an error message, it seems to work but you can't see the output, what? Finally, when you think you have everything correct, and you think it should work, reboot the whole server. then make sure your TV is set to the correct input which has the GPU connected, then try starting the VM. For me, with windows 7, GPU passthru only works one time after a fresh boot of the server. after that, it won't work until I reboot again. windows 8 is better for me, but not always.
November 12, 201411 yr His XML is in the 1st post. His error looks related to his devices not being attached to an IOMMU group. Once he figures that out he'll likely be all set.
November 12, 201411 yr Author This is mu current xml: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>win8passthrough</name> <uuid>75d59834-fafe-4426-9549-5b3f0b65609e</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='1'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='3'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-2.1'>hvm</type> <boot dev='hd'/> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough'> <topology sockets='2' cores='2' threads='1'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none' io='native'/> <source file='/mnt/cache/images/win8/win8pass.qcow2'/> <target dev='vdc' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/cache/images/win8/windows8pro(1).iso'/> <target dev='hda' bus='sata'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/cache/images/win7/virtio-win.iso'/> <target dev='hdd' bus='sata'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='3'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x2'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:99:f5:fc'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </interface> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc52b'/> </source> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </memballoon> </devices> <seclabel type='none' model='none'/> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/> </qemu:commandline> </domain> The VM shows on the - "Virtual Machine Information" page but when I try to start it I get the following errors: Error while starting domain: internal error: early end of file from monitor: possible problem: 2014-11-12T21:16:28.867319Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device 2014-11-12T21:16:28.867351Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed. 2014-11-12T21:16:28.867363Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized I am going to do a server reboot and try again
November 12, 201411 yr Author A restart of the server didnt help His XML is in the 1st post. His error looks related to his devices not being attached to an IOMMU group. Once he figures that out he'll likely be all set. Trying to find info now about attaching my device to an IOMMU group. Let the fun continue
November 12, 201411 yr can you post a screenshot of the pci tab of the devices section/tab on the vmMan page? I just want to confirm the GPU info. Wait: is the CPU in your sig correct? Are you using a i7-3930K 3.2GHz 6-Core CPU? If so, I don't believe the K series processors do VT-d, which is why this isn't/won't work for you.
November 12, 201411 yr I still cannot find the easy way to do this...Grr... Anyhow type this in, readlink /sys/bus/pci/devices/0000:01:00.0/iommu_group It should come back looking something like this, if it does not (and what I am assuming) then vt-d is not properly working. ../../../../kernel/iommu_groups/17 which is showing that my device "01.00.00" is linked to IOMMU group 17, and can be verified which devices are in that group by typing this. ls /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/ Which for me shows this which is the GPU and sound card: 0000:01:00.0@ 0000:01:00.1@ Interesting read about all this stuff here (not all applicable to vfio in Unraid though) http://www.linux-kvm.org/wiki/images/b/b4/2012-forum-VFIO.pdf
November 12, 201411 yr Wait: is the CPU in your sig correct? Are you using a i7-3930K 3.2GHz 6-Core CPU? If so, I don't believe the K series processors do VT-d, which is why this isn't/won't work for you. I looked that up a while ago, he's good. http://ark.intel.com/products/63697
November 12, 201411 yr Wait: is the CPU in your sig correct? Are you using a i7-3930K 3.2GHz 6-Core CPU? If so, I don't believe the K series processors do VT-d, which is why this isn't/won't work for you. http://ark.intel.com/products/63697/Intel-Core-i7-3930K-Processor-12M-Cache-up-to-3_80-GHz?q=3930K -Advanced Technologies Intel® Turbo Boost Technology ‡ 2.0 Intel® vPro Technology ‡ No Intel® Hyper-Threading Technology ‡ Yes Intel® Virtualization Technology (VT-x) ‡ Yes Intel® Virtualization Technology for Directed I/O (VT-d) ‡ Yes
November 12, 201411 yr This gentleman had the same error as you, here http://lime-technology.com/forum/index.php?topic=34939.0 It was related to his CPU not supporting vt-d, which is not the case here, however the same output errors when starting the xml.
November 12, 201411 yr Wait: is the CPU in your sig correct? Are you using a i7-3930K 3.2GHz 6-Core CPU? If so, I don't believe the K series processors do VT-d, which is why this isn't/won't work for you. http://ark.intel.com/products/63697/Intel-Core-i7-3930K-Processor-12M-Cache-up-to-3_80-GHz?q=3930K -Advanced Technologies Intel® Turbo Boost Technology ‡ 2.0 Intel® vPro Technology ‡ No Intel® Hyper-Threading Technology ‡ Yes Intel® Virtualization Technology (VT-x) ‡ Yes Intel® Virtualization Technology for Directed I/O (VT-d) ‡ Yes it also says this... This product includes VT-d support only on the C2 stepping. Stepping information can be viewed via the menu on the left. I'm not sure where to find the spec code, but here is more information... http://ark.intel.com/products/63697/Intel-Core-i7-3930K-Processor-12M-Cache-up-to-3_80-GHz?q=3930K#@ordering
November 12, 201411 yr Interesting, completely overlooked that! CPU-Z in Windows would certainly tell you, I suppose it is possible the BIOS would give this information also, but certainly not as likely.
November 12, 201411 yr Author I still cannot find the easy way to do this...Grr... Anyhow type this in, readlink /sys/bus/pci/devices/0000:01:00.0/iommu_group It should come back looking something like this, if it does not (and what I am assuming) then vt-d is not properly working. ../../../../kernel/iommu_groups/17 which is showing that my device "01.00.00" is linked to IOMMU group 17, and can be verified which devices are in that group by typing this. ls /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/ Which for me shows this which is the GPU and sound card: 0000:01:00.0@ 0000:01:00.1@ Interesting read about all this stuff here (not all applicable to vfio in Unraid though) http://www.linux-kvm.org/wiki/images/b/b4/2012-forum-VFIO.pdf When I ran that I got nothing. Just went straight to the next line. Then root@Tower:~# ls /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/ /bin/ls: cannot access /sys/bus/pci/devices/0000\:01\:00.0/iommu_group/devices/: No such file or directory Here's the pic of my pci tab. Im starting to get worried now that its my cpu and not the ancient gpu I have. My CPU is the i7 3930k C2 stepping SR0KY. So it "should" work. Ill boot up win8 and see if I can find if its enabled from there. Good thinking
November 12, 201411 yr I wouldn't worry that it's your GPU (that may come later, but hopefully not...LOL), your current problem is certainly related to somehow,someway IOMMU not being on, if that's vt-d on the CPU, MB/BIOS, or whatever else. Once I am home (unless someone smarter than I chimes in) I will find the better command to run to verify grouping. However getting nothing back from that first command is already showing the issue you seem to be fighting.
Archived
This topic is now archived and is closed to further replies.