Intent:
This guide will take you through the process of passing through an entire PCI USB Controller instead of doing it individually. One benefit is that USB will be plug n play on your virtual machines. Also, I have found that if you pass through USB devices separately the device name could change which will cause the VM to not start, but if you pass through the entire controller you will avoid this.
Warning!!!:
Please be VERY careful when doing this. You do NOT want to pass through your unRAID USB by mistake. unRAID needs the USB present to function properly.
Prerequisites:
Working VM
CPU and Bios that supports VT-d
USB device (a spare USB Flash drive will work)
Motherboard Manual or paper for notes
Guide:
1. SSH into unRAID
2. Find out how many USB controllers your server has available
lspci | grep USB
Notice that my server has 3 USB controllers
3. Now would be a great time to pull out the Motherboard Manual to take notes on (see mine below). What we are going to do is plug in an USB flash drive in to each spare USB slot to figure out the controller.
4. First lets figure out which USB bus unRAID is on.
lsusb
Note that my unRAID USB is on bus 002 (figure out which USB that relates to on your motherboard manual and write that down).
**THIS MEANS I SHOULD NOT PASS THROUGH BUS 002!!!**
5. Now take a spare USB flash drive that is preferably a different manufacture than your unRAID USB (this will make it easier to identify). Plug it into a spare USB slot and type
lsusb
into your ssh session and write down which "Bus 00#" the spare USB drive appears on. Repeat this step this for all your USB slots.
6. Now you should have identified every USB slot's bus number, so lets figure out which PCI Number the bus belongs to (for me I have 3 USB controllers (Bus 001, 002, & 003). Replace the USB# with your bus number example: bus 001 = usb1 in the below code
readlink /sys/bus/usb/devices/usb1
readlink /sys/bus/usb/devices/usb2
readlink /sys/bus/usb/devices/usb3
7. We have just figured out the USB controller device number (In my server all Bus 001 USB slots are part of 0000:00:1a.0)
8. Now lets make sure that this USB controller doesn't have any other device in its IOMMU Groups
Go to your unRAID gui -> Tools -> System Devices -> IOMMU Groups
9. Match the group with the device number. As you can see below, mine is part of group 5 and that is the only device on group 5. If you have more than one device in the same group you will most likly have to pass them through to your VM as well, YMMV
/sys/kernel/iommu_groups/5/devices/0000:00:1a.0
As of unRAID 6.1 this step is no longer needed. unRAID does this for you automatically.
10. Now lets add that device number to the vfio-pci driver. Open your "go" file under config on your unRAID flash drive and add this line:
/usr/local/sbin/vfio-bind 0000:00:1a.0
and then type the same code into your SSH session to make it active or reboot your server.
11. You can either do step 11.a or 11.b [11.b is the easier option]
11.a Now lets add that to your Windows 8 VM XML to passthrough:
<domain type='kvm' id='2' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>Windows-8-Nvidia</name>
<uuid>cc411d70-4463-4db7-bf36-d364c0cdaa9d</uuid>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<memoryBacking>
<nosharepages/>
<locked/>
</memoryBacking>
<vcpu placement='static'>2</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-q35-2.1'>hvm</type>
<boot dev='hd'/>
<bootmenu enable='no'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<cpu mode='host-passthrough'>
<topology sockets='2' cores='2' threads='1'/>
</cpu>
<clock offset='localtime'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/mnt/vmdisk/vm_images/default/windows8.img'/>
<backingStore/>
<target dev='hda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
</disk>
<controller type='scsi' index='0' model='virtio-scsi'>
<alias name='scsi0'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
</controller>
<controller type='usb' index='0' model='ich9-ehci1'>
<alias name='usb0'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<alias name='usb0'/>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<alias name='usb0'/>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<alias name='usb0'/>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x2'/>
</controller>
<controller type='sata' index='0'>
<alias name='sata0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:00:00:04'/>
<source bridge='br0'/>
<target dev='vnet0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
</interface>
<input type='tablet' bus='usb'>
<alias name='input0'/>
</input>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
</memballoon>
</devices>
<qemu:commandline>
<qemu:arg value='-device'/>
<qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
<qemu:arg value='-device'/>
<qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
<qemu:arg value='-device'/>
<qemu:arg value='vfio-pci,host=00:1a.0,bus=root.1,addr=00.1'/>
</qemu:commandline>
</domain>
This part of the code is needed no matter what and you shouldn't have to change anything (see Hint #1 for more details):
<qemu:commandline>
<qemu:arg value='-device'/>
<qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
This part is my graphics card (yours will be different):
<qemu:arg value='-device'/>
<qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
This part is the code for the USB controller passthrough:
<qemu:arg value='-device'/>
<qemu:arg value='vfio-pci,host=00:1a.0,bus=root.1,addr=00.1'/>
</qemu:commandline>
HINT:
#1 You MUST add the following right after <gemu:commandline> AND then you put in your code for the USB controller
<qemu:arg value='-device'/>
<qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
Then add your USB code so when its all said and done it should look like this if you are only passing through an USB controller without a graphics card (if you are passing through a graphics card as well see my config in step 11 for that example):
<qemu:arg value='-device'/>
<qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/
<qemu:arg value='-device'/>
<qemu:arg value='vfio-pci,host=00:1a.0,bus=root.1,addr=00.0'/>
#2 If you are only passing through a USB controller and nothing else (GPU, etc.) then you need to modify your "addr=" part to "addr=00.0"
Every PCI device you passthrough gets its own addr=00.# and it starts at 0. In my above code my GPU is 00.0 and my USB controller is 00.1
11.b Using Hostdev instead of the qemu:arg
author=saarg link=topic=36768.msg430843#msg430843 date=1451775451
For me it's much easier to use the hostdev tag instead of the qemu:arg. It's just too much trouble with it when you don't know what you are doing.
So for passing through a PCI(e) device with the hostdev tag this is the starting code:
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x00' slot='0x00' function='0x0'/>
</source>
</hostdev>
You then have to modify some parts of it to get it to work. The good thing is that you do not have to care about which bus and address it's supposed to have in the VM. You only need to find out the host PCI address.
The part you change is bus, slot and function. In your case it's 00:14.0.
Let's brake it down.
00 is the bus. You simply exchange the two numbers after the 0x.
14 is the slot. Same method as above.
0 is the function. Her it's also the same method as above.
So in your case the full device tag would be like this:
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x00' slot='0x14' function='0x0'/>
</source>
</hostdev>
After you start the VM you will see that there are some lines added to the tag, but those you don't have to care about. They get created automatically. If you copy a host device tag to pass through a new device, be sure to remove the two lines created after the
</source>
tag, as they are specific to that VM.
<alias name='hostdev2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
12. All done.
What if you only have 1 bus...
To my knowledge there are two paths; I will post what JonP posted HERE below (I revised what Jon said a bit to make sense in this thread):
The Easy Path
The Time Consuming Path