archedraft

Thanks, I know I will need to retrace my steps in the future so I better write it down somewhere. What better place than the forum and hopefully others find it useful.

Intent: This guide will take you through the process of passing through an entire PCI USB Controller instead of doing it individually. One benefit is that USB will be plug n play on your virtual machines. Also, I have found that if you pass through USB devices separately the device name could change which will cause the VM to not start, but if you pass through the entire controller you will avoid this. Warning!!!: Please be VERY careful when doing this. You do NOT want to pass through your unRAID USB by mistake. unRAID needs the USB present to function properly. Prerequisites: Working VM CPU and Bios that supports VT-d USB device (a spare USB Flash drive will work) Motherboard Manual or paper for notes Guide: 1. SSH into unRAID 2. Find out how many USB controllers your server has available lspci | grep USB Notice that my server has 3 USB controllers 3. Now would be a great time to pull out the Motherboard Manual to take notes on (see mine below). What we are going to do is plug in an USB flash drive in to each spare USB slot to figure out the controller. 4. First lets figure out which USB bus unRAID is on. lsusb Note that my unRAID USB is on bus 002 (figure out which USB that relates to on your motherboard manual and write that down). **THIS MEANS I SHOULD NOT PASS THROUGH BUS 002!!!** 5. Now take a spare USB flash drive that is preferably a different manufacture than your unRAID USB (this will make it easier to identify). Plug it into a spare USB slot and type lsusb into your ssh session and write down which "Bus 00#" the spare USB drive appears on. Repeat this step this for all your USB slots. 6. Now you should have identified every USB slot's bus number, so lets figure out which PCI Number the bus belongs to (for me I have 3 USB controllers (Bus 001, 002, & 003). Replace the USB# with your bus number example: bus 001 = usb1 in the below code readlink /sys/bus/usb/devices/usb1 readlink /sys/bus/usb/devices/usb2 readlink /sys/bus/usb/devices/usb3 7. We have just figured out the USB controller device number (In my server all Bus 001 USB slots are part of 0000:00:1a.0) 8. Now lets make sure that this USB controller doesn't have any other device in its IOMMU Groups Go to your unRAID gui -> Tools -> System Devices -> IOMMU Groups 9. Match the group with the device number. As you can see below, mine is part of group 5 and that is the only device on group 5. If you have more than one device in the same group you will most likly have to pass them through to your VM as well, YMMV /sys/kernel/iommu_groups/5/devices/0000:00:1a.0 As of unRAID 6.1 this step is no longer needed. unRAID does this for you automatically. 10. Now lets add that device number to the vfio-pci driver. Open your "go" file under config on your unRAID flash drive and add this line: /usr/local/sbin/vfio-bind 0000:00:1a.0 and then type the same code into your SSH session to make it active or reboot your server. 11. You can either do step 11.a or 11.b [11.b is the easier option] 11.a Now lets add that to your Windows 8 VM XML to passthrough: <domain type='kvm' id='2' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Windows-8-Nvidia</name> <uuid>cc411d70-4463-4db7-bf36-d364c0cdaa9d</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-2.1'>hvm</type> <boot dev='hd'/> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> <pae/> </features> <cpu mode='host-passthrough'> <topology sockets='2' cores='2' threads='1'/> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/vmdisk/vm_images/default/windows8.img'/> <backingStore/> <target dev='hda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/> </disk> <controller type='scsi' index='0' model='virtio-scsi'> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb0'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb0'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb0'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x2'/> </controller> <controller type='sata' index='0'> <alias name='sata0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:00:00:04'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </interface> <input type='tablet' bus='usb'> <alias name='input0'/> </input> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1a.0,bus=root.1,addr=00.1'/> </qemu:commandline> </domain> This part of the code is needed no matter what and you shouldn't have to change anything (see Hint #1 for more details): <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> This part is my graphics card (yours will be different): <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> This part is the code for the USB controller passthrough: <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1a.0,bus=root.1,addr=00.1'/> </qemu:commandline> HINT: #1 You MUST add the following right after <gemu:commandline> AND then you put in your code for the USB controller <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> Then add your USB code so when its all said and done it should look like this if you are only passing through an USB controller without a graphics card (if you are passing through a graphics card as well see my config in step 11 for that example): <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/ <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1a.0,bus=root.1,addr=00.0'/> #2 If you are only passing through a USB controller and nothing else (GPU, etc.) then you need to modify your "addr=" part to "addr=00.0" Every PCI device you passthrough gets its own addr=00.# and it starts at 0. In my above code my GPU is 00.0 and my USB controller is 00.1 11.b Using Hostdev instead of the qemu:arg author=saarg link=topic=36768.msg430843#msg430843 date=1451775451 For me it's much easier to use the hostdev tag instead of the qemu:arg. It's just too much trouble with it when you don't know what you are doing. So for passing through a PCI(e) device with the hostdev tag this is the starting code: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x00' slot='0x00' function='0x0'/> </source> </hostdev> You then have to modify some parts of it to get it to work. The good thing is that you do not have to care about which bus and address it's supposed to have in the VM. You only need to find out the host PCI address. The part you change is bus, slot and function. In your case it's 00:14.0. Let's brake it down. 00 is the bus. You simply exchange the two numbers after the 0x. 14 is the slot. Same method as above. 0 is the function. Her it's also the same method as above. So in your case the full device tag would be like this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x00' slot='0x14' function='0x0'/> </source> </hostdev> After you start the VM you will see that there are some lines added to the tag, but those you don't have to care about. They get created automatically. If you copy a host device tag to pass through a new device, be sure to remove the two lines created after the </source> tag, as they are specific to that VM. <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> 12. All done. What if you only have 1 bus... To my knowledge there are two paths; I will post what JonP posted HERE below (I revised what Jon said a bit to make sense in this thread): The Easy Path The Time Consuming Path

People dislike change. I am willing to bet that a few people are going to complain that the new GUI is dynamix and they are not used to the way it looks. Moral of the story is people be nuts! That said I vote black is default too.

I agree that black looks much better but changing that to default would start a riot.

Also thank you so much for posting your mtab config! These settings fixed an issue I was having with a certain app and 9P shares. The app refused to play nice with the 9P share but now it's working!! I am so excited, VM's are awesome!

Thought I'd chime in with a better way to work around this issue. In your fstab, add _netdev to your options. This will force the VirtFS mounting (even with auto) to wait until the network resources are available, which just so happens to solve the issue where sometimes the automounting would fail. We do this with our OpenELEC unVM and it works perfect. Would you mind posting the contents of your fstab? This is actually in /etc/mtab because we don't need fstab for our OpenELEC VM: storage /storage 9p rw,dirsync,relatime,trans=virtio,version=9p2000.L,posixacl,cache=loose 0 0 Thanks! do you have anything in the "/etc/rc.local" file to mount this share or does it just automount?

Thought I'd chime in with a better way to work around this issue. In your fstab, add _netdev to your options. This will force the VirtFS mounting (even with auto) to wait until the network resources are available, which just so happens to solve the issue where sometimes the automounting would fail. We do this with our OpenELEC unVM and it works perfect. Would you mind posting the contents of your fstab?

Whoa "Stephen's" Smart Report is horrible!

Very true but I don't think Gary knows what an emergency situation is... I hear he has a backup server on the moon! You never know when the earth will blow up after all [emoji1]

Just throwing out wild $#@ guesses but I wonder if it could be a beta 9 issue. You could try install unRAID 5 to rule that out.

I would start looking here: http://lime-technology.com/forum/index.php?topic=31803

There is a plugin called SNAP that should let you mount the hard rives to unRAID. I have never used it before so take that for what it is worth.

I ran the command who -r and it told me run-level 2 That means that all applications should get install on run-level 2 correct?

With editing fstab (like I did above) a few of my apps (sickbeard) loaded before the shares were mounted. To fix this I created a crontab task that delayed the startup of sickbeard for 10 seconds after boot. Does the 9P sharing mount the shares before the apps are loaded in rc.local? The advantage is not using your virtual network especially for intensive operations like, par, rar, usnet and torrents. I had a problem with sickrage and nzbget. I renamed the init symlinks in /etc/rc2.d of my apps from 20 to 99 so they start last. Then changed rc.local from 99 to 98. Check this page it explains the different rc.x runlevels. You'll have to check which level your vm runs or just change them all, rc0.d, rc1.d...rc6.d http://www.linux.com/news/enterprise/systems-management/8116-an-introduction-to-services-runlevels-and-rcd-scripts By doing this you could also make it so one app starts before the other. dmacis, did you go through each rc0-6.d and find all your applications and set them to 99? or when you installed them did you set them up to 99? Currently I am still using a crontab and delaying my applications at boot by 10 seconds but from what I read online this is a dirty way of doing that.

I used it as a precaution because of what dmacias had indicated in his original post. I figured I would start on the side of caution and see how the P9 passthrough works and if I have no issues I may try it without using rc.local and noauto. Although "if it ain't broke don't fix it" comes to mind. So far it is working great without any issues.

So here is a quick write up for anyone who needs a little more guidance (like myself ): 1. First edit your .xml file (source dir = unraid mount) (traget dir = label) <filesystem type='mount' accessmode='passthrough'> <source dir='/mnt/user/MEDIA/MOVIES - HD'/> <target dir='movies'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/mnt/user/MEDIA/TV SERIES'/> <target dir='tv'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/mnt/user/downloads'/> <target dir='downloads'/> </filesystem> 2. Then startup the VM 3. edit fstab sudo nano /etc/fstab movies /home/archedraft/media/movies 9p trans=virtio,version=9p2000.L,_netdev,rw 0 0 tv /home/archedraft/media/tv 9p trans=virtio,version=9p2000.L,_netdev,rw 0 0 downloads /home/archedraft/media/downloads 9p trans=virtio,version=9p2000.L,_netdev,rw 0 0 4. Create the mount directories in your VM Sudo mkdir /home/archedraft/media/movies 5. edit nano [EDIT: You don't need to do this step unless you use this in fstab "trans=virtio,version=9p2000.L,noauto,nobootwait,rw"] sudo nano /etc/rc.local mount /home/archedraft/media/movies mount /home/archedraft/media/tv mount /home/archedraft/media/downloads 6. reboot

Yeah That was the issue. I was mapping a directory that didn't exist.

I got the following error when I tried. qemu-system-x86_64: -device virtio-9p-pci,id=fs0,fsdev=fsdev-fs0,mount_tag=media,bus=pci.2,addr=0x1: Device initialization failed. qemu-system-x86_64: -device virtio-9p-pci,id=fs0,fsdev=fsdev-fs0,mount_tag=media,bus=pci.2,addr=0x1: Device 'virtio-9p-pci' could not be initialized That is what I added to my .xml file <filesystem type='mount' accessmode='passthrough'> <source dir='/mnt/media'/> <target dir='media'/> </filesystem> Do I need to add the following to syslinux.cfg before passthrough will work? label KVM unRAID OS menu default kernel /bzimage append intel_iommu=on quiet vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot acpi_sleep=s3_bios

OK, understood. I'll give this a try this afternoon. Thanks for the example! I am guessing that your "cache" and "user" mounts are the main shares you have setup in unRAID? So for my setup my "cache" share would be "media".

My current /etc/fstab is as follows: //pithos.private/media/MOVIES\040-\040HD /home/archedraft/media/movies cifs noauto,credentials=/home/archedraft/.smbcredentials,noperm,noserverino,uid=nobody,gid=users 0 0 //pithos.private/media/TV\040SERIES /home/archedraft/media/tv cifs noauto,credentials=/home/archedraft/.smbcredentials,noperm,noserverino,uid=nobody,gid=users 0 0 //pithos.private/downloads /home/archedraft/media/downloads cifs noauto,credentials=/home/archedraft/.smbcredentials,noperm,noserverino,uid=nobody,gid=users 0 0 1. I am assuming if I wanted to add this to my .xml it should read like this below? <filesystem type='mount' accessmode='passthrough'> <source dir='/media/MOVIES\040-\040HD'/> <target dir='/home/archedraft/media/movies'/> </filesystem> 2. /etc/fstab should read is as follows: shares /media/MOVIES\040-\040HD 9p trans=virtio,version=9p2000.L,noauto,nobootwait,rw 0 0 3. Do I need to make 3 separate filesystem codes in my .xml for each share or can you add multiple shares to one filesystem code? <filesystem type='mount' accessmode='passthrough'> <source dir='/mnt'/> <target dir='shares'/> </filesystem> 4. I am assuming I do not need to install a 9P program in my ubuntu server? 5. Due to the shares being passthrough, I should not have to have a .smbcredentials file correct?

With editing fstab (like I did above) a few of my apps (sickbeard) loaded before the shares were mounted. To fix this I created a crontab task that delayed the startup of sickbeard for 10 seconds after boot. Does the 9P sharing mount the shares before the apps are loaded in rc.local?

So instead of adding the following to the VM and editing fstab as shown below. You are sharing files directly from the .xml? Is there an advantage to doing it one way vs the other? Edit fstab so we can connect the mounts on boot sudo nano /etc/fstab Edit and paste the following //Tower/Movies /media/Movies cifs noauto,credentials=/home/ubuntuUser/.smbcredentials,noperm,noserverino,uid=nobody,gid=users 0 0 //Tower/TV /media/TV cifs noauto,credentials=/home/ubuntuUser/.smbcredentials,noperm,noserverino,uid=nobody,gid=users 0 0

I would also love that!! For it to be universally useful a flexget container would also need created. I have used rTorrent for as long as I can remember and rutorrent for a 3+f years and TBH i find deluge better now. I got fed up with how stagnant rTorrent is e.g webseeds are available in every client but has an open feature request for 8+ years in rT Do yourself a favour checkout deluge. It used to be crap but now it good now. I think we should change this thread to a poll and use it as a poor mans voting system Your probably right to just use Deluge. The only thing I didn't like about it was viewing it on an iPhone screen. Rutorrent worked much better... But then if I ask myself how often do I actually use my phone to view things and the answer is not that often. I think I read that there was already a Deluge docker. Is that correct?

Would it be possible to have a rtorrent/rutorrent docker?

My GF said geeking out on TCL and docker isn't exactly what she would call a panty dropper. Perhaps I should go wrestle a bear instead. In case you do not hear from me again, you know why. I hear you! My wife looks at me like I have 3 heads when I start talking about the "server"...