Dmtalon Posted September 12, 2014 Share Posted September 12, 2014 I started a similar topic under the Xen forum, but gave up on getting it working, and moved over here with the hopes of a more configurable KVM might allow what I need to happen. Quick Story: HTPC controlled whole house audio via program called Casatunes. It's 1 part Software and 1 part Hardware. I have a PCIE card that is from Casatunes (Xli card). but lspci shows it as a Creative Labs card. That PC's HDD died, and I was already considering trying to virtualize it so here I am! 07:00.0 PCI bridge: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG PCI to PCIe Bridge 08:00.0 Audio device: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG I want to virtualize this PC now but I need that soundcard to work. So, I've gotten a KVM Windows 7 VM up and running. Installed Webvirtmgr as well. I've started pouring through forum posts and I figure it's time for a little help. I'm attaching my current xml file. I have been reading through @peter_sm's posts as he seems to have a lot of information on the subject, however being clueless as I am I'm probably screwing stuff up. I'm trying to follow this thread: http://lime-technology.com/forum/index.php?topic=33927.0 I created the two scripts, and ran the first (on just the 08 device above). The iommu_groups script has the above device grouped into Group 13. The bad part (I think) is this is a large group that includes my network card as well. ### Group 13 ### 00:15.0 PCI bridge: AMD/ATI [Advanced Micro Devices, Inc.] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) 00:15.1 PCI bridge: AMD/ATI [Advanced Micro Devices, Inc.] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1) 00:15.2 PCI bridge: AMD/ATI [Advanced Micro Devices, Inc.] SB900 PCI to PCI bridge (PCIE port 2) 00:15.3 PCI bridge: AMD/ATI [Advanced Micro Devices, Inc.] SB900 PCI to PCI bridge (PCIE port 3) 07:00.0 PCI bridge: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG PCI to PCIe Bridge 08:00.0 Audio device: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG 09:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 09) 0a:00.0 USB controller: ASMedia Technology Inc. Device 1142 0b:00.0 USB controller: ASMedia Technology Inc. Device 1142 Do I have to setup ALL of those devices in my Windows 7 VM to get the one I want? Does that mean I'm going to need another network card for unRaid so I can pass the OB one to Windows7 ? I tried modifying my XML to add the following lines for just the PCIe device: I got this bit of code from : http://libvirt.org/formatdomain.html#elementsHostDev <hostdev mode='subsystem' type='pci' managed='no'> <source> <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </source> When I try to startup the VM this is where I seem to be pointed to the fact that I need to pass the entire group through. I'm certainly not even confident that what I put above is the right way even. I feel pretty far over my head with this stuff and I'm hoping someone can help! WindowsVM.txt Quote Link to comment
archedraft Posted September 12, 2014 Share Posted September 12, 2014 Alright, First do you have a CPU that supports VT-d? Second, do you have VT-d enabled in your bios? Quote Link to comment
Dmtalon Posted September 12, 2014 Author Share Posted September 12, 2014 Alright, First do you have a CPU that supports VT-d? Second, do you have VT-d enabled in your bios? Oh, yes... very good questions :-) I forgot you guys haven't been hanging out at my house the past week or so why I go through this whole thing!! root@NAS1:/etc/libvirt/qemu# dmesg | grep Vi AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 AMD-Vi: Interrupt remapping enabled AMD-Vi: Initialized for Passthrough Mode kvm: Nested Virtualization enabled Quote Link to comment
archedraft Posted September 12, 2014 Share Posted September 12, 2014 Well this is about where my helps ends. Manually entering the KVM XML code is way over my head. If it helps, what I have been doing to get my Windows VM with video pass through: 1. Install a linux VM (I choose Ubuntu) 2. Install "virt-manager" 3. VNC into your linux VM to view virt-manager 4. Deleted the default connection from virt-manger 5. Add a new connection (hint: this should be the same IP as your unRAID server) 6. Connect and you should now see your linux server shown 7. Create the Windows VM and your virt-manager to add your PCI devices. Quote Link to comment
Dmtalon Posted September 12, 2014 Author Share Posted September 12, 2014 Well this is about where my helps ends. Manually entering the KVM XML code is way over my head. If it helps, what I have been doing to get my Windows VM with video pass through: 1. Install a linux VM (I choose Ubuntu) 2. Install "virt-manager" 3. VNC into your linux VM to view virt-manager 4. Deleted the default connection from virt-manger 5. Add a new connection (hint: this should be the same IP as your unRAID server) 6. Connect and you should now see your linux server shown 7. Create the Windows VM and your virt-manager to add your PCI devices. Would I actually need to recreate the windows vm? and does ubuntu need to stay around after? Quote Link to comment
archedraft Posted September 12, 2014 Share Posted September 12, 2014 No if you can see the windows VM in webvirtmgr then virt-manager will also be able to see it and you will not have to recreate the windows VM. After you get the passthrough working you could deleted the ubutnu VM. I personally would just keep it shutdown and if I ever needed it again just turn it on. Quote Link to comment
Dmtalon Posted September 12, 2014 Author Share Posted September 12, 2014 No if you can see the windows VM in webvirtmgr then virt-manager will also be able to see it and you will not have to recreate the windows VM. After you get the passthrough working you could deleted the ubutnu VM. I personally would just keep it shutdown and if I ever needed it again just turn it on. Well, its worth giving it a shot. Installing virt-manager right now on an ubuntu vm... Lets see what happens :-) Quote Link to comment
archedraft Posted September 12, 2014 Share Posted September 12, 2014 Awesome. Report back with the results Quote Link to comment
Dmtalon Posted September 13, 2014 Author Share Posted September 13, 2014 Well, that didn't go so well. I got ubuntu and virt-manager up with help from archedraft. I then tried adding the sound card. Started the VM up, and more or less got the same message as above. Stating I need to include everything in group 13 before its gonna work. Even though I had a bad feeling about it I decided what the hell.. I went ahead and moved it all to the VM. That proceeded to knock my unraid off the network and wipe out flash drive connectivity. I ended up having to reboot from the console w/o stopping the array. w/o the flash drive I couldn't stop samba, therefore couldn't stop the array. <sigh> Maybe there's some trick someone else knows about but I'm not liking my chances of getting this going. I mean I could probably put in another NIC, and move my flash drive to another usb slot, but I'm not convinced thats gonna work. I'm, of course open to any ideas Quote Link to comment
jonp Posted September 13, 2014 Share Posted September 13, 2014 What is in your syslinux.cfg file? Quote Link to comment
Dmtalon Posted September 13, 2014 Author Share Posted September 13, 2014 I never got to see what virt-manager actually put in there when I passed everything through. when unraid dropped the flash drive and network card I had to reboot with the array up. and that .xml got corrupt. root@NAS1:/etc/libvirt/qemu-delete# ls -l /bin/ls: cannot access Windows7.xml: Input/output error total 0 -??? ? ? ? ? ? Windows7.xml But my current one (which is pre-trying to use virt-manager is below) I do have a line at the bottom where I was trying to pass device 8 through. <!-- WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE OVERWRITTEN AND LOST. Changes to this xml configuration should be made using: virsh edit Windows7 or other application using the libvirt API. --> <domain type='kvm'> <name>Windows7</name> <uuid>46940f86-2486-43ed-9248-3f62f9e900a2</uuid> <memory unit='KiB'>2929688</memory> <currentMemory unit='KiB'>1953125</currentMemory> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-2.1'>hvm</type> <loader>/usr/share/qemu/bios-256k.bin</loader> <boot dev='cdrom'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough'> </cpu> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/mnt/cache/VMs/Windows7/windows7.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/ > </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/cache/VMs/Windows7/Windows7ProX64.iso'/> <target dev='sdc' bus='sata'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/cache/VMs/Windows7/virtio-win-0.1-81.iso'/> <target dev='sdd' bus='sata'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='3'/> </disk> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/ > </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/ > </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/ > </controller> <interface type='bridge'> <mac address='52:54:00:4a:af:92'/> <source bridge='br0'/> <model type='rtl8139'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/ > </interface> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0 '> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/ > </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/ > </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:08.0,bus=pcie.0'/> </qemu:commandline> </domain> The one thing that webvirtmgr did is put in some pci devices (address type=pci ... ) those are in there and there are a couple devices that seem to show up (pcie/pci) in virt-manager. I was wondering if I could find the right pcie slot if I could get the card passed through. Quote Link to comment
Dmtalon Posted September 14, 2014 Author Share Posted September 14, 2014 OK OK... Some good progress. I'm not *fixed* but I made progress. I swapped my PCIe card around and found a slot that put it and only it into a group. lspci 05:00.0 PCI bridge: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG PCI to PCIe Bridge 06:00.0 Audio device: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG Running peter's script to find groups: ### Group 17 ### 05:00.0 PCI bridge: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG PCI to PCIe Bridge 06:00.0 Audio device: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG I then used virt-manager from my ubuntu VM to add that device, and it added the following line: <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> When I rebooted Windows 7, I now see the sound card in Device manager. It showed up as a "High Definition Audio Device" which is fine, just needs drivers to identify itself correctly. Driver installed. On with my software setup. That seemed to go well, however not 100%. The application setup page works ok and it seems I can setup things as I should be. But when trying to use the application it has some issues/crashes. (it's a web based IIS driven install). The other notable thing is the sound card (correctly identified in Device Manager) is not showing up under control panel devices. On my bare metal install the device did show up there, and of course the application worked correctly. While none of you likely knows much about Casatunes, below is the error its getting when trying to run. I'm closer... but as before any advice would be appreciated. Quote Link to comment
archedraft Posted September 14, 2014 Share Posted September 14, 2014 Did you had this card to VFIO-PCI? Quote Link to comment
Dmtalon Posted September 15, 2014 Author Share Posted September 15, 2014 Did you had this card to VFIO-PCI? I don't believe I had after swapping the card around. I did rerun peter's script using 06 device, but didn't see any changes. The card, seems to be ok in device manager, but I do not see it under sound devices still. I feel like I'm closer but still can't get this working. I appreciate all the input help! Quote Link to comment
Dmtalon Posted September 17, 2014 Author Share Posted September 17, 2014 So, I think I've got this. It is actually right now playing music using Casatunes out of the sound card and through the house amp and into my son's room. I *believe* a couple things had me tripped up. One... Software (at least partially user error) the "sources" on Casatunes (what you want to play) have a check mark on them to be hidden. I've never used/checked those but all of them were checked. I assumed the lack of sources showing up was due to it not being able to access the actual card. I've installed Casatunes a half dozen times from scratch and never seen them checked. All of them were for some reason. Un-checking them allowed the sources to be seen. As for why the GUI was crashing... i thought I had an explanation for that, but I just tried to prove it, but couldn't. I thought whenever I was RDP into the windows box, the GUI was failing, but when not it wasn't. But it just worked on the server when remote desktop-ed in. I don't know why the GUI started working. I'm not sure if it's related to un-checking those buttons or what. Now, I'm afraid to reboot or touch anything!! Right now I only have one room connected because both unRaid and my amp are in a 42U rack and the are not close to one another. The cables I have don't reach. I just have one extension I had laying around that is allowing me to power one zone/room. I'm not going to mark this solved just yet. I want to make sure this will live through a reboot etc. Quote Link to comment
Dmtalon Posted September 18, 2014 Author Share Posted September 18, 2014 So, I appear to have gotten this up and running even through an unRaid reboot.. I'm attaching my syslinux.cfg, my go script and Windows7.xml KVM config file for reference. I'm super happy now! I am now down to just one box in my server rack. Now I just gotta come up with a decent backup script for my VM Thanks for the help Windows7_xml.txt go.txt Quote Link to comment
EdgarWallace Posted September 18, 2014 Share Posted September 18, 2014 Dmtalon, I'm trying to pass through a NVIDIA Quattro 2000 GPU: iommu_groups from Peter is showing that the GPU is a whole group - great: ### Group 14 ### 01:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1) 01:00.1 Audio device: NVIDIA Corporation GF106 High Definition Audio Controller (rev a1) lspci: 01:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1) go script: #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & # Backup to remote unRAID Server if ! grep ^rsync /etc/inetd.conf > /dev/null ; then cat <<-EOF >> /etc/inetd.conf rsync stream tcp nowait root /usr/sbin/tcpd /usr/bin/rsync --daemon EOF killall -HUP inetd fi cp /boot/custom/rsyncd.conf /etc # Setup PCI passthrough # NVIDIA Video 0000:01:00.0 # NVIDIA Sound 0000:01:00.1 # nGene Multimedia 0000:05:00.0 /usr/local/sbin/vfio-bind 0000:01:00.0 The xml file that I have attached was made by myself with amendments from WebVirtMgr (the memballoon address and all controller items are coming from here). My question is, where you were getting the hostdevice information from: <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </hostdev> I have just used the address from your example but this is in conflict with my servers memballoon address. Thanks for your advise. Quote Link to comment
Dmtalon Posted September 18, 2014 Author Share Posted September 18, 2014 Those parts were filled on by virt-manager from an Ubuntu VM I fired up for only that purpose. Look on page one here... Archedraft gave me that tip. Quote Link to comment
EdgarWallace Posted September 18, 2014 Share Posted September 18, 2014 Thanks for the swift reply Dmtalon. Hmm, installing Ubuntu for just having virt-manager available...I was hoping that I have just overlooked a command that is showing these informations. Will investigate further but I might have to do this as well Quote Link to comment
archedraft Posted September 18, 2014 Share Posted September 18, 2014 Worst case, just install a 10GB Ubuntu VM and once you have the graphics cards successfully passed through, leave the Ubuntu VM off unless you need it in the rare occasion. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.