KVM + Windows 7 + PCIE (PCI) passthrough of Creative Sound Card help


Dmtalon

Recommended Posts

I started a similar topic under the Xen forum, but gave up on getting it working, and moved over here with the hopes of a more configurable KVM might allow what I need to happen.

 

Quick Story:

HTPC controlled whole house audio via program called Casatunes.  It's 1 part Software and 1 part Hardware.  I have a PCIE card that is from Casatunes (Xli card). but lspci shows it as a Creative Labs card.  That PC's HDD died, and I was already considering trying to virtualize it so here I am!

 

    07:00.0 PCI bridge: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG PCI to PCIe Bridge
    08:00.0 Audio device: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG

 

I want to virtualize this PC now but I need that soundcard to work.

 

 

So, I've gotten a KVM Windows 7 VM up and running.  Installed Webvirtmgr as well.  I've started pouring through forum posts and I figure it's time for a little help.  I'm attaching my current xml file.

 

I have been reading through @peter_sm's posts as he seems to have a lot of information on the subject, however being clueless as I am I'm probably screwing stuff up.

 

I'm trying to follow this thread:  http://lime-technology.com/forum/index.php?topic=33927.0

 

I created the two scripts, and ran the first (on just the 08 device above).  The iommu_groups script has the above device grouped into Group 13.  The bad part (I think) is this is a large group that includes my network card as well.

 

### Group 13 ###
    00:15.0 PCI bridge: AMD/ATI [Advanced Micro Devices, Inc.] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0)
    00:15.1 PCI bridge: AMD/ATI [Advanced Micro Devices, Inc.] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1)
    00:15.2 PCI bridge: AMD/ATI [Advanced Micro Devices, Inc.] SB900 PCI to PCI bridge (PCIE port 2)
    00:15.3 PCI bridge: AMD/ATI [Advanced Micro Devices, Inc.] SB900 PCI to PCI bridge (PCIE port 3)
    07:00.0 PCI bridge: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG PCI to PCIe Bridge
    08:00.0 Audio device: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG
    09:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 09)
    0a:00.0 USB controller: ASMedia Technology Inc. Device 1142
    0b:00.0 USB controller: ASMedia Technology Inc. Device 1142

 

Do I have to setup ALL of those devices in my Windows 7 VM to get the one I want?  Does that mean I'm going to need another network card for unRaid so I can pass the OB one to Windows7 ?

 

 

I tried modifying my XML to add the following lines for just the PCIe device:  I got this bit of code from : http://libvirt.org/formatdomain.html#elementsHostDev

 

 

    <hostdev mode='subsystem' type='pci' managed='no'>

      <source>

      <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>

      </source>

 

 

When I try to startup the VM this is where I seem to be pointed to the fact that I need to pass the entire group through.  I'm certainly not even confident that what I put above is the right way even.  I feel pretty far over my head with this stuff and I'm hoping someone can help!

 

mV4DHW5.png

WindowsVM.txt

Link to comment

Alright, First do you have a CPU that supports VT-d?

Second, do you have VT-d enabled in your bios?

 

Oh, yes... very good questions :-)  I forgot you guys haven't been hanging out at my house the past week or so why I go through this whole thing!!

 

root@NAS1:/etc/libvirt/qemu# dmesg | grep Vi
AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
AMD-Vi: Interrupt remapping enabled
AMD-Vi: Initialized for Passthrough Mode
kvm: Nested Virtualization enabled

 

Link to comment

Well this is about where my helps ends. Manually entering the KVM XML code is way over my head.

 

If it helps, what I have been doing to get my Windows VM with video pass through:

1. Install a linux VM (I choose Ubuntu)

2. Install "virt-manager"

3. VNC into your linux VM to view virt-manager

4. Deleted the default connection from virt-manger

5. Add a new connection (hint: this should be the same IP as your unRAID server)

6. Connect and you should now see your linux server shown

7. Create the Windows VM and your virt-manager to add your PCI devices.

Link to comment

Well this is about where my helps ends. Manually entering the KVM XML code is way over my head.

 

If it helps, what I have been doing to get my Windows VM with video pass through:

1. Install a linux VM (I choose Ubuntu)

2. Install "virt-manager"

3. VNC into your linux VM to view virt-manager

4. Deleted the default connection from virt-manger

5. Add a new connection (hint: this should be the same IP as your unRAID server)

6. Connect and you should now see your linux server shown

7. Create the Windows VM and your virt-manager to add your PCI devices.

 

Would I actually need to recreate the windows vm?  and does ubuntu need to stay around after?

Link to comment

No if you can see the windows VM in webvirtmgr then virt-manager will also be able to see it and you will not have to recreate the windows VM. After you get the passthrough working you could deleted the ubutnu VM. I personally would just keep it shutdown and if I ever needed it again just turn it on.

 

Well, its worth giving it a shot.  Installing virt-manager right now on an ubuntu vm... Lets see what happens :-)

Link to comment

Well, that didn't go so well.  I got ubuntu and virt-manager up with help from archedraft.  I then tried adding the sound card.  Started the VM up, and more or less got the same message as above.  Stating I need to include everything in group 13 before its gonna work.

 

Even though I had a bad feeling about it I decided what the hell.. I went ahead and moved it all to the VM.  That proceeded to knock my unraid off the network and wipe out flash drive connectivity.  I ended up having to reboot from the console w/o stopping the array.  w/o the flash drive I couldn't stop samba, therefore couldn't stop the array. <sigh>

 

Maybe there's some trick someone else knows about but I'm not liking my chances of getting this going.  I mean I could probably put in another NIC, and move my flash drive to another usb slot, but I'm not convinced thats gonna work.

 

I'm, of course open to any ideas  :D

 

 

Link to comment

I never got to see what virt-manager actually put in there when I passed everything through.  when unraid dropped the flash drive and network card I had to reboot with the array up.  and that .xml got corrupt. 

 

root@NAS1:/etc/libvirt/qemu-delete# ls -l

/bin/ls: cannot access Windows7.xml: Input/output error

total 0

-??? ? ? ? ?            ? Windows7.xml

 

But my current one (which is pre-trying to use virt-manager is below)  I do have a line at the bottom where I was trying to pass device 8 through.

 

 

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit Windows7
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>Windows7</name>
  <uuid>46940f86-2486-43ed-9248-3f62f9e900a2</uuid>
  <memory unit='KiB'>2929688</memory>
  <currentMemory unit='KiB'>1953125</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-2.1'>hvm</type>
    <loader>/usr/share/qemu/bios-256k.bin</loader>
    <boot dev='cdrom'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'>
  </cpu>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/cache/VMs/Windows7/windows7.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/                  >
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/cache/VMs/Windows7/Windows7ProX64.iso'/>
      <target dev='sdc' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/cache/VMs/Windows7/virtio-win-0.1-81.iso'/>
      <target dev='sdd' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='3'/>
    </disk>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/                  >
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/                  >
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/                  >
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:4a:af:92'/>
      <source bridge='br0'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/                  >
    </interface>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0                  '>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/                  >
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/                  >
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:08.0,bus=pcie.0'/>
  </qemu:commandline>
</domain>

 

The one thing that webvirtmgr did is put in some pci devices (address type=pci ... )  those are in there and there are a couple devices that seem to show up (pcie/pci) in virt-manager.  I was wondering if I could find the right pcie slot if I could get the card passed through.

 

Link to comment

OK OK... Some good progress.  I'm not *fixed*  but I made progress.

 

I swapped my PCIe card around and found a slot that put it and only it into a group.

 

lspci
05:00.0 PCI bridge: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG PCI to PCIe Bridge
06:00.0 Audio device: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG

 

Running peter's script to find groups:

### Group 17 ###
    05:00.0 PCI bridge: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG PCI to PCIe Bridge
    06:00.0 Audio device: Creative Labs [sB X-Fi Xtreme Audio] CA0110-IBG

 

I then used virt-manager from my ubuntu VM to add that device, and it added the following line:

 

   <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>

 

When I rebooted Windows 7, I now see the sound card in Device manager.  It showed up as a "High Definition Audio Device" which is fine, just needs drivers to identify itself correctly.

 

Driver installed.

TMnOfaz.png

 

On with my software setup.  That seemed to go well, however not 100%.  The application setup page works ok and it seems I can setup things as I should be. But when trying to use the application it has some issues/crashes.  (it's a web based IIS driven install).

 

The other notable thing is the sound card (correctly identified in Device Manager) is not showing up under control panel devices.

 

PWFxjXi.png

 

On my bare metal install the device did show up there, and of course the application worked correctly.  While none of you likely knows much about Casatunes, below is the error its getting when trying to run.

 

I'm closer... but as before any advice would be appreciated.

 

 

JA6j15p.png

Link to comment

Did you had this card to VFIO-PCI?

 

I don't believe I had after swapping the card around.  I did rerun peter's script using 06 device, but didn't see any changes.  The card, seems to be ok in device manager, but I do not see it under sound devices still.

 

I feel like I'm closer but still can't get this working.  I appreciate all the input help!

 

 

Link to comment

So,

 

I think I've got this.  It is actually right now playing music using Casatunes out of the sound card and through the house amp and into my son's room.

 

I *believe* a couple things had me tripped up.  One... Software (at least partially user error)  the "sources" on Casatunes (what you want to play) have a check mark on them to be hidden.  I've never used/checked those but all of them were checked.  I assumed the lack of sources showing up was due to it not being able to access the actual card.  I've installed Casatunes a half dozen times from scratch and never seen them checked.  All of them were for some reason.  Un-checking them allowed the sources to be seen.

 

As for why the GUI was crashing... i thought I  had an explanation for that, but I just tried to prove it, but couldn't.  I thought whenever I was RDP into the windows box, the GUI was failing, but when not it wasn't.  But it just worked on the server when remote desktop-ed in.  I don't know why the GUI started working.  I'm not sure if it's related to un-checking those buttons or what. 

 

Now, I'm afraid to reboot or touch anything!! 

 

Right now I only have one room connected because both unRaid and my amp are in a 42U rack and the are not close to one another.  The cables I have don't reach.  I just have one extension I had laying around that is allowing me to power one zone/room.

 

I'm not going to mark this solved just yet.  I want to make sure this will live through a reboot etc.

 

 

 

 

Link to comment

Dmtalon, I'm trying to pass through a NVIDIA Quattro 2000 GPU:

 

iommu_groups from Peter is showing that the GPU is a whole group - great:

### Group 14 ###
    01:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)
    01:00.1 Audio device: NVIDIA Corporation GF106 High Definition Audio Controller (rev a1)

 

lspci:

01:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)

 

go script:

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &

# Backup to remote unRAID Server
if ! grep ^rsync /etc/inetd.conf > /dev/null ; then
cat <<-EOF >> /etc/inetd.conf
rsync   stream  tcp     nowait  root    /usr/sbin/tcpd  /usr/bin/rsync --daemon
EOF
killall -HUP inetd
fi

cp /boot/custom/rsyncd.conf /etc

# Setup PCI passthrough
# NVIDIA Video     0000:01:00.0
# NVIDIA Sound     0000:01:00.1
# nGene Multimedia 0000:05:00.0
/usr/local/sbin/vfio-bind 0000:01:00.0

 

The xml file that I have attached was made by myself with amendments from WebVirtMgr (the memballoon address and all controller items are coming from here). My question is, where you were getting the hostdevice information from:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </hostdev>

 

I have just used the address from your example but this is in conflict with my servers memballoon address.

Thanks for your advise.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.