vfio: error group 5 is not viable, please ensure all devices within the iommu_g

[froboz]

I'm setting up a second VM after having setup another without any issues. I'm recieving the following error message:

 

internal error: early end of file from monitor: possible problem:
2016-03-30T18:26:24.511691Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 5 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
2016-03-30T18:26:24.511703Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 5
2016-03-30T18:26:24.511710Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed
2016-03-30T18:26:24.511715Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

 

PCI Devices
00:00.0 Host bridge: Intel Corporation Sky Lake Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation Sky Lake PCIe Controller (x16) (rev 07)
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)
00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31)
00:17.0 RAID bus controller: Intel Corporation SATA Controller [RAID mode] (rev 31)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #3 (rev f1)
00:1c.3 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #4 (rev f1)
00:1c.4 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #5 (rev f1)
00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #9 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
00:1f.3 Audio device: Intel Corporation Sunrise Point-H HD Audio (rev 31)
00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon R9 290X]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8
02:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04)
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
05:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Amethyst XT [Radeon R9 M295X Mac Edition] (rev f1)
05:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aad8

 

00:00.0 Host bridge: Intel Corporation Sky Lake Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation Sky Lake PCIe Controller (x16) (rev 07)
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)
00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31)
00:17.0 RAID bus controller: Intel Corporation SATA Controller [RAID mode] (rev 31)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #3 (rev f1)
00:1c.3 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #4 (rev f1)
00:1c.4 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #5 (rev f1)
00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #9 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
00:1f.3 Audio device: Intel Corporation Sunrise Point-H HD Audio (rev 31)
00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon R9 290X]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8
02:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04)
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
05:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Amethyst XT [Radeon R9 M295X Mac Edition] (rev f1)
05:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aad8
IOMMU Groups

Warning: Your system has booted with the PCIe ACS Override setting enabled. The below list doesn't not reflect the way IOMMU would naturally group devices. To see natural IOMMU groups for your hardware, go to the VM Settings page and set the PCIe ACS Override setting to No.
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/2/devices/0000:00:14.0
/sys/kernel/iommu_groups/3/devices/0000:00:16.0
/sys/kernel/iommu_groups/4/devices/0000:00:17.0
/sys/kernel/iommu_groups/5/devices/0000:00:1c.0
/sys/kernel/iommu_groups/5/devices/0000:00:1c.3
/sys/kernel/iommu_groups/5/devices/0000:00:1c.4
/sys/kernel/iommu_groups/5/devices/0000:02:00.0
/sys/kernel/iommu_groups/5/devices/0000:04:00.0
/sys/kernel/iommu_groups/5/devices/0000:05:00.0
/sys/kernel/iommu_groups/5/devices/0000:05:00.1
/sys/kernel/iommu_groups/6/devices/0000:00:1d.0
/sys/kernel/iommu_groups/7/devices/0000:00:1f.0
/sys/kernel/iommu_groups/7/devices/0000:00:1f.2
/sys/kernel/iommu_groups/7/devices/0000:00:1f.3
/sys/kernel/iommu_groups/7/devices/0000:00:1f.4
/sys/kernel/iommu_groups/8/devices/0000:01:00.0
/sys/kernel/iommu_groups/8/devices/0000:01:00.1

 

Also this which I dont know what to call it but having looked at other threads it always seems to be relevant to this issue

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=05:00.1,bus=root.1,addr=00.1'/>
  </qemu:commandline>

 

I seem to have a lot of things in group 5 which is the group containing the second pcie graphics card I have installed to use with this VM (listed as Radeon R9 M295X but is actually a radeon r9 380x) After reading similar topics on this forum I tried to apply the PCIe ACS Override but this dosent seem to have made any difference. I have also tried using Q35 machine for the VM which made no difference.

 

I'm very much a beginner at this and dont know very much beyond making a basic setup (started using unraid 2 days ago) so I very much appreciate your help and please keep my inexperience in mind if you have any suggestions for me

[saarg]

It's a known problem that you can't split the devices into separate iommu groups on the skylake platform. I don't think there is anything you can do about it at the moment.

Maybe try moving the second GPU and see if that changes the group?

[froboz]

I'll try this but I'm not sure I can actually fit the other card in the 380's slot because of the physical space limitations in the case (its very big because of a custom cooler)

Just wondering though shouldn't that put the other card in group 5 and then cause the issue with that card instead? I've no idea how the iommu group assignment works

[saarg]

Since I don't know which motherboard you have, it's hard to know if you have any free slots or not 

[froboz]

Well I have the slots, its the actual case thats the limitation here as I think the card wont fit in the second slot but I'll try this weekend. I've tried other servers meanwhile but gosh darnit none seem to work as well as unraid so I really want to get this to work!

[froboz]

Moving the card didnt work either, the problem seems to be with that slot being in a group that the system dosent like because I can put the card in my other PCIE slot and it works fine. Any idea when there will be an update for skylake to fix this problem? Unraid seems to really be the only alternative, I can't get vSphere to run becuase my networkcard is not compatible and when I finally got hyper-v setup I realized too late it dosent even support gpu passthrough...

[jonp]

Just to confirm, when you applied the ACS override, did you reboot the server after?  If you just toggled the setting and hit apply, that doesn't do anything.  You must reboot for that changed to take affect.

 

Please enable the ACS override, reboot, and then if this still doesn't work, navigate to the Tools tab and click on Diagnostics, then click the Download button and upload the zip file you get to a forum post here.

[HrtAttK]

Hello Jon,

 

I'm running into the same issue as well.  I have added a PCIe USB card to be able to pass through an USB controller so I would be able to pass through some USB devices.  If there's anything here I can change, please let me know.

 

Execution error

internal error: early end of file from monitor: possible problem:
2016-04-07T00:31:53.820617Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
2016-04-07T00:31:53.820639Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
2016-04-07T00:31:53.820647Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed
2016-04-07T00:31:53.820655Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

 

What is ACS override and how do I know if it's enabled?  I don't see a toggle in the GUI advanced settings.

 

I did try rebooting as well but that didn't work.

 

Here's some more data if it helps:

 

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Windows</name>
  <uuid>e26b3528-9945-9c04-0b1e-e3ba46aec785</uuid>
  <description>Windows</description>
  <metadata>
    <vmtemplate name="Custom" icon="windows.png" os="windows"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>7</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>
    <vcpupin vcpu='6' cpuset='6'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='7' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/vDisks/Gaming PC/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/ISO/Windows.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/ISO/virtio-win-0.1.112.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:12:26:f0'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/Windows.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>
  </qemu:commandline>
</domain>

 

PCI Devices

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller
00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V
00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2
00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller
00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0)
00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0)
00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1
00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller
00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode]
00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8
02:00.0 USB controller: ASMedia Technology Inc. Device 1242
04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)

 

IOMMU Groups

 

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/1/devices/0000:02:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:14.0
/sys/kernel/iommu_groups/3/devices/0000:00:16.0
/sys/kernel/iommu_groups/4/devices/0000:00:19.0
/sys/kernel/iommu_groups/5/devices/0000:00:1a.0
/sys/kernel/iommu_groups/6/devices/0000:00:1b.0
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/7/devices/0000:00:1c.3
/sys/kernel/iommu_groups/7/devices/0000:04:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1d.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.2
/sys/kernel/iommu_groups/9/devices/0000:00:1f.3

 

USB Devices

 

Bus 002 Device 002: ID 8087:8001 Intel Corp. 
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:8009 Intel Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 002: ID 0930:6544 Toshiba Corp. TransMemory-Mini / Kingston DataTraveler 2.0 Stick (2GB)
Bus 003 Device 004: ID 046d:c24e Logitech, Inc. G500s Laser Gaming Mouse
Bus 003 Device 003: ID 03f0:064a Hewlett-Packard 
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

 

System Log after reboot:

 

Apr  6 20:52:04 unHRTATTK logger:  Updating templates...  Updating info...  Done.
Apr  6 20:52:04 unHRTATTK kernel: EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: (null)
Apr  6 20:52:04 unHRTATTK emhttp: Starting libvirt...
Apr  6 20:52:04 unHRTATTK logger: Starting libvirtd...
Apr  6 20:52:04 unHRTATTK kernel: tun: Universal TUN/TAP device driver, 1.6
Apr  6 20:52:04 unHRTATTK kernel: tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
Apr  6 20:52:04 unHRTATTK kernel: Ebtables v2.0 registered
Apr  6 20:52:04 unHRTATTK kernel: device virbr0-nic entered promiscuous mode
Apr  6 20:52:04 unHRTATTK avahi-daemon[1763]: Joining mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1.
Apr  6 20:52:04 unHRTATTK avahi-daemon[1763]: New relevant interface virbr0.IPv4 for mDNS.
Apr  6 20:52:04 unHRTATTK avahi-daemon[1763]: Registering new address record for 192.168.122.1 on virbr0.IPv4.
Apr  6 20:52:04 unHRTATTK kernel: virbr0: port 1(virbr0-nic) entered listening state
Apr  6 20:52:04 unHRTATTK kernel: virbr0: port 1(virbr0-nic) entered listening state
Apr  6 20:52:04 unHRTATTK dnsmasq[4321]: started, version 2.72 cachesize 150
Apr  6 20:52:04 unHRTATTK dnsmasq[4321]: compile time options: IPv6 GNU-getopt no-DBus i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect
Apr  6 20:52:04 unHRTATTK dnsmasq-dhcp[4321]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
Apr  6 20:52:04 unHRTATTK dnsmasq-dhcp[4321]: DHCP, sockets bound exclusively to interface virbr0
Apr  6 20:52:04 unHRTATTK dnsmasq[4321]: reading /etc/resolv.conf
Apr  6 20:52:04 unHRTATTK dnsmasq[4321]: using nameserver 8.8.8.8#53
Apr  6 20:52:04 unHRTATTK dnsmasq[4321]: using nameserver 8.8.4.4#53
Apr  6 20:52:04 unHRTATTK dnsmasq[4321]: read /etc/hosts - 1 addresses
Apr  6 20:52:04 unHRTATTK dnsmasq[4321]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Apr  6 20:52:04 unHRTATTK dnsmasq-dhcp[4321]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Apr  6 20:52:04 unHRTATTK kernel: virbr0: port 1(virbr0-nic) entered disabled state
Apr  6 20:52:06 unHRTATTK ntpd[1571]: Listen normally on 3 docker0 172.17.42.1:123
Apr  6 20:52:06 unHRTATTK ntpd[1571]: new interface(s) found: waking up resolver
Apr  6 20:52:17 unHRTATTK kernel: docker0: port 1(vethf6dbeb6) entered forwarding state
Apr  6 20:54:08 unHRTATTK php: /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/dockerupdate.php
Apr  6 20:54:15 unHRTATTK kernel: device vnet0 entered promiscuous mode
Apr  6 20:54:15 unHRTATTK kernel: br0: port 2(vnet0) entered forwarding state
Apr  6 20:54:15 unHRTATTK kernel: br0: port 2(vnet0) entered forwarding state
Apr  6 20:54:15 unHRTATTK kernel: vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem
Apr  6 20:54:16 unHRTATTK kernel: br0: port 2(vnet0) entered disabled state
Apr  6 20:54:16 unHRTATTK avahi-daemon[1763]: Withdrawing workstation service for vnet0.
Apr  6 20:54:16 unHRTATTK kernel: device vnet0 left promiscuous mode
Apr  6 20:54:16 unHRTATTK kernel: br0: port 2(vnet0) entered disabled state
Apr  6 21:10:57 unHRTATTK emhttp: shcmd (63): /usr/sbin/hdparm -y /dev/sdc &> /dev/null
Apr  6 21:10:57 unHRTATTK emhttp: shcmd (64): /usr/sbin/hdparm -y /dev/sdg &> /dev/null
Apr  6 21:11:01 unHRTATTK kernel: mdcmd (25): spindown 9
Apr  6 21:11:34 unHRTATTK kernel: kvm: already loaded the other module
Apr  6 21:12:47 unHRTATTK kernel: mdcmd (26): spindown 2
Apr  6 21:12:47 unHRTATTK kernel: mdcmd (27): spindown 3
Apr  6 21:13:21 unHRTATTK kernel: mdcmd (28): spindown 0
Apr  6 21:13:21 unHRTATTK kernel: mdcmd (29): spindown 1

[jonp]

Your issue is that your USB controller is in the same IOMMU group as your video card.  If you are assigning both of these devices to the same VM, that would be ok, but if you are assigning them to different VMs, that won't work unless you enable the PCIe ACS Override.  See the wiki for more information on this feature.  You can enable it from under Settings -> VM Manager, but after you enable it, you will need to reboot your server in order for it to take affect.

[HrtAttK]

That was it!  ACS setting did it.

 

Thanks Jon!