lostwebb Posted February 12, 2016 Share Posted February 12, 2016 Hey all. I have just completed a new server re-build, using a i7 4790k, Z97 Gaming 5, Nvidia 960 GTX and 2x Supermicro AOC-SAS2LP-MV8 and 29TB of WD and Seagate disks. Have been rocking Unraid since Febuary 2011 so I'm really excited to see it come so far in that time. The mobo has 3 PCIe x16 slots, I have the GTX in the top one and the two SAS cards in the other two. One of the main reasons for this setup was for some gaming in a Windows 10 VM however I am having issues with it. The first one being that this board groups the IOMMU so I had to enable ACS Override this seems to work at first. However I am getting varied results not very consistent it seems... The first few days of using the VM was fine, I played an hour of games and all was good. I updated the graphics drivers and then it blue screened and wouldn't boot up. The initial error on the first blue screen is: video tdr failure nvlddmkm.sys Then on the reboot system thread exception not handled The original Windows 10 machine that I setup that worked great for days and let me install and play CSGO and Cities now just bluescreen loops with the same error and will eventually go into system repair: system thread exception not handled I have installed a total of 3 different windows 10s with different settings: All of these have worked fine until I installed the Nvidia drivers now they do as follows: 1. i440fx-2.3 / SeaBIOS / Bootloop (The one that worked for a bit but no doesn't) 2. q35-2.3 / SeaBIOS / Bootloop 3. OVMF / i440fx-2.3 / Boots into windows 10 with a scrambled screen and then reboots 10 seconds later All with 100GB storage on an SSD attached directly to the MOBO, 6GB RAM and VIRBR0 One of them will boot loop a few times and eventually enter Windows and stay up, cant remember which one its either 1 or 2. This was all done with fresh copies of Windows 10 which all work fine until I install the Nvidia drivers and reboot. If at any point I change the Graphics Card to VNC it will boot up and run fine with no issues. I have found these errors in the syslog, they don't sound so good: Feb 9 11:09:19 tower kernel: Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA Feb 9 11:09:19 tower kernel: ACPI: Early table checksum verification disabled Feb 9 11:09:19 tower kernel: spurious 8259A interrupt: IRQ7. Feb 9 11:09:19 tower kernel: ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20150410/hwxface-580) Feb 9 11:09:19 tower kernel: ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20150410/hwxface-580) Feb 9 11:09:19 tower kernel: floppy0: no floppy controllers found Feb 9 11:09:19 tower kernel: ACPI Warning: SystemIO range 0x000000000000F040-0x000000000000F05F conflicts with OpRegion 0x000000000000F040-0x000000000000F04F (\_SB_.PCI0.SBUS.SMBI) (20150410/utaddress-254) Feb 9 11:09:19 tower kernel: ata2.00: READ LOG DMA EXT failed, trying unqueued Feb 9 11:09:19 tower kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1 Feb 9 11:09:19 tower kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1 Feb 9 11:09:19 tower kernel: ata1.00: READ LOG DMA EXT failed, trying unqueued Feb 9 11:09:19 tower kernel: sas: Enter sas_scsi_recover_host busy: 0 failed: 0 Feb 9 11:09:19 tower kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 Feb 9 11:09:19 tower kernel: sas: Enter sas_scsi_recover_host busy: 0 failed: 0 Feb 9 11:09:19 tower kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 Feb 9 11:09:19 tower kernel: sas: Enter sas_scsi_recover_host busy: 0 failed: 0 Feb 9 11:09:19 tower kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 Feb 9 11:09:19 tower kernel: sas: Enter sas_scsi_recover_host busy: 0 failed: 0 Feb 9 11:09:19 tower kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 Feb 9 11:09:19 tower kernel: sas: Enter sas_scsi_recover_host busy: 0 failed: 0 Feb 9 11:09:19 tower kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 Feb 9 11:09:19 tower kernel: sas: Enter sas_scsi_recover_host busy: 0 failed: 0 Feb 9 11:09:19 tower kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 Feb 9 11:09:19 tower kernel: sas: Enter sas_scsi_recover_host busy: 0 failed: 0 Feb 9 11:09:19 tower kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 Feb 9 11:09:19 tower kernel: sas: Enter sas_scsi_recover_host busy: 0 failed: 0 Feb 9 11:09:19 tower kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 Feb 9 11:09:19 tower kernel: sas: Enter sas_scsi_recover_host busy: 0 failed: 0 Feb 9 11:09:19 tower kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 Feb 9 11:09:22 tower rpc.statd[1638]: Failed to read /var/lib/nfs/state: Success Feb 9 11:09:22 tower sshd[1658]: Server listening on 0.0.0.0 port 22. Feb 9 11:09:42 tower logger: WARNING: Package has not been created with 'makepkg' According to this it should work: https://docs.google.com/spreadsheets/d/1LnGpTrXalwGVNy0PWJDURhyxa3sgqkGXmvNCIvIMenk/edit?pli=1#gid=0 PCI Devices 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) 00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06) 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) 00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller 00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1 00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller 00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0) 00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0) 00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller 00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] 00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller 01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) 01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1) 02:00.0 RAID bus controller: Marvell Technology Group Ltd. 88SE9485 SAS/SATA 6Gb/s controller (rev 03) 03:00.0 RAID bus controller: Marvell Technology Group Ltd. 88SE9485 SAS/SATA 6Gb/s controller (rev 03) 05:00.0 Ethernet controller: Qualcomm Atheros Killer E220x Gigabit Ethernet Controller (rev 13) IOMMU Groups /sys/kernel/iommu_groups/0/devices/0000:00:00.0 /sys/kernel/iommu_groups/1/devices/0000:00:01.0 /sys/kernel/iommu_groups/1/devices/0000:00:01.1 /sys/kernel/iommu_groups/1/devices/0000:00:01.2 /sys/kernel/iommu_groups/1/devices/0000:01:00.0 /sys/kernel/iommu_groups/1/devices/0000:01:00.1 /sys/kernel/iommu_groups/1/devices/0000:02:00.0 /sys/kernel/iommu_groups/1/devices/0000:03:00.0 /sys/kernel/iommu_groups/2/devices/0000:00:02.0 /sys/kernel/iommu_groups/3/devices/0000:00:03.0 /sys/kernel/iommu_groups/4/devices/0000:00:14.0 /sys/kernel/iommu_groups/5/devices/0000:00:16.0 /sys/kernel/iommu_groups/6/devices/0000:00:1a.0 /sys/kernel/iommu_groups/7/devices/0000:00:1b.0 /sys/kernel/iommu_groups/8/devices/0000:00:1c.0 /sys/kernel/iommu_groups/8/devices/0000:00:1c.3 /sys/kernel/iommu_groups/8/devices/0000:05:00.0 /sys/kernel/iommu_groups/9/devices/0000:00:1d.0 /sys/kernel/iommu_groups/10/devices/0000:00:1f.0 /sys/kernel/iommu_groups/10/devices/0000:00:1f.2 /sys/kernel/iommu_groups/10/devices/0000:00:1f.3 Can anyone help? Im not sure what else to try now. I just want the damn thing to work. I have already thrown £1,500 at it and it's running everything else fantastic (Plex + all the usual stuff you need for a 29TB array) and I just want to play some Cities Skylines Quote Link to comment
Scrapz Posted February 13, 2016 Share Posted February 13, 2016 Your GPU is sharing groups with 2 of your RAID controllers. Try moving the GPU to a different slot. Edit: Actually I just re-read and saw a bit I missed. Unless someone else has something to say, you might be out of luck, because it looks like all 3 of your PCIe slots are going to be using the same group. Basically, you need to break the GPU away from the group it's in. Quote Link to comment
lostwebb Posted February 13, 2016 Author Share Posted February 13, 2016 Thanks for the reply I have moved the GPU into slot 2 and put the non-raid cards in 1 and 3. Hasn't made any difference to the groups by the looks of things PCI Devices 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) 00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06) 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06) 00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller 00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1 00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller 00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0) 00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0) 00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller 00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] 00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller 01:00.0 RAID bus controller: Marvell Technology Group Ltd. 88SE9485 SAS/SATA 6Gb/s controller (rev 03) 02:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) 02:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1) 03:00.0 RAID bus controller: Marvell Technology Group Ltd. 88SE9485 SAS/SATA 6Gb/s controller (rev 03) 05:00.0 Ethernet controller: Qualcomm Atheros Killer E220x Gigabit Ethernet Controller (rev 13) IOMMU Groups /sys/kernel/iommu_groups/0/devices/0000:00:00.0 /sys/kernel/iommu_groups/1/devices/0000:00:01.0 /sys/kernel/iommu_groups/1/devices/0000:00:01.1 /sys/kernel/iommu_groups/1/devices/0000:00:01.2 /sys/kernel/iommu_groups/1/devices/0000:01:00.0 /sys/kernel/iommu_groups/1/devices/0000:02:00.0 /sys/kernel/iommu_groups/1/devices/0000:02:00.1 /sys/kernel/iommu_groups/1/devices/0000:03:00.0 /sys/kernel/iommu_groups/2/devices/0000:00:02.0 /sys/kernel/iommu_groups/3/devices/0000:00:03.0 /sys/kernel/iommu_groups/4/devices/0000:00:14.0 /sys/kernel/iommu_groups/5/devices/0000:00:16.0 /sys/kernel/iommu_groups/6/devices/0000:00:1a.0 /sys/kernel/iommu_groups/7/devices/0000:00:1b.0 /sys/kernel/iommu_groups/8/devices/0000:00:1c.0 /sys/kernel/iommu_groups/8/devices/0000:00:1c.3 /sys/kernel/iommu_groups/8/devices/0000:05:00.0 /sys/kernel/iommu_groups/9/devices/0000:00:1d.0 /sys/kernel/iommu_groups/10/devices/0000:00:1f.0 /sys/kernel/iommu_groups/10/devices/0000:00:1f.2 /sys/kernel/iommu_groups/10/devices/0000:00:1f.3 Quote Link to comment
Scrapz Posted February 13, 2016 Share Posted February 13, 2016 Edited post just as you replied. Unless someone else has something to say, you might be out of luck, because it looks like all 3 of your PCIe slots are going to be using the same group regardless. Basically, you need to break the GPU away from the group it's in. Somehow. Quote Link to comment
Scrapz Posted February 13, 2016 Share Posted February 13, 2016 Looking at the motherboard online, I see there's 4 1x slots. You might be able to get away with replacing your SAS cards in those? If they exist? Quote Link to comment
lostwebb Posted February 13, 2016 Author Share Posted February 13, 2016 The SAS card slots are too big for the other slots. Why is it that the ACS Override isn't working? I thought it was for situations just like this? Its frustrating as I spent ages choosing this hardware (I don't build PCs very often so my knowledge is always out of date) and I still managed to mess it up. Of all the things I researched IOMMU grouping was not one of them Quote Link to comment
Scrapz Posted February 13, 2016 Share Posted February 13, 2016 ACS override just allows you to make use of things when they are in the same group, when normally you can't. Sometimes it works, sometimes it doesn't. In your case, I'd say you're one of the unlucky ones and your GPU and HDD's are clashing with each other. Provided the 1x slots are in a different group, you could probably trade in the SAS cards for smaller ones. You can trial this with a cheap 1x USB card, and see if it ends up in another group. Maybe search the forums for other Z97 users to see the hoops they jump through. Quote Link to comment
SpaceInvaderOne Posted February 13, 2016 Share Posted February 13, 2016 Probably a long shot but try changing the position of the card in different slots without the acs override setting turned on. Maybe in slot 3 and sas cards in 1 and 2 without acs overide. I would try all combinations possible of slots and acs overide on and off Quote Link to comment
SpaceInvaderOne Posted February 13, 2016 Share Posted February 13, 2016 But you had passthrough working for a few days to start Quote Link to comment
lostwebb Posted February 13, 2016 Author Share Posted February 13, 2016 ACS override just allows you to make use of things when they are in the same group, when normally you can't. Sometimes it works, sometimes it doesn't. In your case, I'd say you're one of the unlucky ones and your GPU and HDD's are clashing with each other. Provided the 1x slots are in a different group, you could probably trade in the SAS cards for smaller ones. You can trial this with a cheap 1x USB card, and see if it ends up in another group. Maybe search the forums for other Z97 users to see the hoops they jump through. Thats a real shame, I think I may look for another motherboard that has an extra PCIe slot or two and sell this one on. What do you think is the best way of finding a mobo that plays nice with IOMMU groups? Probably a long shot but try changing the position of the card in different slots without the acs override setting turned on. Maybe in slot 3 and sas cards in 1 and 2 without acs overide. I would try all combinations possible of slots and acs overide on and off I have tried the only combinations I can with ACS disabled and they are still grouped. But you had passthrough working for a few days to start It seems to be intermittent even now, I am thinking it has something to do with traffic on the SAS cards? Quote Link to comment
SpaceInvaderOne Posted February 13, 2016 Share Posted February 13, 2016 Try enabling msi interupts for the nvidea card. i had some probs with my gtx 960 until i did use this prog https://www.dropbox.com/s/gymaipg6vprd508/MSI_util.zip?dl=0 run it as admin on windows and tick the nvidea stuff enable it for the video and audio parts of card Quote Link to comment
lostwebb Posted February 13, 2016 Author Share Posted February 13, 2016 Try enabling msi interupts for the nvidea card. i had some probs with my gtx 960 until i did use this prog https://www.dropbox.com/s/gymaipg6vprd508/MSI_util.zip?dl=0 run it as admin on windows and tick the nvidea stuff Thank you very much, I'll try it Quote Link to comment
SpaceInvaderOne Posted February 13, 2016 Share Posted February 13, 2016 more info on msi interupts here http://lime-technology.com/wiki/index.php/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support Quote Link to comment
afoard Posted February 13, 2016 Share Posted February 13, 2016 I'm having a similar issue trying to passthrough two video card to two separate VMs. I can start each VM individually and the graphics card works, but when I start the second VM it says it can't start because the resource is busy (because they are in the same IOMMU group. I have also tested with ACS Override on and off and have had no luck. I have the Gigabyte Z97X Gaming-7 board so it may be a limitation with our boards. Tried multiple combinations of this append statement with no luck: append pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot Here are my configs: VM1 <domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Miir-PC10</name> <uuid>9227f105-b1e0-0d54-bf06-6dbd3c8570de</uuid> <description>Win10 PC</description> <metadata> <vmtemplate name="Custom" icon="windows.png" os="windows"/> </metadata> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>3145728</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>3</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='2'/> <vcpupin vcpu='2' cpuset='3'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='3' threads='1'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/disks/Samsung_SSD_850_EVO_250GB_S21NNSAFC96549K-part1/test2/vdisk1.img'/> <backingStore/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/disks/VMs/Miir_10PC/Ddrive/vdisk2.img'/> <backingStore/> <target dev='hdd' bus='virtio'/> <alias name='virtio-disk3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/disks/VMs/Miir_10PC/Gdrive/vdisk3.img'/> <backingStore/> <target dev='hde' bus='virtio'/> <alias name='virtio-disk4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/andy_share/Windows10.iso'/> <backingStore/> <target dev='hda' bus='ide'/> <readonly/> <boot order='2'/> <alias name='ide0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/disks/VMs/appdata/virtual_machines/drivers/virtio-win-0.1.109.iso'/> <backingStore/> <target dev='hdb' bus='ide'/> <readonly/> <alias name='ide0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:3a:5c:e2'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/Miir-PC10.org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1b.0,bus=root.1,addr=01.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=03:00.0'/> </qemu:commandline> </domain> VM2 <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>OSX-El-Capitan-10.11-VNC</name> <uuid>0ba39646-7ba1-4d41-9602-e2968b2fe36d</uuid> <metadata> <type>None</type> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='2'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-2.3'>hvm</type> <kernel>/mnt/cache/appdata/enoch_rev2795_boot</kernel> <boot dev='hd'/> <bootmenu enable='yes'/> </os> <features> <acpi/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>core2duo</model> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/disks/Samsung_SSD_850_EVO_250GB_S21NNSAFC96549K-part1/OSX_El_Captain/ElCaptain.img'/> <target dev='hda' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:00:20:30'/> <source bridge='br0'/> <model type='e1000-82545em'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </interface> <memballoon model='none'/> </devices> <seclabel type='none' model='none'/> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=02:00.1,bus=root.1,addr=00.1'/> <qemu:arg value='-device'/> <qemu:arg value='usb-kbd'/> <qemu:arg value='-device'/> <qemu:arg value='usb-mouse'/> <qemu:arg value='-device'/> <qemu:arg value='isa-applesmc,osk='/> <qemu:arg value='-smbios'/> <qemu:arg value='type=2'/> <qemu:arg value='-device'/> <qemu:arg value='ide-drive,bus=ide.1,drive=MacDVD'/> <qemu:arg value='-drive'/> <qemu:arg value='id=MacDVD,if=none,snapshot=on,file=/mnt/disks/Samsung_SSD_850_EVO_250GB_S21NNSAFC96549K-part1/OSX_El_Captain/ElCaptain.img'/> </qemu:commandline> </domain> IOMMU Groups - Graphics cards are both in Group 1 with ACS override ON /sys/kernel/iommu_groups/0/devices/0000:00:00.0 /sys/kernel/iommu_groups/1/devices/0000:00:01.0 /sys/kernel/iommu_groups/1/devices/0000:00:01.1 /sys/kernel/iommu_groups/1/devices/0000:00:01.2 /sys/kernel/iommu_groups/1/devices/0000:01:00.0 /sys/kernel/iommu_groups/1/devices/0000:01:00.1 /sys/kernel/iommu_groups/1/devices/0000:02:00.0 /sys/kernel/iommu_groups/1/devices/0000:02:00.1 /sys/kernel/iommu_groups/1/devices/0000:03:00.0 /sys/kernel/iommu_groups/2/devices/0000:00:02.0 /sys/kernel/iommu_groups/3/devices/0000:00:03.0 /sys/kernel/iommu_groups/4/devices/0000:00:14.0 /sys/kernel/iommu_groups/5/devices/0000:00:16.0 /sys/kernel/iommu_groups/6/devices/0000:00:1a.0 /sys/kernel/iommu_groups/7/devices/0000:00:1b.0 /sys/kernel/iommu_groups/8/devices/0000:00:1c.0 /sys/kernel/iommu_groups/8/devices/0000:00:1c.2 /sys/kernel/iommu_groups/8/devices/0000:00:1c.3 /sys/kernel/iommu_groups/8/devices/0000:00:1c.4 /sys/kernel/iommu_groups/8/devices/0000:00:1c.5 /sys/kernel/iommu_groups/8/devices/0000:00:1c.7 /sys/kernel/iommu_groups/8/devices/0000:05:00.0 /sys/kernel/iommu_groups/8/devices/0000:06:00.0 /sys/kernel/iommu_groups/8/devices/0000:08:00.0 /sys/kernel/iommu_groups/8/devices/0000:09:00.0 /sys/kernel/iommu_groups/8/devices/0000:0a:00.0 /sys/kernel/iommu_groups/9/devices/0000:00:1d.0 /sys/kernel/iommu_groups/10/devices/0000:00:1f.0 /sys/kernel/iommu_groups/10/devices/0000:00:1f.2 /sys/kernel/iommu_groups/10/devices/0000:00:1f.3 Quote Link to comment
lostwebb Posted April 7, 2016 Author Share Posted April 7, 2016 I'm having a similar issue trying to passthrough two video card to two separate VMs. I can start each VM individually and the graphics card works, but when I start the second VM it says it can't start because the resource is busy (because they are in the same IOMMU group. I have also tested with ACS Override on and off and have had no luck. I have the Gigabyte Z97X Gaming-7 board so it may be a limitation with our boards. Tried multiple combinations of this append statement with no luck: append pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot Here are my configs: VM1 <domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Miir-PC10</name> <uuid>9227f105-b1e0-0d54-bf06-6dbd3c8570de</uuid> <description>Win10 PC</description> <metadata> <vmtemplate name="Custom" icon="windows.png" os="windows"/> </metadata> <memory unit='KiB'>6291456</memory> <currentMemory unit='KiB'>3145728</currentMemory> <memoryBacking> <nosharepages/> <locked/> </memoryBacking> <vcpu placement='static'>3</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='2'/> <vcpupin vcpu='2' cpuset='3'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough'> <topology sockets='1' cores='3' threads='1'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/disks/Samsung_SSD_850_EVO_250GB_S21NNSAFC96549K-part1/test2/vdisk1.img'/> <backingStore/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/disks/VMs/Miir_10PC/Ddrive/vdisk2.img'/> <backingStore/> <target dev='hdd' bus='virtio'/> <alias name='virtio-disk3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/disks/VMs/Miir_10PC/Gdrive/vdisk3.img'/> <backingStore/> <target dev='hde' bus='virtio'/> <alias name='virtio-disk4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/andy_share/Windows10.iso'/> <backingStore/> <target dev='hda' bus='ide'/> <readonly/> <boot order='2'/> <alias name='ide0-0-0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/disks/VMs/appdata/virtual_machines/drivers/virtio-win-0.1.109.iso'/> <backingStore/> <target dev='hdb' bus='ide'/> <readonly/> <alias name='ide0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:3a:5c:e2'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/Miir-PC10.org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=00:1b.0,bus=root.1,addr=01.0'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=03:00.0'/> </qemu:commandline> </domain> VM2 <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>OSX-El-Capitan-10.11-VNC</name> <uuid>0ba39646-7ba1-4d41-9602-e2968b2fe36d</uuid> <metadata> <type>None</type> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='2'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-2.3'>hvm</type> <kernel>/mnt/cache/appdata/enoch_rev2795_boot</kernel> <boot dev='hd'/> <bootmenu enable='yes'/> </os> <features> <acpi/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>core2duo</model> </cpu> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/mnt/disks/Samsung_SSD_850_EVO_250GB_S21NNSAFC96549K-part1/OSX_El_Captain/ElCaptain.img'/> <target dev='hda' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:00:20:30'/> <source bridge='br0'/> <model type='e1000-82545em'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </interface> <memballoon model='none'/> </devices> <seclabel type='none' model='none'/> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=02:00.1,bus=root.1,addr=00.1'/> <qemu:arg value='-device'/> <qemu:arg value='usb-kbd'/> <qemu:arg value='-device'/> <qemu:arg value='usb-mouse'/> <qemu:arg value='-device'/> <qemu:arg value='isa-applesmc,osk='/> <qemu:arg value='-smbios'/> <qemu:arg value='type=2'/> <qemu:arg value='-device'/> <qemu:arg value='ide-drive,bus=ide.1,drive=MacDVD'/> <qemu:arg value='-drive'/> <qemu:arg value='id=MacDVD,if=none,snapshot=on,file=/mnt/disks/Samsung_SSD_850_EVO_250GB_S21NNSAFC96549K-part1/OSX_El_Captain/ElCaptain.img'/> </qemu:commandline> </domain> IOMMU Groups - Graphics cards are both in Group 1 with ACS override ON /sys/kernel/iommu_groups/0/devices/0000:00:00.0 /sys/kernel/iommu_groups/1/devices/0000:00:01.0 /sys/kernel/iommu_groups/1/devices/0000:00:01.1 /sys/kernel/iommu_groups/1/devices/0000:00:01.2 /sys/kernel/iommu_groups/1/devices/0000:01:00.0 /sys/kernel/iommu_groups/1/devices/0000:01:00.1 /sys/kernel/iommu_groups/1/devices/0000:02:00.0 /sys/kernel/iommu_groups/1/devices/0000:02:00.1 /sys/kernel/iommu_groups/1/devices/0000:03:00.0 /sys/kernel/iommu_groups/2/devices/0000:00:02.0 /sys/kernel/iommu_groups/3/devices/0000:00:03.0 /sys/kernel/iommu_groups/4/devices/0000:00:14.0 /sys/kernel/iommu_groups/5/devices/0000:00:16.0 /sys/kernel/iommu_groups/6/devices/0000:00:1a.0 /sys/kernel/iommu_groups/7/devices/0000:00:1b.0 /sys/kernel/iommu_groups/8/devices/0000:00:1c.0 /sys/kernel/iommu_groups/8/devices/0000:00:1c.2 /sys/kernel/iommu_groups/8/devices/0000:00:1c.3 /sys/kernel/iommu_groups/8/devices/0000:00:1c.4 /sys/kernel/iommu_groups/8/devices/0000:00:1c.5 /sys/kernel/iommu_groups/8/devices/0000:00:1c.7 /sys/kernel/iommu_groups/8/devices/0000:05:00.0 /sys/kernel/iommu_groups/8/devices/0000:06:00.0 /sys/kernel/iommu_groups/8/devices/0000:08:00.0 /sys/kernel/iommu_groups/8/devices/0000:09:00.0 /sys/kernel/iommu_groups/8/devices/0000:0a:00.0 /sys/kernel/iommu_groups/9/devices/0000:00:1d.0 /sys/kernel/iommu_groups/10/devices/0000:00:1f.0 /sys/kernel/iommu_groups/10/devices/0000:00:1f.2 /sys/kernel/iommu_groups/10/devices/0000:00:1f.3 Sorry I didn't see this reply. I never did get it working Quote Link to comment
bungee91 Posted April 7, 2016 Share Posted April 7, 2016 If the ACS override is working, you should have exactly what you were expecting, that is: Each device within their own IOMMU group. This would be listed in UnRAID showing exactly that, as it then has no idea what the true grouping is, just that the patch is doing its thing. I have not heard of this patch not working on Z97, so this is something worth investigating or asking further about (VFIO group would be where I'd go to ask this). What is odd to me is that you didn't receive the error when starting the VM about "failed to get group 1, blah blah blah" and that it attempted to run with clearly bad results. Did you use the ACS override option in the GUI, or add the line your syslinux.cfg manually (they both do the same thing, but want to verify that this is actually happening). Quote Link to comment
MikeW Posted April 7, 2016 Share Posted April 7, 2016 Even with the ACS override I had problems with some PCIe devices not being split into their own groups. I fixed it by adding the specific ID of the PCIe switch in front of the devices to the override option. append pcie_acs_override=downstream,id:10b5:8609 initrd=/bzroot Not sure if this is your situation, but I can confirm that enabling ACS override by itself is not always sufficient to break devices into their own groups. The ACS override option in unRAID appears to only set the "downstream" option. You can run the 'lspci -tk' command to visually see the PCI hierarchy. Quote Link to comment
tuna Posted April 8, 2016 Share Posted April 8, 2016 i did solved my system thread bsod by this blog Quote Link to comment
lostwebb Posted April 9, 2016 Author Share Posted April 9, 2016 i did solved my system thread bsod by this blog Amazing thanks. Because it was intermittent for me I think this is the most likely answer. I have installed Windows 5 or more times and it would BSOD at different times every time. But it's possible it was because I was installing drivers in different order. I would be interested in hearing if this helps anyone else. Thank you for all the replies. I'm glad that someone picked this thread back up a few months after it was started. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.