jonp Posted September 8, 2014 Share Posted September 8, 2014 append intel_iommu=on Enables IOMMU (VT-d usage) for Intel processors vfio_iommu_type1.allow_unsafe_interrupts=1 Google it, but it isn't always necessary. Its only for specific cards and also safe for using benchmarking tools. pcie_acs_override=downstream This is really important for PCI pass through with KVM and VFIO. It allows you to discretely pass through PCI devices to disparately and concurrently running VMs. It also solves a bug related to IOMMU grouping that would cause a problem with devices being "grouped" together that weren't really related (e.g. having to pass through tour SATA controller to be able to pass through your GPU ) Hope this helps and clarifies. Quote Link to comment
archedraft Posted September 8, 2014 Share Posted September 8, 2014 So by the sound of it I should be using some if not all of these flags in my syslink file if I am passing through my GPU on my windows KVM? If so, maybe that is why my nivida card wouldn't passthrough properly? Although if tht is the case it is really interesting that my ATI card is working without those flags. Quote Link to comment
jonp Posted September 8, 2014 Share Posted September 8, 2014 So by the sound of it I should be using some if not all of these flags in my syslink file if I am passing through my GPU on my windows KVM? If so, maybe that is why my nivida card wouldn't passthrough properly? Although if tht is the case it is really interesting that my ATI card is working without those flags. Give it a shot and report back what you get with your nVIDIA card! Quote Link to comment
wewantrice Posted September 8, 2014 Share Posted September 8, 2014 What's this for? I could probably google search it but I'd you're passing the audio through the card itself (your 02:00.1 device)... What does this line do for you? Well turns out I don't actually need it...... I could not for the life of me get the HDMI Audio of my GPU to passthrough so I found that line with some google-fu. Did a re-install of Win8.1 after adding the line and the audio worked. Just restarted the VM without the line and it works fine. Quote Link to comment
jonp Posted September 8, 2014 Share Posted September 8, 2014 What's this for? I could probably google search it but I'd you're passing the audio through the card itself (your 02:00.1 device)... What does this line do for you? Well turns out I don't actually need it...... I could not for the life of me get the HDMI Audio of my GPU to passthrough so I found that line with some google-fu. Did a re-install of Win8.1 after adding the line and the audio worked. Just restarted the VM without the line and it works fine. Glad to hear that!! Quote Link to comment
indy5 Posted September 8, 2014 Share Posted September 8, 2014 I'm getting a bit closer here to getting GPU passthrough working (win8.1), or at least, I think I am. I have set up Virt-Manager on a separate Ubuntu VM to experiment. After adding the PCI Host Device, I receive the following attached error message. Error starting domain: unsupported configuration: host doesn't support passthrough of host PCI devices Is there anything else on the host configuration that I need to configure? I appreciate the help Quote Link to comment
jonp Posted September 8, 2014 Share Posted September 8, 2014 I'm getting a bit closer here to getting GPU passthrough working (win8.1), or at least, I think I am. I have set up Virt-Manager on a separate Ubuntu VM to experiment. After adding the PCI Host Device, I receive the following attached error message. Error starting domain: unsupported configuration: host doesn't support passthrough of host PCI devices Is there anything else on the host configuration that I need to configure? I appreciate the help What is your host hardware detail and please post what you have in your syslinux.cfg. Quote Link to comment
archedraft Posted September 8, 2014 Share Posted September 8, 2014 append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot So I added these flags to my syslinux.cfg but my nvidia card refused to install the drive. It tells me that "windows found driver software for your device but encountered an error while attempting to install:. This also happened when I attempted to install a sound usb card. I got the same message. With unRAID do I need to add my devices to pci-stub? It seems like I must be doing something wrong because both devices failed to install... Quote Link to comment
indy5 Posted September 8, 2014 Share Posted September 8, 2014 I'm getting a bit closer here to getting GPU passthrough working (win8.1), or at least, I think I am. I have set up Virt-Manager on a separate Ubuntu VM to experiment. After adding the PCI Host Device, I receive the following attached error message. Error starting domain: unsupported configuration: host doesn't support passthrough of host PCI devices Is there anything else on the host configuration that I need to configure? I appreciate the help What is your host hardware detail and please post what you have in your syslinux.cfg. Host Hardware: Supermicro X9SCM-F, Xeon E31240 syslinux.cfg default /syslinux/menu.c32 menu title Lime Technology prompt 0 timeout 50 label unRAID OS menu default kernel /bzimage append initrd=/bzroot label unRAID OS Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest label Xen/unRAID OS kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot label Xen/unRAID OS Safe Mode (no plugins) kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot unraidsafemode append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot Quote Link to comment
archedraft Posted September 8, 2014 Share Posted September 8, 2014 Your syslinux.cfg is wrong. Use mine: default /syslinux/menu.c32 menu title Lime Technology prompt 0 timeout 50 label unRAID OS kernel /bzimage append initrd=/bzroot label KVM/unRAID OS menu default kernel /bzimage append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot label unRAID OS Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest label Xen/unRAID OS kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot label Xen/unRAID OS Safe Mode (no plugins) kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot unraidsafemode Quote Link to comment
archedraft Posted September 8, 2014 Share Posted September 8, 2014 append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot So I added these flags to my syslinux.cfg but my nvidia card refused to install the drive. It tells me that "windows found driver software for your device but encountered an error while attempting to install:. This also happened when I attempted to install a sound usb card. I got the same message. With unRAID do I need to add my devices to pci-stub? It seems like I must be doing something wrong because both devices failed to install... OK so I loaded up a Windows 7 VM (I have been using Windows 8 ) and Windows 7 installed the USB sound card just fine... The nivida graphics card also installed but gave a 43 error. I wonder why windows 7 will install the drivers but not windows 8? I guess I have to wait until beta 9 to get the official custom built libvirt package Hopefully that will solve a few things. Wanted to address this. The one thing that I'm concerned about right now is the overrwiting of the libvirt package. We custom built libvirt / qemu.conf to do certain things to simplify functionality for users (such as the ACS override patch and default cgroup settings in qemu.conf. We also patched QEMU 2.1 to include a patch from Alex Williamson at Red Hat to hide the KVM flag from nVIDIA devices. I am testing now to see if this plugin breaks any of that. http://lime-technology.com/forum/index.php?topic=34875.msg325830#msg325830 Quote Link to comment
jonp Posted September 9, 2014 Share Posted September 9, 2014 append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot So I added these flags to my syslinux.cfg but my nvidia card refused to install the drive. It tells me that "windows found driver software for your device but encountered an error while attempting to install:. This also happened when I attempted to install a sound usb card. I got the same message. With unRAID do I need to add my devices to pci-stub? It seems like I must be doing something wrong because both devices failed to install... You should not be getting error 43 with beta 8. append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot So I added these flags to my syslinux.cfg but my nvidia card refused to install the drive. It tells me that "windows found driver software for your device but encountered an error while attempting to install:. This also happened when I attempted to install a sound usb card. I got the same message. With unRAID do I need to add my devices to pci-stub? It seems like I must be doing something wrong because both devices failed to install... OK so I loaded up a Windows 7 VM (I have been using Windows 8 ) and Windows 7 installed the USB sound card just fine... The nivida graphics card also installed but gave a 43 error. I wonder why windows 7 will install the drivers but not windows 8? I guess I have to wait until beta 9 to get the official custom built libvirt package Hopefully that will solve a few things. Wanted to address this. The one thing that I'm concerned about right now is the overrwiting of the libvirt package. We custom built libvirt / qemu.conf to do certain things to simplify functionality for users (such as the ACS override patch and default cgroup settings in qemu.conf. We also patched QEMU 2.1 to include a patch from Alex Williamson at Red Hat to hide the KVM flag from nVIDIA devices. I am testing now to see if this plugin breaks any of that. http://lime-technology.com/forum/index.php?topic=34875.msg325830#msg325830 Quote Link to comment
archedraft Posted September 9, 2014 Share Posted September 9, 2014 You should not be getting error 43 with beta 8. So even though I am using peters_sm custom libvirt 1.2.8 file, I still shouldn't be getting that error with beta 8? Quote Link to comment
jonp Posted September 9, 2014 Share Posted September 9, 2014 You should not be getting error 43 with beta 8. So even though I am using peters_sm custom libvirt 1.2.8 file, I still shouldn't be getting that error with beta 8? Hmm, I can't say for certain. The patch for nVIDIA I think is part of QEMU, but I think we built-in QEMU with libvirt, so potentially... No worries... BIG patch coming for this imminently with beta 9 that will resolve all this nonsense. Libvirt 1.2.8 with packages for networking/interfaces control / etc. I have tested with dmacias plugin as-is and it works solid. Hell, scared the shit out of me because his latest version seems to reset the default network on webvirtmgr to your bridge address. Real smart! Like it! Beta 9 will also allow all this config with libvirt to persist upon reboots. Basically think the docker.img now for libvirt, but the libvirt.img file is only going to be ~ 30MB. Quote Link to comment
JustinChase Posted September 9, 2014 Author Share Posted September 9, 2014 ... BIG patch coming for this imminently with beta 9 that will resolve all this nonsense... Well, hurry the hell up with it then!! Seriously, I'm looking forward to it. I'll install the nVidia card into the server tonight. Quote Link to comment
jonp Posted September 9, 2014 Share Posted September 9, 2014 ... BIG patch coming for this imminently with beta 9 that will resolve all this nonsense... Well, hurry the hell up with it then!! Seriously, I'm looking forward to it. I'll install the nVidia card into the server tonight. Dealing with the reiserfs fix first. That is priority #1. Thankfully the other stuff is already "done." Quote Link to comment
JustinChase Posted September 9, 2014 Author Share Posted September 9, 2014 perfect. I honestly didn't know about it when I wrote that. I just set mover to monthly, so you have 20 days before I write to the array again. Seriously though, fixing potentially serious issues obviously comes first, playing with passthru is a far off second. thanks again for keeping us updated. Quote Link to comment
archedraft Posted September 9, 2014 Share Posted September 9, 2014 You should not be getting error 43 with beta 8. So even though I am using peters_sm custom libvirt 1.2.8 file, I still shouldn't be getting that error with beta 8? Hmm, I can't say for certain. The patch for nVIDIA I think is part of QEMU, but I think we built-in QEMU with libvirt, so potentially... No worries... BIG patch coming for this imminently with beta 9 that will resolve all this nonsense. Libvirt 1.2.8 with packages for networking/interfaces control / etc. I have tested with dmacias plugin as-is and it works solid. Hell, scared the shit out of me because his latest version seems to reset the default network on webvirtmgr to your bridge address. Real smart! Like it! Beta 9 will also allow all this config with libvirt to persist upon reboots. Basically think the docker.img now for libvirt, but the libvirt.img file is only going to be ~ 30MB. Looking forward to it! We are getting so close to having a relatively easy way to create and config VM's. Quote Link to comment
archedraft Posted September 9, 2014 Share Posted September 9, 2014 Basically think the docker.img now for libvirt, but the libvirt.img file is only going to be ~ 30MB. So then we will not have to symlink libvirt to our cache drives any longer? Quote Link to comment
jonp Posted September 9, 2014 Share Posted September 9, 2014 Basically think the docker.img now for libvirt, but the libvirt.img file is only going to be ~ 30MB. So then we will not have to symlink libvirt to our cache drives any longer? Correct. Quote Link to comment
indy5 Posted September 10, 2014 Share Posted September 10, 2014 Your syslinux.cfg is wrong. Use mine: default /syslinux/menu.c32 menu title Lime Technology prompt 0 timeout 50 label unRAID OS kernel /bzimage append initrd=/bzroot label KVM/unRAID OS menu default kernel /bzimage append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot label unRAID OS Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest label Xen/unRAID OS kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot label Xen/unRAID OS Safe Mode (no plugins) kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot unraidsafemode Updated my syslinux.cfg with your example. Still getting the same error when attempting start of VM unsupported configuration: host doesn't support passthrough of host PCI devices dmesg | grep -iE "dmar|iommu" Command line: intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot BOOT_IMAGE=/bzimage Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA Kernel command line: intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot BOOT_IMAGE=/bzimage Quote Link to comment
jonp Posted September 10, 2014 Share Posted September 10, 2014 Your syslinux.cfg is wrong. Use mine: default /syslinux/menu.c32 menu title Lime Technology prompt 0 timeout 50 label unRAID OS kernel /bzimage append initrd=/bzroot label KVM/unRAID OS menu default kernel /bzimage append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot label unRAID OS Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest label Xen/unRAID OS kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot label Xen/unRAID OS Safe Mode (no plugins) kernel /syslinux/mboot.c32 append /xen --- /bzimage --- /bzroot unraidsafemode Updated my syslinux.cfg with your example. Still getting the same error when attempting start of VM unsupported configuration: host doesn't support passthrough of host PCI devices dmesg | grep -iE "dmar|iommu" Command line: intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot BOOT_IMAGE=/bzimage Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA Kernel command line: intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot BOOT_IMAGE=/bzimage Have you checked your bios? Is VTd enabled? Quote Link to comment
archedraft Posted September 10, 2014 Share Posted September 10, 2014 Basically think the docker.img now for libvirt, but the libvirt.img file is only going to be ~ 30MB. So then we will not have to symlink libvirt to our cache drives any longer? Correct. To clairfy I will be able to remove my S00libvirt.sh in my "\flash\config\plugins\powerdown\rc.unRAID.d" folder. Do I still want to keep the K00libvirt.sh file? It appears to just be shutting down the VM's when the array is stopped: CODE: #!/bin/bash # Stop libvirt and all domains /etc/rc.d/rc.libvirt stop Quote Link to comment
jonp Posted September 11, 2014 Share Posted September 11, 2014 Basically think the docker.img now for libvirt, but the libvirt.img file is only going to be ~ 30MB. So then we will not have to symlink libvirt to our cache drives any longer? Correct. To clairfy I will be able to remove my S00libvirt.sh in my "\flash\config\plugins\powerdown\rc.unRAID.d" folder. Do I still want to keep the K00libvirt.sh file? It appears to just be shutting down the VM's when the array is stopped: CODE: #!/bin/bash # Stop libvirt and all domains /etc/rc.d/rc.libvirt stop We may actually pull that back out. rc.libvirt stop actually kills the domains forcefully after a while. Probably not a good idea to do that... Quote Link to comment
jonp Posted September 11, 2014 Share Posted September 11, 2014 Also beta 9 will NOT have the libvirt changes. That will be beta 9a. We need to get this reiserfs bug fixed first before we patch the livirt stuff in. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.