What do we need to do for successful passthru now on beta8?


Recommended Posts

append intel_iommu=on

 

Enables IOMMU (VT-d usage) for Intel processors

 

vfio_iommu_type1.allow_unsafe_interrupts=1

 

Google it, but it isn't always necessary. Its only for specific cards and also safe for using benchmarking tools.

 

pcie_acs_override=downstream

 

This is really important for PCI pass through with KVM and VFIO.  It allows you to discretely pass through PCI devices to disparately and concurrently running VMs.  It also solves a bug related to IOMMU grouping that would cause a problem with devices being "grouped" together that weren't really related (e.g. having to pass through tour SATA controller to be able to pass through your GPU )

 

Hope this helps and clarifies.

Link to comment
  • Replies 55
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

So by the sound of it I should be using some if not all of these flags in my syslink file if I am passing through my GPU on my windows KVM?

 

If so, maybe that is why my nivida card wouldn't passthrough properly? Although if tht is the case it is really interesting that my ATI card is working without those flags.

Give it a shot and report back what you get with your nVIDIA card!

 

Link to comment

 

What's this for?  I could probably google search it but I'd you're passing the audio through the card itself (your 02:00.1 device)...  What does this line do for you?

 

 

Well turns out I don't actually need it...... I could not for the life of me get the HDMI Audio of my GPU to passthrough so I found that line with some google-fu. Did a re-install of Win8.1 after adding the line and the audio worked. Just restarted the VM without the line and it works fine.

Link to comment

 

What's this for?  I could probably google search it but I'd you're passing the audio through the card itself (your 02:00.1 device)...  What does this line do for you?

 

 

Well turns out I don't actually need it...... I could not for the life of me get the HDMI Audio of my GPU to passthrough so I found that line with some google-fu. Did a re-install of Win8.1 after adding the line and the audio worked. Just restarted the VM without the line and it works fine.

Glad to hear that!!

Link to comment

I'm getting a bit closer here to getting GPU passthrough working (win8.1), or at least, I think I am.  I have set up Virt-Manager on a separate Ubuntu VM to experiment.  After adding the PCI Host Device, I receive the following attached error message.

 

Error starting domain: unsupported configuration: host doesn't support passthrough of host PCI devices

 

Is there anything else on the host configuration that I need to configure? 

 

I appreciate the help

virt_error.PNG.cf33a3d3df87ed7538c600b3168aa615.PNG

Link to comment

I'm getting a bit closer here to getting GPU passthrough working (win8.1), or at least, I think I am.  I have set up Virt-Manager on a separate Ubuntu VM to experiment.  After adding the PCI Host Device, I receive the following attached error message.

 

Error starting domain: unsupported configuration: host doesn't support passthrough of host PCI devices

 

Is there anything else on the host configuration that I need to configure? 

 

I appreciate the help

 

What is your host hardware detail and please post what you have in your syslinux.cfg.

Link to comment

  append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot

 

 

So I added these flags to my syslinux.cfg but my nvidia card refused to install the drive. It tells me that "windows found driver software for your device but encountered an error while attempting to install:. This also happened when I attempted to install a sound usb card. I got the same message. With unRAID do I need to add my devices to pci-stub? It seems like I must be doing something wrong because both devices failed to install...

Link to comment

I'm getting a bit closer here to getting GPU passthrough working (win8.1), or at least, I think I am.  I have set up Virt-Manager on a separate Ubuntu VM to experiment.  After adding the PCI Host Device, I receive the following attached error message.

 

Error starting domain: unsupported configuration: host doesn't support passthrough of host PCI devices

 

Is there anything else on the host configuration that I need to configure? 

 

I appreciate the help

 

What is your host hardware detail and please post what you have in your syslinux.cfg.

 

Host Hardware: Supermicro X9SCM-F, Xeon E31240

 

syslinux.cfg

default /syslinux/menu.c32
menu title Lime Technology
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append initrd=/bzroot
label unRAID OS Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Memtest86+
  kernel /memtest
label Xen/unRAID OS
  kernel /syslinux/mboot.c32
  append /xen --- /bzimage --- /bzroot
label Xen/unRAID OS Safe Mode (no plugins)
  kernel /syslinux/mboot.c32
  append /xen --- /bzimage --- /bzroot unraidsafemode

append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot

Link to comment

Your syslinux.cfg is wrong. Use mine:

 

default /syslinux/menu.c32
menu title Lime Technology
prompt 0
timeout 50
label unRAID OS
  kernel /bzimage
  append initrd=/bzroot
label KVM/unRAID OS
  menu default
  kernel /bzimage
  append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot
label unRAID OS Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Memtest86+
  kernel /memtest
label Xen/unRAID OS
  kernel /syslinux/mboot.c32
  append /xen --- /bzimage --- /bzroot
label Xen/unRAID OS Safe Mode (no plugins)
  kernel /syslinux/mboot.c32
  append /xen --- /bzimage --- /bzroot unraidsafemode

Link to comment

  append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot

 

 

So I added these flags to my syslinux.cfg but my nvidia card refused to install the drive. It tells me that "windows found driver software for your device but encountered an error while attempting to install:. This also happened when I attempted to install a sound usb card. I got the same message. With unRAID do I need to add my devices to pci-stub? It seems like I must be doing something wrong because both devices failed to install...

 

OK so I loaded up a Windows 7 VM (I have been using Windows 8 ) and Windows 7 installed the USB sound card just fine... The nivida graphics card also installed but gave a 43 error. I wonder why windows 7 will install the drivers but not windows 8? I guess I have to wait until beta 9 to get the official custom built libvirt package  ;D Hopefully that will solve a few things.

 

Wanted to address this.  The one thing that I'm concerned about right now is the overrwiting of the libvirt package.  We custom built libvirt / qemu.conf to do certain things to simplify functionality for users (such as the ACS override patch and default cgroup settings in qemu.conf.  We also patched QEMU 2.1 to include a patch from Alex Williamson at Red Hat to hide the KVM flag from nVIDIA devices.  I am testing now to see if this plugin breaks any of that.

http://lime-technology.com/forum/index.php?topic=34875.msg325830#msg325830

Link to comment

  append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot

 

 

So I added these flags to my syslinux.cfg but my nvidia card refused to install the drive. It tells me that "windows found driver software for your device but encountered an error while attempting to install:. This also happened when I attempted to install a sound usb card. I got the same message. With unRAID do I need to add my devices to pci-stub? It seems like I must be doing something wrong because both devices failed to install...

You should not be getting error 43 with beta 8.

  append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot

 

 

So I added these flags to my syslinux.cfg but my nvidia card refused to install the drive. It tells me that "windows found driver software for your device but encountered an error while attempting to install:. This also happened when I attempted to install a sound usb card. I got the same message. With unRAID do I need to add my devices to pci-stub? It seems like I must be doing something wrong because both devices failed to install...

 

OK so I loaded up a Windows 7 VM (I have been using Windows 8 ) and Windows 7 installed the USB sound card just fine... The nivida graphics card also installed but gave a 43 error. I wonder why windows 7 will install the drivers but not windows 8? I guess I have to wait until beta 9 to get the official custom built libvirt package  ;D Hopefully that will solve a few things.

 

Wanted to address this.  The one thing that I'm concerned about right now is the overrwiting of the libvirt package.  We custom built libvirt / qemu.conf to do certain things to simplify functionality for users (such as the ACS override patch and default cgroup settings in qemu.conf.  We also patched QEMU 2.1 to include a patch from Alex Williamson at Red Hat to hide the KVM flag from nVIDIA devices.  I am testing now to see if this plugin breaks any of that.

http://lime-technology.com/forum/index.php?topic=34875.msg325830#msg325830

 

Link to comment

You should not be getting error 43 with beta 8.

 

So even though I am using peters_sm custom libvirt 1.2.8 file, I still shouldn't be getting that error with beta 8?

 

Hmm, I can't say for certain.  The patch for nVIDIA I think is part of QEMU, but I think we built-in QEMU with libvirt, so potentially...

 

No worries...  BIG patch coming for this imminently with beta 9 that will resolve all this nonsense.  Libvirt 1.2.8 with packages for networking/interfaces control / etc.  I have tested with dmacias plugin as-is and it works solid.  Hell, scared the shit out of me because his latest version seems to reset the default network on webvirtmgr to your bridge address.  Real smart!  Like it!

 

Beta 9 will also allow all this config with libvirt to persist upon reboots.  Basically think the docker.img now for libvirt, but the libvirt.img file is only going to be ~ 30MB.

Link to comment

...  BIG patch coming for this imminently with beta 9 that will resolve all this nonsense...

 

Well, hurry the hell up with it then!!  ;);D:P

 

Seriously, I'm looking forward to it.  I'll install the nVidia card into the server tonight.

 

Dealing with the reiserfs fix first.  That is priority #1.  Thankfully the other stuff is already "done."

Link to comment

You should not be getting error 43 with beta 8.

 

So even though I am using peters_sm custom libvirt 1.2.8 file, I still shouldn't be getting that error with beta 8?

 

Hmm, I can't say for certain.  The patch for nVIDIA I think is part of QEMU, but I think we built-in QEMU with libvirt, so potentially...

 

No worries...  BIG patch coming for this imminently with beta 9 that will resolve all this nonsense.  Libvirt 1.2.8 with packages for networking/interfaces control / etc.  I have tested with dmacias plugin as-is and it works solid.  Hell, scared the shit out of me because his latest version seems to reset the default network on webvirtmgr to your bridge address.  Real smart!  Like it!

 

Beta 9 will also allow all this config with libvirt to persist upon reboots.  Basically think the docker.img now for libvirt, but the libvirt.img file is only going to be ~ 30MB.

 

;D  Looking forward to it! We are getting so close to having a relatively easy way to create and config VM's.

Link to comment

Your syslinux.cfg is wrong. Use mine:

 

default /syslinux/menu.c32
menu title Lime Technology
prompt 0
timeout 50
label unRAID OS
  kernel /bzimage
  append initrd=/bzroot
label KVM/unRAID OS
  menu default
  kernel /bzimage
  append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot
label unRAID OS Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Memtest86+
  kernel /memtest
label Xen/unRAID OS
  kernel /syslinux/mboot.c32
  append /xen --- /bzimage --- /bzroot
label Xen/unRAID OS Safe Mode (no plugins)
  kernel /syslinux/mboot.c32
  append /xen --- /bzimage --- /bzroot unraidsafemode

 

Updated my syslinux.cfg with your example.  Still getting the same error when attempting start of VM

 

unsupported configuration: host doesn't support passthrough of host PCI devices

 

dmesg | grep -iE "dmar|iommu" 

Command line: intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot BOOT_IMAGE=/bzimage
Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
Kernel command line: intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot BOOT_IMAGE=/bzimage

Link to comment

Your syslinux.cfg is wrong. Use mine:

 

default /syslinux/menu.c32
menu title Lime Technology
prompt 0
timeout 50
label unRAID OS
  kernel /bzimage
  append initrd=/bzroot
label KVM/unRAID OS
  menu default
  kernel /bzimage
  append intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot
label unRAID OS Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Memtest86+
  kernel /memtest
label Xen/unRAID OS
  kernel /syslinux/mboot.c32
  append /xen --- /bzimage --- /bzroot
label Xen/unRAID OS Safe Mode (no plugins)
  kernel /syslinux/mboot.c32
  append /xen --- /bzimage --- /bzroot unraidsafemode

 

Updated my syslinux.cfg with your example.  Still getting the same error when attempting start of VM

 

unsupported configuration: host doesn't support passthrough of host PCI devices

 

dmesg | grep -iE "dmar|iommu" 

Command line: intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot BOOT_IMAGE=/bzimage
Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
Kernel command line: intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream initrd=/bzroot BOOT_IMAGE=/bzimage

Have you checked your bios?  Is VTd enabled?

Link to comment

Basically think the docker.img now for libvirt, but the libvirt.img file is only going to be ~ 30MB.

 

So then we will not have to symlink libvirt to our cache drives any longer?

Correct.

 

To clairfy I will be able to remove my S00libvirt.sh in my "\flash\config\plugins\powerdown\rc.unRAID.d" folder. Do I still want to keep the K00libvirt.sh file? It appears to just be shutting down the VM's when the array is stopped: CODE:

#!/bin/bash
# Stop libvirt and all domains
/etc/rc.d/rc.libvirt stop

Link to comment

Basically think the docker.img now for libvirt, but the libvirt.img file is only going to be ~ 30MB.

 

So then we will not have to symlink libvirt to our cache drives any longer?

Correct.

 

To clairfy I will be able to remove my S00libvirt.sh in my "\flash\config\plugins\powerdown\rc.unRAID.d" folder. Do I still want to keep the K00libvirt.sh file? It appears to just be shutting down the VM's when the array is stopped: CODE:

#!/bin/bash
# Stop libvirt and all domains
/etc/rc.d/rc.libvirt stop

 

We may actually pull that back out.  rc.libvirt stop actually kills the domains forcefully after a while.  Probably not a good idea to do that...

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.