October 29, 20187 yr Hi, I have been trying to get my one LSI SAS2008 passed through to a VM. I have two of these HBA controllers in my system: IOMMU group 18: [1000:0072] 0b:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) IOMMU group 19: [8086:3420] 80:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 22) IOMMU group 20: [8086:3408] 80:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22) IOMMU group 21: [8086:340a] 80:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22) IOMMU group 22: [8086:340c] 80:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 22) IOMMU group 23: [8086:340e] 80:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22) IOMMU group 24: [8086:3410] 80:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 22) IOMMU group 25: [8086:342d] 80:13.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 22) IOMMU group 26: [8086:342e] 80:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 22) [8086:3422] 80:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22) [8086:3423] 80:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22) [8086:3438] 80:14.3 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers (rev 22) IOMMU group 27: [8086:3430] 80:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) [8086:3431] 80:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) [8086:3432] 80:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) [8086:3433] 80:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) [8086:3429] 80:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) [8086:342a] 80:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) [8086:342b] 80:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) [8086:342c] 80:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22) IOMMU group 28: [1000:0072] 83:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) (Just copied the part of the device listing that contains the two devices) I have tried the following: 1) assigning just 0b:00.0 to the VM by adding the following: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/> </source> </hostdev> 2) stubbing it in the Syslinux config as well as in the VM XML mod in 1) above. Below is the snip-it of code from my SysLinux Config: label unRAID OS menu default kernel /bzimage append pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1000:0072 initrd=/bzroot label unRAID OS GUI Mode 3) As can be seen from the above snip-it, I have also added workaround to allow the unsafe interrupts None of the above have helped to get the card assigned to the VM. Below is the whole VM config: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>FreeBSD</name> <uuid>c372360e-6aa2-eab7-82f3-69ddd6081f15</uuid> <metadata> <vmtemplate xmlns="unraid" name="FreeBSD" icon="freebsd.png" os="freebsd"/> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='6'/> <vcpupin vcpu='2' cpuset='1'/> <vcpupin vcpu='3' cpuset='7'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='2' threads='2'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/disk1/domains/FreeBSD/vdisk1.img'/> <target dev='hdc' bus='scsi'/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/FreeBSD-11.2-RELEASE-amd64-bootonly.iso'/> <target dev='hda' bus='sata'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:f1:3a:9d'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> </devices> </domain> When I try run the VM I get the following errors: In the GUI: Execution error internal error: qemu unexpectedly closed the monitor: 2018-10-29T01:13:37.821096Z qemu-system-x86_64: -device vfio-pci,host=0b:00.0,id=hostdev0,bus=pci.0,addr=0x9: vfio error: 0000:0b:00.0: failed to setup container for group 18: failed to set iommu for container: Operation not permitted ! Not valid! Command prompt: root@VMServer:~# virsh start FreeBSD error: Failed to start domain FreeBSD error: internal error: qemu unexpectedly closed the monitor: 2018-10-29T01:14:34.453671Z qemu-system-x86_64: -device vfio-pci,host=0b:00.0,id=hostdev0,bus=pci.0,addr=0x9: vfio error: 0000:0b:00.0: failed to setup container for group 18: failed to set iommu for container: Operation not permitted Of course, if I remove the card from the VM XML, the VM boots (without the HBA "attached") One odd thing I have noticed. Under Settings -> VM Manager I still have the yellow message "System must be rebooted for changes to take effect!" showing, despite having rebooted. Not sure if this is pointing to something.... Before someone asks "have you tried the other LSI HBA?". Yes I have..... Same outcome. And one last thing: HVM: Enabled IOMMU: Enabled Any help would be appreciated Thanks Sandi Edited October 29, 20187 yr by romihs Forgot to include the system VM flags
November 1, 20187 yr Author Any ideas what might be going on here? Has anyone else come across such an issue? The only reason I can think of is that the controllers currently have HDD's which are "reserved" by UNRAID (one card has a single HDD on it used for the array, the other has the 5 ZFS HDD's, which I am not using for UNRAID) and therefore the system does not allow the controller(s) to be passed through to the VM... Thanks
November 1, 20187 yr I had an issue with having 2 of the same USB card being passed through. If you can, try removing one of the cards and see if the other passes through.
November 1, 20187 yr Author The one I "need" to pass through is on the motherboard.... What I can try is to remove the card in the PCIe slot and see what happens... Thanks for the idea.
November 7, 20187 yr Author Okay, I have managed to make a step in the correct direction. If one is using the GUI, one needs to make the required changes to the applicable section in the Syslinux Configuration. Now onto the next hurdle. I have two LSI HBA's in my system, the one is on the motherboard, the other is a PCIe card, but unfortunately both have the same vendor/device ID (1000:0072), so when I want to stub one by adding pci-stub.ids=1000:0072 to syslinux config, I remove both from the unRaid system. My intention is to have one LSI HBA for the unRaid array, and the other for my FreeBSD array. Is there any way that I can stub a PCIe device using its bus address (83:00.0)? Also, where can I find a description of the syslinux configuration options? Thanks Edited November 7, 20187 yr by romihs
November 7, 20187 yr Author Okay, so this is still not working. I've removed the PCIe LSI card so that I now only have the LSI controller on the motherboard available to the system. I have blacklisted the LSI controller by adding vfio-pci.ids=1000:0072 to the syslinux config, and now the HDD's attached to the controller do not show up under the "Unassigned Devices" in the Main tab, telling me that the controller is blacklisted from the unRaid system. Adding the controller to the VM results in the "Operation not permitted" error when I launch the VM. Giving up on unRaid slowly.....
Archived
This topic is now archived and is closed to further replies.