November 14, 201510 yr Hi Everyone, So i have posted a few times but finally bit the bullet on new parts but i am getting a weird vm issue. Here is my list of system devices 00:00.0 Host bridge: Intel Corporation Xeon E5 v3/Core i7 DMI2 (rev 02) 00:01.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02) 00:02.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02) 00:03.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02) 00:05.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 02) 00:05.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Hot Plug (rev 02) 00:05.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 02) 00:05.4 PIC: Intel Corporation Xeon E5 v3/Core i7 I/O APIC (rev 02) 00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05) 00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05) 00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05) 00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05) 00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05) 00:1b.0 Audio device: Intel Corporation C610/X99 series chipset HD Audio Controller (rev 05) 00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5) 00:1c.1 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #2 (rev d5) 00:1c.2 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #3 (rev d5) 00:1c.3 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #4 (rev d5) 00:1c.4 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 (rev d5) 00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05) 00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05) 00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05) 02:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) 03:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) 03:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) 05:00.0 VGA compatible controller: NVIDIA Corporation GT215 [GeForce GT 240] (rev a2) 05:00.1 Audio device: NVIDIA Corporation High Definition Audio Controller (rev a1) 06:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) 07:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) 07:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) 08:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2) 08:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1) 09:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1) 09:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1) 0b:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 0c:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 0d:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) 0e:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a802 (rev 01) ff:0b.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02) ff:0b.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02) ff:0b.2 Performance counters: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02) ff:0c.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0f.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02) ff:0f.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02) ff:0f.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02) ff:0f.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02) ff:0f.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02) ff:10.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02) ff:10.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02) ff:10.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02) ff:10.6 Performance counters: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02) ff:10.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02) ff:12.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Home Agent 0 (rev 02) ff:12.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 Home Agent 0 (rev 02) ff:13.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02) ff:13.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02) ff:13.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02) ff:13.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02) ff:13.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02) ff:13.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02) ff:13.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast (rev 02) ff:13.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02) ff:14.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control (rev 02) ff:14.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control (rev 02) ff:14.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers (rev 02) ff:14.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers (rev 02) ff:14.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02) ff:14.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02) ff:15.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control (rev 02) ff:15.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control (rev 02) ff:15.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers (rev 02) ff:15.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers (rev 02) ff:16.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02) ff:16.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast (rev 02) ff:16.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02) ff:17.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control (rev 02) ff:17.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02) ff:17.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02) ff:17.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02) ff:17.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02) ff:1e.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1e.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1e.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1e.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1e.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1f.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 VCU (rev 02) ff:1f.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 VCU (rev 02) and here is my IOMMU groupings /sys/kernel/iommu_groups/0/devices/0000:ff:0b.0 /sys/kernel/iommu_groups/0/devices/0000:ff:0b.1 /sys/kernel/iommu_groups/0/devices/0000:ff:0b.2 /sys/kernel/iommu_groups/1/devices/0000:ff:0c.0 /sys/kernel/iommu_groups/1/devices/0000:ff:0c.1 /sys/kernel/iommu_groups/1/devices/0000:ff:0c.2 /sys/kernel/iommu_groups/1/devices/0000:ff:0c.3 /sys/kernel/iommu_groups/1/devices/0000:ff:0c.4 /sys/kernel/iommu_groups/1/devices/0000:ff:0c.5 /sys/kernel/iommu_groups/1/devices/0000:ff:0c.6 /sys/kernel/iommu_groups/1/devices/0000:ff:0c.7 /sys/kernel/iommu_groups/2/devices/0000:ff:0f.0 /sys/kernel/iommu_groups/2/devices/0000:ff:0f.1 /sys/kernel/iommu_groups/2/devices/0000:ff:0f.4 /sys/kernel/iommu_groups/2/devices/0000:ff:0f.5 /sys/kernel/iommu_groups/2/devices/0000:ff:0f.6 /sys/kernel/iommu_groups/3/devices/0000:ff:10.0 /sys/kernel/iommu_groups/3/devices/0000:ff:10.1 /sys/kernel/iommu_groups/3/devices/0000:ff:10.5 /sys/kernel/iommu_groups/3/devices/0000:ff:10.6 /sys/kernel/iommu_groups/3/devices/0000:ff:10.7 /sys/kernel/iommu_groups/4/devices/0000:ff:12.0 /sys/kernel/iommu_groups/4/devices/0000:ff:12.1 /sys/kernel/iommu_groups/5/devices/0000:ff:13.0 /sys/kernel/iommu_groups/5/devices/0000:ff:13.1 /sys/kernel/iommu_groups/5/devices/0000:ff:13.2 /sys/kernel/iommu_groups/5/devices/0000:ff:13.3 /sys/kernel/iommu_groups/5/devices/0000:ff:13.4 /sys/kernel/iommu_groups/5/devices/0000:ff:13.5 /sys/kernel/iommu_groups/5/devices/0000:ff:13.6 /sys/kernel/iommu_groups/5/devices/0000:ff:13.7 /sys/kernel/iommu_groups/6/devices/0000:ff:14.0 /sys/kernel/iommu_groups/6/devices/0000:ff:14.1 /sys/kernel/iommu_groups/6/devices/0000:ff:14.2 /sys/kernel/iommu_groups/6/devices/0000:ff:14.3 /sys/kernel/iommu_groups/6/devices/0000:ff:14.6 /sys/kernel/iommu_groups/6/devices/0000:ff:14.7 /sys/kernel/iommu_groups/7/devices/0000:ff:15.0 /sys/kernel/iommu_groups/7/devices/0000:ff:15.1 /sys/kernel/iommu_groups/7/devices/0000:ff:15.2 /sys/kernel/iommu_groups/7/devices/0000:ff:15.3 /sys/kernel/iommu_groups/8/devices/0000:ff:16.0 /sys/kernel/iommu_groups/8/devices/0000:ff:16.6 /sys/kernel/iommu_groups/8/devices/0000:ff:16.7 /sys/kernel/iommu_groups/9/devices/0000:ff:17.0 /sys/kernel/iommu_groups/9/devices/0000:ff:17.4 /sys/kernel/iommu_groups/9/devices/0000:ff:17.5 /sys/kernel/iommu_groups/9/devices/0000:ff:17.6 /sys/kernel/iommu_groups/9/devices/0000:ff:17.7 /sys/kernel/iommu_groups/10/devices/0000:ff:1e.0 /sys/kernel/iommu_groups/10/devices/0000:ff:1e.1 /sys/kernel/iommu_groups/10/devices/0000:ff:1e.2 /sys/kernel/iommu_groups/10/devices/0000:ff:1e.3 /sys/kernel/iommu_groups/10/devices/0000:ff:1e.4 /sys/kernel/iommu_groups/11/devices/0000:ff:1f.0 /sys/kernel/iommu_groups/11/devices/0000:ff:1f.2 /sys/kernel/iommu_groups/12/devices/0000:00:00.0 /sys/kernel/iommu_groups/13/devices/0000:00:01.0 /sys/kernel/iommu_groups/14/devices/0000:00:02.0 /sys/kernel/iommu_groups/15/devices/0000:00:03.0 /sys/kernel/iommu_groups/16/devices/0000:00:05.0 /sys/kernel/iommu_groups/16/devices/0000:00:05.1 /sys/kernel/iommu_groups/16/devices/0000:00:05.2 /sys/kernel/iommu_groups/16/devices/0000:00:05.4 /sys/kernel/iommu_groups/17/devices/0000:00:11.0 /sys/kernel/iommu_groups/17/devices/0000:00:11.4 /sys/kernel/iommu_groups/18/devices/0000:00:14.0 /sys/kernel/iommu_groups/19/devices/0000:00:16.0 /sys/kernel/iommu_groups/20/devices/0000:00:1a.0 /sys/kernel/iommu_groups/21/devices/0000:00:1b.0 /sys/kernel/iommu_groups/22/devices/0000:00:1c.0 /sys/kernel/iommu_groups/23/devices/0000:00:1c.1 /sys/kernel/iommu_groups/24/devices/0000:00:1c.2 /sys/kernel/iommu_groups/25/devices/0000:00:1c.3 /sys/kernel/iommu_groups/26/devices/0000:00:1c.4 /sys/kernel/iommu_groups/27/devices/0000:00:1d.0 /sys/kernel/iommu_groups/28/devices/0000:00:1f.0 /sys/kernel/iommu_groups/28/devices/0000:00:1f.3 /sys/kernel/iommu_groups/29/devices/0000:02:00.0 /sys/kernel/iommu_groups/30/devices/0000:03:08.0 /sys/kernel/iommu_groups/31/devices/0000:03:10.0 /sys/kernel/iommu_groups/32/devices/0000:05:00.0 /sys/kernel/iommu_groups/32/devices/0000:05:00.1 /sys/kernel/iommu_groups/33/devices/0000:06:00.0 /sys/kernel/iommu_groups/34/devices/0000:07:08.0 /sys/kernel/iommu_groups/35/devices/0000:07:10.0 /sys/kernel/iommu_groups/36/devices/0000:08:00.0 /sys/kernel/iommu_groups/36/devices/0000:08:00.1 /sys/kernel/iommu_groups/37/devices/0000:09:00.0 /sys/kernel/iommu_groups/37/devices/0000:09:00.1 /sys/kernel/iommu_groups/38/devices/0000:0b:00.0 /sys/kernel/iommu_groups/39/devices/0000:0c:00.0 /sys/kernel/iommu_groups/40/devices/0000:0d:00.0 /sys/kernel/iommu_groups/41/devices/0000:0e:00.0 The first vm i am trying to pass 09:00.0 and 09:00.1. These are the only devices in group 37 so should have no issues. My system is also reporting vt-d and iommu support. Upon trying to start the vm i get the following error internal error: early end of file from monitor: possible problem: 2015-11-14T00:52:29.584664Z qemu-system-x86_64: -device vfio-pci,host=09:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/37: Operation not permitted 2015-11-14T00:52:29.584683Z qemu-system-x86_64: -device vfio-pci,host=09:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 37 2015-11-14T00:52:29.584690Z qemu-system-x86_64: -device vfio-pci,host=09:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed 2015-11-14T00:52:29.584697Z qemu-system-x86_64: -device vfio-pci,host=09:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized My system has 3 gpu's installed, a gt240 which unraid uses connected to 1 screen, my 780 connected to anther, and a 750ti not yet connected to a screen. i have tried to enabled ACS Ovveride which seems to have no change this at all, and i have tried altering my syslinux.conf to the following which also did not help default /syslinux/menu.c32 menu title Lime Technology prompt 0 timeout 50 label unRAID OS menu default kernel /bzimage append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot label unRAID OS Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest I am currently running a free trial to test this before buying, hopefully this is not having an effect? I am unable to move the cards around in the system but even switching to giving access to the 750ti shows the same message but for the other group. My system is: Intel 5960x 32GB DDR4 Asrock X99 ws-e Help would be hugely appreciated. Regards, Jamie
November 14, 201510 yr Limetech thought it was enough with 32 groups, so you will have to add a device for all your iommu groups in /etc/libvirt/qemu.conf under cgroup_device_acl. The file should survive reboots after you edit it.
November 14, 201510 yr Author Hi Saarg, thanks for the reply. I edited the file under vi but on a reboot it reverted back to default. I also tried adding group 37 to the file, and starting the vm without a system reboot and i came across the same issue. I'm aware the second part probably wouldn't work, and i dont have the knowledge to create a new bzroot and bzimage file with the changes in place. Any further ideas would be appreciated. Regards, Jamie Edit: I just stumbled across the following post with the same issue: https://lime-technology.com/forum/index.php?topic=43428.0 Some how his config changes persisted, and whilst i know they said it would be patched in the next release, the last release was almost 2 months ago and i havent been able to find any beta release links to try
November 14, 201510 yr It's persistent here also. Are you sure you saved the file? I'm not sure if you have to restart libvirt to activate the changes, but I guess you have to. So try to modify the file again and check that it's really saved and the restart libvirt.
November 14, 201510 yr Author Hi saarg, I just tried to modify the file again. Here are my exact steps for doing so: Login to unraid as root Type "cd /etc/libvirt" Type "vi qemu.conf" Press "i" to enter insert mode Modify the file so it looks as the image below Press "Esc" to enter command mode Type ":w" and receive the saved message Type ":q" to exit the file I have tried going back into the file after saving and the changes are there. After a reboot the changes are no longer there. I have tried to start the array before editing the files just in case but that made no different. Is there something i have done wrong above or maybe something with my USB? i am using a brand new 16GB Corsair Voyager VEGA usb, formatted so it was called "UNRAID", ran the make_bootable.bat file, and unraid boots fine off it and works Perhaps there is something else i need to do? Regards, Jamie
November 14, 201510 yr Author I just booted up and ran "mount" and this is the response i get. I have not started the array at this point Sorry about all the bad images, my phone hates pictures of screens Regards, Jamie
November 14, 201510 yr You need to have the array started and VM Manager enabled before making those edits.
November 14, 201510 yr Author I have just tried the changes again a number of times, each time starting the array, ensuring vm manager is enabled. Then running the edit... On the 3rd attempt, all changes saved and the system booted and started my new windows 10 vm without a problem. I have no idea why the 3rd time saved, it must be an issue with my usb or something but its installing windows 10 now. Now to wait and see how it runs Thanks so much for your help jonp and saarg
November 14, 201510 yr I also posted an alternative fix here: http://lime-technology.com/forum/index.php?topic=44010.0
Archived
This topic is now archived and is closed to further replies.