[SOLVED] Problems building Windows VM's and Hardware Passthrough (vfio-pci)

[chris1259]

Hardware of host machine

 

Asus Rampage III Extreme BIOS v1502

Intel Core i7 950 at 3.07Ghz Quad Core

24GB DDR3 at 1066Mhz

Blueray drive

1 X Dell Mouse

1 X Dell Keyboard

2 ViewSonic Monitors

1 X EVGA Geforce GTX 960 SuperClocked Gaming (04G-P4-1962-KR) 4GB

1 X EVGA Geforce GTX 570 Classified (012-P3-1578-AR) 1.2GB

1 X Intel Corporation 82574L Gigabit (temead, PCI-e card)

1 X Intel Corporation 82567V-2 Gigabit (teamed, intergrated)

4 X Hitachi HDS5C3030ALA630 3TB - SATA 3

2 X Hitachi HDS722020ALA330 2TB - SATA 2

1 X SK Hynix SC300 256GB - SSD - SATA 3 (used for cache)

1 X SanDisk Cruzer Edge 8GB - USB

 

Software and BIOS configuration of host machine

 

unRAID v6.1.9

virtio-win drivers ISO stable v0.1.102

SATA configured as AHCI

Intel Virtualization Tech Enabled

Intel HT Technology Enabled

Intel VT-d Enabled

 

Graphics Cards in host machine

 

03:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)

04:00.0 VGA compatible controller: NVIDIA Corporation GF110 [GeForce GTX 570] (rev a1)

 

Software tested

 

unRAIDServer-6.2.0-rc2-x86_64

Non-GUI & GUI stops at - DMAR: dmar0: Using Queued invalidation

 

unRAIDServer-6.2.0-rc1-x86_64

Non-GUI & GUI stops at - DMAR: dmar0: Using Queued invalidation

 

unRAIDServer-6.1.9-x86_64 Works

 

Error if sound is enabled after booting a VM. When set to disabled it works.

 

internal error: early end of file from monitor: possible problem:

2016-07-15T01:56:34.301689Z qemu-system-x86_64: -device vfio-pci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x6: vfio: failed to set iommu for container: Operation not permitted

2016-07-15T01:56:34.301743Z qemu-system-x86_64: -device vfio-pci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x6: vfio: failed to setup container for group 8

2016-07-15T01:56:34.301751Z qemu-system-x86_64: -device vfio-pci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x6: vfio: failed to get group 8

2016-07-15T01:56:34.301762Z qemu-system-x86_64: -device vfio-pci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x6: Device initialization failed

2016-07-15T01:56:34.301772Z qemu-system-x86_64: -device vfio-pci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x6: Device 'vfio-pci' could not be initialized

 

Error if either graphics card is enabled after booting a VM. When set to VNC it works.

 

internal error: early end of file from monitor: possible problem:

2016-07-15T02:03:21.215651Z qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to set iommu for container: Operation not permitted

2016-07-15T02:03:21.215680Z qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 16

2016-07-15T02:03:21.215689Z qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 16

2016-07-15T02:03:21.215699Z qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed

2016-07-15T02:03:21.215709Z qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

 

Here is what i tried

 

- Many different VM configurations (e.g. number of CPU's, memory, have the VM run in cache, etc). Enabling 1 thing at a time to troubleshoot the issue. Including enable/disable PCIe ACS Override.

 

- The domain.img and qemu.conf fix here: http://lime-technology.com/forum/index.php?topic=44010.0g

 

- Updating \\tower\flash\syslinux.conf becasue i received Operation_not_permitted previously: http://lime-technology.com/wiki/index.php/UnRAID_6/VM_Management#Help.21_Failed_to_set_iommu_for_container:_Operation_not_permitted

 

AND IT WORKED...for a short period of time. No more errors, but the screens went black.

 

Enabling 2 VM's at the same time disconnects me from unRaid web console. But on the second monitor it appears that unRaid is still running (the cursor is still blinking).

 

I ended up installing Ubuntu Mate 16.04 by setting the Graphics Card to VNC. When it was complete, i was able to output the video to either graphics card.

 

Windows 7 SP1 Ent x64 on the other hand will boot the ISO, but halt to a black screen after 15 seconds. I haven't tried WinXP, 8.1 or 10 yet.

 

Are there any other tweaks or forum guides i can follow. Right now the only thing that works is the file server component.

 

Thank you.

[chris1259]

*Update 1*

 

Software tested

 

unRAIDServer-6.2.0-rc3-x86_64

Non-GUI & GUI stops at - DMAR: dmar0: Using Queued invalidation

 

unRAID v6.1.9

Still works. Formated and started from scratch

 

Removed: virtio-win drivers ISO stable v0.1.102

Added:  virtio-win drivers ISO latest v0.1.118

 

Hardware of host machine

 

Removed: SK Hynix SC300 256GB - SSD - SATA 3 (used for cache)

Added:  2 X SAMSUNG SSD RBX 5D15 64GB - SSD - SATA 2 (used for cache)

 

VM's

 

Ubuntu Mate 16.04 x64, Windows 7 SP1 Ent x64 and Windows XP SP3 Pro x86 all boot fine off the ISO. But choosing to STOP the VM never works. Only FORCE STOP works.

 

I can now connect to all 3 VM's at the same time without losing connection to the unRaid server.

 

Windows 7 could not find the hard drive until i found the driver on the virtio-win ISO under E:\Viostor\w7\amd64.

 

Everytime i make a change to the VM i have to set the VNC password again.

 

VM Results

 

Windows 7 SP1 Ent x64 installed fine.

 

Ubuntu Mate 16.04 x64 installed fine.

 

Windows XP SP3 Pro x86 only booted off the ISO on the first attempt. I had to delete and recreate the VM for it to boot off of the ISO again. LIke Windows 7, it didn't see the HDD until i followed the instrustions here : https://lime-technology.com/forum/index.php?topic=42256.0 Then it finished succesfully.

 

Graphic Card results

 

Same errors in the first post so i applied this fix again http://lime-technology.com/wiki/index.php/UnRAID_6/VM_Management#Help.21_Failed_to_set_iommu_for_container:_Operation_not_permitted

 

I installed Ubuntu Mate 16.04 x64 while watching through VNC. When i tried to change the video output to one of the 2 video cards the screen went black and i was dissconnected from the unRaid server.

[chris1259]

*Update 2*

 

Hardware of host machine

 

Removed: EVGA Geforce GTX 570

Removed: EVGA Geforce GTX 960

Added:  Nvidia Quadro K2000

 

03:00.0 VGA compatible controller: NVIDIA Corporation GK107GL [Quadro K2000] (rev a1)

 

The screen is still black.

 

I removed Nvidia Quadro K2000 added back EVGA Geforce GTX 960. I then downloaded the BIOS for the graphics card from here https://www.techpowerup.com/vgabios/ and followed the instructions on how to add it to the XML here http://lime-technology.com/wiki/index.php/UnRAID_6/VM_Management#Help.21_I_can_start_my_VM_with_a_GPU_assigned_but_all_I_get_is_a_black_screen_on_my_monitor.21

 

The screen is still black.

 

Added this to the end of the XML before the closing <domain> tag http://lime-technology.com/wiki/index.php/UnRAID_6/VM_Management#VM_starts_up_and_outputs_to_monitor_but_crashes_soon_after

 

The screen is still black.

 

Removed Nvidia Quadro K2000, moved EVGA Geforce GTX 960 to PCI-e slot #3 and EVGA Geforce GTX 570 to PCI-e slot #1. Video to monitor 1 now works but the keyboard does not. Ubuntu Mate 16.04 x64 does not finish loading and i lost contact with the unRaid server.

 

Removed Intel Corporation 82574L Gigabit and disabled network bonding. Tried Windows 7 VM on monitor 1 and did not add EVGA Geforce GTX 960 BIOS to the XML. The graphics card started to output video but stopped loading the Windows 7 ISO and i lost contact with the unRaid server.

 

Installed Windows 7 with video set to VNC successfully. Changed video to EVGA Geforce GTX 960 and Windows 7 froze on the Starting Windows screen. I also lost contact to the unRaid server.

 

Disabled VM's from running in cache. Windows 7 VM froze on the Starting Windows screen and I lost contact to the unRaid server. Adding the EVGA Geforce GTX 960 BIOS to the XML has the same result.

[chris1259]

*Update 3*

 

I think the cause of the VM freezing was the audio device i was selecting. I have 3 choices:

 

Intel 82801JI (Ich10 Family) HD Audio Controller  <-causes VM to freeze during Starting Windows screen

NVidia GF110 High Definition Audio Controller  <--Causes both monitors to go black

Nvidia Device  <--works

 

Here are some other settings i changed:

 

Shares

 

VM share - Use cache disk: Yes (if set to ONLY there is no video output to the monitor)

 

Settings/VM Manger

 

PCIe ACS Override: Enabled

 

VM's

 

Windows 7

 

Placed a checkmark next to all cores.

Machine: Q35-2.3

BIOS: SeaBIOS

Hyper-V: yes

 

Device Manager

 

Windows 7 reports there is a problem with EVGA Geforce GTX 960 and has stopped this device. I was able to set the resolution to 1920 x 1080 and everything looks fine at the moment. The OS is unpatched so i will will run Windows Updates to bring the OS up to date.

 

Testing

 

Installing Steam now and will run a test with the new Doom from 2016 tonight.

 

The VM is not feasible. No matter how many cpu cores i provide, memory or running in cache, the VM is very slow and could not finish searching for Windows Updates or install 1 game. After a few hours neither task made any progress. Without further testing it's unclear if the problem lays in KVM, GPU passthrough or Nvidia cards in general. Reading many forum posts it appears that using GPU passthrough with an AMD device would provide more desirable results.

[chris1259]

*Update 4*

 

New Host Machine

 

ASRock X99 Extreme4

Intel i7-5820K Haswell-E 3.30Ghz 6 Core BIOS vP2.10

CORSAIR Vengeance LPX 32GB DDR4 @ 2133Mhz

1 X Dell Mouse

1 X Dell Keyboard

1 X ViewSonic Monitor

1 X Intel Corporation I218-V Gigabit (teamed, intergrated)

1 X Intel Corporation 82574L Gigabit (temead, PCI-e card)

1 X Nvidia Geforce GT430 1GB (used for unRaid)

1 X EVGA Geforce GTX 960 SuperClocked Gaming (04G-P4-1962-KR) 4GB (used for the virtual machine)

4 X Hitachi HDS5C3030ALA630 3TB - SATA 3

2 X Hitachi HDS722020ALA330 2TB - SATA 2

1 X Western Digital WD20EARS-60M AB51 2TB - SATA 2

1 X Samsung 850 500GB SSD - SATA 3 (used for cache)

1 X SanDisk Cruzer Edge 8GB - USB

 

01:00.0 VGA compatible controller: NVIDIA Corporation GF108 [GeForce GT 430] (rev a1)

04:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)

 

Configuration

 

Shares

 

VM share - Use cache disk: Yes

 

Settings/VM Manger

 

PCIe ACS Override: Enabled

 

VM

 

Windows 10 Ent x64 v1607

 

CPU Mode: Host Passthrough

Cores: 4 (8 threads)

Memory: 16GB

Machine: Q35-2.3

BIOS: SeaBIOS

Hyper-V: No (i changed this to yes, but the VM changed it to No after Windows installed successfully)

 

Windows 10 installed without issue and all devices installed successfully including the Nvidia drivers.

 

Crystal DiskMark Shows:

 

read 1807 MB/s and write 1234 MB/s 1GB test

read 1919 MB/s and write 1286 MB/s 2GB test

read 1813 MB/s but no write speed would display 4GB test

 

Installed Steam and Doom (2016) to test. On medium settings with Graphics API set to opengl 4.5 the game was a solid 60FPS. When i changed the settings to Ultra and the Graphis API to Vulkan the game was 40-60FPS depending on amount of enemies on the screen. Over all the game looked great and responded well.

 

Fraps v3.5.99 testing

Diablo 3: Reaper Of Souls (2014) - Highest settings - 80-120FPS depending on the amount of enemies on the screen.

Hard Reset Redux Edition (2016) - Highest settings - 160-260FPS

Need For Speed: Most Wanted (2012) - Highest settings - 20-26FPS - Turned down Super Sampling (anti-aliasing) to 2x - 30-45FPS - Turned off Super Sampling - 60FPS

[bungee91]

PCIe ACS Override: Enabled

 

Given your hardware, there should be NO reason to need this enabled, if anything it could cause issues.

The ACS override is for systems without ACS on root ports, which your 5820k has natively within the CPU.

[chris1259]

I changed this to disabled since my VM's feel a little bit sluggish. I notice it the most when navigating the Start Menu in Windows 10. I don't notice any slow down when a game is running.

 

It also appears to me that the longer the VM runs the better it runs.  So far so good with the setting changed to disabled.

 

I also had an issue passing through 3 USB devices on the same USB bus as the unRaid USB boot stick. But the suggestion here http://lime-technology.com/forum/index.php?topic=41922.msg493278#msg493278 fixed my issue.

[chris1259]

I was having an issue were the VM (virtual machine) was a bit sluggish due to 100% disk usage. So I upgraded unRaid to version 6.2.0-rc4 from v6.1.9. This fixed my issue right away. And i no longer needed the XML fix to pass through 3 USB devices that i did here http://lime-technology.com/forum/index.php?topic=41922.msg493278#msg493278

 

I used the Windows 10 template to test the default settings for a Windows 10 VM vs my settings. Here were my results.

 

Crystal Disk Mark v5.1.2.2016

 

NEW VM

 

1821 read and 1099 write on the 1GB test

1758 read and 1069 write on the 2GB test

1631 read and 498 write on the 4GB test

 

OLD VM

 

10362 read and 1350 write on the 1GB test

10250 read and 1336 write on the 2GB test

9879 read and 575 write on the 4GB test

 

PassMark v8.0.1054.0 (Benchmark software)

 

OLD VM

Score: 3869

 

NEW VM

Score: 3449

 

Differences in the VM's

 

OLD VM

 

BIOS: OVMF

Primary vDisk Bus: SATA

 

NEW VM

 

BIOS: SeaBIOS

Primary vDisk Bus: Virtio

[chris1259]

I created new VM's on v6.20. RC5 and ran more tests. The tests below are with the cache drive enabled. I have now disabled this option since my VM is 1TB and my cache drive is 500GB. My final VM, #3, not listed below, became corrupted and would no longer boot.

 

VM #1

 

BIOS: Seabios

vDisk Bus: Virtio

Machine: Q35-2.5

 

PassMark v8.0.1054.0 - 3 passes (Benchmark software)

 

Average Score: 4222

 

Crystal Disk Mark v5.1.2.2016 - 3 passes each

 

Average Score 1GB: 7765 read and 3463 write

Average Score 2GB: 8049 read and 859 write

 

VM #2

 

BIOS: Seabios

vDisk Bus: SATA

Machine: i440fx-2.5

 

PassMark v8.0.1054.0 - 3 passes (Benchmark software)

 

Average Score: 3522

 

Crystal Disk Mark v5.1.2.2016 - 3 passes each

 

Average Score 1GB: 2194 read and 1849 write

Average Score 2GB: 1866 read and 1019 write

 

Deploying Windows Images

 

MDT 2013 Update 2 (aka Microsoft Deployment Toolkit) works very well deploying a WIM (image file). I injected the Windows 10 Virtio drivers into the boot disc (LiteTouchPE_x64.iso) and imported them into the deployment share. Along with the Nvidia drivers i was able to automatically inject the Virtio drivers from the deployment share. This means that when the build was complete i didn't need to install any additional drivers.

 

OVMF Issues

 

When deploying from MDT 2013, with a 500GB drive, the machine would not boot into Windows. It booted to a shell command line. I read on here that my video card might not support UEFI support. I don't know if my card support this.

 

I could not deploy an image to a drive that was 1TB. I had to create a 500GB primary partition instead. It fails to inject drivers because it can't detect the drive size correctly. Running diskpart at a command prompt in WinPE states the the Disk is uninitialized, initializing it to GPT. But it never does. I could not manually create or format a primary partition either.