Hey Everyone,

     I love unRAID. I finally got the server up and running, everything looks to be working ok.......ish......

 

So one of my VMs is a Gaming VM that I use for nVidia GameStream. its got a GTX 660Ti 3GB card on it. I have the Audio portion of the card passed to it too.

 

When I first setup the VM it worked great, was getting over 100 FPS on games like Devil May Cry 4 Special Edition (its got a benchmark tool on the main menu of the game) and Streat Fighter 4 Ultimate (same kind benchmark on the main menu). Pretty much I got bare metal performance.

One day the VM crashed, I rebooted it and found that the GPU was stuck in Code 43, I passed the GPU's BIOS to it, didn't work. I had to remove the GPU from the VM setting it back to VNC and then putting it back and then the GPU started working again only this time the GPU's performance is horrible.

I'm lucky to hold a 40 FPS average in the same to games.

In my troubleshooting, I wiped the VM of the system completely, re-installed Windows, installed the drivers, passed the GPU and its HDMI Audio back over, and got the same bad performance.

I noticed that CPU Performance was kinda bad in the VM at this time so I attempted to dedicate the threads to the VM and that didn't seem to do anything.

Sometimes if I'm lucky the VM will just work at full speed and be great but then the VM will crash and I'll have to un-pass and then re-pass the GPU for the VM to be able to use the GPU again.

I have even went through and enabled the MSI on the GPU and the HDMI Audio.

I have even tried using a friends GTX 770 and get the same results.

 

The System is a HP DL370 G6

2x 5570 CPUs

96GB of RAM

 

At this point, I don't know what to do I have tried everything I can google about this.

 

Here are my CPU and Thread pairings.

 

cpu 0 <===> cpu 16
cpu 2 <===> cpu 18
cpu 4 <===> cpu 20
cpu 6 <===> cpu 22
cpu 8 <===> cpu 24
cpu 10 <===> cpu 26
cpu 12 <===> cpu 28
cpu 14 <===> cpu 30

 

The syslinux.cfg

default /syslinux/menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append isolcpus=4,12,20,28 vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label unRAID OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

The XML for the VM

<domain type='kvm' id='1'>
  <name>Windows 10 Gaming VM GTX 660 Ti</name>
  <uuid>9d972ce5-59ae-9cfb-73d2-1a1f64326548</uuid>
  <description>Gaming VM GTX 660 Ti</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows_teamgreen.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='12'/>
    <vcpupin vcpu='2' cpuset='20'/>
    <vcpupin vcpu='3' cpuset='28'/>
    <emulatorpin cpuset='2,18'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/9d972ce5-59ae-9cfb-73d2-1a1f64326548_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='2' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10 Gaming VM GTX 660 Ti/vdisk1.img'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Windows/Desktop/Windows10_InsiderPreview_Client_x64_en-us_15063.iso'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.126-2.iso'/>
      <backingStore/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:e8:23:36'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-Windows 10 Gaming VM/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x21' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x21' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <rom file='/mnt/user/system/vBIOS/EVGAGTX660Ti.dump'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

Any Help anyone can give would be amazing!

Edited June 3, 20179 yr by AnnabellaRenee87

So updates.

 

I was home sick last night from work so had some time to play around with this.

 

The performance issue was solved by going to unRAID 6.4 RC2

It fixed my odd Core pairings, I also turned Hyper-V on as I seen that @gridrunner left his on in his last video series for setting up a Windows 10 VM.

So that was the performance issue solved.

 

The odd crash.

I was able to play games on GameStream for about an hour until the VM hard locks. The only way I can get the VM back up without code 43 was to reboot the server.

I was able to repeat the issue after like 45 minutes. So two more times of that I have been borrowing a friends GTX 770 so I put that in there and was able to play Devil May Cry 4 SE all the way through, I was able to play Bayonetta about half way till I went to bed.

So my GTX 660 Ti (with 3GB of vRAM and flashed to support UEFI from EVGA) apparently has an issue with being on KVM, the card works fine when I take out the GTX 1080 Ti in my Desktop and put the GTX 660 Ti in and GameStream works fine on my Bare Metal Desktop.

 

So anyone know if the 1060 (6GB version) works ok with unRAID?

The 1060 should work absolutely fine. Really any card should be okay if it supports reset and EFI boot. So I am surprised you are having an issue.

So you physically flashed the GPU with EFI compatible bios. You didn't just pass through an EFI compatible bios to the VM?

I notice you are using unsafe interrupts on your server so iommu isn't quite right. So to be sure that this ist the problem for your GPU problems, I would take out your 1080ti from your desktop and put it in the server and see if its stable. If it is, get a 1060. If its not stable with the 1080ti it will not work with the 1060.

If you only have one GPU in the server or it is primary you must pass through the vbios to the VM.

Hope that helps.

From the looks of it the 770 is stable.
I updated the BIOS in the 660 ti back when it was my desktops main gpu. I wanted uefi boot on my desktop so I contacted evga and they sent me the bios for it

Sent from my HTC 10 using Tapatalk

That moment you realise the craziness of your experiment that you stand back and take pictures.

Sent from my HTC 10 using Tapatalk

So Nvidia requires a monitor connected for GameStream on the 10 series cards now, connected a monitor and it was as amazing as you would expect, time to go and buy the 1060 this pay day, gonna pick up this one for the blower design cooler

https://www.evga.com/products/product.aspx?pn=06G-P4-5161-KR

 

And a dummy plug.

Sent from my HTC 10 using Tapatalk
 

Edited June 10, 20179 yr by AnnabellaRenee87

wont fit in my proliant. I had to trim the plastic shroud of the gtx 760 to get it to barely fit.... i'm a bit jealous of your spaciousness.

 

 

also had to hack the bracket....

 

 

 

 

that bracket is now used on the 1050 card since it already worked. I should have cut the server instead, but I couldn't bring myself to do that.