Hi Guys,

 

I am well aware that SLI not tested/supported on UNRAID, have noted the message @jonp posted way back in 2015.

 

Things have obviously moved on since then and I thought I would start this post to discuss what I am seeing on my machine in the hopes to generate some discussion and hopefully a resolution?

 

I am currently runnings UNRAID 6.3.2 and have several VM's one which is a Windows 10 Pro (Q35 v2.7) machine with the following passed through:

 

• EVGA 1080 GTX FTW 3.0 x 2 (00:05.0) and (00:06.0)
• EVGA 1080 GTX HDMI x 2 (00:05.1) and (00:06.1)
• Intel C610/X99 series chipset USB xHCI Host Controller | USB controller (00:14.0)
• Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 | Non-Volatile memory controller 1 TB (00:02.0)

 

I have been in general happy with the performance and started exploring why SLI is not an option in the Nvidia Control Panel. I downloaded GPU-Z and queried both cards and noticed that the cards are being reported by GPU-Z as 'PCI' and not PCI-E x 16. I then started to surmise that perhaps this is why SLI was not available as an option in the Nvidia driver as the Q35 BIOS is reporting the GPU's as PCI capable only; PCIE being a requirement for SLI.

 

I then started to check my GPU's and they are indeed set to run at the full PCIE 3.0 X 16 in each slot: My Asus X99E-WS motherboard has a PLX chip so the X99 board along with the PLX chip is capable of running 4 GPUs @ PCIE 3.0 x 16. GPU 1 is in PCIE Slot 1 and GPU 2 is in PCIE Slot 3.

 

There was a registry hack that you could apply to the video drivers to force the PCI bus to PCIE 3.0 this was in the days of Windows 8.1 when PCIE 3.0 was new and not enabled by default in the Nvidia drivers. I applied that 'fix' to the VM but it made no difference. It does seem that the BIOS is reporting the GPUs as PCI.

 

Question: Is it possible for  PCIE devices to be reported as PCIE devices through KVM / Q35 / OVMF? I believe that Seabios / x440f only supports PCI pass-through.

 

Please see my attached screenshot showing PCI as the bus, in GPU-Z. Also here is the XML of that VM; it has not been optimised yet.

<domain type='kvm' id='3'>
  <name>Windows 10 Pro</name>
  <uuid>6cdeacc6-d49b-da2b-a62c-f948822dec92</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='5'/>
    <vcpupin vcpu='2' cpuset='6'/>
    <vcpupin vcpu='3' cpuset='7'/>
    <vcpupin vcpu='4' cpuset='12'/>
    <vcpupin vcpu='5' cpuset='13'/>
    <vcpupin vcpu='6' cpuset='14'/>
    <vcpupin vcpu='7' cpuset='15'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/mnt/user/domains/Windows 10 Pro/OVMF-pure-efi.fd</loader>
    <nvram>/mnt/user/domains/Windows 10 Pro/OVMF_VARS-pure-efi.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/UNRAID Drivers/virtio-win-0.1.126-2.iso'/>
      <backingStore/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source dev='/dev/disk/by-id/scsi-ARC-1883-VOL001_10feb80341957892'/>
      <backingStore/>
      <target dev='hdd' bus='virtio'/>
      <alias name='virtio-disk3'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x0a' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='nec-xhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:ca:4d:57'/>
      <source bridge='br0'/>
      <target dev='vnet2'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-3-Windows 10 Pro/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/domains/Video ROM/evgagtx1080ftw.dump'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x14' function='0x0'/>
      </source>
      <alias name='hostdev4'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev5'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x08' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

 

 

Any advice you can provide or support would be greatly appreciate. I am keen to try and understand if this can be fixed.

 

Thanks,

 

Enver
 

Edited March 29, 20179 yr by Enver

lspci -vv shows both cards LnkCap and LnkStat as x16 speed.

 

05:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1) (prog-if 00 [VGA controller])

    Subsystem: eVga.com. Corp. Device 6286

    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+

    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

    Latency: 0, Cache Line Size: 32 bytes

    Interrupt: pin A routed to IRQ 78

    NUMA node: 0

    Region 0: Memory at fa000000 (32-bit, non-prefetchable)

    Region 1: Memory at c0000000 (64-bit, prefetchable)

    Region 3: Memory at d0000000 (64-bit, prefetchable)

    Region 5: I/O ports at c000

    Expansion ROM at 000c0000 [disabled]

    Capabilities: [60] Power Management version 3

        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

    Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+

        Address: 00000000fee00938  Data: 0000

    Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00

        DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us

            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-

        DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+

            MaxPayload 256 bytes, MaxReadReq 512 bytes

        DevSta:    CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

        LnkCap:    Port #16, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us

            ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+

        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk-

            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

        LnkSta:    Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

        DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Via message

        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-

             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-

             Compliance De-emphasis: -6dB

        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+

             EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-

    Capabilities: [100 v1] Virtual Channel

        Caps:    LPEVC=0 RefClk=100ns PATEntryBits=1

        Arb:    Fixed- WRR32- WRR64- WRR128-

        Ctrl:    ArbSelect=Fixed

        Status:    InProgress-

        VC0:    Caps:    PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-

            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-

            Ctrl:    Enable+ ID=0 ArbSelect=Fixed TC/VC=01

            Status:    NegoPending- InProgress-

    Capabilities: [250 v1] Latency Tolerance Reporting

        Max snoop latency: 0ns

        Max no snoop latency: 0ns

    Capabilities: [128 v1] Power Budgeting <?>

    Capabilities: [420 v2] Advanced Error Reporting

        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-

        CESta:    RxErr+ BadTLP+ BadDLLP- Rollover- Timeout+ NonFatalErr-

        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+

        AERCap:    First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-

    Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>

    Capabilities: [900 v1] #19

    Kernel driver in use: vfio-pci

 

Edited March 29, 20179 yr by Enver

I doubt this issue is related to PCI vs PCIe. My reason? We have seen folks get crossfire to work with AMD devices (in some circumstances).  It is more likely some voodoo black magic that NVIDIA just doesn't expose us to so we can't recreate it on the virtual side of things. 

1 hour ago, jonp said:

I doubt this issue is related to PCI vs PCIe. My reason? We have seen folks get crossfire to work with AMD devices (in some circumstances).  It is more likely some voodoo black magic that NVIDIA just doesn't expose us to so we can't recreate it on the virtual side of things. 

Hi Jon,

 

Thanks for the reply.

My approach was to ascertain the differences of what is being exposed through the hypervisor. This machine is currently duel boot so I am able to compare the differences in Windows 10 native vs Windows 10 KVM.

 

One of the obvious differences from an driver perspective seems to be the be the way the bus is being reported to the Windows 10 OS; at least that is the case for me.

 

I did see posts from other users who have been able to 'see' the correct PCIE x 8 / 16 bus speeds through GPU-Z, however these were just single GPU VMs who were trying to diagnose pass-through issues.

 

I did notice in the Nvidia System Info Panel that the Nvidia driver reports the GPU bus as 'PCIE x 0' which isn't correct.

 

Whilst I agree with you that there is probably some 'voodoo' as you say, the only way we will be able to ascertain what that may be is to start looking at the differences and eliminating them.

 

I would appreciate some advice and insight to members who are more experienced than me to assist me with my investigation.

 

I personally believe if Lime-Technology can work this out it will probably be a huge lift in uptake and marketing of the UNRAID product.

 

Thanks,

 

Enver

 

 

 

You do know that this is a qemu/libvirt limitation and nothing LT can do much about. They just include the hypervisor and make a nice gui to create/manage VM's. 

Might be better to go to the qemu mailing list to discuss sli and how to get it implemented.