GPU Passthrough Best Practices

[Arbadacarba]

I've read little references in a few other posts about optional ways to pass a GPU through to a VM... I've had a Nvidia GPU passed though to my Gaming VM (Passed through NVMe as well) and been very happy with the result when I'm playing Games on the VM.

 

But that is not all that often... Sadly...

 

I'm wondering if there is a better way to pass a GPU through that would let me use the GPU for other things in Dockers or even other VMs when I'm not gaming?

[Shan]

Hi @Arbadacarba could you please let me now how you were able to passthrough the GPU, I keep getting code 43?

Thanks,

Shan

[RiDDiX]

On 4/16/2023 at 3:03 AM, Arbadacarba said:

I've read little references in a few other posts about optional ways to pass a GPU through to a VM... I've had a Nvidia GPU passed though to my Gaming VM (Passed through NVMe as well) and been very happy with the result when I'm playing Games on the VM.

 

But that is not all that often... Sadly...

 

I'm wondering if there is a better way to pass a GPU through that would let me use the GPU for other things in Dockers or even other VMs when I'm not gaming?

You are able to do so. Install the Nvidia-Plugin which ich777 maintains. Then you should be able to use this GPU also for Dockers, when the VM is completely off.

[Shan]

How do I go about setting up the Nvidia-Plugin? Per the plug-in install, it shouldn't be installed for pass-through to VM.

[Arbadacarba]

That stopped me too...

 

I may have to give it a shot.

 

Question is: Does the VM Booting cause the dockers to fail or do they just lose the resource? Do they recover when the VM is shut down?

 

As for how I got my GPU Passed Through in the first place I'm afraid I just did as little as I could.

 

What Hardware are you running? Intel CPU Nvidia GPU?

 

Have you got the video card bound to VFIO at boot?

 

 

[Arbadacarba]

So I removed the VFIO Binding for my video card and rebooted the server... Then reset the Video card in my VM (Just in case) and it booted to the 4080 just fine... I did not expect that... I had always thought the Binding was required...

 

I guess next, I will try loading the NVIDIA Driver and trying it again...

 

[Shan]

I’ve got mine working but if I have hdmi cable plugged in it before starting up my VM it throws code 43 error. I can only connect the hdmi cable after the VM is up and running. It worked with Windows 10 before the upgrade without binding but after the upgrade to 6.11.5 and running windows 11 it only worked by binding the GPU. 

[Arbadacarba]

And you DID install the NVIDIA Driver App in Unraid?

 

[Shan]

No, that’s only if you want to use your GPU with a docker. If the GPU is being used by a docker then you won’t be able to do a pass-through to a vm, it’s either used with docker or passed through to a VM but not both. 
 

For use with a docker  - Requires the NVIDIA Driver App

 

For use with VM as a pass-through - Don’t install the NVIDIA App
 

[Arbadacarba]

That was my general understanding as well... But I've seen several references (By people who are fairly experienced with Unraid) talking about being able to use the card on Dockers and VM's... Not simultaneously but asynchronously without a reboot.

 

On the upside, my experimentation required that I play games for 3 hours last night... So it's not a total loss.

[shpitz461]

As long as you don't bind the GPU to VFIO on boot, it is available to unraid/docker when GPU is not used by VM (i.e. VM is turned off).

From my experience there are glitches where the card is still bound to VFIO even after the VM is turned off, so for me, I just added several GPUs to be used for other tasks and have a dedicated GPU for VM use.

If you issue the following command (my VM GPU is at address 85:00.0):

lspci -vnn -s 85:00.0

I get the following output:

Quote

85:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Hewlett-Packard Company GA102 [GeForce RTX 3090] [103c:88d5]
        Flags: bus master, fast devsel, latency 0, IRQ 598, IOMMU group 37
        Memory at f2000000 (32-bit, non-prefetchable)
        Memory at 47000000000 (64-bit, prefetchable)
        Memory at 47800000000 (64-bit, prefetchable)
        I/O ports at b000
        Expansion ROM at f3000000 [disabled]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Virtual Channel
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] Secondary PCI Express
        Capabilities: [bb0] Physical Resizable BAR
        Capabilities: [c1c] Physical Layer 16.0 GT/s <?>
        Capabilities: [d00] Lane Margining at the Receiver <?>
        Capabilities: [e00] Data Link Feature <?>
        Kernel driver in use: vfio-pci
        Kernel modules: nvidia_drm, nvidia

Note the 'Kernel driver in use', 'vfio-pci' means it is bound to it.

You can bind/unbind pcie devices without rebooting using the following commands:

Quote

echo  -n "0000:85:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo  -n "0000:85:00.0" > /sys/bus/pci/drivers/nvidia/bind

------------------------------------

echo  -n "0000:85:00.0" > /sys/bus/pci/drivers/nvidia/unbind
echo  -n "0000:85:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

 

[Shan]

Thanks, I'll try this and report back.