Hotplug guest GPU

[derbestimmer]

Hi all. To be honest I am not completely sure if I am in the right space here since my problem has only remotely to do with the VM Engine (KVM) but I didn't find a more suitable category. So if I misplaced the post please point me to the correct place and I will move the post.
Here is my problem:

I want to be able to hotplug a gpu which I only use within a guest VM. To further describe the problem I should firstly describe my "unusual" setup:

• Case/Board: Asrock X300 Mini PC
• APU: AMD Ryzen 7 Pro 4750G (Vega 8 as host iGPU, only used for transcoding in jellyfin)
• RAM: 64GB PC3200 (2x 32GB SO-DIMM)
• Cache: Viper VPN100 2TB M.2 NVME
• Array: 2x Seagate 5TB 2.5'' (data drives), 1x Toshiba S300 5TB 3,5'' (parity drive)
• dGPU: Nvidia GeForce GTX 1660 Super (passed through to a Win10 gaming VM)
• pci-e 3.0 x4 to m.2 nvme adapter: ADT-Link R43SG (since there is no pci-e slot on the board to host a dGPU) -> dGPU is powered by a dedicated 200W dell power supply

 

First of all passthrough is not an issue and worked out of the box. The dGPU and all its sub components are bound to VFIO and have their own IOMMU group. Force PCIe ACS is set to both and Allow unsafe interrupts is set to yes. The dGPU host driver is not installed since it is only used within the VM.

As mentioned I want to be able to make use of the advantage of having a dedicated power source for my guest GPU. Specifically I want to remotely power down the GPU after a gaming session to save power and power it up before I want to play a game. Most importantly I want to achieve this during the runtime of Unraid without a restart.

 

What did I do so far:

- first of all I shut down the gaming vm my dGPU is passed through

- then used the following commands from @SpaceInvaderOnes vBIOS dump script:

echo "1" | tee -a /sys/bus/pci/devices/$disconnectid/remove

- power on the gpu

echo "1" | tee -a /sys/bus/pci/rescan

result: when starting the VM i get "Unknown PCI header type '127'" which seems to indicate that the GPU couldn't be reset correctly. A reboot fixes the problem but I want to be able to do this without shutting down all running services.... if actually possible.

 

There are a lot of posts in this forum regarding switching guest GPUs binding and unbinding the devices from the host driver / vfio driver. But nothing seemed to match my case exactly.

 

I appreciate your help.

Edited July 21, 2021 by derbestimmer
missing words