Nvidia-smi can't see GPU once it's passed through to a VM

[Rikumo1978]

Hello all,

 

So I'm pretty good with servers and server related activities. But something keeps on eluding me, and I don't know how to fix it. So, I have a Ubuntu Virtual Machine setup for a Plex Server. I have a GPU that is being passed through to this Server for Hardware Transcoding. Before I did any of this, I installed the Nvidia drivers and installed nvidia-smi. Now, this is what I see before I start up the VM with the attached GPU.

 

All seemed fine, so I setup the VM, and passed through the GPU, and started it. Once I setup Plex, all is working fine. I can transcode using the GPU just fine, but then I noticed that I no longer can see the stats of the GPU in GPU Statistics. This is what I see

 

So I was messing around, turned the VM off, and read all sorts of stuff, and one common thing I saw was to Bind the GPU to VFIO, so I did that, rebooted my server, and now, whether the VM is started or not started, this is what I see.

 

 

I unbound the card from VFIO since it felt like I was closer before I did that. I will attach the diagnostics stuff with the GPU not bound to the VFIO.

 

So, what I want to know is, is there a way that I can Pass this GPU through to the Plex VM, and still see the GPU statistics while it's being used? I see other people doing it, but I don't know how they are doing it. My original thought is, maybe it's because they are using Plex with a Docker, but I don't want to do that. I want to use 1 VM for Plex, and another VM for the downloading/media management. Please help me haha

 

nautilus-diagnostics-20220414-1249.zip

[itimpi]

If you pass the GPU through to a VM then it is no longer visible to Unraid.

[Sneyek]

@itimpi I currently have the same "Issue", so I understand that, when a VM is started, I can't monitor the GPU usage right ?

That weird, it's kind of the whole point of this, but ok, anyway, on the other hand, when I shutdown the VM, the issue persist, I can't see the stats anymore, my GPU is not recognized and I have ~5% CPU usage because of log flood:

Jun 13 14:21:45 Yggdrasil kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x23:0x65:1426)
Jun 13 14:21:45 Yggdrasil kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0

PS: Yggdrasil is my server name

 

Any idea how I could fix this ??

[SimonF]

3 minutes ago, Sneyek said:

@itimpi I currently have the same "Issue", so I understand that, when a VM is started, I can't monitor the GPU usage right ?

That weird, it's kind of the whole point of this, but ok, anyway, on the other hand, when I shutdown the VM, the issue persist, I can't see the stats anymore, my GPU is not recognized and I have ~5% CPU usage because of log flood:

Jun 13 14:21:45 Yggdrasil kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x23:0x65:1426)
Jun 13 14:21:45 Yggdrasil kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0

PS: Yggdrasil is my server name

 

Any idea how I could fix this ??

Have you bound the gpu to vfio if yes the correct driver will not be bound to the card.

[Sneyek]

3 minutes ago, SimonF said:

Have you bound the gpu to vfio if yes the correct driver will not be bound to the card.

 

Should I do it ? How can I do this ?

[SimonF]

2 minutes ago, Sneyek said:

 

Should I do it ? How can I do this ?

Can you provide screen print of system devices in tools? GPU check boxes must not be checked if using nvidia driver.

[Sneyek]

It's not checked.

Right now I have stats showing, but of course my GPU is in idle, when a VM is started with the GPU passthrough it just doesn't monitor anything anymore. It's not even able to display the GPU name or temperature...

And when I stop the VM, it doesn't work again, instead I have the message mentioned previously flooded in the Logs and ~5% CPU usage. (But also my entire Unraid seems to be slow, it takes almost 1 minute to change the current page.) The only way to fix the log flood, cpu usage and slowness it to reboot the entire system...

[SimonF]

1 minute ago, Sneyek said:

It's not checked.

Right now I have stats showing, but of course my GPU is in idle, when a VM is started with the GPU passthrough it just doesn't monitor anything anymore. It's not even able to display the GPU name or temperature...

And when I stop the VM, it doesn't work again, instead I have the message mentioned previously flooded in the Logs and ~5% CPU usage. (But also my entire Unraid seems to be slow, it takes almost 1 minute to change the current page.) The only way to fix the log flood, cpu usage and slowness it to reboot the entire system...

That is expected as the gpu is exclusivily allocated to the vm. it is no longer visible to the host.

 

Why do you have iommu group 12 bound to vfio? post diagnostics,

[Sneyek]

IOMMU group 12 is because I wasn't able to get sound from a VM using only the iGPU, this was the only way, I don't even know why.

 

I've added the diagnostics, but I downloaded them after a reboot. If I reproduce my issue I can't download them, the server is freezing and it's not downloading. I hope it helps.

yggdrasil-diagnostics-20230613-1959.zip

[Sneyek]

@SimonF Look like I was able to reproduce the issue and download the diagnostic right after, this one may be more useful:

yggdrasil-diagnostics-20230613-2017.zip

[Sneyek]

Hey @SimonF any idea about this, It's still an issue, I wasn't able to find a solution, especially now with Reddit being private.

Thank you for your help !