Can't pass both GPUs to Ubuntu VM

[FourMoBro]

My goal is to have an Ubuntu VM running headless in which BOTH of my 1080tis are available for GPU processing. I have 26 days remaining on my trial and would gladly purchase a license if this can be solved before the trial expires.

 

1. I created the VM using VNC as the primary Graphics Card to setup the OS.
2. Once the install was complete, I "reboot" the VM and can then access the VM via ssh. I installed my files and everything works fine.
3. After a VM stop, I can add one of my 1080tis as the second Graphics card, without any BIOS changes and without sound card options. I can start the VM, run a program, and verify that python is using the GPU for processing via nvidia-smi. I am halfway there!
4. If I want, I can stop VM, switch cards in the GUI (from 02:00.0 to 03:00.0) for the second graphics card and go back in. Everything works just like before. python can use the GPU that is passed.

 

In order to parallelize the processing, I need access to both GPUs. I have done this in a standalone version of Ubuntu and was hoping to do this within a VM in UNRAID. However, I can't for the life of me get both cards to passthru at the same time. I tried:

1. just adding the second GPU as a 3rd card
2. removing VNC and just have the nvidia cards
3. changed order of cards, etc

 

No matter what I do, when the Ubuntu VM is started, it never fully boots no matter how long I wait. I cannot ping the VM, nor login via ssh. I have to Force Stop the VM and change the configuration back to VNC primary with a single GPU as the "2nd card".

 

Some additional info:

• There are no other VMs created or running
• Only one docker container is running (minecraft server) but behavior is the same if it is not running
• Card 02:00.0 is in IOMMU group 30
• Card 03:00.0 is in IOMMU group 31
• I do have Unraid Nvidia installed. When the VM is not running, nvidia-smi from the unraid terminal shows both devices available to unraid. When 1 GPU is passed, nvidia-smi shows the remaining device available to unraid. When both devices are attempted to be passed, nvidia-smi command returns "Failed to initialize NVML: Unknown Error". Which should be expected as the devices are being consumed, or at least trying to be consumed.
• Unraid (6.8.2) runs headless, no GUI, on the hardware and is accessed via web browser to make changes*
• I can create another Ubuntu VM, pass the unused GPU, and have both VMs running concurrently with their respective GPUs.**

 

Any help would be appreciated. If you need more info, please tell me where to look and I will provide it.

 

*added after first post

** second edit

 

 

Edited February 29, 2020 by FourMoBro
**

[FourMoBro]

C'mon forum community. Don't let me down. 80+ views but no replies in a week from the original post. Give me something to try or look into!

[FourMoBro]

two week bump. last time, i swear, as my trial is just about up!!!!

[JonathanM]

Have you looked through this thread?

https://forums.unraid.net/topic/39981-multiple-gpus-in-a-single-vm/

 

[fearlessknight]

Hi Unraid Community,

 

I just wanted to inform those that I also experienced this recently. I have 2 VMs. One is mainly online as my Primary whereas I use the other VM for travel (laptop) which is mostly off. 

I suspected I may have had a bad card. I went through the physical process of removing each of the cards and 2x checking their  status. All is verified by Unraid and 1 of the VMs, but not both. Despite the Unraid Nvidia Driver package indicating it see's both cards. I've also attempted to create a new VM from scratch thinking my xml file may be corrupt . 

I went back and forth checking the passthrough on both GPUs in each VM. I can ping and access 1 VM with Passthrough but not the other.

 

Have you come across a solution for this by chance? I'll keep digging around in the meantime.

 

Thanks,

 

Edited February 27, 2021 by fearlessknight