[SOLVED] Help with GPU passthrough Ubuntu


12 posts in this topic Last Reply

Recommended Posts

I need help getting GPU passthrough on my newly created ubuntu server 20.04.2. The GPU I try to passthrough is a NVIDIA P400. 

I have added the edited rom file and selected both the GPU and "audio" from nvidia in unraid. 

Every time I boot it with the GPU it will not start. I can't connect with PuTTy. If I change it to VNC I can connect with PuTTY after a minute. 

 

Hope someone can assist me alittle, unraid is still very new for me.


VM:

Bios: OVMF

Machine: Q35-5.1
 

SERVER Spec:

Software: Unraid 6.9.1
MB: Supermicro x9drd-if
RAM: 32GB 1333 ECC
GPU onboard: Matrox Electronics Systems Ltd. MGA G200eW WPCM45
GPU PCIe x16: NVIDIA Quadro P400

 

IOMMU group 63:[10de:1cb3] 82:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P400] (rev a1)

[10de:0fb9] 82:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)

 

Edited by julianbr
Problem solved
Link to post

Okay, so I finally figured it out. It was the network adapter in ubuntu that caused all the issues. After resolving it, everything turned out fine!

 

Anyway, I also found out that dumping the vbios myself is a much better approach then using one from tpu and edit it. I had to use the "put to sleep" mode even when the card was not being used in the dump vbios script.

Link to post
  • julianbr changed the title to [SOLVED] Help with GPU passthrough Ubuntu
On 3/31/2021 at 5:13 AM, julianbr said:

Okay, so I finally figured it out. It was the network adapter in ubuntu that caused all the issues. After resolving it, everything turned out fine!

 

Anyway, I also found out that dumping the vbios myself is a much better approach then using one from tpu and edit it. I had to use the "put to sleep" mode even when the card was not being used in the dump vbios script.

 

Julianbr,

I hope you are open to some follow-up questions.  I have a Supermicro X10SRH-CF with a XEON CPU. I am trying to passthrough an Nvidia Geforce 1030 card. And I'm using the Ubuntu template to setup a VM in my UnRaid. My guidebook is the SpaceInvaderOne YouTube in the link below:

 

https://www.youtube.com/watch?v=FWn6OCWl63o

 

The full details to the problems I am having are described in the link below:

 

https://forums.unraid.net/topic/105179-please-help-vbios-dump-script-does-not-work/?_fromLogout=1

 

How did you dump the vBIOS yourself?  I am using the SpaceInvader One vBIOS dump script and it does not work for me.  Did you have two GPUs installed on your Supermicro motherboard?  My Supermicro MB only has a single PCI slot big enough for a single GPU.    Can you explain how you fixed the Ubuntu network problem?  Any tips you can offer will be greatly appreciated. Thank you for your time.

 

 

 

Link to post
On 4/5/2021 at 5:38 AM, slipstream said:

 

Julianbr,

I hope you are open to some follow-up questions.  I have a Supermicro X10SRH-CF with a XEON CPU. I am trying to passthrough an Nvidia Geforce 1030 card. And I'm using the Ubuntu template to setup a VM in my UnRaid. My guidebook is the SpaceInvaderOne YouTube in the link below:

 

https://www.youtube.com/watch?v=FWn6OCWl63o

 

The full details to the problems I am having are described in the link below:

 

https://forums.unraid.net/topic/105179-please-help-vbios-dump-script-does-not-work/?_fromLogout=1

 

How did you dump the vBIOS yourself?  I am using the SpaceInvader One vBIOS dump script and it does not work for me.  Did you have two GPUs installed on your Supermicro motherboard?  My Supermicro MB only has a single PCI slot big enough for a single GPU.    Can you explain how you fixed the Ubuntu network problem?  Any tips you can offer will be greatly appreciated. Thank you for your time.

 

 

 

 

I only have 1 GPU in my system atm. I'm using the default settings for ubuntu VM 
 

Bios: OVMF

Machine: Q35-5.1

 

Does everything work when you set GPU to VNC? Before using the vbios script, change forcereset="no" to yes. Then, run the script and press the physical power button on your server when asked. After a while it will be done and unraid is back at it. The vbios should be dumped now. 

 

Attach the dumped vbios file to your VM and set GPU and GPU soundcard in VM.
image.png.129ce9cb7696cd126a1ecbc1e7a1a1f7.png

 

This should work. Tell me how it goes

 

Link to post
9 hours ago, julianbr said:

 

I only have 1 GPU in my system atm. I'm using the default settings for ubuntu VM 
 

Bios: OVMF

Machine: Q35-5.1

 

Does everything work when you set GPU to VNC? Before using the vbios script, change forcereset="no" to yes. Then, run the script and press the physical power button on your server when asked. After a while it will be done and unraid is back at it. The vbios should be dumped now. 

 

Attach the dumped vbios file to your VM and set GPU and GPU soundcard in VM.
image.png.129ce9cb7696cd126a1ecbc1e7a1a1f7.png

 

This should work. Tell me how it goes

 

 

 

Julianbr,
Thank very much for replying back. I appreciate your help a lot.

 

To answer your question, YES, when I have it set to VNC I am allowed to make a "VNC Remote" menu selection which envokes my Ubuntu VM inside a VNC window on my Windows10 client. However, if I change VNC to Nvidia GT1030 on my Ubuntu template the "VNC Remote" menu selection button disappears and I do not see my Ubuntu VM on either my Windows10 Client monitor or my UnRaid Server Monitor which is connected to my Nvidia GT1030.

 

image.png.f42a79ac805090e91645ce8d3785192e.png

 

I noticed my Machine parameter is set to Q35-4.2 (as shown below) and you are set to Q35-5.1. I will make this change tonight to my Ubuntu template so it matches what you have.

 

image.png.e77dd27fc3ea1f9ef46eb6de1570aebd.png
 
The troubleshooting I have performed is listed below and it has been crazy:

  1. I have tweaked the "PCIEe ACS override" parameter to "Downstream", "MultiFunction", "Both" and "None".  This did not help.
  2. I tried editing a GT1030 ROM file in a HEX editor and this did not work.
  3. I also tried the STUBBING approach where I added the my Nvidia GT1030 IOMMU ID number to the UnRaid OS Flash Drive and this did not fix the problem.
  4. I upgraded from 6.8.3 to 6.9.1 without applying UEFI option and this did not fix the problem.
  5. I changed my SuperMicro BIOS from UEFI to Legacy boot and this did not work.

And currently I am trying to get the SpaceInvader One vBIOS dump script to work but no success so far. In short, this is the most insanely complex UnRaid problem I have so far encountered.

 

The one thing I have not tried yet is making the code change shown below to the vBIOS script:
 

forcereset="no" to yes

Tonight when I get home from work I will make this vBIOS script change to see if it fixes my problem. I pray that by making this script change I am finally able to see my Ubuntu VM on my UnRaid Server monitor setup with the GT1030 card.

 

Also some good news is how you only have a single GPU installed because that matches what I got on my SuperMicro MB. I only have a single slot for a GPU and not two. So I am glad I don't have to buy a MB with two GPU PCI slots because a YouTube video mentions you need to install to graphic cards in order for the ROM dump to work successfully.

 

Again thanks for your help.  And I will test it out tonight.

 

 

Edited by slipstream
Link to post

Julianbr,

Thank you for your help! I got my Ubuntu VM to work through my Nvidia GT1030 card.  However, I am now having an audio problem. Within my Ubuntu VM I hear a loud noise that sounds like a rapid fire beeping machine gun that does not stop.  Did you have this audio problem in your Ubuntu VM?  If yes how did you fix it?  Thanks for your help.

Link to post

Julianbr,

One more question. When you bootup your UnRaid Server do you have it setup to go straight into GUI mode?  If yes, do you see your UnRaid OS initial code boot process display on your monitor connected to your Quadro P400 card? And then when your UnRaid OS is done with the pre-boot process do you see your UnRaid OS GUI display on the same monitor?  In my case things are not working the way I want it to work.  I have setup my UnRaid OS to boot straight into GUI mode as shown in screen capture below, and I do not see the UnRaid OS pre-boot code or GUI appear on my monitor connected to my GT1030 card.

 

image.thumb.png.5e715a896f687c3903efaa43cc64d66c.png

 

The only way I can get control of my UnRaid OS is remotely through my Windows10 client PC. In short, my UnRaid OS is behaving as though it were a headless unit eventhough I have it setup to function as a stand alone PC. Do you have any tips on what change I need to make so my UnRaid OS GUI appears straight into the monitor that is connected to the Nvidia GT1030 card because my goal is to use it as a standalone PC and not a headless unit?   Any help from you or anybody in the community will be welcome.  Thank you for your time.

 

 

 

Link to post
Posted (edited)
15 hours ago, slipstream said:

Julianbr,

One more question. When you bootup your UnRaid Server do you have it setup to go straight into GUI mode?  If yes, do you see your UnRaid OS initial code boot process display on your monitor connected to your Quadro P400 card? And then when your UnRaid OS is done with the pre-boot process do you see your UnRaid OS GUI display on the same monitor?  In my case things are not working the way I want it to work.  I have setup my UnRaid OS to boot straight into GUI mode as shown in screen capture below, and I do not see the UnRaid OS pre-boot code or GUI appear on my monitor connected to my GT1030 card.

 

image.thumb.png.5e715a896f687c3903efaa43cc64d66c.png

 

The only way I can get control of my UnRaid OS is remotely through my Windows10 client PC. In short, my UnRaid OS is behaving as though it were a headless unit eventhough I have it setup to function as a stand alone PC. Do you have any tips on what change I need to make so my UnRaid OS GUI appears straight into the monitor that is connected to the Nvidia GT1030 card because my goal is to use it as a standalone PC and not a headless unit?   Any help from you or anybody in the community will be welcome.  Thank you for your time.

 

 

 

 

I'm glad to hear that it's working! I used a few hours myself to figure it out! Btw, I have "PCIe ACS override" set to disabled.

 

Regarding GUI:

 

I don't use unraid in GUI mode, only commandline, however you should connect you monitor to the VGA output on the supermicro motherboard, this will use the internal GPU and everything should be working. I'm using VGA on the motherboard for unraid and displayport on the P400 for Ubuntu. It's connected for troubleshooting only. I mostly work via PuTTy, TeamViewer/RDP.

Edited by julianbr
Link to post

I think my problem is I am thinking about all of the this in the wrong way. I am expecting my Nvidia GT 1030 card to support both my Ubuntu VM video output and also my UnRaid GUI video output. When I plug my monitor to the VGA port on my SuperMicro Motherboard I see the hundreds of lines of code the Unraid OS shows in the very beginning of the boot process. But then my VGA turns black and does not show my Unraid GUI. So I can't use my UnRaid server as a stand alone PC. I have to get on my Windows 10 client PC and connect to my UnRaid Server using my FireFox browser.  If I want to start my Ubuntu VM I can start it from my UnRaid OS GUI inside my FIreFox browser and then eventually I see my Ubuntu VM on my monitor that is connected to my Nvidia GT1030 card. But I see the VM within a noVNC. In short, right now I have two monitors connected to my UnRaid Server. One monitor is connected to my Supermicro VGA port and then  I have a second monitor that is connected to the HDMI port on my GT1030 card. My GT1030 monitor only supports my Ubuntu VM launched from my Windows 10 client PC.  My VGA monitor does not get me to the UnRaid OS GUI. It turns black after all of the text code comes to a stop.

 

In a perfect world I would like to have a single monitor connected to my GT1030 card that supports both my UnRaid OS in GUI mode and also my Ubuntu VM.  Is this thinking right or wrong?  And is it necessary for me two have two separate monitors connected to my UnRaid server?

 

I will try disabling my "PCIe ACS override" parameter because right now I have it set to "Multifunction".  I hope this will fix my VGA monitor problem going black and showing me my UnRaid OS in GUI mode.

 

 

 

 

Link to post

Are you loading the ast video driver?

 

Also if you boot in non gui mode do you get login prompt on vga monitor.

 

You cant use the 1030 for both outputs. 

 

 

Link to post

SimonF,

Thank you for your post.  To answer your question I have not downloaded the AST (Aspeed Technologies) video driver.  I can tell you my Supermicro motherboard has an integrated Aspeed 2400 BMC (Baseboard Management) Controller to provide IPMI 2.O/KVM support on the motherboard. The VGA allows you to see the IPMI and the BIOS when there is a VGA capable monitor connected to the VGA port.  Can you share how such an AST driver would get installed? I imagine Supermicro would have to package it together as part of a BIOS update.  Do you think installing a separate AST driver will help my UnRaid OS show me the GUI on my VGA Monitor?

 

image.png.b9a7a7b55bb6997948815256f48080e1.png

 

To answer your second question.  YES, if I boot in non GUI mode and wait for all of the code in white text to scroll to the very end it it ends up sitting at a login prompt something like:  TOWER>

 

To clarify, if I input the correct account information at this login prompt is the UnRaid OS GUI designed to appear on my VGA monitor?  

 

You say I cannot use my Nvidia GT1030 for both my Ubuntu VM output and my VGA output.  Is what you say still true if I replace my SuperMicro Motherboards and GT1030 card with different hardware? In other words, do you say I can't see both outputs because that is the way the UnRaid OS is designed or is it because you think I am using the wrong motherboard and GPU?  Below is the Block Diagram for my Supermicro motherboard. I hope you can take a look at it so you can tell my why you think I cannot see both video outputs on a single monitor connected to my GT1030 card.  Thank you for your time.

 

image.thumb.png.f1bbab17a3cf8e2d192b0d1a38c8bef3.png

 

System Info:

SuperMicro X10SRH-CF Motherboard

128GB ECC RAM

Intel Xeon E5-2640 v3 CPU

UnRaid OS v6.9.1

Nvidia GT1030 GPU

Noctua Fans

Seagate 18TB Hard Drives

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.