6.10RC1-2 Kills my Nvidia Primary GPU Passthrough

[bigbangus]

Has anyone experienced issues passing a primary Nvidia GPU once upgrading to 6.10RC1 or RC2? In my case it's a 1660Ti in the primary slot.

 

Everything was working fine in 6.9.2 using a combination of methods (below), but nothing can fix my Code 43 in Windows 10 in Unraid 6.10. The VM log doesn't report any issues, but the Code 43 persists. If I roll back to 6.9.2 the VM works fine.

 

What I've done to make my primary GPU passthrough work in 6.9.2

• IOMMU: Enabled, vfio bind all the GPU devices (VGA+Audio+USB+USB)
• append "video=efifb:off" to the syslinux configuration (this was the game changer for 6.9 and prior)
• dump the vbios using @SpaceInvaderOne 's awesome script (thanks!)
• Tried ifx440 and Q35 no difference
• Did the VM xml edit multifunction trick with preserving the slot assignment to the VM for all 4 GPU devices again thx @SpaceInvaderOne

 

Nothing can get it working in 6.10. I'm stumped. 

 

[GavinCampbell]

Mine stopped working when I upgraded to the Nvidia driver v495.29.05.  I had to roll back to v470.74 for it to work.  Not sure if thats related.

 

I haven't tried the latest though.

[bigbangus]

I don't think the Nvidia driver matters when you are passing the GPU through from what I gather. It's just for when you are using a GPU in dockers and such like Plex/Frigate etc...

[ich777]

3 hours ago, GavinCampbell said:

Mine stopped working when I upgraded to the Nvidia driver v495.29.05.  I had to roll back to v470.74 for it to work.  Not sure if thats related.

Are you talking about the Nvidia Driver plugin on unRAID or in a VM?

What card do you own?

Please remember 600 and 700 Series cards where dropped from Nvidia with the new driver versions, the v470 drivers are the last ones that support those cards.

[GavinCampbell]

Just now, ich777 said:

Are you talking about the Nvidia Driver plugin on unRAID or in a VM?

What card do you own?

Please remember 600 and 700 Series cards where dropped from Nvidia with the new driver versions, the v470 drivers are the last ones that support those cards.

 

Yes.  The Nvidia driver plugin on unRAID.  Its an Nvidia GT 1030.  Its an old card but mainly use is because something in the VM requires a physical card for it to function.

 

I haven't dug in deep enough but was that card also dorpped in the newer driver versions?

 

I also have a GT710 in the box which also makes sense now why it stopped working.  Maybe its time to upgrade my cards to something a bit newer.

[ich777]

18 minutes ago, GavinCampbell said:

Its an Nvidia GT 1030. 

The GT1030 should work just fine, you can always check what cards are supported here, simply click on "Supported Products" (in this case it's for driver v495.44

 

19 minutes ago, GavinCampbell said:

GT710

This card will only work with the v470 drivers.

[bigbangus]

Hey @ich777 any thoughts on what I can try to fix my passthrough in 6.10? Should I try to make a custom 6.10 with an older libvert version if that's even possible using your docker?

[ich777]

8 minutes ago, bigbangus said:

Should I try to make a custom 6.10 with an older libvert version if that's even possible using your docker?

No, that won't solve nothing also I don't think that's the root cause of the issue, have you somewhere your Diagnostics?

You also have to know I'm not the specialist when it comes to VMs and Passthrough, what card is the primary display output? The 1050Ti from your signature?

If you have two cards in your server I think you don't even need video=efifb:off if you have set the primary card to the 1050Ti in your BIOS.

[bigbangus]

31 minutes ago, ich777 said:

No, that won't solve nothing also I don't think that's the root cause of the issue, have you somewhere your Diagnostics?

You also have to know I'm not the specialist when it comes to VMs and Passthrough, what card is the primary display output? The 1050Ti from your signature?

If you have two cards in your server I think you don't even need video=efifb:off if you have set the primary card to the 1050Ti in your BIOS.

 

So my BIOS doesn't seem to expose an option to select primary GPU so I've had to keep video=efifb:off for that reason I think.

 

See Diagnostics attached. Thank you for your help.

unraidnas-diagnostics-20211103-1340.zip

[ich777]

2 minutes ago, bigbangus said:

So my BIOS doesn't seem to expose an option to select primary GPU so I've had to keep video=efifb:off for that reason I think.

Can you maybe try to disable this option and see which card is the console output? I think it would be best to use the 1050Ti for console output since you are using this card in Docker containers.

If the console output from unRAID happens on the wrong card, simply try to swap the PCIe slots for the cards physically.

[bigbangus]

2 minutes ago, ich777 said:

Can you maybe try to disable this option and see which card is the console output? I think it would be best to use the 1050Ti for console output since you are using this card in Docker containers.

If the console output from unRAID happens on the wrong card, simply try to swap the PCIe slots for the cards physically.

 

It's definitely the 1660Ti on the console when I boot, but I'm trying to maintain my 1660Ti on the primary slot for best performance in VM (x16 slot). All was working in 6.9.2 and then stopped in 6.10. Just don't know what to look at to solve it. Would rather not make the 1050Ti primary. Seems backwards to put the fast card in the slower slot?

[ich777]

Just now, bigbangus said:

It's definitely the 1660Ti on the console when I boot, but I'm trying to maintain my 1660Ti on the primary slot for best performance in VM (x16 slot).

This will make no difference even if you put it in a PCIe 3.0 8x slot, same goes for my 1070Ti.

[ich777]

2 minutes ago, bigbangus said:

Seems backwards to put the fast card in the slower slot?

Is the second slot not shared with the first slot in your system, like it is the case on Intel systems?

[bigbangus]

From the manual



Sent from my iPhone using Tapatalk

[ich777]

24 minutes ago, bigbangus said:

From the manual

Oh, that's too bad, but I'm not sure how the 1660 would act in a x4 slot, it always depends...

Are you sure that you don't have a option like "PEG/PCI/PCIE" in your BIOS to set the main graphics card?

[bigbangus]

15 hours ago, ich777 said:

Oh, that's too bad, but I'm not sure how the 1660 would act in a x4 slot, it always depends...

Are you sure that you don't have a option like "PEG/PCI/PCIE" in your BIOS to set the main graphics card?

Yeah pretty sure. It's an option on some other B550M boards but not mine  Also tried legacy boot this morning to see if that solves it, but no luck.

[ich777]

1 hour ago, bigbangus said:

Yeah pretty sure. It's an option on some other B550M boards but not mine  Also tried legacy boot this morning to see if that solves it, but no luck.

Can you try to do a new VM "test" installation without a GPU BIOS file, I think @alturismo runs a system with two cards and also tested it without vfio and it just works fine.

[bigbangus]

Can you try to do a new VM "test" installation without a GPU BIOS file, I think @alturismo runs a system with two cards and also tested it without vfio and it just works fine.

I’m game for whatever. When you say new VM without vfio are you saying create a new template, new template and image or new template and/or image without vfio binding any GPU?


Sent from my iPhone using Tapatalk

[ich777]

5 minutes ago, bigbangus said:

I’m game for whatever. When you say new VM without vfio are you saying create a new template, new template and image or new template and/or image without vfio binding any GPU?

Exactly, I really don't know the AMD hardware and how it works exactly with Virtualization, but I think it's worth a try to to test a VM from scratch without a BIOS file and the card not bound to VFIO and another one also without a BIOS file and bound to VFIO.

[alturismo]

1 hour ago, ich777 said:

Can you try to do a new VM "test" installation without a GPU BIOS file, I think @alturismo runs a system with two cards and also tested it without vfio and it just works fine.

indeed, but, i have a igpu and im not in the video=efifb:off mode ... so hard to compare, also on a intel system which makes life way more easier

 

19 hours ago, bigbangus said:

Seems backwards to put the fast card in the slower slot?

i can just also say, you wont feel any difference ... my RTX3070 "lost" ~ 1 % performance in benchmarks when running in x8 mode, i couldnt say that i really recognized anything, just in bench values ... same experience what @ich777 pointed out.

[bigbangus]

31 minutes ago, alturismo said:

i can just also say, you wont feel any difference ... my RTX3070 "lost" ~ 1 % performance in benchmarks when running in x8 mode, i couldnt say that i really recognized anything, just in bench values ... same experience what @ich777 pointed out.

 

According to my motherboard, it's x4 only in the second slot. So from what I can read that's probably a more significant loss when playing games?

[bigbangus]

2 hours ago, ich777 said:

Exactly, I really don't know the AMD hardware and how it works exactly with Virtualization, but I think it's worth a try to to test a VM from scratch without a BIOS file and the card not bound to VFIO and another one also without a BIOS file and bound to VFIO.

 

This didn't work either. Did a full factorial on all the options here. Even created a Win11 VM and it shows code43 too. I've lost all hope. I'm going back to 6.9.2 that seems to work just fine. Thanks for the help @ich777 @alturismo

[ich777]

2 minutes ago, bigbangus said:

 

This didn't work either. Did a full factorial on all the options here. Even created a Win11 VM and it shows code43 too. I've lost all hope. I'm going back to 6.9.2 that seems to work just fine. Thanks for the help @ich777 @alturismo

I would recommend that you open up a thread in the Bug Support sub forums with the Diagnostics attached from above so that the team can take a look at it.

[bigbangus]

8 minutes ago, ich777 said:

I would recommend that you open up a thread in the Bug Support sub forums with the Diagnostics attached from above so that the team can take a look at it.

Done and thank you again for your help and all you've done for the Unraid community. Much appreciated.

[CodingMicrobe]

@bigbangus wondering if you've had any progress on this issue? I upgrades from 6.9.2 to 6.10-rc3 yesterday and while everything else works fine, suddenly my GTX 760 GPU which is being used for my Windows 10 VM is throwing error 43, and VM is stuck at 800x600 resolution. I tried reinstalling Nvidia drivers (v470), which now officially support virtualization, without any success. Also made a brand new Windows 10 VM, same issue.