Passthrough Asus RTX2080-ti with IOMMU brake up......


frodr

Recommended Posts

SIO is saying do not brake up IOMMU groups unless you have to. 

 

Any chance to get the RTX 2080 working:

 

IOMMU group 73:[10de:1e07] af:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A] (rev a1)

[10de:10f7] af:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1)

[10de:1ad6] af:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1)

[10de:1ad7] af:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1)

 

In the video Advanced GPU passthrough SIO show xml editing. My xml says different bus, not virtual slot.

 

 

source>
        <address domain='0x0000' bus='0xaf' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0xaf' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>

 

 

This is the error message as is, with braking up IOMMU groups.

1249802159_Screenshot2020-02-26at14_15_51.png.1885a4fff6f31fc6320a4ae6e0d5eef1.png

 

 

 

After ACS override, Multifunction, the kvm start but with black screen. 

 

My questions are: 

1) Is it possible/worth while setting GPU without braking up IOMMU groups?

 

2) If not, how to fix the black screen?

This is said to happened when Nvidia is primary GPU. But, in my server this GPU is not the only GPU. An Aspeed GPU is in a PCIe slot and set as onboard in bios. 

 

IOMMU group 27:[1a03:1150] 01:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)

[1a03:2000] 02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)

 

 

 

 

 

Cheers,

 

 

Frode

 

Asus C621E Sage/BMC | dual Xeon Silver 4116 | Samsung 4 x 32GB DDR4-2933 RDIMM | 4 x Corsair Force MP510 (960GB) NvME ssd on Asus Hyper M.2 Card V2 l | 4 x WD and Seagate hdd l Sapphire Pulse Radeon RX 580 8GB as Mac WS l ASUS GeForce RTX 2080 Ti ROG Strix Advance in gaming kvm l Samsung 970 Pro in M.2 port for gaming kvm  l Asmedia PCIe USB card

 

 

WinPro kvm xml.rtf Server System Devices.rtf

Link to comment

The RTX 2080 has FOUR devices (GPU, HDMI audio, 2x USB devices i.e. everything you see in IOMMU group 73) that must be passed through together for it to work, regardless of IOMMU grouping (i.e. regardless of ACS Override).

Your xml only includes the GPU and HDMI audio and is missing the other 2 devices.

 

Go to the app store and look for VFIO PCI Cfg plugin and install it.

Then Settings -> VFIO PCI Cfg -> tick all 4 devices in group 73 -> build VFI-PCI.cfg -> reboot.

The 2 USB devices should now show up in the Other PCI Devices section in your VM template for you to select.

Link to comment
2 hours ago, testdasi said:

The RTX 2080 has FOUR devices (GPU, HDMI audio, 2x USB devices i.e. everything you see in IOMMU group 73) that must be passed through together for it to work, regardless of IOMMU grouping (i.e. regardless of ACS Override).

Your xml only includes the GPU and HDMI audio and is missing the other 2 devices.

 

Go to the app store and look for VFIO PCI Cfg plugin and install it.

Then Settings -> VFIO PCI Cfg -> tick all 4 devices in group 73 -> build VFI-PCI.cfg -> reboot.

The 2 USB devices should now show up in the Other PCI Devices section in your VM template for you to select.

Thanks, brilliant little plugin. Disabled ACS override, ticked all devices in Group 73, marked green. Kvm starts, but still black screen. Can it be that the motherboard sees the Nvidia GPU as main GPU?

Link to comment
9 minutes ago, frodr said:

Thanks, brilliant little plugin. Disabled ACS override, ticked all devices in Group 73, marked green. Kvm starts, but still black screen. Can it be that the motherboard sees the Nvidia GPU as main GPU?

 

Are you passing through the vBIOS to the VM as well? If you are, pull a new vBIOS from the card and try that. I had a lot of issue getting a working vBIOS for my 1070, but once I got it the black screen issue went away.

Link to comment
 
Are you passing through the vBIOS to the VM as well? If you are, pull a new vBIOS from the card and try that. I had a lot of issue getting a working vBIOS for my 1070, but once I got it the black screen issue went away.


No, I haven't. Do I have to do the "hex editor thing" explained by SIO?


Sent from my iPhone using Tapatalk
Link to comment
1 minute ago, frodr said:

 


No, I haven't. Do I have to do the "hex editor thing" explained by SIO?


Sent from my iPhone using Tapatalk

 

 

If you have a second video card to boot from you can just dump the vBIOS from the card. If not, yes that is the other way to do it. That is how I did it, but I had a spare PCIe video card to boot from.

 

 

Link to comment
4 minutes ago, frodr said:

No, I didn't. Scared of everything that starts with "hex.....". That's next. It's like the first bath in the Summer. Terribly cold at first, and very nice if you stay long enough.

You can't really break anything here by doing it. Worst case maybe hang unraid, but more than likely all you'll get is the black screen again.

Link to comment
2 minutes ago, frodr said:

I have modded the vbios downloaded from Techpowerup. Hopefully the correct version. Still black screen. I tried booting with a AMD card. It starts, but black screen also with this card. That shouldn't happened??

 

 

 

You have an AMD card in and the Nvidia one in too? If so, dump the vBIOS from your Nvidia one with the AMD card in as boot. I never was able to get a techpowerup vBIOS to work after editing. Nvidia seems to always need a vBIOS, unless I'm mistaken.

Link to comment
 
You have an AMD card in and the Nvidia one in too? If so, dump the vBIOS from your Nvidia one with the AMD card in as boot. I never was able to get a techpowerup vBIOS to work after editing. Nvidia seems to always need a vBIOS, unless I'm mistaken.


But it is black screen with the AMD as well.


Sent from my iPhone using Tapatalk
Link to comment

Are you booting with both cards? Which card is unraid  booting to? If you can run both cards, dump the vBIOS from the Nvidia. This is what I had to do to get the Nvidia to work at all. I've not tried an AMD card, so can't really help there. But I believe they don't need a vBIOS, but some of them have issues called the AMD reset bug. 

Link to comment

I tried the AMD GPU, still black monitor. Setting this GPU back to the Mac kvm I can passthrough, but MacOS do not recognise the GPU. Tried with and without VFIO-PCI Config as well at VNC/passthrough and restarting the server. 

 

I VFIO-PCI Config it says the AMD GPU is reset compatible. Anyone knows how to reset an AMD GPU? 

 

2064059459_Screenshot2020-02-27at16_43_38.thumb.png.40d2322459c18ca356664653370e3915.png

Link to comment
11 minutes ago, frodr said:

I tried the AMD GPU, still black monitor. Setting this GPU back to the Mac kvm I can passthrough, but MacOS do not recognise the GPU. Tried with and without VFIO-PCI Config as well at VNC/passthrough and restarting the server. 

 

I VFIO-PCI Config it says the AMD GPU is reset compatible. Anyone knows how to reset an AMD GPU? 

 

2064059459_Screenshot2020-02-27at16_43_38.thumb.png.40d2322459c18ca356664653370e3915.png

 

There was a script running around to reset the card, but I'm not sure that is 100% needed anymore. Still getting an NVidia card to pass through is easier. You just need the right vBIOS. Can you not boot the system with the two cards installed and pull the vBIOS from the NVidia one? Maybe ask if someone from the fourm here can supply a working NVidia vBIOS for a 2080? I bet there are a few of them here using one.

Link to comment
1 hour ago, Chess said:

 

There was a script running around to reset the card, but I'm not sure that is 100% needed anymore. Still getting an NVidia card to pass through is easier. You just need the right vBIOS. Can you not boot the system with the two cards installed and pull the vBIOS from the NVidia one? Maybe ask if someone from the fourm here can supply a working NVidia vBIOS for a 2080? I bet there are a few of them here using one.

Only black screen when booting from onboard or AMD GPU. 

Link to comment

From here I'm not sure what to suggest. Booting with the on board for unraid and then passing through a second video card to a  vm usually works without issue. That you are not getting an error when starting the vm says that unraid is allowing the pass through and that IMMOU is being used. So from here I'd start to look around and see if anyone has the same mainboard and cards ever got this to work. It could be a bug in the BIOS version you are on, but getting a new BIOS for a board that old is unlikely. I don't like doing BIOS down grades, but that is something I might consider if this was my own setup.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.