GPU passthrough in Ubuntu desktop 22.04 VM not recognized

[bmartino1]

On 4/6/2024 at 5:10 PM, JKunraid said:

 

Results of running (dmesg | grep -e DMAR -e IOMMU) on host system.

 

[    0.507012] pci 0000:60:00.2: AMD-Vi: IOMMU performance counters supported
[    0.508237] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported
[    0.513697] pci 0000:20:00.2: AMD-Vi: IOMMU performance counters supported
[    0.515574] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.518364] pci 0000:60:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.518532] pci 0000:40:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.518699] pci 0000:20:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.518867] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.521408] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    0.521522] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).
[    0.521637] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank).
[    0.521747] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank).
[    1.748930] AMD-Vi: AMD IOMMUv2 loaded and initialized

 

 

Results of (dmesg | grep 'remapping') on host,

 

 0.519033] AMD-Vi: Interrupt remapping enabled

 

Installed "disable security mitigations" with no luck. Is this enabled or disabled by default?

 

You mention add line (options vfio_iommu_type1 allow_unsafe_interrupts=1) to grub. Am I supposed to append it to it's own line  somewhere specific?

(e.g  append it to the line I previously added?)

 

label GPU passthrough mode
  menu default
  kernel /bzimage
  append initrd=/bzroot video=vesafb:off,efifb:off,simplefb:off,astdrmfb initcall_blacklist=sysfb_init pci=noaer pcie_aspm=off pcie_acs_override=downstream,multifunction options vfio_iommu_type1 allow_unsafe_interrupts=1

 

btw - thanks for the help. Hopefully we can figure this out but even if not I'm learning a lot just trying different things.

 

found my notes for the grub otion.

 

 

pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1

your 
Label GPU passthrough mode
  menu default
  kernel /bzimage
  append initrd=/bzroot video=vesafb:off,efifb:off,simplefb:off,astdrmfb initcall_blacklist=sysfb_init pci=noaer pcie_aspm=off pcie_acs_override=downstream,multifunction options vfio_iommu_type1 allow_unsafe_interrupts=1

so add the vfio_iommu_type1.allow_unsafe_interrupts=1. test adding all three iommu options as well. 

amd_iommu=on iommu=pt intel_iommu=on

so
 

Label GPU passthrough mode
  menu default
  kernel /bzimage
  append initrd=/bzroot video=vesafb:off,efifb:off,simplefb:off,astdrmfb initcall_blacklist=sysfb_init pci=noaer pcie_aspm=off pcie_acs_override=downstream,multifunction options vfio_iommu_type1 allow_unsafe_interrupts=1 vfio_iommu_type1.allow_unsafe_interrupts=1 amd_iommu=on iommu=pt intel_iommu=on

 

[JKunraid]

On 4/11/2024 at 12:13 AM, bmartino1 said:

found my notes for the grub otion.

 

 

pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1

your 
Label GPU passthrough mode
  menu default
  kernel /bzimage
  append initrd=/bzroot video=vesafb:off,efifb:off,simplefb:off,astdrmfb initcall_blacklist=sysfb_init pci=noaer pcie_aspm=off pcie_acs_override=downstream,multifunction options vfio_iommu_type1 allow_unsafe_interrupts=1

so add the vfio_iommu_type1.allow_unsafe_interrupts=1. test adding all three iommu options as well. 

amd_iommu=on iommu=pt intel_iommu=on

so
 

Label GPU passthrough mode
  menu default
  kernel /bzimage
  append initrd=/bzroot video=vesafb:off,efifb:off,simplefb:off,astdrmfb initcall_blacklist=sysfb_init pci=noaer pcie_aspm=off pcie_acs_override=downstream,multifunction options vfio_iommu_type1 allow_unsafe_interrupts=1 vfio_iommu_type1.allow_unsafe_interrupts=1 amd_iommu=on iommu=pt intel_iommu=on

 

 

 

Success!

 

My existing Ubuntu VM still doesn't see it. And I tried with multiple new templates of Ubuntu but I tried with your suggested PopOS VM and it recognizes the GPU. Thanks so much for the help. 

 

[bmartino1]

Awesome. Glad that worked for you.

Pop is built ontop of the Ubuntu system with Debian. This tells me that the Ubuntu VM instance you have doesn't have the correct kernel module or that linux driver nouveau is in the way. in ubuntu you should be able to use nvidia-smi to get the card.


To potential fix your ubuntu VM

 

I also recommend installing and using dkms

 

apt-get install dkms

 

Before you download, the nvidia run file to install the ubuntu driver.

https://www.nvidia.com/download/driverResults.aspx/223426/en-us/
as of writing:
 

Linux X64 (AMD64/EM64T) Display Driver

 

Version:550.67

Release Date:2024.3.19

Operating System:Linux 64-bit

Language:English (US)

File Size:292.53 MB

form terminal:
then chmod +x NVIDIA-Linux-x86_64-550.67.run

./NVIDIA-Linux-x86_64-550.67.run

answering the prompts.

 

you may need to use similar grub options that we used with unraid to Ubuntu boot grub commands and blacklist driver trying to touch it...

such as:

video=vesafb:off,efifb:off,simplefb:off,astdrmfb initcall_blacklist=sysfb_init to the ubuntu grub boot


cd /etc/modprobe.d
nano blacklist.conf

blacklist nouveau

 

Don't forget to update the kernal and grub:

nano /etc/default/grub

update-grub

 

if you touch anything in mod probe
update-initramfs -u -k al