Jump to content

Cant passthrough nvidia gpu anymore to vm, used to in previous unraid versions

sully316

Recommended Posts

sully316 Noob

sully316

Posted
Posted (edited)

So I had a gaming vm with passedthrough gtx 1070 few months ago but was using virtual disk and wasnt great so deleted it, I have since got a 1tb ssd now and wanted to passthrough to get better performance so follwed steps here and also SpaceInvaderOne videos and got it working and can connect via vnc and rdp. I then tried to get same gpu passed through but when I do it says started but cant rdp to it (I can when using virtual gpu) I have the nvidia gpu in its own iomuu with its audio part as follows 

IOMMU group 0:			 	[8086:9bc8] 00:02.0 VGA compatible controller: Intel Corporation CometLake-S GT2 [UHD Graphics 630] (rev 03)
IOMMU group 1:				[8086:9b53] 00:00.0 Host bridge: Intel Corporation Comet Lake-S 6c Host Bridge/DRAM Controller (rev 03)
IOMMU group 2:				[8086:1901] 00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 03)
IOMMU group 3:			 	[8086:1911] 00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
IOMMU group 4:			 	[8086:43ed] 00:14.0 USB controller: Intel Corporation Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller (rev 11)
Bus 001 Device 001 Port 1-0		ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002 Port 1-2		ID 8087:0032 Intel Corp. AX210 Bluetooth
Bus 001 Device 003 Port 1-3		ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
Bus 001 Device 004 Port 1-7		ID 0781:556b SanDisk Corp. Cruzer Edge
Bus 001 Device 005 Port 1-11		ID 05e3:0608 Genesys Logic, Inc. Hub
Bus 001 Device 006 Port 1-14		ID 1462:7d17 Micro Star International MYSTIC LIGHT
Bus 002 Device 001 Port 2-0		ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 002 Port 2-2		ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
 	[8086:43ef] 00:14.2 RAM memory: Intel Corporation Tiger Lake-H Shared SRAM (rev 11)
IOMMU group 5:			 	[8086:43e0] 00:16.0 Communication controller: Intel Corporation Tiger Lake-H Management Engine Interface (rev 11)
IOMMU group 6:			 	[8086:43d2] 00:17.0 SATA controller: Intel Corporation Device 43d2 (rev 11)
[2:0:0:0]    disk    ATA      KINGSTON SEDC500 J2.7  /dev/sdb    480GB
[4:0:0:0]    disk    ATA      MK000960GWCFA    HPG2  /dev/sdc    960GB
[5:0:0:0]    disk    ATA      MK0400GCTZA      HPG4  /dev/sdd    400GB
IOMMU group 7:				[8086:43c4] 00:1b.0 PCI bridge: Intel Corporation Device 43c4 (rev 11)
IOMMU group 8:				[8086:43bc] 00:1c.0 PCI bridge: Intel Corporation Tiger Lake-H PCI Express Root Port #5 (rev 11)
IOMMU group 9:				[8086:43bd] 00:1c.5 PCI bridge: Intel Corporation Device 43bd (rev 11)
IOMMU group 10:			 	[8086:4387] 00:1f.0 ISA bridge: Intel Corporation B560 LPC/eSPI Controller (rev 11)
 	[8086:f0c8] 00:1f.3 Audio device: Intel Corporation Device f0c8 (rev 11)
 	[8086:43a3] 00:1f.4 SMBus: Intel Corporation Tiger Lake-H SMBus Controller (rev 11)
 	[8086:43a4] 00:1f.5 Serial bus controller: Intel Corporation Tiger Lake-H SPI Controller (rev 11)
IOMMU group 11:			 	[10de:1b81] 01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
 	[10de:10f0] 01:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
IOMMU group 12:			 	[1000:0072] 02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
[1:0:0:0]    disk    ATA      WDC WD40EFAX-68J 0A82  /dev/sde   4.00TB
[1:0:1:0]    disk    ATA      ST4000DM000-1F21 CC52  /dev/sdf   4.00TB
[1:0:2:0]    disk    ATA      WDC WD120EDAZ-11 0A81  /dev/sdg   12.0TB
[1:0:3:0]    disk    ATA      WDC WD80EDAZ-11T 0A81  /dev/sdh   8.00TB
[1:0:4:0]    disk    ATA      ST10000DM0004-1Z DN01  /dev/sdi   10.0TB
[1:0:5:0]    disk    ATA      Hitachi HUA72302 A840  /dev/sdj   2.00TB
[1:0:6:0]    disk    ATA      Hitachi HUA72302 A840  /dev/sdk   2.00TB
IOMMU group 13:			 	[10ec:8125] 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
IOMMU group 14:			 	[8086:2725] 04:00.0 Network controller: Intel Corporation Wi-Fi 6E(802.11ax) AX210/AX1675* 2x2 [Typhoon Peak] (rev 1a)

See VFIO log  

Loading config from /boot/config/vfio-pci.cfg
BIND=0000:01:00.0|10de:1b81 0000:01:00.1|10de:10f0---
Processing 0000:01:00.0 10de:1b81
Vendor:Device 10de:1b81 found at 0000:01:00.0

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/0000:01:00.0
/sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/0000:01:00.1

Binding...

success...

Device 10de:1b81 at 0000:01:00.0 bound to vfio-pci
---
Processing 0000:01:00.1 10de:10f0
Vendor:Device 10de:10f0 found at 0000:01:00.1

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:01:00.1/iommu_group/devices/0000:01:00.0
/sys/bus/pci/devices/0000:01:00.1/iommu_group/devices/0000:01:00.1

Binding...
0000:01:00.0 already bound to vfio-pci
0000:01:00.1 already bound to vfio-pci

success...

Device 10de:10f0 at 0000:01:00.1 bound to vfio-pci
---
Devices listed in /sys/bus/pci/drivers/vfio-pci:
lrwxrwxrwx 1 root root    0 May  2 14:56 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root    0 May  2 14:56 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
vfio-pci binding complete

 

Here is the config screenshot and xml image.png.c5f0d9b21664d25fbc3a84a9d95a2302.png

  

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>gaming</name>
  <uuid>a8e735cf-1bce-e365-5e2c-a151fe2c35e9</uuid>
  <description>win gaming</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='8'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='9'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='10'/>
    <vcpupin vcpu='6' cpuset='5'/>
    <vcpupin vcpu='7' cpuset='11'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-7.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='4' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source dev='/dev/disk/by-id/ata-MK000960GWCFA_S3EANX0K519910'/>
      <target dev='hdc' bus='virtio'/>
      <serial>vdisk1</serial>
      <boot order='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/win/Win10_20H2_v2_English_x64.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.248-1.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:17:03:08'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

Here is vm log and it stays there for about 5-10 min before i kill it again 

image.thumb.png.35dcac3d050a640eadc2379831ab4849.png

 

Any help will be allreciated

 

Edited by sully316
Link to comment
sully316 Noob

sully316

Posted
  • Author
Posted

Worth noting I tested creating win10 vm with virtual disk (so no hdd passthrough and it worked)

 

Here is the xml for that 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='9'>
  <name>Windows 10</name>
  <uuid>cc544cc4-a7c6-f79f-c074-b9369a9e826a</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>10485760</memory>
  <currentMemory unit='KiB'>10485760</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='8'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='9'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='10'/>
    <vcpupin vcpu='6' cpuset='5'/>
    <vcpupin vcpu='7' cpuset='11'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-7.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/cc544cc4-a7c6-f79f-c074-b9369a9e826a_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='4' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img' index='3'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <serial>vdisk3</serial>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/win/Win10_20H2_v2_English_x64.iso' index='2'/>
      <backingStore/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.248-1.iso' index='1'/>
      <backingStore/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='sata0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:f8:e6:10'/>
      <source bridge='br0'/>
      <target dev='vnet8'/>
      <model type='virtio-net'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-9-Windows 10/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

 

Link to comment
sully316 Noob

sully316

Posted
  • Author
Posted (edited)

Also worth noting ssd passthrough using SeaBIOS as per other docs and vm using OVMF, not sure this matters

also this is ssd via data not nvme

 

sorry for all the updates

Edited by sully316
Link to comment
bmartino1 Collaborator

bmartino1

Posted
Posted (edited)

Review over the years notes:

Ok I see that you have an Intel platform and have an onboard GPU which is good:
Please note that if you stubbed the NVIDIA card it will only be available to the vm side and not for any docker!
--The dockers would then be using your intel UHD for any graphic related tasks...
[UHD Graphics 630]


I see your Nvidia G-Card:
Per text data, IOMMU group 11:


[10de:1b81] 01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
[10de:10f0] 01:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)


If installed... Since we are stubbing I recommend removing the NVIDIA Driver plugin. 

So lets review. You may have a bad stub!

You appear to be binding IOMMU Group 11..

per vfio-pci Log:

If this was the NVIDIA card, it would be the 

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:01:00.1/iommu_group/devices/0000:01:00.0
/sys/bus/pci/devices/0000:01:00.1/iommu_group/devices/0000:01:00.1


Per you above post this is done correctly.
Even with correct. There may be other area that failed. 

So we will start by:

Please unsub all your devices and stub only the Graphic card.

Per data above, Your Group 11 would look something like this:
image.thumb.png.549111da3eab5acc2ef32acdc4c22b4d.png

After a reboot in Unraid Termainl run the command 

lspci -v

Scroll to the G-Card you should see:

Kernel driver in use: vfio-pci

Other Recommended Optional options:
I would also recommend running the multifunction group option via vm settings.

Here is an example of my advance setting that I run for both nvidia and amd:
image.thumb.png.3d1529a389860711ec64537d0e9ca081.png


I recommend that you enable PCIe ACS Override and set option to both

This configures some things in unraid and sets this option to the grub boot menu: 

pcie_acs_override=downstream,multifunction


It also messes with IOMMU groups layout.

after another reboot and stubbing.
I recommend visiting the Flash drive grub options for some additional kernel parameters to help ease the G card for VM use.

/Main/Settings/Flash?name=flash
image.thumb.png.dff7979e8e8712c4bcbe0e062f28ff55.png

This is my recommended Grub options to assist with pcie power management and to stop FB on G-card that would prevent stubbing and make a easier transition to the VM 

kernel /bzimage
append initrd=/bzroot pcie_acs_override=downstream,multifunction video=vesafb:off,efifb:off,simplefb:off,astdrmfb initcall_blacklist=sysfb_init pci=noaer pcie_aspm=off


I have the info form testing with ubunut/Proxmox and some other Unraid to correctly setup VFIO and edit the modprobe configs to use a G-Card.
Option can be obscured in Unraid. But here are some of my notes when I try to explain my grub options/Modprobe options:


More for running Proxmox but same commands and settings are set to touch pcie cards.

Quote

Grub Config Options:

https://wiki.ubuntu.com/Kernel/KernelBootParameters
https://manpages.ubuntu.com/manpages/bionic/man7/kernel-command-line.7.html
save default grub: https://manpages.ubuntu.com/manpages/bionic/man8/grub-set-default.8.html (to boot to pve 5.15 kernel)

nano /etc/default/grub
Example:
GRUB_CMDLINE_LINUX_DEFAULT="initcall_blacklist=sysfb_init libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 kvm.ignore_msrs=1 intel_iommu=on pcie_acs_override=downstream,multifunction nvme_core.default_ps_max_latency_us=5500 default_hugepagesz=1G hugepagesz=1G transparent_hugepage=always rootflags=noatime pci=noaer pcie_aspm=off intremap=no_x2apic_optout video=vesafb:off,efifb:off,simplefb:off,astdrmfb vfio-pci.ids= "

###############################
So why theses kerneal linux comands:
###############################
*Order is everything!

initcall_blacklist=sysfb_init 
*is a fix for the simple FB that is sent to the G card at first IRQ registration and initiantes a FB / ?X-x11 session to get card informatoin. to use a G card as passthorugh to a VM  NO Framebuffer or active session are to be on the card.
it more a fix for issues with video=simplefb:off

video=vesafb:off,efifb:off,simplefb:off,astdrmfb
*this is the line used to disable and turn off Frame Buffers so they don't touch the card we will use for passthough.

libata.allow_tpm=1 
is a tpm and sata fix from truenas to help with HBA and sata operation.

*YES AMD THREADSRIPEER AND EPIC REQUIRE BOTH AMD AND INTEL IOMMU ON!...
amd_iommu=on iommu=pt intel_iommu=on pcie_acs_override=downstream,multifunction
*this turns on IOMMU / SRV-IO for the ability to passthrough cards via Memory address.

vfio-pci.ids=
* Lspci -v and other comands to get the IOMMU and PCI device ID to be used with vfio kerneal driver.
-- the devcie especail a G card bue only be in use with vfio kerneal drvier for passthorugh to work.

kvm_amd.npt=1 kvm_amd.avic=1 kvm.ignore_msrs=1
* this is a truenas and other KVM erro log fix.

nvme_core.default_ps_max_latency_us=5500 default_hugepagesz=1G hugepagesz=1G transparent_hugepage=always 
* these are fixes and settings for NVME and storay / memory fixes...

rootflags=noatime 
* this is a file sytem setting to stop some time adding when a file is accessed.

pci=noaer pcie_aspm=off 
*these are other PCI settings such as power mangment and other advance reprots that have been known to spam the log.

intremap=no_x2apic_optout
* this is a fix for usb2.0 and 3.0 to mange usb passthrough.
###############################

PCIE modprobe stubbing:

https://pve.proxmox.com/wiki/PCI_Passthrough

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf
echo "options kvm ignore_msrs=1 report_ignored_msrs=0" > /etc/modprobe.d/kvm.conf
#echo "options snd-hda-intel enable_msi=1" >> /etc/modprobe.d/snd-hda-intel.conf

*TO set kernel level drives from starting as the host should not use them for GPU passthrough.
#lspci -v to get ci drive in use -- this can be dangerous...

nano blacklist.conf 
blacklist nouveau
blacklist amdgpu
blacklist radeon
blacklist rivafb
blacklist nvidiafb
blacklist rivatv
blacklist nvidia
blacklist nvidia_drm
blacklist i2c_nvidia_gpu
blacklist nvidia-gpu

*TO fix intel sound and pulse audio passthrough of sound. mainly turn off power save sleep settings...

nano alsa-base.conf
# Disable snd-hda-intel power saving
options snd-hda-intel power_save=0 power_save_controller=N

*Force SRV-IO
nano iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

*Fix kvm spam of message log:
nano kvm.conf
options kvm ignore_msrs=1

*Your VFIO LSPCI device to pass to VM
nano vfio.conf

( if g card sould have this: )
options vfio-pci ids= disable_vga=1

(if usb card / HBA should have this:)
options vfio-pci ids= disable_idle_d3=1 enable_sriov disable_denylist

##################################

List VIFO PCIDs:
lspci -v: to list drive in use
lspci -n -s 01:00 to list vfio Hardware ID

Example VFIO option in config:
root@BMM-PVE:~# lspci -n -s 36:00.0
to get vfio id

VFIO Comand options:

parm:           ids: Initial PCI IDs to add to the vfio driver, format is "vendor:device[:subvendor[:subdevice[:class[:class_mask]]]]" and multiple comma separated entries can be specified (string)

parm:           nointxmask: Disable support for PCI 2.3 style INTx masking.  If this resolves problems for specific devices, report lspci -vvvxxx to [email protected] so the device can be fixed automatically via the broken_intx_masking flag. (bool)

parm:           disable_vga: Disable VGA resource access through vfio-pci (bool)

parm:           disable_idle_d3: Disable using the PCI D3 low power state for idle, unused devices (bool)

parm:           enable_sriov: Enable support for SR-IOV configuration.  Enabling SR-IOV on a PF typically requires support of the userspace PF driver, enabling VFs without such support may result in non-functional VFs or PF. (bool)

parm:           disable_denylist: Disable use of device denylist. Disabling the denylist allows binding to devices with known errata that may lead to exploitable stability or security issues when accessed by untrusted users. (bool)

Don't forget to update the kernel and grub:
nano /etc/modprobe.d/vfio.conf
update-initramfs -u -k al
REBOOT!

Step 6 Side fixes in VM configs.
User have reported that NVIDIA Kepler K80 GPUs need this in vmid.conf:
args: -machine pc,max-ram-below-4g=1G

#https://forum.proxmox.com/threads/quadro-gpu-passthrough-to-windows-shows-code-43.88788/
For Nvidia Passthorugh to fix some error code 43:
###########
qemu - XML config options for nvdia errors
////////////////////////////////////
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
cpu: host,hidden=1,flags=+pcid

///////////////////////////////////

and softmod to get linux driver to use VFIO :
Also See:
https://forum.proxmox.com/threads/pci-gpu-passthrough-on-proxmox-ve-8-installation-and-configuration.130218/

Kerneal 6 New Modproble to fix vfio driver load:

Example:
echo "options vfio-pci ids=1002:67df,1002:aaf0" >> /etc/modprobe.d/vfio.conf
For AMD
echo "softdep radeon pre: vfio-pci" >> /etc/modprobe.d/vfio.conf
echo "softdep amdgpu pre: vfio-pci" >> /etc/modprobe.d/vfio.conf

For Nvidia
echo "softdep nouveau pre: vfio-pci" >> /etc/modprobe.d/vfio.conf
echo "softdep nvidia pre: vfio-pci" >> /etc/modprobe.d/vfio.conf
echo "softdep nvidiafb pre: vfio-pci" >> /etc/modprobe.d/vfio.conf
echo "softdep nvidia_drm pre: vfio-pci" >> /etc/modprobe.d/vfio.conf
echo "softdep drm pre: vfio-pci" >> /etc/modprobe.d/vfio.conf

For Intel
echo "softdep snd_hda_intel pre: vfio-pci" >> /etc/modprobe.d/vfio.conf
echo "softdep snd_hda_codec_hdmi pre: vfio-pci" >> /etc/modprobe.d/vfio.conf
echo "softdep i915 pre: vfio-pci" >> /etc/modprobe.d/vfio.conf



Additional. We can in advance add a VFIO and confirm this NVIDIA G-card to be a good stub. Add to help ease kernel drive in use. this works similar to unraid bind. and before bind tool and other implementation this was how to grab a PCIE device. since 6.12.?4-8? Unraid gave us access to modprobe...

 

/Tools/SysDrivers

Top right:

image.png.72644d6ae91ff6885663f2dadad1541c.png

 

and find VFIO:

 

image.thumb.png.8c231c680301fbcf1d14959083d9d119.png

 

Here we can edit the mod probe config and add some extra option to easy the G card for VM use.

This would be your VFIO config options: 

#Prepare Nvdia Kerneal Driver:
softdep nouveau pre: vfio-pci
softdep nvidia pre: vfio-pci
softdep nvidiafb pre: vfio-pci
softdep nvidia_drm pre: vfio-pci
softdep drm pre: vfio-pci

#GPU: Nvidia: GeForce GTX 1070
options vfio-pci ids=10de:1b81,10de:10f0 disable_idle_d3=1 disable_vga=1

#USB PCI Card
#options vfio-pci ids= disable_idle_d3=1 enable_sriov disable_denylist


The goal here is to ease all the NVIDIA kernel drivers in use and to set the vfio-pci as the driver to be used later in the virtual machine.

######################################################
In unraids past revision, I found that setting the PCIE acls was more than enough. With changes to the Linux kernel, theses additional options and settings are sometimes warned to fix things like the dreaded NVIDIA error code 43.

 

Theirs some xml vm edits to fix error codes, but that would come after we can confirm PCI Stubbing is correct.

This is where I would have you watch space invaders video and confirm the proper XML edit to the VM:


Warning! IF you edit the XML and then make a GUI edit, THIS WILL ERASE ANY CUSTOM EDITS TO XML!
To use a NVIDIA G-card you must set the functions correctly and have them on the same qemu virtual bus!

There are other xml edits such as turning off KVM or hiding KVM, but that after we see the g-card in the VM space...


 

Edited by bmartino1
Documentation - Notes
Link to comment
bmartino1 Collaborator

bmartino1

Posted
Posted (edited)

Your hardware steps checklist:

Step 1 is verifying your IOMMU options:

On 4/4/2024 at 11:16 AM, bmartino1 said:

Verifying IOMMU parameters

  

dmesg | grep -e DMAR -e IOMMU


Intel will show something similar:

image.png.811138c5b4308713bdd3eddee1bbd471.png

 

There should be a line that looks like "DMAR: IOMMU enabled".

If there is no output, something is wrong.

 

Verify IOMMU interrupt remapping is enabled

 

It is not possible to use PCI passthrough without interrupt remapping. Device assignment will fail with 'Failed to assign device "[device name]": Operation not permitted' or 'Interrupt Remapping hardware not found, passing devices to unprivileged domains is insecure.'.

All systems using an Intel processor and chipset that have support for Intel Virtualization Technology for Directed I/O (VT-d), but do not have support for interrupt remapping will see such an error. Interrupt remapping support is provided in newer processors and chipsets (both AMD and Intel).

To identify if your system has support for interrupt remapping:

  

dmesg | grep 'remapping'

 

If you see one of the following lines:

AMD-Vi: Interrupt remapping enabled

DMAR-IR: Enabled IRQ remapping in x2apic mode ('x2apic' can be different on old CPUs, but should still work)

then remapping is supported.

If your system doesn't support interrupt remapping, you can allow unsafe interrupts
^enabled under unraid VM settings...

image.png.e7e44d62d49c51c603a819d864b16deb.png

 


Step 2 is verified your Stub PCEI device
Green Circles, check box. VFIO PCI Log.

Step 3 is additional IOMMU options to help ease the G-card into vfio-pci

optional edits to fix other things. but mainly
confirm that vfio-pci drive in the kernel driver in use.
unraid terminal command lspci -v

Step 4 is proper VM/XML data
--I Have yet to review your XML code. This is where you appear to need some help.
XMl code can be edited to fix or change qemu options to a VM.

Edited by bmartino1
Documentation
Link to comment
bmartino1 Collaborator

bmartino1

Posted
Posted (edited)

Software/Vm hyperv notes over the years:

You need to fix your XML Code to pass the G-card properly. You also wish to pass a hard disk directly to the VM?
Then we will need to get the PC ID of that drive. That drive can not be in the unraid array or use with the unattended drive plugin you would see it there until we pass it by disk by ID.

So lets review Proper VM setting and setup:

step 1 to have gpu passthrough working is to use only the PCIE G-Card.

lets create a new vm to review:
If we create a default VM no gui chagnes this is what we get:

image.thumb.png.2c612d6c08f482c9b98b54077ea1e832.png

image.thumb.png.5ec5d707af8f673ae76d71713bbb3a9b.png

Copy the XML code to review... As you can see this is a default Template for KVM- QUEMU for windows 10
To use Windows 10 and a G-card there are some settings we want to change and take a look at.

When making a VM we want to select a emulated machine type. q35 / i440fx:


It is recommended when using pcie to use the q35

The next thing we want to confirm is are we using uefi or classic bios:


are choices are OVM or Seabios:


It is recommended to use OVM as it is a UEFI bios.
Thanks to Windows 11 and Manufacture moving to firmware tpm you can also emulate a TPM with a Virtual Machine. So lets continue to make these changes via the form view to the default no options change:

image.png.83bbf7790e3b683f203e0a785b1e48de.png

So what changes in our xml? Note these options are set once at first VM creation. It is a pain to go and fix the entire XML:
image.thumb.png.8df1ec515b8c8f656f9fe15efb4be98a.png

so let's re-create with the correct options and choices for our VM.

Here is my recommend VM settings for windows 7 and above:
image.png.f23834550fb553fb78fb2a710040bc79.png

 

I also recommend at least 120 GB HDD if using a virtual file as a Disk
image.png.9f1bcf3d18e643b57acdf0fce84d654d.png

*Here is where you would use disk by id to pass a physical disk... select the disk not in use or select none, and we will add the disk latter:
Review:

https://serverfault.com/questions/896027/kvm-pass-through-disk-partition-by-label-or-uuid#:~:text=<disk type%3D'block' device%3D'disk'> <source dev%3D'/dev/disk/by-id/ata-Corsair_CSSD-F120GB2_1109650632000461003B' /> <target dev%3D'hdc' bus%3D'virtio' /> </disk>

To get you disk by id: 

ls -la /dev/disk/by-id/

Example output:
image.png.b5cd4cb2bb55d956b529e1d4ff0b4a91.png

Next is the G-Card XML edits:

^ Please watch as SPACE INVADER! Has done many great video covering this!
 

So I recommend installing the OS then make this edit as preferred as OS setup may have problems :
image.png.2ee8b7686bbe3b89d76c61b3cd9940f6.png

This way you have a base installation with a VNC qemu virtual display. Note under VMS you will see:

image.png.ff80871ce199d9bc4e10dcb22ca98269.png
VNC auto for virtual display or the G-card you pass when done correctly!

Example:
image.thumb.png.d3b5c8732198efbdc03126a198cccd57.png

xml example: 

    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
    </hostdev>

 

It is possible to use both, but the G-card must be first and the primary device before the VNC screen!

so, lets take a look at your XML code. I assume the 1st XML code is what you want edited and fixed...

Edited by bmartino1
Documentation - Notes
Link to comment
bmartino1 Collaborator

bmartino1

Posted
Posted (edited)

The problem is in how you were trying to handle disk by id passthorugh.

Review:

 

Quote

<?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>gaming</name> <uuid>a8e735cf-1bce-e365-5e2c-a151fe2c35e9</uuid> <description>win gaming</description> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='8'/> <vcpupin vcpu='2' cpuset='3'/> <vcpupin vcpu='3' cpuset='9'/> <vcpupin vcpu='4' cpuset='4'/> <vcpupin vcpu='5' cpuset='10'/> <vcpupin vcpu='6' cpuset='5'/> <vcpupin vcpu='7' cpuset='11'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-7.2'>hvm</type> </os> <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='4' threads='2'/> <cache mode='passthrough'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source dev='/dev/disk/by-id/ata-MK000960GWCFA_S3EANX0K519910'/> <target dev='hdc' bus='virtio'/> <serial>vdisk1</serial> <boot order='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/win/Win10_20H2_v2_English_x64.iso'/> <target dev='hda' bus='sata'/> <readonly/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.248-1.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:17:03:08'/> <source bridge='br0'/> <model type='virtio-net'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'> <listen type='address' address='0.0.0.0'/> </graphics> <audio id='1' type='none'/> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </memballoon> </devices> </domain>

 


Here is the necessary edits that I have found to use this XML for GPU pass through. Based on information provided.
*Its been a while since I tried to pass a disk by id on unraid. your xml seems to be sanitized as well, missing nloader etc... so review edits and make changes at your own risk.

Things I see problematic you are:

VM type you are using i440fx... Use q35...

Process type / cpu pinning
I personally hate how the default pins parts of the cpu. The load balancer should be doing that...
You can delete cputune section, set vcpu and set cores/thread correctly for load balancer to uses all cores when processing vm stuff.... 

<topology sockets='1' dies='1' cores='8' threads='1'/>

or

<topology sockets='1' dies='1' cores='4' threads='2'/>

Depends on hardware/process cores and layout.



Your Disk passthrough is wrong...
dev disk by id (looks like a nvme drive) should be using scsi not virtio.
?address werid you seem to be trying to use/pass it via pcie?
Notes:
is a NVMe in the very top slot, yes?

(great its direct to PCIE processor buss and can be stubbed via iommu!) You can bind and stub it under bottom options.

Is it a random sata drive that is seen by unassigned and not in use by the main tab for a pool/array?
(great it can be passed via disk by id....) XML code varies and should use buss SCSI...

Its been a while for disk by id on unraid. ATM I don't have a test system in place to go over these options. Working tests were in proxmox 6/7 when doing this...
  

	<disk type='block' device='disk'>
	    <driver name='qemu' type='raw' cache='writeback'/>
        <source dev='/dev/disk/by-id/ata-MK000960GWCFA_S3EANX0K519910' />
        <target dev='hdc' bus='scsi' />
		<boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>		
    </disk>
	
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/win/Win10_20H2_v2_English_x64.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    
	<disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.248-1.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>

^ this looks better but may error out...

You have yet to pass the g-card can't help or fix gamin xml file... 

 <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>


############

alot wrong in that vm. I recommend stating that VM over...

Entire xml code with my edits: 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>gaming</name>
  <uuid>a8e735cf-1bce-e365-5e2c-a151fe2c35e9</uuid>
  <description>win gaming</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-7.1'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='8' threads='1'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
	
	<disk type='block' device='disk'>
	    <driver name='qemu' type='raw' cache='writeback'/>
        <source dev='/dev/disk/by-id/ata-MK000960GWCFA_S3EANX0K519910' />
        <target dev='hdc' bus='scsi' />
		<boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>		
    </disk>
	
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/win/Win10_20H2_v2_English_x64.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    
	<disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.248-1.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
	
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:17:03:08'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>


Good luck and have fun.

 

Edited by bmartino1
Link to comment
sully316 Noob

sully316

Posted
  • Author
Posted

Thanks all for the info, not had time to go over this but will in next few days. 

Just a note the sdd is not nvme but a 2.5" connected to sata3 port on the mbx

I have the IOMUU setup correctly as when i dont try to pass through the disk by id ie and instead use a vdisk1.img the gpu asses through no problem

Then if I use disk-by-id and dont pass through gpu it works

 

Ill review how I setup the VMs and config both via gui and xml and see what I am missing thank aall

Link to comment
bmartino1 Collaborator

bmartino1

Posted
Posted (edited)
On 5/9/2024 at 4:59 AM, sully316 said:

So I redid a vm using vdisk1.img and can pass through gpu no problem. which I am then assuming that my IOMMU are the nsetup correctly.

 

Il start working on the disk by id next 


Awesome. Its easier to pass a disk to vm at first vm creation.
image.thumb.png.0e98669035274203cc2b4eb513ac7a81.png

  

ls -la /dev/disk/by-id/


will give you all your partitions and disk id #
in my case, I want to pass my 16TB disk as example
it is seen as /dev/sdb

image.png.57e87bc98d4d907c5166e2ae30762a8a.png

I then compared my disk ID by physical Linux path /dev/sdb:
Not I have other partitions on this i don't want the part i want the full disk!
 

lrwxrwxrwx 1 root root   9 May  9 01:16 wwn-0x5000c500dd148fc2 -> ../../sdb
lrwxrwxrwx 1 root root  10 May  9 01:18 wwn-0x5000c500dd148fc2-part1 -> ../../sdb1

 

So my disk passthorugh by id code is:

/dev/disk/by-id/wwn-0x5000c500dd148fc2

at disk selection in VM Click manual.
image.png.dcba125e89cca84fb1189b853cb6016e.png

The disk can not be used for unraid what so ever!
if its is a pool device, you will break many things!

you would want to see it under:
image.thumb.png.24c8f69e086ff5e765394d3e41e1907b.png

but not in use with the unsigned device plugin. As the VM will take if from VM host.

XML code:
  

    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source dev='/dev/disk/by-id/wwn-0x5000c500dd148fc2'/>
      <target dev='hdc' bus='scsi'/>
      <serial>vdisk1</serial>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>

 

Edited by bmartino1
Link to comment
sully316 Noob

sully316

Posted
  • Author
Posted (edited)

Yeah so I made a vm with the disk by id like in my first post again and it worked but Ill play around with not using virtio as this is ssd using the sata connections on mbx it will require some fiddling. ill post my updates and let ye know

 

Also yes I use UD plugin to format it only but I dont mount it via UD sa then its not recognizable for what I need

Edited by sully316
  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing
×
×
  • Create New...