<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>5c38b6aa-7111-fa30-830e-12d193d9c66f</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='10'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='11'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-6.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/5c38b6aa-7111-fa30-830e-12d193d9c66f_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='2' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/en_windows_10_consumer_editions_version_20h2_updated_jan_2021_x64_dvd_00152b46.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:82:32:92'/>
      <source bridge='br1'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0c' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/domains/vbios/PNY.RTX3070.8192.200928.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0c' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x4'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1a2c'/>
        <product id='0x4c5e'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>
 

 

Above is the VM XML.  I followed Space Invaders how to videos and I still get a black screen when it boots.  vBios have been downloaded and the vBios modified for Nvidia graphics card.  I put all the different variables in the flash boot that he mentions to fix it.  I also have a P2000 i use for transcode process with plex and was wondering if I made the p2000 primary would it cause issues with the docker container for plex.  Essentially freeing up my GTX 3070 for passthru.  Thoughts any other options.  

 

Note:  I did not dumb my own vbios.

 

System Info:

Model: Custom

M/B: ASUSTeK COMPUTER INC. ROG CROSSHAIR VIII DARK HERO Version Rev X.0x - s/n: 210383114601949

BIOS: American Megatrends Inc. Version 3801. Dated: 07/30/2021

CPU: AMD Ryzen 5 3600X 6-Core @ 3800 MHz

HVM: Enabled

IOMMU: Enabled

Cache: 384 KiB, 3 MB, 32 MB

Memory: 32 GiB DDR4 (max. installable capacity 128 GiB)

Network: eth0: 1000 Mbps, full duplex, mtu 1500
 eth1: 1000 Mbps, full duplex, mtu 1500

Kernel: Linux 5.13.8-Unraid x86_64

OpenSSL: 1.1.1k

Uptime: 

 

Flash Info:

default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label Unraid OS
  menu default
  kernel /bzimage
  append vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction initrd=/bzroot
label Unraid OS GUI Mode
  kernel /bzimage
  append pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot,/bzroot-gui
label Unraid OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Unraid OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

 

VM Settings:

PCIe ACS override:

Downstream

VFIO allow unsafe interrupts:

Yes

Here are the logs from starting the VM

 

 

 

2021-09-17T21:10:51.237582Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b0220, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237588Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b0228, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237593Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b0230, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237599Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b0238, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237604Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b1220, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237610Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b1228, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237616Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b1230, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237622Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b1238, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237627Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b2220, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237633Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b2228, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237639Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b2230, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237645Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b2238, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237651Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b3220, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237657Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b3228, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237662Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b3230, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237668Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b3238, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237674Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b4220, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237680Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b4228, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237685Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b4230, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237690Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b4238, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237696Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b5220, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237702Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b5228, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237708Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b5230, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237714Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b5238, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237719Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b6220, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237725Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b6228, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237731Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b6230, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237737Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b6238, 0x0,8) failed: Device or resource busy
2021-09-17T21:10:51.237742Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b7220, 0x98989800989898,8) failed: Device or resource busy
2021-09-17T21:10:51.237748Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b7228, 0x98989800989898,8) failed: Device or resource busy
2021-09-17T21:10:51.237754Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b7230, 0x98989800989898,8) failed: Device or resource busy
2021-09-17T21:10:51.237760Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b7238, 0x98989800989898,8) failed: Device or resource busy
2021-09-17T21:10:51.237765Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b8220, 0x98989800989898,8) failed: Device or resource busy
2021-09-17T21:10:51.237771Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b8228, 0x98989800989898,8) failed: Device or resource busy
2021-09-17T21:10:51.237777Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b8230, 0x98989800989898,8) failed: Device or resource busy
2021-09-17T21:10:51.237782Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b8238, 0x98989800989898,8) failed: Device or resource busy
2021-09-17T21:10:51.237788Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b9220, 0x98989800989898,8) failed: Device or resource busy
2021-09-17T21:10:51.237794Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b9228, 0x98989800989898,8) failed: Device or resource busy
2021-09-17T21:10:51.237800Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b9230, 0x98989800989898,8) failed: Device or resource busy
2021-09-17T21:10:51.237806Z qemu-system-x86_64: vfio_region_write(0000:0c:00.0:region1+0x1b9238, 0x98989800989898,8) failed: Device or resource busy

1. Attach diagnostic file

2. Paste the output of

cat /proc/iomem

3. It is strongly recommended to dump your own vbios

root@Foundation:~# cat /proc/iomem
00000000-00000fff : Reserved
00001000-0009ffff : System RAM
000a0000-000fffff : Reserved
  000a0000-000bffff : PCI Bus 0000:00
  000c0000-000dffff : PCI Bus 0000:00
  000f0000-000fffff : System ROM
00100000-09cfefff : System RAM
  04000000-04a02046 : Kernel code
  04c00000-04f8efff : Kernel rodata
  05000000-051ac9bf : Kernel data
  055ec000-057fffff : Kernel bss
09cff000-09ffffff : Reserved
0a000000-0a1fffff : System RAM
0a200000-0a210fff : ACPI Non-volatile Storage
0a211000-97ef5017 : System RAM
97ef5018-97f1b457 : System RAM
97f1b458-97f1c017 : System RAM
97f1c018-97f2d067 : System RAM
97f2d068-97f2e017 : System RAM
97f2e018-97f3b867 : System RAM
97f3b868-97f3c017 : System RAM
97f3c018-97f5ba57 : System RAM
97f5ba58-a3295fff : System RAM
a3296000-a3296fff : Reserved
a3297000-a6bbffff : System RAM
a6bc0000-a6bc0fff : Reserved
a6bc1000-aa4e0fff : System RAM
aa4e1000-aa81cfff : Reserved
  aa7fb000-aa7fefff : MSFT0101:00
  aa7ff000-aa802fff : MSFT0101:00
aa81d000-aa969fff : ACPI Tables
aa96a000-aacc6fff : ACPI Non-volatile Storage
aacc7000-ab9fefff : Reserved
ab9ff000-acffffff : System RAM
ad000000-afffffff : Reserved
b0000000-fec02fff : PCI Bus 0000:00
  b0000000-c1ffffff : PCI Bus 0000:0c
    b0000000-bfffffff : 0000:0c:00.0
      b0000000-b167ffff : efifb
    c0000000-c1ffffff : 0000:0c:00.0
  d0000000-e1ffffff : PCI Bus 0000:02
    d0000000-e1ffffff : PCI Bus 0000:03
      d0000000-e1ffffff : PCI Bus 0000:05
        d0000000-dfffffff : 0000:05:00.0
        e0000000-e1ffffff : 0000:05:00.0
  f0000000-f7ffffff : PCI MMCONFIG 0000 [bus 00-7f]
    f0000000-f7ffffff : Reserved
      f0000000-f7ffffff : pnp 00:00
  f9000000-fa9fffff : PCI Bus 0000:02
    f9000000-fa9fffff : PCI Bus 0000:03
      f9000000-fa0fffff : PCI Bus 0000:05
        f9000000-f9ffffff : 0000:05:00.0
          f9000000-f9ffffff : nvidia
        fa000000-fa07ffff : 0000:05:00.0
        fa080000-fa083fff : 0000:05:00.1
      fa200000-fa3fffff : PCI Bus 0000:09
        fa200000-fa2fffff : 0000:09:00.3
          fa200000-fa2fffff : xhci-hcd
        fa300000-fa3fffff : 0000:09:00.1
          fa300000-fa3fffff : xhci-hcd
      fa400000-fa4fffff : PCI Bus 0000:0b
        fa400000-fa4007ff : 0000:0b:00.0
          fa400000-fa4007ff : ahci
      fa500000-fa5fffff : PCI Bus 0000:0a
        fa500000-fa5007ff : 0000:0a:00.0
          fa500000-fa5007ff : ahci
      fa600000-fa6fffff : PCI Bus 0000:08
        fa600000-fa603fff : 0000:08:00.0
      fa700000-fa7fffff : PCI Bus 0000:07
        fa700000-fa71ffff : 0000:07:00.0
          fa700000-fa71ffff : igb
        fa720000-fa723fff : 0000:07:00.0
          fa720000-fa723fff : igb
      fa800000-fa8fffff : PCI Bus 0000:06
        fa800000-fa80ffff : 0000:06:00.0
          fa800000-fa80ffff : r8169
        fa810000-fa813fff : 0000:06:00.0
      fa900000-fa9fffff : PCI Bus 0000:04
        fa900000-fa903fff : 0000:04:00.0
          fa900000-fa903fff : nvme
  fb000000-fc0fffff : PCI Bus 0000:0c
    fb000000-fbffffff : 0000:0c:00.0
      fb000000-fbffffff : nvidia
    fc000000-fc07ffff : 0000:0c:00.0
    fc080000-fc083fff : 0000:0c:00.1
  fc200000-fc4fffff : PCI Bus 0000:0e
    fc200000-fc2fffff : 0000:0e:00.3
      fc200000-fc2fffff : xhci-hcd
    fc300000-fc3fffff : 0000:0e:00.1
      fc300000-fc3fffff : ccp
    fc400000-fc407fff : 0000:0e:00.4
    fc408000-fc409fff : 0000:0e:00.1
      fc408000-fc409fff : ccp
  fc500000-fc5fffff : PCI Bus 0000:01
    fc500000-fc503fff : 0000:01:00.0
      fc500000-fc503fff : nvme
  fd200000-fd2fffff : Reserved
    fd200000-fd2fffff : pnp 00:01
  fd380000-fd3fffff : amd_iommu
  fd400000-fd5fffff : Reserved
  fea00000-fea0ffff : Reserved
  feb80000-fec01fff : Reserved
    fec00000-fec003ff : IOAPIC 0
    fec01000-fec013ff : IOAPIC 1
fec10000-fec10fff : Reserved
  fec10000-fec10fff : pnp 00:04
fed00000-fed00fff : Reserved
  fed00000-fed003ff : HPET 0
    fed00000-fed003ff : PNP0103:00
fed40000-fed44fff : Reserved
fed80000-fed8ffff : Reserved
  fed81500-fed818ff : AMDI0030:00
fedc0000-fedc0fff : pnp 00:04
fedc2000-fedcffff : Reserved
fedd4000-fedd5fff : Reserved
fee00000-ffffffff : PCI Bus 0000:00
  fee00000-fee00fff : Local APIC
    fee00000-fee00fff : pnp 00:04
  ff000000-ffffffff : Reserved
    ff000000-ffffffff : pnp 00:04
100000000-84f37ffff : System RAM
84f380000-84fffffff : Reserved
root@Foundation:~# 

 

 

Guess I could get a USB ISO of windows to get access to the vbios unless i can dumb from inside unraid

 

foundation-diagnostics-20210918-0811.zip

Before dumping the vbios:

 

vfio-pci 0000:0c:00.0: BAR 1: can't reserve [mem 0xb0000000-0xbfffffff 64bit pref]

Memory can't be mapped because it's in use by efifb (look at /proc/iomem)

 

Append video=efifb:off to syslinux

label Unraid OS
  menu default
  kernel /bzimage
  append video=efifb:off vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction initrd=/bzroot
label Unraid OS GUI Mode
  kernel /bzimage
  append video=efifb:off pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot,/bzroot-gui

 

(Attach to vfio iommu group 32 (0c:00.0 and 0c:00.1)) <-- may be needed, you can try without at first

 

Reboot unraid and start the vm

Edited September 18, 20214 yr by ghost82

15 hours ago, TCMapes said:

Note:  I did not dumb my own vbios

 

22 minutes ago, TCMapes said:

unless i can dumb from inside unraid

It's dump not dumb

Dumb means stupid ahah

Grammar Police...   Thanks I am attempting the fix now.  Fingers Crossed.  I post again if that did the trick

GHOST82.... Thanks for the assist.  It worked.  I didnt even need to present the vm with rom.