Jump to content

I'm having an odd issue with AMD gpu passthrough to macOS VM. The GPU is sort of working, never seen this issue before.

relink

By relink
in VM Engine (KVM)

Recommended Posts

relink Explorer

relink

Posted
Posted

Some help would be greatly appreciated, I have been at this for over a day now, and tried 2 different GPU's with the exact same results.

 

My unraid host is:

Ryzen 5 2600

80GB DDR4

Unraid 6.10-rc4

 

I have setup macOS Monterey using the @SpaceInvaderOne Macinabox container. I left everything to default, and got the VM up and running no problem. I used the built in VNC to get macOS setup, create an account, etc. I then changed the allocations to 4 cpus and 12GB of ram, ran the helper script, and started the VM to verify the changes took, and they did. So using VNC macOS runs fine.

 

I am trying to passthrough my AMD HD7870 to the macOS VM, this card is still fully and officially supported in the latest versions of macOS, and I have used it in several bare-metal Hackintoshes before with zero issues.

 

I have the ACS override patch enabled and set to both, I went to "System Devices" and checked both the GPU and its Audio to stub to vfio and rebooted. I then dumped the vbios and added the GPU, vbios, and audio to the VM.

 

Here is the problem, if I remove the VNC graphics adapter and start the VM I just get a blank screen on the monitor attached to the HD7870, the monitor doesn't even wake up. I have tried both HDMI and VGA (Using DVI to VGA adapter), Id try DVI but I cant find my DVI cable right now.

 

However if I keep both the VNC adapter and the HD7870 then I get the boot picker and macOS boot process on the VNC adapter and while the VNC adapter is showing the boot process the HD7870 is doing nothing, the monitor doesn't even come out of standby. Once macOS has finished booting the VNC adapter just continues to show the last part of the boot process forever, but now the monitor attached to the HD7870 wakes up but all I see is the default macOS Monterey purple wallpaper.

 

I installed the AMD vendor reset patch and rebooted unraid, now when I boot with just the HD7870 I still get nothing same as before. But when I boot with VNC and the HD7870, I still get nothing on the HD7870, and the VNC adapter now just shows the message "Guest has not initialized the display (yet)."

 

I tried the exact same steps with an RX570 and got identical results. I'm not really sure what else to try at this point, either of these GPUs should have literally been plug and play with macOS...

 

Below is the VM log and unraid diag with the VM running using VNC and the HD7870.

  

XDG_CONFIG_HOME='/var/lib/libvirt/qemu/domain-3-Macinabox Monterey/.config' \
/usr/local/sbin/qemu \
-name 'guest=Macinabox Monterey,debug-threads=on' \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-3-Macinabox Monterey/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/system/custom_ovmf/Macinabox_CODE-pure-efi.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/5ad5f709-3506-4953-b339-152a9fa799c3_VARS-pure-efi.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-4.2,usb=off,dump-guest-core=off,mem-merge=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-accel kvm \
-cpu host,migratable=on,topoext=on,host-cache-info=on,l3-cache=off \
-m 12288 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":12884901888}' \
-overcommit mem-lock=off \
-smp 4,sockets=1,dies=1,cores=2,threads=2 \
-uuid 5ad5f709-3506-4953-b339-152a9fa799c3 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=35,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":8,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x1"}' \
-device '{"driver":"pcie-root-port","port":9,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x1.0x1"}' \
-device '{"driver":"pcie-root-port","port":10,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x1.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"pcie-root-port","port":11,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x1.0x3"}' \
-device '{"driver":"pcie-pci-bridge","id":"pci.7","bus":"pci.1","addr":"0x0"}' \
-device '{"driver":"ich9-usb-ehci1","id":"usb","bus":"pcie.0","addr":"0x7.0x7"}' \
-device '{"driver":"ich9-usb-uhci1","masterbus":"usb.0","firstport":0,"bus":"pcie.0","multifunction":true,"addr":"0x7"}' \
-device '{"driver":"ich9-usb-uhci2","masterbus":"usb.0","firstport":2,"bus":"pcie.0","addr":"0x7.0x1"}' \
-device '{"driver":"ich9-usb-uhci3","masterbus":"usb.0","firstport":4,"bus":"pcie.0","addr":"0x7.0x2"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.2","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Macinabox Monterey/Monterey-opencore.img","node-name":"libvirt-3-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"}' \
-device '{"driver":"ide-hd","bus":"ide.2","drive":"libvirt-3-format","id":"sata0-0-2","bootindex":1,"write-cache":"on"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/Monterey-install.img","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
-device '{"driver":"ide-hd","bus":"ide.3","drive":"libvirt-2-format","id":"sata0-0-3","write-cache":"on"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Macinabox Monterey/macos_disk.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.4","addr":"0x0","drive":"libvirt-1-format","id":"virtio-disk4","write-cache":"on"}' \
-netdev tap,fd=37,id=hostnet0,vhost=on,vhostfd=38 \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:3b:a6:7d","bus":"pci.3","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0"}' \
-chardev socket,id=charchannel0,fd=39,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:0,websocket=5700,audiodev=audio1 \
-k en-us \
-device '{"driver":"qxl-vga","id":"video0","ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"max_outputs":1,"bus":"pci.7","addr":"0x1"}' \
-device '{"driver":"vfio-pci","host":"0000:07:00.0","id":"hostdev0","bus":"pci.5","addr":"0x0","romfile":"/mnt/user/isos/vbios/HD7870_vbios.rom"}' \
-device '{"driver":"vfio-pci","host":"0000:07:00.1","id":"hostdev1","bus":"pci.6","addr":"0x0"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/001/005","id":"hostdev2","bus":"usb.0","port":"2"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/001/003","id":"hostdev3","bus":"usb.0","port":"3"}' \
-usb \
-device usb-kbd,bus=usb-bus.0 \
-device '************************' \
-smbios type=2 \
-cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+fma4,+bmi1,+bmi2,+xsave,+xsaveopt,+rdrand,check \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
2022-04-20T14:58:25.992004Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2022-04-20T14:58:25.992094Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
2022-04-20T14:58:25.993776Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2022-04-20T14:58:25.993795Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
2022-04-20T14:58:25.995221Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2022-04-20T14:58:25.995239Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
2022-04-20T14:58:25.996604Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2022-04-20T14:58:25.996621Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.fma4 [bit 16]
qxl_send_events: spice-server bug: guest stopped, ignoring
2022-04-20T15:01:55.412615Z qemu-system-x86_64: libusb_release_interface: -4 [NO_DEVICE]

 

serverus-diagnostics-20220420-1104.zip

Link to comment
ghost82 Proficient

ghost82

Posted
Posted (edited)

Mac os doesn't like qxl video on a bus different than 0 (i.e.: built-in), that's probably why you are getting the "guest has not initialized the display"; moreover looking at your xml you attached the qxl device to a "pcie to pci bridge", instead of a "pcie root port". The whole layout needs to be re-arranged.

Moreover the gpu (digital audio/video) is not set as multifunction.

I don't know if mac os likes qxl (and vnc) + gpu passthrough (it depends on the smbios you chose, if the real mac you are emulating has an integrated gpu then it's ok, otherwise not), if it doesn't work try also to delete vnc+qxl blocks from the xml, i.e. delete this from the xml in this post, below: 

    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>

 

Replace the whole xml with this: 

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit 5ad5f709-3506-4953-b339-152a9fa799c3
or other application using the libvirt API.
-->

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Macinabox Monterey</name>
  <uuid>5ad5f709-3506-4953-b339-152a9fa799c3</uuid>
  <description>MacOS Monterey</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="Monterey.png" os="osx"/>
  </metadata>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>12582912</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='6'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/mnt/user/system/custom_ovmf/Macinabox_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/5ad5f709-3506-4953-b339-152a9fa799c3_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='2' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Macinabox Monterey/Monterey-opencore.img'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/isos/Monterey-install.img'/>
      <target dev='hdd' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='3'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/mnt/user/domains/Macinabox Monterey/macos_disk.img'/>
      <target dev='hde' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:3b:a6:7d'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/HD7870_vbios.rom'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x04f2'/>
        <product id='0x1053'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1a2c'/>
        <product id='0x2c27'/>
      </source>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-usb'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='usb-kbd,bus=usb-bus.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='isa-applesmc,osk=...
    <qemu:arg value='-smbios'/>
    <qemu:arg value='type=2'/>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+fma4,+bmi1,+bmi2,+xsave,+xsaveopt,+rdrand,check'/>
  </qemu:commandline>
</domain>

 

Readd the osk at the bottom.

Edited by ghost82
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing
×
×
  • Create New...