Hi,

 

I've got issue with MacOS Big Sur VM, I've followed SpaceInvader's One guide to the letter but no succes when passing through GPU.

The VM starts successfully with GPU and sound passthrough but (hdmi) display does NOT detect any signal at all from the VM. And for example on windows 11 VM everything works just fine with passthrough etc. but not MacOS.

 

How can I fix the issue? or rather what is the problem that prevents display output?

 

Here is a log from the VM, there is a few errors:

 

-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=34,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-device 'vfio-pci,host=0000:0e:00.0,id=hostdev0,bus=pci.3,addr=0x0,romfile=/mnt/user/isos/vbios/rx 480v2.rom' \
-device vfio-pci,host=0000:0e:00.1,id=hostdev1,bus=pci.4,addr=0x0 \
-device usb-host,hostbus=5,hostaddr=5,id=hostdev2,bus=usb.0,port=2 \
-usb \
-device usb-kbd,bus=usb-bus.0 \
-device '************************' \
-smbios type=2 \
-cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-07-21 14:48:35.448+0000: Domain id=12 is tainted: high-privileges
2021-07-21 14:48:35.448+0000: Domain id=12 is tainted: custom-argv
2021-07-21 14:48:35.448+0000: Domain id=12 is tainted: host-cpu
char device redirected to /dev/pts/3 (label charserial0)
2021-07-21T14:48:35.516052Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.517561Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.518593Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.519424Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.520368Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.521435Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.522396Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.523457Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.524552Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.525444Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.526346Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.527252Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.528167Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.529115Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.530040Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:35.530979Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-21T14:48:39.912666Z qemu-system-x86_64: vfio_err_notifier_handler(0000:0e:00.1) Unrecoverable error detected. Please collect any data possible and then kill the guest
2021-07-21T14:48:39.912778Z qemu-system-x86_64: vfio_err_notifier_handler(0000:0e:00.0) Unrecoverable error detected. Please collect any data possible and then kill the guest
2021-07-21T14:53:42.531522Z qemu-system-x86_64: libusb_release_interface: -4 [NO_DEVICE]
2021-07-21T14:53:42.531630Z qemu-system-x86_64: vfio_err_notifier_handler(0000:0e:00.1) Unrecoverable error detected. Please collect any data possible and then kill the guest
2021-07-21T14:53:42.532672Z qemu-system-x86_64: vfio_err_notifier_handler(0000:0e:00.0) Unrecoverable error detected. Please collect any data possible and then kill the guest

Edited January 6, 20224 yr by SimpleDino

All I can see from the data you attached is wrong topology:

18 hours ago, SimpleDino said:

-device 'vfio-pci,host=0000:0e:00.0,id=hostdev0,bus=pci.3,addr=0x0,romfile=/mnt/user/isos/vbios/rx 480v2.rom' \
-device vfio-pci,host=0000:0e:00.1,id=hostdev1,bus=pci.4,addr=0x0 \

You have video on bus 3 and audio on bus 4.

 

Replace with this and make sure to dump your own rom and maybe the unrecoverable errors will go away (or may be related to amd reset bug, but starting from a fresh unraid boot you should see at least some video output on first vm startup) :

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/rx 480v2.rom'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
    </hostdev>

 

Edited July 22, 20214 yr by ghost82

Hi thanks for finding the error, I will try the change now.

Regarding vbios and amd bug, I have delt with those issues, the vbios is being dumped with the Spaceinvader method and amd gpu reset bug also.

Is there any extra change needed for the vbios? I thought not when dumping your own?

 

Br,

 

3 minutes ago, SimpleDino said:

I thought not when dumping your own?

it's ok.

Fix the bus and try.

Hi,

 

I tried it now after the changes and a amd gpu bug reset but no display output , one thing to mention is that the gpu is in the primary gpu slot. Does that affect anything?

 

Here is the new log:

 

-device e1000-82545em,netdev=hostnet0,id=net0,mac=52:54:00:e4:da:ce,bus=pci.1,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=34,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-device 'vfio-pci,host=0000:0e:00.0,id=hostdev0,bus=pci.3,multifunction=on,addr=0x0,romfile=/mnt/user/isos/vbios/rx 480v2.rom' \
-device vfio-pci,host=0000:0e:00.1,id=hostdev1,bus=pci.3,addr=0x0.0x1 \
-device usb-host,hostbus=5,hostaddr=6,id=hostdev2,bus=usb.0,port=2 \
-usb \
-device usb-kbd,bus=usb-bus.0 \
-device '************************' \
-smbios type=2 \
-cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-07-22 11:37:52.547+0000: Domain id=13 is tainted: high-privileges
2021-07-22 11:37:52.547+0000: Domain id=13 is tainted: custom-argv
2021-07-22 11:37:52.547+0000: Domain id=13 is tainted: host-cpu
char device redirected to /dev/pts/2 (label charserial0)
2021-07-22T11:37:52.614986Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.616758Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.618153Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.619304Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.620615Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.621913Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.622898Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.623867Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.624902Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.625784Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.626708Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.627566Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.628506Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.629434Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.630358Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:52.631286Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2021-07-22T11:37:58.257929Z qemu-system-x86_64: vfio_err_notifier_handler(0000:0e:00.1) Unrecoverable error detected. Please collect any data possible and then kill the guest
2021-07-22T11:37:58.258039Z qemu-system-x86_64: vfio_err_notifier_handler(0000:0e:00.0) Unrecoverable error detected. Please collect any data possible and then kill the guest
2021-07-22T11:38:44.521096Z qemu-system-x86_64: terminating on signal 15 from pid 28679 (/usr/sbin/libvirtd)
2021-07-22 11:38:46.723+0000: shutting down, reason=destroyed

Do you think maybe multifunction=on should be applied on the sound also in the xml?

8 minutes ago, SimpleDino said:

one thing to mention is that the gpu is in the primary gpu slot

 

8 minutes ago, SimpleDino said:

2021-07-22T11:37:58.258039Z qemu-system-x86_64: vfio_err_notifier_handler(0000:0e:00.0) Unrecoverable error detected. Please collect any data possible and then kill the guest

 

Maybe...the unrecoverable error doesn't sound good.

The hdmi sound must not have multifunction, multifunction is applied to video and function is changed to audio as it is in the snipped I attached.

Do you have only one video card?

Does unraid uses this gpu and then that gpu tries to switch to the vm?

Can you attach a diagnostic unraid file after you start the vm, so that it includes in the log that unrecoverable error?

 

Does the vm work with vnc (no gpu passthrough)?

 

Can you also try with no rom file at all?

Edited July 22, 20214 yr by ghost82

No, the system has two video cards.

• 1st primary slot - AMD RX 480
  • Connected to display with HDMI cable (works perfectly when win11 vm is running)
• 2nd secondary slot - Nvidia P2000 with DP Dummy Plug (used for plex transcoding only)

I absolutely have no idea if unraid switches between these, all I can mention when no vm is running the display shows no signal detected and as mentioned above same message is diplayed if macos is running or not.

 

Please see attached diag.

tower-diagnostics-20210722-1350.zip

I don't know...I'm sorry I can't be of any help..If I were you I would try to change the pcie slot as a last chance..

On the contrary, you've been of great help!

I will try the slot change when I have the time.

 

Thanks!

@ghost82

The issue has been solved, again thanks for the help.

 

The changes I made and steps taken:

 

1. Changed the GPU to the 2nd slot and re-dumped the vbios.

2. Input the edit text from you and changed of course from 0e to 0f accordingly to correct slot.

On 7/22/2021 at 11:46 AM, ghost82 said:

All I can see from the data you attached is wrong topology:

You have video on bus 3 and audio on bus 4.

 

Replace with this and make sure to dump your own rom and maybe the unrecoverable errors will go away (or may be related to amd reset bug, but starting from a fresh unraid boot you should see at least some video output on first vm startup) :

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/rx 480v2.rom'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
    </hostdev>

 

3. Removed the USB mouse also and replaced it with a USB Logitec unifier with mouse and keyboard. (maybe this also eliminated some errors?)

4. RUN AMD reset bug Script so no reboot needed.

4.Always made sure to RUN the macos vm_helper script after changes.

5. Succes! display output works perfectly!

6. For remote access (outside of home network), I've setup teamviewer which works flawlessly. Better than NoMachine, no port forwarding needed.

 

Issues left, how to make the vm autoboot without me needing to press enter in bootloader.

 

Best regards,

 

27 minutes ago, SimpleDino said:

@ghost82

The issue has been solved, again thanks for the help.

 

The changes I made and steps taken:

 

1. Changed the GPU to the 2nd slot and re-dumped the vbios.

2. Input the edit text from you and changed of course from 0e to 0f accordingly to correct slot.

3. Removed the USB mouse also and replaced it with a USB Logitec unifier with mouse and keyboard. (maybe this also eliminated some errors?)

4. RUN AMD reset bug Script so no reboot needed.

4.Always made sure to RUN the macos vm_helper script after changes.

5. Succes! display output works perfectly!

6. For remote access (outside of home network), I've setup teamviewer which works flawlessly. Better than NoMachine, no port forwarding needed.

 

Issues left, how to make the vm autoboot without me needing to press enter in bootloader.

 

Best regards,

 

happy you solved it.

To hide opencanopy (the bootloader gui) and automatically boot mount the efi in mac os, go to EFI/OC open config.plist with a text editor, search for ShowPicker and set it to false.

Aha thanks, will make sure to try that when I'm near the Unraid PC again, can not do it remotely because of the amd reset bug unfortunately:(.

 

Or do you think it's better to make a delay in " Misc > Boot > Timeout?

Edited July 23, 20214 yr by SimpleDino
other solution

12 hours ago, SimpleDino said:

Or do you think it's better to make a delay in " Misc > Boot > Timeout?

It's your choice.

I have ShowPicker=false too, if you need the picker to show you can just press (let it pressed) esc or 0 "zero" when the vm boots and the picker will show (0 (zero) was added later because if you press esc too early it will show the ovmf bios settings, so better to press 0 (zero)).

Otherwise you can let ShowPicker=true and opencanopy will always show, and the default boot entry will automatically boot after xx seconds, defined in Timeout.

In both ways, if something goes wrong, you will always able to show the picker.

Edited July 24, 20214 yr by ghost82

12 hours ago, SimpleDino said:

can not do it remotely because of the amd reset bug

There's the vendor-reset patch:

https://github.com/gnif/vendor-reset/

 

It's a patch for the kernel, there's also a kernel helper/builder for unraid to build a kernel with the included patch:

 

However note that this may not work for mac os guests, depending on the gpu.

14 minutes ago, ghost82 said:

However note that this may not work for mac os guests, depending on the gpu.

This also works for OSX VMs I heard...

@ghost82

Sorry to bother you with even more troubleshooting. I have some issues with the VM not outputting any display picture again.

 

What has has changed since last I got the display output working with your help, are basically these:

• I split up the IOMMU grouping in stages per SP1 instruction video until I got the BT devices separate and then VFIO bind. After that I disabled PCIe ACS override again.
• Added the onboard Wi-Fi & BT to the VM (Intel AX200), run the script and afterward made the necessary changes in the XML again. (works perfectly in VNC mode)

Now even if I remove the Wi-Fi & BT devices I do not get the display output back (of course script run and the xml change as before). But works when in VNC mode.

 

I have added the XML, Template and vm logs.

XML-Monterey+GPU+WiFi.xml Logs--Monterey+GPU+WiFi.txt

Edited December 22, 20214 yr by SimpleDino

Please attach the diagnostics file after you run the vm with passthrough(s).

@ghost82

Of course, here it is.tower-diagnostics-20211223-1017.zip

20 hours ago, SimpleDino said:

I split up the IOMMU grouping in stages

 

20 hours ago, SimpleDino said:

After that I disabled PCIe ACS override again

I think your issue is this.

Why did you disable after you set all the things?

Either disable acs override and set your things, or enable acs override and do the same.

Do not enable, set and disable, or disable, set and enable!

 

General rules:

1. enable or disable acs override: I would enable it, it was never an issue for me

2. attach to vfio (checkmarks) all the devices you want to passthrough to the vm

3. manually edit the xml to change multifunction: for gpu (audio and video) and for the usb controller (the structure of your xml looks good, but check changes in hardware addresses once you set everything up again)

 

Reboot and try

Yet again, thanks for the input. I will give this a try now and see how it goes.

I re-did everything but it did not help. Thinking of doing backup of the vm and then start fresh with a new install all over again.

2 hours ago, SimpleDino said:

I re-did everything but it did not help

do what suggested and attach diagnostics again.

@ghost82

Sorry for the delay: I tried with 6.10.0 RC2 but that was hopeless because everything crashes, also UnRAID server immediately when I try to access anything that has to with VMS.

 

I have done as you suggested and here are the info plus diagnostics.

 

tower-diagnostics-20220103-0853.zipbigsur XML.xmlBigSur VM Logs.txt

Can you attach to vfio at boot also the gpu (0f:00.0 and 0f:00.1)?

You have only the usb controller and the network controller (this however is inside a iommu group with other devices).

When it worked, did you passthrough the usb controller?