vigs

Dropping my solution in here for everyone. I've only been testing this for the last few hours but it seems to survive reboots and work automatically. Few changes from the solution above. I use a udev rule to bring up the thunderbolt interface any time a thunderbolt cable is connected. That same rule also adds the interface to br0 so it has access to the rest of the network through the unraid box. That has the added benefit of not needing to use static ips on the thunderbolt interface or the mac. The Mac can just get an ip from DHCP and you don't need to mess with separate subnets. Finally, it sets the mtu of the thunderbolt interface to 9000 which squeezes out a little extra performance especially at 40g speeds. I also set the bridge to spanning tree just in case there could be any loops since the mac would also be connected to the network with wifi. Unsure if that's necessary but it's working for me so I'm rolling with it. my udev rule that i save at /boot/rules/99-thunderbolt.rules SUBSYSTEM=="net", KERNEL=="thunderbolt0", ACTION=="add", \ RUN+="/sbin/ip link set thunderbolt0 up", \ RUN+="/sbin/ip link set thunderbolt0 master br0", \ RUN+="/sbin/ip link set dev br0 type bridge stp_state 1", \ RUN+="/sbin/ip link set thunderbolt0 mtu 9000" I added the following to my /boot/config/go to do all the same stuff at boot as well as copy over the udev rule and enable it so that it works until next reboot # Ensure the thunderbolt0 interface is up and part of the br0 bridge /sbin/ip link set thunderbolt0 up /sbin/ip link set thunderbolt0 master br0 /sbin/ip link set dev br0 type bridge stp_state 1 /sbin/ip link set thunderbolt0 mtu 9000 # Copy the thunderbolt rules file cp /boot/rules/99-thunderbolt.rules /etc/udev/rules.d/ chmod 644 -R /etc/udev/rules.d/99-thunderbolt.rules 2> /dev/null # reload udev rules udevadm control --reload-rules udevadm trigger So far it's working for me and I hope it works for you. Please improve on it if you know more than I do! (I don't know much...this is a combination of ChatGPT and some other helpful posts I found)

I realize this is a rare use case but I'd also like to see this. Would love to have it show up as an interface on the networking page and allow bridging so the downstream device can get access to the network through the unraid box.

I had the same question and found some guidance here:

I feel like this thread is mostly dead but I've got two separate questions. a) I can't figure out how to get macos to boot on my system with "-cpu host" in the qemu args. Boots fine with the old school Penryn values. With -cpu host, hangs at the exitbs screen, in other words, never gets out of the boot screen, i think. b) updating to anything newer than 6.10.3, my working VM gives me a black screen, seems to be in some sort of loop where one core maxes, goes quiet, then all cores max, goes quiet, etc. Never get display out and can't screen share from another mac on the network. This xml works fine in 6.10.3 But if i change to <qemu:arg value='-cpu'/> <qemu:arg value='host,kvm=on,+hypervisor,+invtsc'/> it hangs on EXITBS And it just always hang if i upgrade to anything newer than 6.10.3. Current config.plist

Hey all, I had a perfectly fine, functioning Big Sur install running directly on a passed through NVME with RX 580 passed through. I was mucking around with trying to upgrade my cache drive (didn't actually finish because it wasn't working for me), restored everything back to their cache locations. My three other VMs are working just fine (windows, arch, hassio) but Big Sur is now kicking back a "operation failed: unable to find any master var store for loader: /mnt/user/system/custom_ovmf/Macinabox_CODE-pure-efi.fd" error. I've confirmed both the CORE and VARS efis are in the proper location. I also tried downloading them from the SIO github and replacing them to make sure they weren't corrupt. Still no dice. For what it's worth, I've searched for the nvram efi associated with the UUID of the VM and I can't find it anywhere on the machine. Is that the source of my problem? Should I create a new VM to create a new UUID and NVRAM and then just port over my existing working xml? Any ideas? EDIT SOLVED I answered my own question....it was the missing machine nvram. Once I had some time to tinker, I recreated the VM, brought over all my custom XML and poof it's working. I'm starting to get a hang of this.

I know you've all been waiting for me to chime back in but we have success! Windows, MacOS, Ubuntu all up and running. Switched to an RX 580 to avoid the reset bug issue with the R9 290X and it seems to work fine. Only a couple more wrinkles to iron out. The HDMI audio doesn't appear in any of the VMs despite passing through as multifunction. Ultimately, I'll be able to live without it but would be nice to get it working. Doesn't appear in Device manager in windows, system profiler in MacOS, nothing. Also, in MacOS, the GPU is showing up as an RX 480 w/ 4GB rather than my 580 8GB. I've tried passing thought the bios dumped from the card and no bios. Same result. Any ideas? Windows 10 xml: MacOS xml:

Okay, some research has taught me it's not the reset bug since I do not get that execution error. Tried SI1's script anyway, hoping that it would mimic whatever rebooting the server is going. Didn't work. Still get the Code 43 and need to reboot the server to get the GPU passed through again. So, since other's seem to have luck with passthrough with a 290x, I'm hoping it's something simpler. I'll attach my latest xml and diagnostics. My next step, unrelated to this problem specifically, is to disable the GPU during boot so I, hopefully, don't need the spare GPU in there. Then I can start attempt booting macOS again and see if I have and better luck after figuring out the BIOS problem. Any help with the Code 43 after a graceful windows shutdown would be most appreciated! <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows 10_2</name> <uuid>d953a484-fee6-229f-f7da-347bb388981f</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>16</vcpu> <cputune> <vcpupin vcpu='0' cpuset='20'/> <vcpupin vcpu='1' cpuset='48'/> <vcpupin vcpu='2' cpuset='21'/> <vcpupin vcpu='3' cpuset='49'/> <vcpupin vcpu='4' cpuset='22'/> <vcpupin vcpu='5' cpuset='50'/> <vcpupin vcpu='6' cpuset='23'/> <vcpupin vcpu='7' cpuset='51'/> <vcpupin vcpu='8' cpuset='24'/> <vcpupin vcpu='9' cpuset='52'/> <vcpupin vcpu='10' cpuset='25'/> <vcpupin vcpu='11' cpuset='53'/> <vcpupin vcpu='12' cpuset='26'/> <vcpupin vcpu='13' cpuset='54'/> <vcpupin vcpu='14' cpuset='27'/> <vcpupin vcpu='15' cpuset='55'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-6.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/d953a484-fee6-229f-f7da-347bb388981f_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='8' threads='2'/> <cache mode='passthrough'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='writeback'/> <source file='/mnt/user/domains/Windows 10_2/vdisk1.img'/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso'/> <target dev='hdb' bus='sata'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xd'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:e1:38:8e'/> <source bridge='br0'/> <model type='virtio-net'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/r9_290x_vbios.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc52b'/> </source> <address type='usb' bus='0' port='1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x05ac'/> <product id='0x024f'/> </source> <address type='usb' bus='0' port='2'/> </hostdev> <memballoon model='none'/> </devices> </domain> tower-diagnostics-20220107-1824.zip

Big update! After trying seemingly every permutation of ovmf/seabios/machine type and then updating to 6.10rc2 just to install the AMD Vendor Reset plugin because why not....I tried playing around with other GPUs. With two GPUs in machine, I now saw the primary graphics option in my bios. With another GPU selected as primary, my 290x will now passthrough to Windows successfully.....once. Off a fresh reboot of unraid, I can launch Windows and passthrough the 290x. Once I shutdown the VM and try again, no dice. When I log in with splashtop I see it's showing the Code 43 again. After a reboot of unraid, I can passthrough again. I feel like I'm now running into the reset bug? Is that possible? I see several reports of people passing through a 290x with nary a mention of the reset bug but I'm not sure what else to consider at this point. I have the vendor reset plugin installed but where the 290x isn't listed as a supported GPU, well, I suspect it's not supported and the plugin is doing nothing for me. Have I hit the end of the road? Is anyone else successfully passing through a R9 290X to windows without this issue? Furthermore, how do I get around needing another GPU? I've already been binding the 290x to VFIO. I gather I could prevent the GPU from loading into Unraid entirely but, at least during this setup/troubleshooting phase, it's pretty useful to have some output at boot.

Ah, I grabbed that from techpowerup. I've dumped my own but It gives me the same issue regardless of the rom. That happened to be the one I was testing that time. I've tried several roms from there and roms pulled from the card. The card has a rom switch, I don't know what it's supposed to be in either setting. I thought one was supposed to be a UEFI only rom but in either position it shows output while booting unraid in legacy mode until the VFIO grabs it.

Alright, I'm back, pumped and ready for the new year. Thanks for your help so far and I tried to figure out all I could before coming back for more. And here I am. Tried bumping the ram and the boot arg but neither helped but i don't think they mattered. Needed to install some splashtop drivers for headless operation while the GPU wasn't being recognized properly but now I'm getting code 43 errors Running Q35 chipsets and OMVF since I hope to adapt any working config to a future macos install. I've gone through every SI1 video as well as countless threads and I feel like Ive tried the normal stuff...bioses native and from techpowerup. Starting to think my 290x just hates this box and should try to get my hands on another gpu but I'm really trying to get what I have in hand working before investing more in other hardware. I'm also thinking (but Im not sure) that it's an 8gb version of the 290x which might make it not play nice with many drivers? But I also know it works just fine on bare metal windows. Hoping I can get it going but I also suspect that might cause troubles with macos. I know Ive seen people on here talking about using unraid based machines in production environments using davinci and/or premiere/ae and maybe at this point I could use a tag to those folks. Since I'm building this on a z840, should be the kind of machine production folks have dealt with a lot. Open to any/all advice! Latest diagnostics attached. tower-diagnostics-20220102-2238.zip

Ooops, thought I attached. Should be attached now. Have attached the GPU to vfio in system devices. I should note that in ubuntu, it happily runs with or without the rom file attached. I've rebuilt the windows VM several times as well as reinstalled windows a couple times just to make sure it's as clean as possible. I've tried to copy the working xml from the ubuntu VM for windows and it just keeps doing the black screen thing. syslog.txt

Hey gang...long time listener, first time caller. Goal: Video editing machine w/ various virtualized extras since there's a powerful machine sitting doing nothing most of the time. Extra goal: macos. Hardware: HP Z840 Sapphire Radeon R9 290X Tri-X OC (can/will change to something more modern just tryna get it going to make sure what i want is possible) (also have an old AMD and NVIDIA card kicking around...super old, amd 2600xt, dunno about the nvidia right now but assume worse) Had been trying to get the 290x working on it's own, just added the 2600xt back in to try to figure things out Successes: Booting and running ubuntu 20.04 w/ 290x passthrough!!! 😀😀😀 Booting and running windows w/ VNC Booting and running macos w/ 2600xt, i think....generally have put the mac side of it aside while trying to figure out windows. Failures: Everything else. Namely: Windows black screen w/ 290x passthrough. Responds to virsh commands, etc, just no graphics. VNC (via tightVNC) and Splashtop just show a black screen. Macos featured similar failure but I figured it'd be easier to figure out windows first since ubuntu seems happy enough. Caveats: Im a super unraid/virtualization noob yet moderately linux/command line experienced. The important takeaway is that in ubuntu, it works fine. In windows, nothing. Below is my typical section for the gpu. <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/r9_290x_vbios.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc52b'/> </source> <address type='usb' bus='0' port='1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x05ac'/> <product id='0x024f'/> </source> <address type='usb' bus='0' port='2'/> </hostdev> Love you all....hugs and kisses