So this thread originally fixed my issue, putting the following in my go file made passthrough work:
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
But now after upgrading to 6.9.2 I can't use it anymore, something about that third command sends my tower in to complete lockdown and I have to manually shut it down, I had to take out my USB and remove the lines from the Go file to get it to even boot properly. I'm still getting the MMAP errors because the GPU is being used by Unraid and I can't unbind it.