Skitals

The rebar patch was backported to unraid 6.11.5 if you want to tinker with it sooner.

Yes, your gpu has a usb controller but no port. Some gpus have usb-c for vr and other uses. There is no adapter that will give you access to it. And yes, any devices connected to the controller would only be visible/usable inside a VM if the controller is bound to vfio-pci. So unfortunately no way to make use of it. Best practice is still to passthrough all sub-devices together with multifunction on (54:00.0, 54:00.1, 54:00.2, 54:00.3 on the same bus, etc). Typically they would be in the same iommu group, what do your groups look like with acs overrides disabled?

Thanks. Was getting "Not available" for Dozzle and this fixed it.

Do you have Above 4G Decoding enabled in your bios?

Like I said above, you are trying to set the bar to 16GB when you have 8GB VRAM. Read the reddit post and learn your cards bar options and learn to manipulate it manually before trying to use a script.

Is the pci address of your GPU really the same as mine? The GPU needs to be unbound. The previous line in my script unbinds it from vfio, but it also can't be bound to an Nvidia driver or anything else. I also believe you need Above 4G Decoding enabled in your bios, but I'm not sure if that would give you the error you are seeing. Before referencing my script you should really read and understand the reddit post I linked. You need to be able to read and manipulate the bar sizes on your own because the addresses, paths, number of bars etc will be different than mine. Edit: I just looked up your video card and see it has 8GB VRAM. Why are you trying to set the bar size to 16GB? Like I said you really need to understand the reddit post I linked before preceding.

Also, you can check if windows is using the larger bar size from Device Manager. Open the GPU and go to the Resources tab. Look for a "Large Memory Range". You can calculate the size by subtracting the second number in the range from the first in a hex calculator. For me that is 13FFFFFFFF - 1000000000 = 17179869183 bytes = 16 gigabytes.

The patch allows you to manipulate the bar size through sysfs. See the example how to use it in this reddit post. This is the User Script I created to set the bar size to 16GB. You would obviously need to tweak this to the proper bar size, pice address, folder path, device id, etc. #!/bin/bash echo -n "0000:0d:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind echo 14 > /sys/bus/pci/devices/0000\:0d\:00.0/resource1_resize echo -n "10de 2782" > /sys/bus/pci/drivers/vfio-pci/new_id || echo -n "0000:0d:00.0" > /sys/bus/pci/drivers/vfio-pci/bind # Bit Sizes # 1 = 2MB # 2 = 4MB # 3 = 8MB # 4 = 16MB # 5 = 32MB # 6 = 64MB # 7 = 128MB # 8 = 256MB # 9 = 512MB # 10 = 1GB # 11 = 2GB # 12 = 4GB # 13 = 8GB # 14 = 16GB # 15 = 32GB As stated the patch and the script works, but it was ultimately unneeded with my 4070 Ti since passthrough works when I have Above 4G and Resize Bar enabled in my bios. With those enabled the 4070 Ti defaults to a 16GB bar size, so there is no need to manually change it. Other setups/GPUs will give a code 43 or not passthrough at all with Resize Bar enabled in the bios, which is where this patch would be helpful to manually resize the bar. In that case I'm not sure what GPU-Z would report. The only way to know if it's making a difference is to benchmark a game known to benefit from resize bar with and without manipulating the bar size.

Thanks. Merged and should be live in Community Apps. Edit: Should be live now.

I was curious so I did a few benchmarks passing all 32 cores/threads to the VM. Forza Horizon 5 1440p Extreme (No DLSS) VM Rebar OFF 20cpu: 116 fps VM Rebar ON 20cpu: 129 fps VM Rebar ON 32cpu: 134 fps Bare metal Rebar ON: 144 fps Cyberpunk 1440p RT Ultra (DLSS): VM Rebar OFF 20cpu: 81.07 fps VM Rebar ON 20cpu: 95.29 VM Rebar ON 32cpu: 98.26 Bare metal Rebar ON: 102.21 That's pretty dang close to bare metal performance with full resizable bar, given the extra overhead from unraid and vfio. Hitting 129 fps in the vm in Forza is amazing when with the 7900XTX I could never beat 114 fps with identical settings.

I did a little more testing. It looks like the patch isn't needed with the RTX 40 Series. If I boot unraid with Resize Bar Support OFF, bar size is 256MB. If I boot unraid with Resize Bar Support ON, bar size is the max 16GB even when the gpu is bound to vfio on boot. With the patch I can leave Resize Bar Support OFF (but Above 4G Decoding ON) and manually resize the bar and have the same effect. This is another huge win for Nvidia over the AMD XTX which I couldn't passthrough at all when Resize Bar Support was enabled in the bios. Same as resizing the bar manually, sometimes I need to reboot the windows vm to see the larger bar. From the Device Manager > select your GPU > Resources tab look for a "Large Memory Range"

I ditched the 7900 XTX. It was too many headaches, worse than the navi reset bug. I picked up a 4070 Ti today and passthrough just works, first time every time. I've been team red for a long time, but the 7000 series is not cut out for vms.

I ditched the 7900 XTX for a 4070 Ti and I'm seeing encouraging results. Forza Horizon 5 1440p Extreme (No DLSS) VM 256MB Bar: 116 fps VM 16GB Bar: 129 fps Baremetal Rebar Off: 127 fps Baremetal Rebar On: 144 fps Cyberpunk 1440p RT Ultra (DLSS): VM 256MG Bar: 81 fps VM 16GB Bar: 95 fps Baremetal Rebar Off: 94 fps Baremetal Rebar On: 102 fps Of note I went from testing in baremetal windows (with rebar on), rebooted into unraid and the bar size was still set to 16GB. I'm not sure what negotiated that or if it remembers the last setting. Setting the bar to 16GB basically brings it to parity with bare metal when rebar is OFF. Still a big performance delta, which I think it largely due to fewer cpu cores in the VM. Time Spy Graphics scores are identical in VM and baremetal (23k).

Do you ever see the Tianocore screen when booting a vm? I always did with my 5700 XT, I never see it with the 7900 XTX. When it works the first thing I see is the Windows login screen. Finicky is an understatement. I lived with the 5700 XT for 2 years with the dreaded reset bug and that was a joy in comparison. I am also seeing massive instability. I've had multiple crashes. Once the screen went black while launching 3dmark, and it recovered with Andrenalin not running. Another time the Windows VM fully locked up while I was using Chrome. Looking at the unraid dashboard there were a couple cores pegged at 100%. I had to Force stop the VM. Never seen anything like this in 2 years of using the 5700 XT and a Windows vm as my primary desktop computer. Not sure if it's driver instability or issues with using the card in a virtualized environment. I haven't used it enough on bare metal windows to know if there are stability issues there, too. The resizable bar thing is another issue. I'm used to games running within a percentage inside a vm. Games that make use of resizable bar are seeing 20% better fps on bare metal.

That seems to pertain to relocating the MSI-X bar, which isn't relevant. I gave it a try anyway and it didn't help.

I think I found the problem. Looks like by default OVMF PCI MMIO BAR allocation is 32GB, hence there isn't enough space for the GPU + other devices. Edit: Adding the xml from the link above I was able to pass the gpu with 32GB bar, but windows is only seeing 256MB. This is not unlike the first time I passed through with 8GB bar where subsequent reboots showed the full bar in windows. Except at 32GB it will not reboot and getting it to passthrough even once is finicky.

Some interesting testing and a roadblock. When resize bar is enabled and I boot windows on baremetal, the bar is being sized to the max 32GB/256MB. This sees HUGE performance gains in Forza Horizon 5. Resize bar off: 113fps Resize bar on: 139fps With the resize bar patch I can easily resize the bars to the max, however passthrough fails when bar 0 i set to 32GB. I don't get any errors, it just doesn't work. I can set bar 0 to any other size and passthrough works. With the bars set to 16GB/256MB I'm getting 114fps and improved latency. Really frustrated I can't get the vm to boot when set to 32GB to match baremetal.

I can confirm the above patch works. Below is a script to unbind my gpu from vfio, set the bar size to 8GB, and rebind to vfio: GPU=0e:00 GPU_ID="1002 744c" echo -n "0000:${GPU}.0" > /sys/bus/pci/drivers/vfio-pci/unbind || echo "Failed to unbind gpu from vfio-pci" cd /sys/bus/pci/devices/0000\:0e\:00.0/ echo 13 > resource0_resize echo -n "$GPU_ID" > /sys/bus/pci/drivers/vfio-pci/new_id The bar size goes from 256MB to 8GB, and stays there after rebinding and passing through the gpu. HOWEVER, the memory range in Windows is unchanged and I'm seeing no gains in Forza 5 (known to benefit from resizable bar). Edit: I rebooted the VM and for whatever reason on the second boot the gpu was assigned the proper 8GB address range in Windows. This time I did see a performance change... the fps in forza 5 went DOWN from 114 to 101 fps. Rebooted unraid, bar back to 256MB and fps back to 114. Time to play around with different bar sizes and see what happens.

Thanks... I got it working. What I did different I have no idea. I swear I tried this exact setup like 5 minutes in. Reset also seems to be working. That's a big relief as there are multiple reports of the xtx having the reset bug on reddit. I have stopped and started the vm multiple times and it works every time.

I back ported the resizable bar patch to unraid 6.11.5 (kernel 5.19.17). It applies and builds cleanly with the other unraid patches and unraid .config file. I attached the patch and a precompiled kernel. I can confirm it's working (resourceN_resize property present), but making it useful is another trick. resize-bar.patch unraid-6.11.5-kernel.zip

I'm having the toughest time. 7900 XTX won't pass at all no matter what I've tried. Can you share exactly what you did? Are you binding all 4 devices to vfio from the system devices page?

Did you hex edit it, or used exactly as dumped from gpu-z?

what method did you use to save your vbios?

I second this. Resizable BAR is now being leveraged by AMD, Nvidia, and Intel. More games are seeing massive performance improvements with it's implementation. Please make this a priority. I would love to see a a beta on kernel 6.1 so we can start playing around with this.

Re: AMD Vendor Reset I have been using this wonderful plugin since it was available, and was building a custom kernel with gnif's patch before that. I've never had any issues with my 5700 XT until yesterday. It seems like it crashed hard. At first it was stalling at the Tianocore splash screen, after some reboots and troubleshooting it would boot but the GPU had "stopped working" with Code 43 (low res output, drivers wouldn't load). I tried everything under the sun to fix the code 43, but no beans. The one and only fix was to boot windows on bare metal (Windows is installed on an NVME drive I pass to the VM). The GPU/AMD driver immediately worked, I rebooted into unraid and it immediately worked in the VM again. My best guess is the GPU was in some unrecoverable state. The only thing that fixed it was booting it natively. A word of advice if anyone finds themselves in a similar situation where the card suddenly stops working with no change.