GPU resizable BAR support is coming in Linux kernel 6.1. It'd be great to support this natively in Unraid, somehow in the VM GUI or a setting or at least a documented script or how-to.
The patch allows you to manipulate the bar size through sysfs. See the example how to use it in this reddit post.
This is the User Script I created to set the bar size to 16GB. You would obviously need to tweak this to the proper bar size, pice address, folder path, device id, etc.
#!/bin/bash
echo -n "0000:0d:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo 14 > /sys/bus/pci/devices/0000\:0d\:00.0/resource1_resize
echo -n "10de 2782" > /sys/bus/pci/drivers/vfio-p