Skitals
Community Developer-
Posts
201 -
Joined
-
Last visited
-
Days Won
3
Skitals last won the day on January 31 2023
Skitals had the most liked content!
Recent Profile Visitors
The recent visitors block is disabled and is not being shown to other users.
Skitals's Achievements
Explorer (4/14)
100
Reputation
-
[6.11.5] docker becomes unusable after recreating wireguard interface
Skitals commented on Skitals's report in Stable Releases
This bug continues to be the bane of my existence in 6.11.5 as I need to re-import a new wg conf every few weeks. You can get around having to recreate all your docker containers if you DISABLE docker before recreating your wg interface, but I have forgotten a few times and needed to recreate 20+ containers. -
The rebar patch was backported to unraid 6.11.5 if you want to tinker with it sooner.
-
Yes, your gpu has a usb controller but no port. Some gpus have usb-c for vr and other uses. There is no adapter that will give you access to it. And yes, any devices connected to the controller would only be visible/usable inside a VM if the controller is bound to vfio-pci. So unfortunately no way to make use of it. Best practice is still to passthrough all sub-devices together with multifunction on (54:00.0, 54:00.1, 54:00.2, 54:00.3 on the same bus, etc). Typically they would be in the same iommu group, what do your groups look like with acs overrides disabled?
-
Fix to a small issue with updating OCI docker images
Skitals replied to m33ts4k0z's topic in General Support
Thanks. Was getting "Not available" for Dozzle and this fixed it. -
VM GPU passthrough resizable BAR support in 6.1 kernel
Skitals replied to Ancalagon's topic in Feature Requests
Do you have Above 4G Decoding enabled in your bios? -
VM GPU passthrough resizable BAR support in 6.1 kernel
Skitals replied to Ancalagon's topic in Feature Requests
Like I said above, you are trying to set the bar to 16GB when you have 8GB VRAM. Read the reddit post and learn your cards bar options and learn to manipulate it manually before trying to use a script. -
VM GPU passthrough resizable BAR support in 6.1 kernel
Skitals replied to Ancalagon's topic in Feature Requests
Is the pci address of your GPU really the same as mine? The GPU needs to be unbound. The previous line in my script unbinds it from vfio, but it also can't be bound to an Nvidia driver or anything else. I also believe you need Above 4G Decoding enabled in your bios, but I'm not sure if that would give you the error you are seeing. Before referencing my script you should really read and understand the reddit post I linked. You need to be able to read and manipulate the bar sizes on your own because the addresses, paths, number of bars etc will be different than mine. Edit: I just looked up your video card and see it has 8GB VRAM. Why are you trying to set the bar size to 16GB? Like I said you really need to understand the reddit post I linked before preceding. -
VM GPU passthrough resizable BAR support in 6.1 kernel
Skitals replied to Ancalagon's topic in Feature Requests
Also, you can check if windows is using the larger bar size from Device Manager. Open the GPU and go to the Resources tab. Look for a "Large Memory Range". You can calculate the size by subtracting the second number in the range from the first in a hex calculator. For me that is 13FFFFFFFF - 1000000000 = 17179869183 bytes = 16 gigabytes. -
VM GPU passthrough resizable BAR support in 6.1 kernel
Skitals replied to Ancalagon's topic in Feature Requests
The patch allows you to manipulate the bar size through sysfs. See the example how to use it in this reddit post. This is the User Script I created to set the bar size to 16GB. You would obviously need to tweak this to the proper bar size, pice address, folder path, device id, etc. #!/bin/bash echo -n "0000:0d:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind echo 14 > /sys/bus/pci/devices/0000\:0d\:00.0/resource1_resize echo -n "10de 2782" > /sys/bus/pci/drivers/vfio-pci/new_id || echo -n "0000:0d:00.0" > /sys/bus/pci/drivers/vfio-pci/bind # Bit Sizes # 1 = 2MB # 2 = 4MB # 3 = 8MB # 4 = 16MB # 5 = 32MB # 6 = 64MB # 7 = 128MB # 8 = 256MB # 9 = 512MB # 10 = 1GB # 11 = 2GB # 12 = 4GB # 13 = 8GB # 14 = 16GB # 15 = 32GB As stated the patch and the script works, but it was ultimately unneeded with my 4070 Ti since passthrough works when I have Above 4G and Resize Bar enabled in my bios. With those enabled the 4070 Ti defaults to a 16GB bar size, so there is no need to manually change it. Other setups/GPUs will give a code 43 or not passthrough at all with Resize Bar enabled in the bios, which is where this patch would be helpful to manually resize the bar. In that case I'm not sure what GPU-Z would report. The only way to know if it's making a difference is to benchmark a game known to benefit from resize bar with and without manipulating the bar size. -
Thanks. Merged and should be live in Community Apps. Edit: Should be live now.
-
VM GPU passthrough resizable BAR support in 6.1 kernel
Skitals replied to Ancalagon's topic in Feature Requests
I was curious so I did a few benchmarks passing all 32 cores/threads to the VM. Forza Horizon 5 1440p Extreme (No DLSS) VM Rebar OFF 20cpu: 116 fps VM Rebar ON 20cpu: 129 fps VM Rebar ON 32cpu: 134 fps Bare metal Rebar ON: 144 fps Cyberpunk 1440p RT Ultra (DLSS): VM Rebar OFF 20cpu: 81.07 fps VM Rebar ON 20cpu: 95.29 VM Rebar ON 32cpu: 98.26 Bare metal Rebar ON: 102.21 That's pretty dang close to bare metal performance with full resizable bar, given the extra overhead from unraid and vfio. Hitting 129 fps in the vm in Forza is amazing when with the 7900XTX I could never beat 114 fps with identical settings. -
VM GPU passthrough resizable BAR support in 6.1 kernel
Skitals replied to Ancalagon's topic in Feature Requests
I did a little more testing. It looks like the patch isn't needed with the RTX 40 Series. If I boot unraid with Resize Bar Support OFF, bar size is 256MB. If I boot unraid with Resize Bar Support ON, bar size is the max 16GB even when the gpu is bound to vfio on boot. With the patch I can leave Resize Bar Support OFF (but Above 4G Decoding ON) and manually resize the bar and have the same effect. This is another huge win for Nvidia over the AMD XTX which I couldn't passthrough at all when Resize Bar Support was enabled in the bios. Same as resizing the bar manually, sometimes I need to reboot the windows vm to see the larger bar. From the Device Manager > select your GPU > Resources tab look for a "Large Memory Range" -
I ditched the 7900 XTX. It was too many headaches, worse than the navi reset bug. I picked up a 4070 Ti today and passthrough just works, first time every time. I've been team red for a long time, but the 7000 series is not cut out for vms.
-
VM GPU passthrough resizable BAR support in 6.1 kernel
Skitals replied to Ancalagon's topic in Feature Requests
I ditched the 7900 XTX for a 4070 Ti and I'm seeing encouraging results. Forza Horizon 5 1440p Extreme (No DLSS) VM 256MB Bar: 116 fps VM 16GB Bar: 129 fps Baremetal Rebar Off: 127 fps Baremetal Rebar On: 144 fps Cyberpunk 1440p RT Ultra (DLSS): VM 256MG Bar: 81 fps VM 16GB Bar: 95 fps Baremetal Rebar Off: 94 fps Baremetal Rebar On: 102 fps Of note I went from testing in baremetal windows (with rebar on), rebooted into unraid and the bar size was still set to 16GB. I'm not sure what negotiated that or if it remembers the last setting. Setting the bar to 16GB basically brings it to parity with bare metal when rebar is OFF. Still a big performance delta, which I think it largely due to fewer cpu cores in the VM. Time Spy Graphics scores are identical in VM and baremetal (23k). -
Do you ever see the Tianocore screen when booting a vm? I always did with my 5700 XT, I never see it with the 7900 XTX. When it works the first thing I see is the Windows login screen. Finicky is an understatement. I lived with the 5700 XT for 2 years with the dreaded reset bug and that was a joy in comparison. I am also seeing massive instability. I've had multiple crashes. Once the screen went black while launching 3dmark, and it recovered with Andrenalin not running. Another time the Windows VM fully locked up while I was using Chrome. Looking at the unraid dashboard there were a couple cores pegged at 100%. I had to Force stop the VM. Never seen anything like this in 2 years of using the 5700 XT and a Windows vm as my primary desktop computer. Not sure if it's driver instability or issues with using the card in a virtualized environment. I haven't used it enough on bare metal windows to know if there are stability issues there, too. The resizable bar thing is another issue. I'm used to games running within a percentage inside a vm. Games that make use of resizable bar are seeing 20% better fps on bare metal.