Skitals

Community Developer
 View Profile  See their activity

  • Posts

    201

  • Joined

  • Last visited

  • Days Won

    3

 Content Type 

Posts posted by Skitals

    [Plugin] VFIO-PCI Config

    in Plugin Support

    Posted

    5 hours ago, JhonScript said:

     

    Hello, I would like to understand what this means in other words? I don't know if you can help me with my case.

    I have two GPUs (TUF-GTX1660S), and in SysDevs this appears, my Motherboard is b550 Vision D

     

    image.thumb.png.d14ce310e3a060dfaf40819d2ec424ce.png

     

    image.png.6b0ad5ebccc626ea7169563fd9600244.png

     

    image.thumb.png.95eeb610f5258356f3fc7925c074dc5f.png

     

    My GPU does not have any USB c or USB 3.1 port

     

    363310336_GPUGTX.jpg.c436a5bae9a0cdebda48ce7fbd2db4a7.jpg

     

    My Motherboard does have a usb c port and usb 3.1 ports

     

    1431715401_visiond.png.b653951b2bc44ad5014b1418aa47cbfe.png

     

    The first question is because this text appears "This controller is linked to vfio, the connected USB devices are not seen", this means that if I managed to connect a usb, and if I have made a passthrough from the GPU to the VM, the usb will it connect directly to the VM and not be seen on the Unraid server?

     

    The second question is, because my GPU indicates that it has a USB 3.1 and USB c controller, if it does not have any physical port, is there an adapter to convert HDMI to USB?

    https://www.amazon.com/Adapter-Converter-Thunderbolt-MacBook-Microsoft/dp/B08VDT3YGK

     

    https://www.amazon.com/ELUTENG-Thunderbolt-Compatible-Converter-Connector/dp/B07Z4G911X

     

     

     

     

     

    Yes, your gpu has a usb controller but no port. Some gpus have usb-c for vr and other uses. There is no adapter that will give you access to it. And yes, any devices connected to the controller would only be visible/usable inside a VM if the controller is bound to vfio-pci.

     

    So unfortunately no way to make use of it. Best practice is still to passthrough all sub-devices together with multifunction on (54:00.0, 54:00.1, 54:00.2, 54:00.3 on the same bus, etc). Typically they would be in the same iommu group, what do your groups look like with acs overrides disabled?

    VM GPU passthrough resizable BAR support in 6.1 kernel

    in Feature Requests

    Posted

    3 hours ago, JesterEE said:

     

    Yup, messed that up in the copypasta while experimenting.

     

    Anyway, not a big deal...it works for me if I want to set the ReBAR to acceptable values lower than the default 256MB (for my card [64MB, 128MB, 256MB]) ... But it will not set them higher (for my card [512MB, 1GB, 2GB, 4GB, 8GB]). If I try and set it to a value lower than 64MB or higher than 256MB I will get the error. 

      

    # -bash: echo: write error: Device or resource busy

     

    Here is the is the memory allocation info for my card

      

    # lspci -vvvs 0b:00.0
0b:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070 Lite Hash Rate] (rev a1) (prog-if 00 [VGA controller])
...
        Region 0: Memory at fb000000 (32-bit, non-prefetchable) [size=16M]
        Region 1: Memory at d0000000 (64-bit, prefetchable) [size=256M]
        Region 3: Memory at c8000000 (64-bit, prefetchable) [size=32M]
...
Physical Resizable BAR
                BAR 0: current size: 16MB, supported: 16MB
                BAR 1: current size: 256MB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB
                BAR 3: current size: 32MB, supported: 32MB

     

    Thanks for publishing the patch and modified kernel even though it didn't work for me completely. Hope others give it a shot too to report their mileage. 

     

    Do you have Above 4G Decoding enabled in your bios?

    VM GPU passthrough resizable BAR support in 6.1 kernel

    in Feature Requests

    Posted

    8 hours ago, JesterEE said:

     

    No, I changed the addressed on my side, but I posted the command so it could easily be referenced from your post.  Here is my version:

      

    #!/bin/bash
echo -n "0000:0b:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo 14 > /sys/bus/pci/devices/0000\:0b\:00.0/resource1_resize  # <<<< Gets stuck here
echo -n "10de 2488" > /sys/bus/pci/drivers/vfio-pci/new_id || echo -n "0000:0b:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

     

     

    Like I said above, you are trying to set the bar to 16GB when you have 8GB VRAM. Read the reddit post and learn your cards bar options and learn to manipulate it manually before trying to use a script.

    VM GPU passthrough resizable BAR support in 6.1 kernel

    in Feature Requests

    Posted · Edited by Skitals

    46 minutes ago, JesterEE said:

    @Skitals So I tried the script commands you specified in your previous post, but got stuck when actually sizing the ReBar with:

      

    # echo 14 > /sys/bus/pci/devices/0000\:0d\:00.0/resource1_resize
# -bash: echo: write error: Device or resource busy

     

    Did some searching and I couldn't find a way to correct this. Not looking for tech support necessarily, just reporting my experience.

     

    On my system, the video card is bound to VFIO and the system is booting with a syslinux config including 

    ... video=efifb:off ...

     

    Is the pci address of your GPU really the same as mine? The GPU needs to be unbound. The previous line in my script unbinds it from vfio, but it also can't be bound to an Nvidia driver or anything else. I also believe you need Above 4G Decoding enabled in your bios, but I'm not sure if that would give you the error you are seeing. 

     

    Before referencing my script you should really read and understand the reddit post I linked. You need to be able to read and manipulate the bar sizes on your own because the addresses, paths, number of bars etc will be different than mine.

     

    Edit: I just looked up your video card and see it has 8GB VRAM. Why are you trying to set the bar size to 16GB? Like I said you really need to understand the reddit post I linked before preceding. 

    VM GPU passthrough resizable BAR support in 6.1 kernel

    in Feature Requests

    Posted · Edited by Skitals

    9 hours ago, JesterEE said:

     

    Tried your patch today with my ASUS KO RTX 3070 and a Windows 10 VM.  GPU-Z is still reporting Resizeable Bar as Disabled.

     

    Was there any additional setup needed to set the initial state of the Bar or should it be on by default with the patched kernel?

     

    The patch allows you to manipulate the bar size through sysfs. See the example how to use it in this reddit post.

     

    This is the User Script I created to set the bar size to 16GB. You would obviously need to tweak this to the proper bar size, pice address, folder path, device id, etc.

      

    #!/bin/bash
echo -n "0000:0d:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
echo 14 > /sys/bus/pci/devices/0000\:0d\:00.0/resource1_resize
echo -n "10de 2782" > /sys/bus/pci/drivers/vfio-pci/new_id || echo -n "0000:0d:00.0" > /sys/bus/pci/drivers/vfio-pci/bind


# Bit Sizes
# 1 = 2MB
# 2 = 4MB
# 3 = 8MB
# 4 = 16MB
# 5 = 32MB
# 6 = 64MB
# 7 = 128MB
# 8 = 256MB
# 9 = 512MB
# 10 = 1GB
# 11 = 2GB
# 12 = 4GB
# 13 = 8GB
# 14 = 16GB
# 15 = 32GB

     

    As stated the patch and the script works, but it was ultimately unneeded with my 4070 Ti since passthrough works when I have Above 4G and Resize Bar enabled in my bios. With those enabled the 4070 Ti defaults to a 16GB bar size, so there is no need to manually change it. Other setups/GPUs will give a code 43 or not passthrough at all with Resize Bar enabled in the bios, which is where this patch would be helpful to manually resize the bar. In that case I'm not sure what GPU-Z would report. The only way to know if it's making a difference is to benchmark a game known to benefit from resize bar with and without manipulating the bar size.

    • Thanks 3

    VM GPU passthrough resizable BAR support in 6.1 kernel

    in Feature Requests

    Posted

    I was curious so I did a few benchmarks passing all 32 cores/threads to the VM.

     

    Forza Horizon 5 1440p Extreme (No DLSS)

    VM Rebar OFF 20cpu: 116 fps

    VM Rebar ON 20cpu: 129 fps

    VM Rebar ON 32cpu: 134 fps

    Bare metal Rebar ON: 144 fps

     

    Cyberpunk 1440p RT Ultra (DLSS):

    VM Rebar OFF 20cpu: 81.07 fps

    VM Rebar ON 20cpu: 95.29

    VM Rebar ON 32cpu: 98.26

    Bare metal Rebar ON: 102.21

     

    That's pretty dang close to bare metal performance with full resizable bar, given the extra overhead from unraid and vfio.

     

    Hitting 129 fps in the vm in Forza is amazing when with the 7900XTX I could never beat 114 fps with identical settings.

     

     

    • Like 4
    • Thanks 1

    VM GPU passthrough resizable BAR support in 6.1 kernel

    in Feature Requests

    Posted

    I did a little more testing. It looks like the patch isn't needed with the RTX 40 Series.

     

    If I boot unraid with Resize Bar Support OFF, bar size is 256MB.

    If I boot unraid with Resize Bar Support ON, bar size is the max 16GB even when the gpu is bound to vfio on boot.

     

    With the patch I can leave Resize Bar Support OFF (but Above 4G Decoding ON) and manually resize the bar and have the same effect.

     

    This is another huge win for Nvidia over the AMD XTX which I couldn't passthrough at all when Resize Bar Support was enabled in the bios.

     

    Same as resizing the bar manually, sometimes I need to reboot the windows vm to see the larger bar. From the Device Manager > select your GPU > Resources tab look for a "Large Memory Range" 

    Radeon RX 7900 XT passthrough

    in VM Engine (KVM)

    Posted

    2 hours ago, RustyClark said:

    I've tried everything in this thread but can't seem to get my XTX passthrough to post.  It and the VNC says "...has not initialized the display (yet)".  When I run it with just the VNC it boots, when I passthrough my secondary Nvidia 3080 it works.  Tried disabling resize bar and setting the XTX as the non primary gpu in the bios, and nothing happens.  I've run out of ideas.

     

    I ditched the 7900 XTX. It was too many headaches, worse than the navi reset bug. I picked up a 4070 Ti today and passthrough just works, first time every time.

     

    I've been team red for a long time, but the 7000 series is not cut out for vms.

    VM GPU passthrough resizable BAR support in 6.1 kernel

    in Feature Requests

    Posted

    I ditched the 7900 XTX for a 4070 Ti and I'm seeing encouraging results.

     

    Forza Horizon 5 1440p Extreme (No DLSS)

    VM 256MB Bar: 116 fps

    VM 16GB Bar: 129 fps

    Baremetal Rebar Off: 127 fps

    Baremetal Rebar On: 144 fps

     

    Cyberpunk 1440p RT Ultra (DLSS):

    VM 256MG Bar: 81 fps

    VM 16GB Bar: 95 fps

    Baremetal Rebar Off: 94 fps

    Baremetal Rebar On: 102 fps

     

    Of note I went from testing in baremetal windows (with rebar on), rebooted into unraid and the bar size was still set to 16GB. I'm not sure what negotiated that or if it remembers the last setting.

     

    Setting the bar to 16GB basically brings it to parity with bare metal when rebar is OFF. Still a big performance delta, which I think it largely due to fewer cpu cores in the VM. Time Spy Graphics scores are identical in VM and baremetal (23k).

     

     

     

    • Thanks 1

    Radeon RX 7900 XT passthrough

    in VM Engine (KVM)

    Posted

    On 1/4/2023 at 11:26 PM, Ancalagon said:

    Great to hear.

     

    After I got this working, I was messing around with it more to see if there were other ways I could get it working without needing the vbios, but never found one. But afterwards reverting the changes, it wasn't consistent getting it to work again, even after using the same steps (booting into CSM mode first, etc.). But once it works, it seems to continue working, restarting the VM multiple times and never had it stop working.

     

    When it wasn't working (always code 43), I found booting the VM with only the GPU passed through without other passed through hardware worked. Whether that was a coincidence or not, I can't say. It's definitely more finicky than the RX 6800 XT was.

     

    Do you ever see the Tianocore screen when booting a vm? I always did with my 5700 XT, I never see it with the 7900 XTX. When it works the first thing I see is the Windows login screen.

     

    Finicky is an understatement. I lived with the 5700 XT for 2 years with the dreaded reset bug and that was a joy in comparison.

     

    I am also seeing massive instability. I've had multiple crashes. Once the screen went black while launching 3dmark, and it recovered with Andrenalin not running. Another time the Windows VM fully locked up while I was using Chrome. Looking at the unraid dashboard there were a couple cores pegged at 100%. I had to Force stop the VM. Never seen anything like this in 2 years of using the 5700 XT and a Windows vm as my primary desktop computer.

     

    Not sure if it's driver instability or issues with using the card in a virtualized environment. I haven't used it enough on bare metal windows to know if there are stability issues there, too.

     

    The resizable bar thing is another issue. I'm used to games running within a percentage inside a vm. Games that make use of resizable bar are seeing 20% better fps on bare metal.

    VM GPU passthrough resizable BAR support in 6.1 kernel

    in Feature Requests

    Posted · Edited by Skitals

    1 hour ago, SimonF said:

    Maybe the virtio driver does support 32GB yet?

     

    I think I found the problem. Looks like by default OVMF PCI MMIO BAR allocation is 32GB, hence there isn't enough space for the GPU + other devices.

     

    Edit: Adding the xml from the link above I was able to pass the gpu with 32GB bar, but windows is only seeing 256MB. This is not unlike the first time I passed through with 8GB bar where subsequent reboots showed the full bar in windows. Except at 32GB it will not reboot and getting it to passthrough even once is finicky.

    VM GPU passthrough resizable BAR support in 6.1 kernel

    in Feature Requests

    Posted

    Some interesting testing and a roadblock.

     

    When resize bar is enabled and I boot windows on baremetal, the bar is being sized to the max 32GB/256MB. This sees HUGE performance gains in Forza Horizon 5.

     

    Resize bar off: 113fps

    Resize bar on: 139fps

     

    With the resize bar patch I can easily resize the bars to the max, however passthrough fails when bar 0 i set to 32GB. I don't get any errors, it just doesn't work. I can set bar 0 to any other size and passthrough works.

     

    With the bars set to 16GB/256MB I'm getting 114fps and improved latency. Really frustrated I can't get the vm to boot when set to 32GB to match baremetal.

    • Thanks 1

    VM GPU passthrough resizable BAR support in 6.1 kernel

    in Feature Requests

    Posted · Edited by Skitals

    I can confirm the above patch works. Below is a script to unbind my gpu from vfio, set the bar size to 8GB, and rebind to vfio:

      

    GPU=0e:00
GPU_ID="1002 744c"
echo -n "0000:${GPU}.0" > /sys/bus/pci/drivers/vfio-pci/unbind || echo "Failed to unbind gpu from vfio-pci"
cd /sys/bus/pci/devices/0000\:0e\:00.0/
echo 13 > resource0_resize
echo -n "$GPU_ID" > /sys/bus/pci/drivers/vfio-pci/new_id

     

    The bar size goes from 256MB to 8GB, and stays there after rebinding and passing through the gpu.

     

    HOWEVER, the memory range in Windows is unchanged and I'm seeing no gains in Forza 5 (known to benefit from resizable bar).

     

    Edit: I rebooted the VM and for whatever reason on the second boot the gpu was assigned the proper 8GB address range in Windows. This time I did see a performance change... the fps in forza 5 went DOWN from 114 to 101 fps. Rebooted unraid, bar back to 256MB and fps back to 114.

     

    Time to play around with different bar sizes and see what happens.

    • Thanks 1

    Radeon RX 7900 XT passthrough

    in VM Engine (KVM)

    Posted

    34 minutes ago, Ancalagon said:

    I'm using the original vbios dump, no edits.

     

    Yes, I am binding all 4 devices to vfio. After adding to them the VM, I'm manually editing the XML to put them on the same bus and slot as a multifunction device.

     

    Here's the GPU portion of my XML: 

    <hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x0c' slot='0x00' function='0x0'/>
  </source>
  <alias name='hostdev0'/>
  <rom file='/mnt/user/domains/GPU ROM/AMD.RX7900XT.022.001.002.008.000001.rom'/>
  <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x0c' slot='0x00' function='0x1'/>
  </source>
  <alias name='hostdev1'/>
  <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x0c' slot='0x00' function='0x2'/>
  </source>
  <alias name='hostdev2'/>
  <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x2'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <driver name='vfio'/>
  <source>
    <address domain='0x0000' bus='0x0c' slot='0x00' function='0x3'/>
  </source>
  <alias name='hostdev3'/>
  <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x3'/>
</hostdev>

     

    Have you been able to boot the VM after booting the host in legacy CSM mode?

     

    Thanks... I got it working. What I did different I have no idea. I swear I tried this exact setup like 5 minutes in. 

     

    Reset also seems to be working. That's a big relief as there are multiple reports of the xtx having the reset bug on reddit. I have stopped and started the vm multiple times and it works every time.

    Radeon RX 7900 XT passthrough

    in VM Engine (KVM)

    Posted

    On 12/28/2022 at 6:02 AM, Ancalagon said:

    I got this working! First, I was able to successfully pass through the GPU after switching to legacy CSM boot. I was then able to save my vbios rom. Then after adding the vbios rom to the VM XML and switching back to EFI boot, the GPU passthrough works! I can also confirm the GPU is properly reset when rebooting the VM as well.

    what method did you use to save your vbios?

    [Support] ich777 - AMD Vendor Reset, CoralTPU, hpsahba,...

    in Plugin Support

    Posted

    Re: AMD Vendor Reset

     

    I have been using this wonderful plugin since it was available, and was building a custom kernel with gnif's patch before that. I've never had any issues with my 5700 XT until yesterday. It seems like it crashed hard. At first it was stalling at the Tianocore splash screen, after some reboots and troubleshooting it would boot but the GPU had "stopped working" with Code 43 (low res output, drivers wouldn't load). I tried everything under the sun to fix the code 43, but no beans.

     

    The one and only fix was to boot windows on bare metal (Windows is installed on an NVME drive I pass to the VM). The GPU/AMD driver immediately worked, I rebooted into unraid and it immediately worked in the VM again.

     

    My best guess is the GPU was in some unrecoverable state. The only thing that fixed it was booting it natively. A word of advice if anyone finds themselves in a similar situation where the card suddenly stops working with no change.

    • Like 1