Salty2011

hey @mjorud Sorry hadnt realised people replied to my post. Definately possible to change to 12GB if your using the script I made, Should be easy enought to do the following, Where you see "32GB") echo 15 ;; Add a new line after it and put in "12GB") echo 12 ;; Then set you vram size to variable to VRAM_SIZE="12GB" Given my script doesn't have 12GB entry currently, and your saying it setting to 16GB.... that sounds like you may not have set the VRAM_SIZE variable at the beginning of the script. Its currently set to 16GB cause of the card I was using at the time. Additionally be sure you have a copy of vbios to pass through as found that was required

Howdy All After reading all the posts, I now have my 6900XT passing through and in GPU-Z it says resize bar is enabled. With that said there are the following issues i am getting: For AMD if you open up the AMD Driver console and view the Smart Access Memory status under "Smart Technology" it says unsupported. Not sure if this is just an AMD / VM bug and its safe to ignore For the user script that creates the resizebar resource (in this guide below) i get the following error Not sure why when rebinding the device its saying it exist despite it being unbound in the beginning. If anyone can shed some like on this will be appreciated and I will update this post Now for everyone else, this is what I did in order to get it working to this point PreRequisits In the BIOS make sure you have these enabled AMD-V or Intel VT-d SR-IOV Enable Above 4G Decoding Do not enable Resize BAR Unraid Ensure you have community app installed Ensure you have the user scripts plugin In the UNRAID WebUI you will need to ensure that you have bound the gfx card to vfio so that unraid doesnt reserve it preventing the card from being used. (I am using UNRAID 6.12.10) To do this: Goto Tools > System Devices Check all the PCI devices that are you graphics card. Not a Graphics card can be listed as multiple devices, In the example below the 6900XT has 4 devices In the pic above I have highlighted two things that you will need to note for later - GPU Address ( thats the 0a:00.0) - GPU ID ( thats the 1002:73bf) Yours will be different to the ones I have listed so make sure to note your down. Additionally note i ticked serval devices that all share the same address space of 0a:00.x, x being .0 /1 /2/3. For your setup make sure you have ticked all other devices in the same address space. Youll notice the descriptions will also loosely correlate. When done scroll to the bottom and click "BIND SELECTED DEVICES TO VFIO AT BOOT", you will then need to reboot the server before continuing. Initial Steps Create a normal Windows 11 VM, do not pass any additional hardware Go through the standard Windows Install Perform all updates as Microsoft suck and don't update there ISO and AMD and Likely NVIDIA drivers will need some of the latest updates for these to even install Download the latest full install of either nvidia or amd (do not use the autodetect installed, this can act weird). do not install at this time. that will come later Show down the machine when done GPU Passthrough - No Resize BAR Do this first just to ensure its all working without ReSize BAR enabled on the VM In Unraid now go edit your VM In Form view make the following changes, Add a second GPU, set the sound card to be the GPU For AMD and maybe NVIDIA check "other pci deivces" and attached the additional devices related to the card. For example the 6900XT with show as 4 devices, the GPU, Sound and two USB device/controller. Save configuration Boot up VM and use the VNC console. Check that you see you gpu in device manager (it may have error 43, this is ok ignore it) Now install your GPU drivers and restart the vm Load up some games and test card is working within expectations (not its a VM so wont get exact performance.. well at least not without some additional tuning) If this all working, congrats you have working GPU passthrough and now for the fun of getting ReSize bar working Getting ResizeBAR Working Switch back to the unraid webui for now Reboot unraid server and enable ReSize BAR and Above 4G decode in BIOS Now open the unraid console Open up user scripts Create a new User Script and schedule it to run on array start. Add the following script, take note that you will need to alter the GPU, GPU_ID and VRAM for the card your passing through. As the beginning of this post these were the values we noted down. ##### NOTE ##### The script I have written below will select the correct bit size for your card vram, determine which GPU (AMD/NVIDIA) based on the gpu address and then set the resize values accordingly. For AMD cards resource2_resize needs to be less than 8MB for windows to boot, else you get black screen, this doesnt seem to apply to nvidia gpu's based on what I could find. I discovered the AMD resource2_resize trick over here and they have a good article explaining the setup. VFIO: How to enable Resizeable BAR (ReBAR) in your VFIO Virtual Machine REPO: https://github.com/salty2011/unraid-userscsripts #!/bin/bash # User-configurable variables GPU=0000:0a:00.0 # GPU PCI address GPU_ID="1002 73bf" # GPU ID for re-binding VRAM_SIZE="16GB" # Set VRAM size based on your GPU # Define a function to determine the bit size for rebar based on VRAM size function get_rebar_size { local vram_size=$1 case $vram_size in "2MB") echo 1 ;; "4MB") echo 2 ;; "8MB") echo 3 ;; "16MB") echo 4 ;; "32MB") echo 5 ;; "64MB") echo 6 ;; "128MB") echo 7 ;; "256MB") echo 8 ;; "512MB") echo 9 ;; "1GB") echo 10 ;; "2GB") echo 11 ;; "4GB") echo 12 ;; "8GB") echo 13 ;; "16GB") echo 14 ;; "32GB") echo 15 ;; *) echo "Unsupported VRAM size: $vram_size"; exit 1 ;; esac } # Use lspci to check the GPU manufacturer MANUFACTURER=$(lspci -v -s ${GPU} | grep 'VGA compatible controller') # Unbind GPU from vfio-pci driver echo "Unbinding GPU from vfio-pci driver..." echo ${GPU} > /sys/bus/pci/drivers/vfio-pci/unbind sleep 1 # Get the correct rebar size bit value REBAR_SIZE=$(get_rebar_size $VRAM_SIZE) # Check if the GPU is from AMD or NVIDIA and apply settings accordingly if echo "${MANUFACTURER}" | grep -q "NVIDIA"; then echo "Detected NVIDIA GPU. Applying NVIDIA-specific settings..." echo "Setting the rebar size to $VRAM_SIZE" if ! echo $REBAR_SIZE > /sys/bus/pci/devices/${GPU}/resource1_resize; then echo "Failed to set rebar size. Please check if the VRAM size variable ($VRAM_SIZE) is correct for your GPU." exit 1 fi elif echo "${MANUFACTURER}" | grep -q "AMD"; then echo "Detected AMD GPU. Applying AMD-specific settings..." echo "Setting the rebar 0 size to $VRAM_SIZE" if ! echo $REBAR_SIZE > /sys/bus/pci/devices/${GPU}/resource0_resize; then echo "Failed to set rebar 0 size. Please check if the VRAM size variable ($VRAM_SIZE) is correct for your GPU." exit 1 fi echo "Setting the rebar 2 size to 8MB" if ! echo 3 > /sys/bus/pci/devices/${GPU}/resource2_resize; then echo "Failed to set rebar 2 size to 8MB." exit 1 fi else echo "GPU manufacturer not recognized. Exiting script." exit 1 fi # Wait for settings to apply sleep 2 # Re-bind GPU echo ${GPU_ID} > /sys/bus/pci/drivers/vfio-pci/new_id || echo -n "${GPU}" > /sys/bus/pci/drivers/vfio-pci/bind echo "Configuration complete." sleep 1 Restart your UNRAID Server so that this script runs and were ready for next step To check if the script applied the setting you can lspci -vvvs GPU (eg 0a:00.0) | grep "BAR" Importantly for AMD card BAR2 is 8MB, for me it was defaulting to 265MB and this stops windows from booting. Edit the VM and remove the Virtual Graphics card and ensure the only gpu is the card you are wanting to pass through switch to XML View. At the top, line 2 you will need to edit <domain type='kvm'> to be <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> Scroll to the bottom of the XML code and above where it says </domain> insert: <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> Click update configuration Boot the VM (make sure a Monitor is connected to the passthrough GPU), you should see Windows boot up to the desktop Download GPU-Z, run it and on the main screen ResizeBAR should say enabled. or check the GPU in device manager and you see large memory allocation NOTES For the AMD card I have both Device Manager and GPU-Z say resize bar is enabled. However in the AMD Driver under Smart Technology I am seeing Smart Access Memory as unsupported. Not sure if this a quirk or something else. but if anyone knows let me know. As for the script.. i take no responsibility for any issues it causes, ive tested on my setup and it works. happy to make some amendments if other bugs occurs.

Hey I am attempting to run Games on Whales (Wolf) on Unraid and noticed when I run the command below, the mode set is currently set to know but the doco I am ready need sudo cat /sys/module/nvidia_drm/parameters/modeset N But I need this enabled, It does indicate this can be done by adding the following to the grub boot nvidia-drm.modeset=1 However being new to unraid and all wanted to field the question on best approach to enabling this. Is it possible I can enable this post grub boot, or is there an appropriate way to do this for unraid. For context I am following this guide Games on Whales Wolf Setup

well last night I went into the BIOS and disabled the AMD Global C state settings and since then I no longer see any of the AMD GPU errors posted into the log. only been 10 hours, but in my experience I have usually seen these issues appear by now. So will be interesting if this leads to solving the crashing.

yeah thats fair. ill try the c-states suggestion and see if helps. if not will take out the AMD card. the use cases for it arent a must and ill wait till the next version of nvidia cards to come out and either get something on sale or the new option.

thanks @ich777 ill take a look at disabling the c-states. As for the duel gpu's, this is server I built up with parts I had laying around. I opted to put a second gpu in the rig as I wanted to test / poc moving to a virtualized gaming rig. There are two approaches i was going to look at trying to achieve this, either using Steam headless / Games On Whales (Wolf) or a VM with GPU pass through and just have sunshine stream installed. end goal is to completely consolidate nas/daily pc and home assistant into the one box. However im quickly finding that for transcoding and game streaming, and literally everything nvidia seems to the better option to use for allot of this. Was kinda surprised given from a linux desktop standpoint nvidia has always been a royal pain. (i noticed the transcode performance of the 1080 was equal to the 6900xt but at way less power draw)

Dont have the GPU passed through to any VM, but do have it passed into the emby container, also that does seem to detect it. Did test using plex / unmanic and that works without issue and see load on the card using radeontop (so suspect its just the emby container) Ive attached the diagnostic zip. tower-diagnostics-20240528-2203.zip

Hey, I have the driver installed with a Radeon 6900XT and noticing these errors, checking the temps using gpu stats it sitting at 40c and I havent been able to find much regarding the other errors listed, on top my unraid server seem to hard lock up, every few days. not sure what the cause is, just assuming these errors might be related to that. Still currently trying to get a copy of the logs when the hard crash occurs. In the mean time anyone able to shed light on the issue below? May 28 21:33:28 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: Fail to disable thermal alert! May 28 21:33:28 Tower kernel: [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP block <smu> failed -22 May 28 21:33:28 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: free PSP TMR buffer May 28 21:33:29 Tower kernel: [drm] PCIE GART of 512M enabled (table at 0x00000083FEB00000). May 28 21:33:29 Tower kernel: [drm] PSP is resuming... May 28 21:33:29 Tower kernel: [drm] reserve 0xa00000 from 0x83fd000000 for PSP TMR May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: SMU is resuming... May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: smu driver if version = 0x00000040, smu fw if version = 0x00000041, smu fw program = 0, version = 0x003a5800 (58.88.0) May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: SMU driver if version not matched May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: dpm has been enabled May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: SMU is resumed successfully! May 28 21:33:29 Tower kernel: [drm] DMUB hardware initialized: version=0x02020020 May 28 21:33:29 Tower kernel: [drm] kiq ring mec 2 pipe 1 q 0 May 28 21:33:29 Tower kernel: [drm] VCN decode and encode initialized successfully(under DPG Mode). May 28 21:33:29 Tower kernel: [drm] JPEG decode initialized successfully. May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 5 on hub 0 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 6 on hub 0 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 7 on hub 0 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 8 on hub 0 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 9 on hub 0 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 10 on hub 0 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring kiq_2.1.0 uses VM inv eng 11 on hub 0 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring sdma1 uses VM inv eng 13 on hub 0 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring sdma2 uses VM inv eng 14 on hub 0 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring sdma3 uses VM inv eng 15 on hub 0 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring vcn_dec_0 uses VM inv eng 0 on hub 1 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring vcn_enc_0.0 uses VM inv eng 1 on hub 1 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring vcn_enc_0.1 uses VM inv eng 4 on hub 1 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring vcn_dec_1 uses VM inv eng 5 on hub 1 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring vcn_enc_1.0 uses VM inv eng 6 on hub 1 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring vcn_enc_1.1 uses VM inv eng 7 on hub 1 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: amdgpu: ring jpeg_dec uses VM inv eng 8 on hub 1 May 28 21:33:29 Tower kernel: amdgpu 0000:0a:00.0: [drm] Cannot find any crtc or sizes

@GeekFreak, just for clarity is the stream deck itself connected to the unraid server and passed through to the vm and on switch would be passed through to the vm switching to?

Curious as to how you got D2R working. For me everytime I launch the game it crashes straight away