KptnKMan Posted June 16, 2023 Share Posted June 16, 2023 21 hours ago, jonathanselye said: how do i get my ideal bar size? You will need to determine that yourself, using the provided scales in the script they are written out, but essentially you need to pick one that your card memory fits in (and is supported). You haven't stated if your Arc A770 is 8GB or 16GB so I would suggest starting there for a hint. You can also try listing the compatible Bar sizes with with this command: lspci -vvvs {YOUR_GPU_ID_HERE} | grep "BAR" You should see something like this: root@unraid1:~# lspci -vvvs 0b:00.0 | grep "BAR" Capabilities: [bb0 v1] Physical Resizable BAR BAR 0: current size: 16MB, supported: 16MB BAR 1: current size: 32GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB BAR 3: current size: 32MB, supported: 32MB 21 hours ago, jonathanselye said: will it work if i already unbinded the gpu to unraid?[using vfio-pcie] I might be wrong, but I don't think you need to unbind it. That is what the script is for, I believe, running at Array Startup. 21 hours ago, jonathanselye said: and i only have </devices> line, is that the same? I'm not sure what you mean, if you have </devices> then you will certainly have <devices> somewhere, or you've got bigger problems. Follow the instructions, and read your XML config carefully, line by line. Read upwards from each line beginning to find the opening <devices>. 21 hours ago, jonathanselye said: in device manager there is a new device called "NF I2C Slave Device" when i tried updating the driver both online and latest virtio the drivers seems not available yet so i think it is working the driver just isnt available yet. Doing a quick google search led to this reddit post which indicates this device is an RGB controller on the Arc card. This is nothing to do with virtio drivers, its something on your card. Depending on the manufacturer, download their drivers or accompanying config application for their RGB control. Quote Link to comment
KptnKMan Posted June 21, 2023 Share Posted June 21, 2023 On 6/15/2023 at 12:41 PM, KptnKMan said: Hi @Skitals I took another look at the kernel patch you linked earlier. As far as I can tell, this patch is present in kernel 6.1 since v6.1-rc1 which is now present in unraid 6.12. So I wanted to ask, on the surface do you know if there would be any obvious reason that this wouldn't be present, or not work in unraid 6.12? Also, I'm yet to test 6.12 so I haven't personally verified any issues of missing ReBar in unraid 6.12 using kernel 6.1. @jonathanselye can you confirm and describe what you've done to test on unraid 6.12, what hardware you use, and what your exact results were? @Skitals and @jonathanselye, I just upgraded from 6.11.5 direct to 6.12.1. I can confirm that everything works on the Unraid 6.12.1 stock kernel 6.1.34. My daily VM has ReBar enabled. I can report any issues encountered, but everything seems to work as of right now. Quote Link to comment
ELP1 Posted July 11, 2023 Share Posted July 11, 2023 (edited) Hello, I am trying to enable Resizable Bar in my VM with Radeon 6650XT, but till now no luck. I am running Unraid 6.12. I did all steps according checklist from previous post: - Host BIOS Enable ReBAR support - Host BIOS Enable 4G Decoding - Enable & Boot Custom Kernel syslinux configuration (near beginning of this thread) - Boot Unraid in UEFI Mode - VM must use UEFI BIOS - VM must have the top line of XML from <domain type='kvm'> to: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> - VM must have added the following (after the </device> line, before the </domain> line): <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> I have GPU bind to VFIO at boot. Standard GPU passthrough is working properly. What can I try? I found some script but as far as I understand it is not needed for Unraid 6.12. Thank you for any help! Edited July 11, 2023 by ELP1 Quote Link to comment
letrain Posted August 9, 2023 Share Posted August 9, 2023 (edited) On 2/2/2023 at 11:52 AM, Trozmagon said: Just a heads up, the custom kernel wasn't necessary for me with my RTX 3090, just adding the additional xml to my VM config was enough. ok what did i do wrong? its enabled in bios. asrock calls it Clever access memory. it is on in the bios. the xml is edited. everything is on. i ran the user script.... and the vbios is up to date for the 3090.. @KptnKMan it works in bare metal..but i just can't get it to work in the VM. Edited August 10, 2023 by letrain update Quote Link to comment
letrain Posted August 10, 2023 Share Posted August 10, 2023 On 6/21/2023 at 10:20 AM, KptnKMan said: @Skitals and @jonathanselye, I just upgraded from 6.11.5 direct to 6.12.1. I can confirm that everything works on the Unraid 6.12.1 stock kernel 6.1.34. My daily VM has ReBar enabled. I can report any issues encountered, but everything seems to work as of right now. So what the final solution here? Everything in the list needed? Custom kernel? Rom resize script? Everything works bare metal. But it won't work in the vm. Unraid 6.12.3 here, is 6.12.1 needed? Was there regression? Host BIOS Enable ReBAR support - Host BIOS Enable 4G Decoding - Enable & Boot Custom Kernel syslinux configuration (near beginning of this thread) - Boot Unraid in UEFI Mode - VM must use UEFI BIOS - VM must have the top line of XML from <domain type='kvm'> to: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> - VM must have added the following (after the </device> line, before the </domain> line): <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> Quote Link to comment
letrain Posted August 10, 2023 Share Posted August 10, 2023 On 1/4/2023 at 6:39 PM, Skitals said: I back ported the resizable bar patch to unraid 6.11.5 (kernel 5.19.17). It applies and builds cleanly with the other unraid patches and unraid .config file. I attached the patch and a precompiled kernel. I can confirm it's working (resourceN_resize property present), but making it useful is another trick. resize-bar.patch 7.22 kB · 52 downloads unraid-6.11.5-kernel.zip 5.97 MB · 46 downloads How do you apply this? Quote Link to comment
jfoxwoosh Posted August 16, 2023 Share Posted August 16, 2023 Tried this on my system: Ryzen 5900x MSI B550 Motherboard RX 6800XT UnRaid 6.12.3 I followed the instructions posted here, and tried it on one of my existing Win11 VM, which ended it with Code43. I then tried creating a Win11 VM from scratch, and it caused my host to reboot during the VM OS installation. After some more tries, it always end up causing the host to reboot. To avoid host reboot during Win11 VM creation, I had to disable ReBAR in bios. Quote Link to comment
KptnKMan Posted August 16, 2023 Share Posted August 16, 2023 To everyone responding in this thread, READ THE THREAD, all of it. Read the instructions I left and the responses. Read the linked Reddit post. The latest unraid 6.12.x DOES NOT REQUIRE THE CUSTOM BIOS in the first few comments. I left an updated summary just a little bit back, that definitely works on both my systems. A few extra pointers: -make sure your VM works without REBAR enabled first, mine did. -make sure to use the unraid VM REBAR bios, without checking, ita not rhe SeaBIOS, the other one. Make sure to use the UEFI-enabled VM BIOS. -make sure you are passing through both the gfx card AND the cards sound device. I would suggest no other audio devices in the VM. -read other threads about passthrough FIRST and get that working, this thread isn't diagnosing basic passthrough issues like code 43. -these instructions REQUIRE manual XML editing, if you then go to update the VM in GUI mode, the custom REQUIRED XML edits will be lost. Be aware. Quote Link to comment
mjorud Posted August 25, 2023 Share Posted August 25, 2023 Hi, I have been tinkering with ReBAR and general performance in a W11 Gaming VM. Hardware is i7-12700K, Asus PRIME Z690M-PLUS D4 and Asus RTX 3060 Ti Dual Mini v2. The GPU does support ReBAR. The VM has 12 CPUs and 32 GB RAM allocated. Samsung 990 Pro is passed through as the only disk. The VM is working fine. Not great, but fine. I say fine because it's a hugh step down from bare metal in regard of benchmarks. A little overhead is expected though. I have followed this and other threads and I think I have done everything correctly. GPU-Z shows that ReBAR is enabled... but that the GPU is unsupported. The VM in installed on a NVMe so when booting bare metal the all is good; the GPU is supported. Looking in the Device Manager, Large Memory Range is there. I have to admit that I have not yet tried the user script... This is my results from PerformanceTest. Best regards J 1 Quote Link to comment
mjorud Posted August 28, 2023 Share Posted August 28, 2023 (edited) I'm trying the patch but for some reason I'm missing the "resource1_resize" under "/sys/bus/pci/devices/0000\:01\:00.0/". "cat resource1_resize" only gives me "cat: resource1_resize: No such file or directory". I assume that my problem lies there? But if i run "lspci -vvvs 01:00.0 | grep "BAR"" I get: Capabilities: [bb0 v1] Physical Resizable BAR BAR 0: current size: 16MB, supported: 16MB BAR 1: current size: 8GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB BAR 3: current size: 32MB, supported: 32MB So by the look of that, everything looks OK. Edited August 28, 2023 by mjorud lspci Quote Link to comment
mjorud Posted August 30, 2023 Share Posted August 30, 2023 I updated from 6.12.3 to 6.12.4-rc19 today and now I have the resource1_resize folder. Ran the script, but I still have "Unsupported GPU" in GPU-Z. A bonus is that I have seen a 13% increase for both 2D and 3D when running PassMark PerformanceTest after the update. 1 Quote Link to comment
opentux Posted December 5, 2023 Share Posted December 5, 2023 (edited) Hello, I have a question before I get started, in an unraid configuration to do gpu passthrough with rebar... under unraid 6.12.5 or 6.12.6. is it always necessary to patch the kernel? or is it supposed to work straight away. i see that some have had to patch on certain versions and others where it worked straight away. is this random? i currently have two nas under unraid without gpu. and i'm about to install my main machine (x570 ryzen 9 5950x 128gb ram and nvidia 3090) under unraid. i've seen some people do a conf with two gpu's and others only one. is there anything i should know before embarking on this idea? i've already tested unraid on my main machine and it works pretty well. but i've never tested gpu passthrough. and today i have the rebar active on windows... but i'd like to switch to full unraid to have several test operating systems (windows and linux) with access to the gpu with preferably the rebar working under windows for gaming... sorry in advance if my question seems silly. thank you in advance for your feedback or opinion on the matter. bye update * I've just come across a guide on the same subject. I'll try it out. but I'm still interested in your opinions ... thanks in advance Edited December 5, 2023 by opentux find a guide on unraid forum Quote Link to comment
reb0rn Posted December 12, 2023 Share Posted December 12, 2023 I have 5900x 64G, resbar enabled in bios, uefi boot and 3x 7900 xfx, this worked for me, but I had to adjust rebar to 4MB/32G or I get black screen, I have no idea why more can not work I am on acs 6.3 kernel as I need to have 3 GPU in 3 VM so their own group each maybe I need to adjust opt/ovmf/X-PciMmio64Mb,string=65536 for 24G GPU or I need to set more ram per VM (atm 14G) Quote Link to comment
opentux Posted December 20, 2023 Share Posted December 20, 2023 (edited) i will test with acs on 6.3 kernel have you a guide for this ? because i test this guide : https://gitlab.com/stefan99353/unraid-custom-kernel/-/blob/main/README.md but i have an error root@ryzen:/mnt/ssd/Data/unraid/linux# make oldconfig make: error while loading shared libraries: libffi.so.7: cannot open shared object file: No such file or directory i have install libfii and other with un-get, but i still this error. Do I have to do this from another operating system, vm or docker? thanks Edited December 20, 2023 by opentux modify precision Quote Link to comment
seyar Posted January 4 Share Posted January 4 (edited) Hi ! Stuck with this although I followed steps provided earlier : Config : Intel Arc 380 Unraid 6.12.3 here, with @thor2002ro's 6.2 kernel Machine: Q35-7.1 BIOS : OVMF-TPM Followed the following steps : - Host BIOS Enable ReBAR support - Host BIOS Enable 4G Decoding - Enable & Boot Custom Kernel syslinux configuration (near beginning of this thread) apparently not needed anymore - VM must have the top line of XML from <domain type='kvm'> to: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> - VM must have added the following (after the </device> line, before the </domain> line): <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> Only steps I'm unsure are the following : - Boot Unraid in UEFI Mode - VM must use UEFI BIOS Pretty sure it's a yes, but how can I be sure of it ? Attached some screenshots of config+ BIOS config + VM XML here : https://pastebin.com/6Re8fs6Z Any clue ? Thanks for your help! Edited January 4 by seyar Quote Link to comment
JesterEE Posted February 18 Share Posted February 18 With the Unraid 6.12 series on Linux kernel 6.1 natively, I decided to finally revisit this topic with my update to 6.12.8. After the OS update, I checked the lspci output to see if the OS was correctly assigning the correct memory size allocation for my ASUS KO GeForce RTX 3070 V2 OC Edition 8GB. I was pleasantly surprised that without doing anything, it was assigning the resource space to the maximum video memory allotment my card is able to provide (i.e. 8GB) (see full lspci output at bottom of this post). # lspci -vvvs 0c:00.0 0c:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070 Lite Hash Rate] (rev a1) (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. GA104 [GeForce RTX 3070 Lite Hash Rate] Capabilities: [bb0 v1] Physical Resizable BAR BAR 0: current size: 16MB, supported: 16MB BAR 1: current size: 8GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB BAR 3: current size: 32MB, supported: 32MB Note the BAR 1 size is set to 8GB. Before the kernel update (and with the kernel patch referenced in the earlier pages of this thread), it was set to a default of 256MB. All is looking good so far! I followed the these baseline steps ✅ Host BIOS UEFI w/o CSM ✅ Host BIOS Enable ReBAR support ✅ Host BIOS Enable 4G Decoding ⬛ Enable & Boot Custom Kernel syslinux configuration (near beginning of this thread) not needed anymore Before modifying my Windows 10 Pro VM configuration, I booted up the VM to see if anything was needed for the Guest OS to recognize ReBAR. I did make sure my guest bios was set to OVMF TPM (regular OVMF provided the same result as shown below though). Windows booted without issue and I ran both GPU-Z 2.57.0 and the NVIDIA Control Panel to check ReBAR support: This is what I saw: GPU-Z reported ReBAR as Enabled, but when I went into the Advanced settings, 4G Decode was shown as Disabled in BIOS. NVIDIA Control Panel shows ReBAR an Enabled and shows it's correctly allocating 8GB of dedicated video memory with an additional 16GB of shared memory for 24GB total. If I close the apps and relaunch them, GPU-Z reports differently, showing ReBAR as Disabled with the same advanced details (NVIDIA Control Panel stays reporting ReBAR Enabled with the same details). I shut down the VM and tried the XML edits noted in this thread and other online spaces talking about VFIO ReBAR: <domain type='kvm'> ➡️ <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> After relaunching the VM, I found the results to be the same. So, this is interesting in that the XML may not be required for ReBAR anymore either. However, since I'm getting inconsistent reporting using GPU-Z and the NVIDIA Control Panel, I can't be sure. I think I trust NVIDIA Control Panel more than GPU-Z on this one even though GPU-Z has never steered me wrong in the past. I figure the hardware vendors driver information software probably knows better and GPU-Z is looking at some inconsistent information and reporting incorrectly. But, I think putting a synthetic benchmark to test Host BIOS setting differences is probably called for in this scenario (ReBar and 4G Decoding On vs Off). I'll report some of that in a follow-up post. Anyone else see something similar to what I'm seeing and have verified ReBAR functional in their VM? -JesterEE # lspci -vvvs 0c:00.0 0c:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070 Lite Hash Rate] (rev a1) (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. GA104 [GeForce RTX 3070 Lite Hash Rate] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 141 IOMMU group: 30 Region 0: Memory at fb000000 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at 7c00000000 (64-bit, prefetchable) [size=8G] Region 3: Memory at 7e00000000 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at f000 [size=128] Expansion ROM at fc000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00000 Data: 0000 Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s (downgraded), Width x8 (downgraded) TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range AB, TimeoutDis+ NROPrPrP- LTR- 10BitTagComp+ 10BitTagReq+ OBFF Via message, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS- LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+ EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Capabilities: [258 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=255us PortTPowerOnTime=10us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=0ns L1SubCtl2: T_PwrOn=10us Capabilities: [128 v1] Power Budgeting <?> Capabilities: [420 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Capabilities: [900 v1] Secondary PCI Express LnkCtl3: LnkEquIntrruptEn- PerformEqu- LaneErrStat: 0 Capabilities: [bb0 v1] Physical Resizable BAR BAR 0: current size: 16MB, supported: 16MB BAR 1: current size: 8GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB BAR 3: current size: 32MB, supported: 32MB Capabilities: [c1c v1] Physical Layer 16.0 GT/s <?> Capabilities: [d00 v1] Lane Margining at the Receiver <?> Capabilities: [e00 v1] Data Link Feature <?> Kernel driver in use: vfio-pci Kernel modules: nvidia_drm, nvidia Quote Link to comment
mjorud Posted February 26 Share Posted February 26 (edited) I have see the same thing. GPU-Z shows that Resizable BAR is enabled, even though "Resizable BAR enabled in BIOS" shows Unsupported GPU. Nvidia shows that Resizable BAR is enabled with 8192 MB GDDR6 dedicated video memory and 24572 MB total available graphics memory. Does ReBAR work? Not sure, and I'm not sure how to confirm it in-game. Edited February 26 by mjorud Quote Link to comment
Noggers Posted March 16 Share Posted March 16 Hi Guys - I have been trying to enable resizable bar in a windows vm with and AMD 6900XT GPU. Has anyone actually managed to get resizable bar working for AMD GPUs in Unraid, or is it a case of it only works with nVidia GPUs? I have rebar enabled in bios as well as above 4g decoding. Have the array start script to set the resizable bar to 16GB which this card is capable of. Have also made the extra xml changes to the vm definition. I see no errors when the script runs, and the systems starts to boot but when it tries to load the video driver, the screen goes black and turns off. I haven't seen any posts so far with AMD card working - so was wondering if I am wasting my time or if indeed its possible to get an AMD card to work with REBAR enabled in unraid? if i turn rebar off in bios, vm boots normally and windows loads without issue. VM Def: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>wksRod</name> <uuid>xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx</uuid> <description>Rods Workstation</description> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/> </metadata> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='36'/> <vcpupin vcpu='2' cpuset='5'/> <vcpupin vcpu='3' cpuset='37'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='38'/> <vcpupin vcpu='6' cpuset='7'/> <vcpupin vcpu='7' cpuset='39'/> <emulatorpin cpuset='3,35'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-7.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader> <nvram>/etc/libvirt/qemu/nvram/d62a5da1-46ea-cbf7-dce7-a57944565dd7_VARS-pure-efi-tpm.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='4' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback' discard='unmap'/> <source file='/mnt/user/domains/wksRod/vdisk1.img'/> <target dev='hdc' bus='scsi'/> <serial>vdisk1</serial> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/en-us_windows_11_consumer_editions_version_22h2_updated_nov_2022_x64_dvd_c148a37b.iso'/> <target dev='hda' bus='ide'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.248-1.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:3d:14:8d'/> <source bridge='br0'/> <model type='virtio-net'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <tpm model='tpm-tis'> <backend type='emulator' version='2.0' persistent_state='yes'/> </tpm> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x23' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x23' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x25' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x46' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x47' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x47' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x47' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x4a' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> </domain> User Script: #!/bin/bash #echo -n "0000:23:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind #echo 14 > /sys/bus/pci/devices/0000\:23\:00.0/resource0_resize #echo -n "1002 73af" > /sys/bus/pci/drivers/vfio-pci/new_id || echo -n "0000:23:00.0" > /sys/bus/pci/drivers/vfio-pci/bind If I tried to use resource1_resize, i would get an error that file already exists. VGA PCI Any thoughts on what I might be doing wrong? Cheers Rod. Quote Link to comment
Noggers Posted March 16 Share Posted March 16 (edited) Hi Guys, quick update - Just for sh!ts and giggles, I turned off REBAR in BIOS, and I adjusted the REBAR size from 16GB to 8GB in the script and the VM now launches and shows the large memory addressing in the GPU. So even though the card is capable of 16GB REBAR size, the VM didn't like it. Moved it to the next one down, and it worked. Here is the updated user script. #!/bin/bash echo -n "0000:23:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind echo 13 > /sys/bus/pci/devices/0000\:23\:00.0/resource0_resize echo -n "1002 73af" > /sys/bus/pci/drivers/vfio-pci/new_id || echo -n "0000:23:00.0" > /sys/bus/pci/drivers/vfio-pci/bind Remember to adjust the id's and addresses based on your system configuration. Hope this helps some others - it absolutely works with AMD cards and works very well. Buttery smooth with REBAR enabled. I did some testing with Tiny Tinas Wonderlands on the Video benchmarking and that no longer stutters, whereas before it would stutter quite a bit. Cheers Rod. UPDATE - I just realised the script was commented out - updated accordingly, and gave myself an upper cut while at it. UPDATE 2 - I was able to get the full 16GB REBAR enabled and the machine is working fine. So using the script above, as well as in my case disabling REBAR In Bios solved the issue and the its working nicely. Your mileage may vary. Edited March 18 by Noggers Adding an update after more tinkering. Quote Link to comment
Salty2011 Posted June 6 Share Posted June 6 (edited) Howdy All After reading all the posts, I now have my 6900XT passing through and in GPU-Z it says resize bar is enabled. With that said there are the following issues i am getting: For AMD if you open up the AMD Driver console and view the Smart Access Memory status under "Smart Technology" it says unsupported. Not sure if this is just an AMD / VM bug and its safe to ignore For the user script that creates the resizebar resource (in this guide below) i get the following error Not sure why when rebinding the device its saying it exist despite it being unbound in the beginning. If anyone can shed some like on this will be appreciated and I will update this post Now for everyone else, this is what I did in order to get it working to this point PreRequisits In the BIOS make sure you have these enabled AMD-V or Intel VT-d SR-IOV Enable Above 4G Decoding Do not enable Resize BAR Unraid Ensure you have community app installed Ensure you have the user scripts plugin In the UNRAID WebUI you will need to ensure that you have bound the gfx card to vfio so that unraid doesnt reserve it preventing the card from being used. (I am using UNRAID 6.12.10) To do this: Goto Tools > System Devices Check all the PCI devices that are you graphics card. Not a Graphics card can be listed as multiple devices, In the example below the 6900XT has 4 devices In the pic above I have highlighted two things that you will need to note for later - GPU Address ( thats the 0a:00.0) - GPU ID ( thats the 1002:73bf) Yours will be different to the ones I have listed so make sure to note your down. Additionally note i ticked serval devices that all share the same address space of 0a:00.x, x being .0 /1 /2/3. For your setup make sure you have ticked all other devices in the same address space. Youll notice the descriptions will also loosely correlate. When done scroll to the bottom and click "BIND SELECTED DEVICES TO VFIO AT BOOT", you will then need to reboot the server before continuing. Initial Steps Create a normal Windows 11 VM, do not pass any additional hardware Go through the standard Windows Install Perform all updates as Microsoft suck and don't update there ISO and AMD and Likely NVIDIA drivers will need some of the latest updates for these to even install Download the latest full install of either nvidia or amd (do not use the autodetect installed, this can act weird). do not install at this time. that will come later Show down the machine when done GPU Passthrough - No Resize BAR Do this first just to ensure its all working without ReSize BAR enabled on the VM In Unraid now go edit your VM In Form view make the following changes, Add a second GPU, set the sound card to be the GPU For AMD and maybe NVIDIA check "other pci deivces" and attached the additional devices related to the card. For example the 6900XT with show as 4 devices, the GPU, Sound and two USB device/controller. Save configuration Boot up VM and use the VNC console. Check that you see you gpu in device manager (it may have error 43, this is ok ignore it) Now install your GPU drivers and restart the vm Load up some games and test card is working within expectations (not its a VM so wont get exact performance.. well at least not without some additional tuning) If this all working, congrats you have working GPU passthrough and now for the fun of getting ReSize bar working Getting ResizeBAR Working Switch back to the unraid webui for now Reboot unraid server and enable ReSize BAR and Above 4G decode in BIOS Now open the unraid console Open up user scripts Create a new User Script and schedule it to run on array start. Add the following script, take note that you will need to alter the GPU, GPU_ID and VRAM for the card your passing through. As the beginning of this post these were the values we noted down. ##### NOTE ##### The script I have written below will select the correct bit size for your card vram, determine which GPU (AMD/NVIDIA) based on the gpu address and then set the resize values accordingly. For AMD cards resource2_resize needs to be less than 8MB for windows to boot, else you get black screen, this doesnt seem to apply to nvidia gpu's based on what I could find. I discovered the AMD resource2_resize trick over here and they have a good article explaining the setup. VFIO: How to enable Resizeable BAR (ReBAR) in your VFIO Virtual Machine REPO: https://github.com/salty2011/unraid-userscsripts #!/bin/bash # User-configurable variables GPU=0000:0a:00.0 # GPU PCI address GPU_ID="1002 73bf" # GPU ID for re-binding VRAM_SIZE="16GB" # Set VRAM size based on your GPU # Define a function to determine the bit size for rebar based on VRAM size function get_rebar_size { local vram_size=$1 case $vram_size in "2MB") echo 1 ;; "4MB") echo 2 ;; "8MB") echo 3 ;; "16MB") echo 4 ;; "32MB") echo 5 ;; "64MB") echo 6 ;; "128MB") echo 7 ;; "256MB") echo 8 ;; "512MB") echo 9 ;; "1GB") echo 10 ;; "2GB") echo 11 ;; "4GB") echo 12 ;; "8GB") echo 13 ;; "16GB") echo 14 ;; "32GB") echo 15 ;; *) echo "Unsupported VRAM size: $vram_size"; exit 1 ;; esac } # Use lspci to check the GPU manufacturer MANUFACTURER=$(lspci -v -s ${GPU} | grep 'VGA compatible controller') # Unbind GPU from vfio-pci driver echo "Unbinding GPU from vfio-pci driver..." echo ${GPU} > /sys/bus/pci/drivers/vfio-pci/unbind sleep 1 # Get the correct rebar size bit value REBAR_SIZE=$(get_rebar_size $VRAM_SIZE) # Check if the GPU is from AMD or NVIDIA and apply settings accordingly if echo "${MANUFACTURER}" | grep -q "NVIDIA"; then echo "Detected NVIDIA GPU. Applying NVIDIA-specific settings..." echo "Setting the rebar size to $VRAM_SIZE" if ! echo $REBAR_SIZE > /sys/bus/pci/devices/${GPU}/resource1_resize; then echo "Failed to set rebar size. Please check if the VRAM size variable ($VRAM_SIZE) is correct for your GPU." exit 1 fi elif echo "${MANUFACTURER}" | grep -q "AMD"; then echo "Detected AMD GPU. Applying AMD-specific settings..." echo "Setting the rebar 0 size to $VRAM_SIZE" if ! echo $REBAR_SIZE > /sys/bus/pci/devices/${GPU}/resource0_resize; then echo "Failed to set rebar 0 size. Please check if the VRAM size variable ($VRAM_SIZE) is correct for your GPU." exit 1 fi echo "Setting the rebar 2 size to 8MB" if ! echo 3 > /sys/bus/pci/devices/${GPU}/resource2_resize; then echo "Failed to set rebar 2 size to 8MB." exit 1 fi else echo "GPU manufacturer not recognized. Exiting script." exit 1 fi # Wait for settings to apply sleep 2 # Re-bind GPU echo ${GPU_ID} > /sys/bus/pci/drivers/vfio-pci/new_id || echo -n "${GPU}" > /sys/bus/pci/drivers/vfio-pci/bind echo "Configuration complete." sleep 1 Restart your UNRAID Server so that this script runs and were ready for next step To check if the script applied the setting you can lspci -vvvs GPU (eg 0a:00.0) | grep "BAR" Importantly for AMD card BAR2 is 8MB, for me it was defaulting to 265MB and this stops windows from booting. Edit the VM and remove the Virtual Graphics card and ensure the only gpu is the card you are wanting to pass through switch to XML View. At the top, line 2 you will need to edit <domain type='kvm'> to be <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> Scroll to the bottom of the XML code and above where it says </domain> insert: <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> Click update configuration Boot the VM (make sure a Monitor is connected to the passthrough GPU), you should see Windows boot up to the desktop Download GPU-Z, run it and on the main screen ResizeBAR should say enabled. or check the GPU in device manager and you see large memory allocation NOTES For the AMD card I have both Device Manager and GPU-Z say resize bar is enabled. However in the AMD Driver under Smart Technology I am seeing Smart Access Memory as unsupported. Not sure if this a quirk or something else. but if anyone knows let me know. As for the script.. i take no responsibility for any issues it causes, ive tested on my setup and it works. happy to make some amendments if other bugs occurs. Edited June 7 by Salty2011 Correcting some information and providing an updated script Quote Link to comment
mjorud Posted June 24 Share Posted June 24 Hi Salty2011! Nice work with gathering the information, and script. I have Zotac RTX 4070 Super with 12 GB of VRAM. UR sets BAR 1 to 16GB. And I think that is the reason ReBar fails. Do you know if it is possible to set it to 12 GB of VRAM? Best regards Quote Link to comment
mjorud Posted June 27 Share Posted June 27 With the new version, v7.0.0 Beta 1, GPU-Z shows that ReBar is enabled. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.