KptnKMan Posted June 16, 2023 Share Posted June 16, 2023 21 hours ago, jonathanselye said: how do i get my ideal bar size? You will need to determine that yourself, using the provided scales in the script they are written out, but essentially you need to pick one that your card memory fits in (and is supported). You haven't stated if your Arc A770 is 8GB or 16GB so I would suggest starting there for a hint. You can also try listing the compatible Bar sizes with with this command: lspci -vvvs {YOUR_GPU_ID_HERE} | grep "BAR" You should see something like this: root@unraid1:~# lspci -vvvs 0b:00.0 | grep "BAR" Capabilities: [bb0 v1] Physical Resizable BAR BAR 0: current size: 16MB, supported: 16MB BAR 1: current size: 32GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB BAR 3: current size: 32MB, supported: 32MB 21 hours ago, jonathanselye said: will it work if i already unbinded the gpu to unraid?[using vfio-pcie] I might be wrong, but I don't think you need to unbind it. That is what the script is for, I believe, running at Array Startup. 21 hours ago, jonathanselye said: and i only have </devices> line, is that the same? I'm not sure what you mean, if you have </devices> then you will certainly have <devices> somewhere, or you've got bigger problems. Follow the instructions, and read your XML config carefully, line by line. Read upwards from each line beginning to find the opening <devices>. 21 hours ago, jonathanselye said: in device manager there is a new device called "NF I2C Slave Device" when i tried updating the driver both online and latest virtio the drivers seems not available yet so i think it is working the driver just isnt available yet. Doing a quick google search led to this reddit post which indicates this device is an RGB controller on the Arc card. This is nothing to do with virtio drivers, its something on your card. Depending on the manufacturer, download their drivers or accompanying config application for their RGB control. Quote Link to comment
KptnKMan Posted June 21, 2023 Share Posted June 21, 2023 On 6/15/2023 at 12:41 PM, KptnKMan said: Hi @Skitals I took another look at the kernel patch you linked earlier. As far as I can tell, this patch is present in kernel 6.1 since v6.1-rc1 which is now present in unraid 6.12. So I wanted to ask, on the surface do you know if there would be any obvious reason that this wouldn't be present, or not work in unraid 6.12? Also, I'm yet to test 6.12 so I haven't personally verified any issues of missing ReBar in unraid 6.12 using kernel 6.1. @jonathanselye can you confirm and describe what you've done to test on unraid 6.12, what hardware you use, and what your exact results were? @Skitals and @jonathanselye, I just upgraded from 6.11.5 direct to 6.12.1. I can confirm that everything works on the Unraid 6.12.1 stock kernel 6.1.34. My daily VM has ReBar enabled. I can report any issues encountered, but everything seems to work as of right now. Quote Link to comment
ELP1 Posted July 11, 2023 Share Posted July 11, 2023 (edited) Hello, I am trying to enable Resizable Bar in my VM with Radeon 6650XT, but till now no luck. I am running Unraid 6.12. I did all steps according checklist from previous post: - Host BIOS Enable ReBAR support - Host BIOS Enable 4G Decoding - Enable & Boot Custom Kernel syslinux configuration (near beginning of this thread) - Boot Unraid in UEFI Mode - VM must use UEFI BIOS - VM must have the top line of XML from <domain type='kvm'> to: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> - VM must have added the following (after the </device> line, before the </domain> line): <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> I have GPU bind to VFIO at boot. Standard GPU passthrough is working properly. What can I try? I found some script but as far as I understand it is not needed for Unraid 6.12. Thank you for any help! Edited July 11, 2023 by ELP1 Quote Link to comment
letrain Posted August 9, 2023 Share Posted August 9, 2023 (edited) On 2/2/2023 at 11:52 AM, Trozmagon said: Just a heads up, the custom kernel wasn't necessary for me with my RTX 3090, just adding the additional xml to my VM config was enough. ok what did i do wrong? its enabled in bios. asrock calls it Clever access memory. it is on in the bios. the xml is edited. everything is on. i ran the user script.... and the vbios is up to date for the 3090.. @KptnKMan it works in bare metal..but i just can't get it to work in the VM. Edited August 10, 2023 by letrain update Quote Link to comment
letrain Posted August 10, 2023 Share Posted August 10, 2023 On 6/21/2023 at 10:20 AM, KptnKMan said: @Skitals and @jonathanselye, I just upgraded from 6.11.5 direct to 6.12.1. I can confirm that everything works on the Unraid 6.12.1 stock kernel 6.1.34. My daily VM has ReBar enabled. I can report any issues encountered, but everything seems to work as of right now. So what the final solution here? Everything in the list needed? Custom kernel? Rom resize script? Everything works bare metal. But it won't work in the vm. Unraid 6.12.3 here, is 6.12.1 needed? Was there regression? Host BIOS Enable ReBAR support - Host BIOS Enable 4G Decoding - Enable & Boot Custom Kernel syslinux configuration (near beginning of this thread) - Boot Unraid in UEFI Mode - VM must use UEFI BIOS - VM must have the top line of XML from <domain type='kvm'> to: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> - VM must have added the following (after the </device> line, before the </domain> line): <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> Quote Link to comment
letrain Posted August 10, 2023 Share Posted August 10, 2023 On 1/4/2023 at 6:39 PM, Skitals said: I back ported the resizable bar patch to unraid 6.11.5 (kernel 5.19.17). It applies and builds cleanly with the other unraid patches and unraid .config file. I attached the patch and a precompiled kernel. I can confirm it's working (resourceN_resize property present), but making it useful is another trick. resize-bar.patch 7.22 kB · 52 downloads unraid-6.11.5-kernel.zip 5.97 MB · 46 downloads How do you apply this? Quote Link to comment
jfoxwoosh Posted August 16, 2023 Share Posted August 16, 2023 Tried this on my system: Ryzen 5900x MSI B550 Motherboard RX 6800XT UnRaid 6.12.3 I followed the instructions posted here, and tried it on one of my existing Win11 VM, which ended it with Code43. I then tried creating a Win11 VM from scratch, and it caused my host to reboot during the VM OS installation. After some more tries, it always end up causing the host to reboot. To avoid host reboot during Win11 VM creation, I had to disable ReBAR in bios. Quote Link to comment
KptnKMan Posted August 16, 2023 Share Posted August 16, 2023 To everyone responding in this thread, READ THE THREAD, all of it. Read the instructions I left and the responses. Read the linked Reddit post. The latest unraid 6.12.x DOES NOT REQUIRE THE CUSTOM BIOS in the first few comments. I left an updated summary just a little bit back, that definitely works on both my systems. A few extra pointers: -make sure your VM works without REBAR enabled first, mine did. -make sure to use the unraid VM REBAR bios, without checking, ita not rhe SeaBIOS, the other one. Make sure to use the UEFI-enabled VM BIOS. -make sure you are passing through both the gfx card AND the cards sound device. I would suggest no other audio devices in the VM. -read other threads about passthrough FIRST and get that working, this thread isn't diagnosing basic passthrough issues like code 43. -these instructions REQUIRE manual XML editing, if you then go to update the VM in GUI mode, the custom REQUIRED XML edits will be lost. Be aware. Quote Link to comment
mjorud Posted August 25, 2023 Share Posted August 25, 2023 Hi, I have been tinkering with ReBAR and general performance in a W11 Gaming VM. Hardware is i7-12700K, Asus PRIME Z690M-PLUS D4 and Asus RTX 3060 Ti Dual Mini v2. The GPU does support ReBAR. The VM has 12 CPUs and 32 GB RAM allocated. Samsung 990 Pro is passed through as the only disk. The VM is working fine. Not great, but fine. I say fine because it's a hugh step down from bare metal in regard of benchmarks. A little overhead is expected though. I have followed this and other threads and I think I have done everything correctly. GPU-Z shows that ReBAR is enabled... but that the GPU is unsupported. The VM in installed on a NVMe so when booting bare metal the all is good; the GPU is supported. Looking in the Device Manager, Large Memory Range is there. I have to admit that I have not yet tried the user script... This is my results from PerformanceTest. Best regards J 1 Quote Link to comment
mjorud Posted August 28, 2023 Share Posted August 28, 2023 (edited) I'm trying the patch but for some reason I'm missing the "resource1_resize" under "/sys/bus/pci/devices/0000\:01\:00.0/". "cat resource1_resize" only gives me "cat: resource1_resize: No such file or directory". I assume that my problem lies there? But if i run "lspci -vvvs 01:00.0 | grep "BAR"" I get: Capabilities: [bb0 v1] Physical Resizable BAR BAR 0: current size: 16MB, supported: 16MB BAR 1: current size: 8GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB BAR 3: current size: 32MB, supported: 32MB So by the look of that, everything looks OK. Edited August 28, 2023 by mjorud lspci Quote Link to comment
mjorud Posted August 30, 2023 Share Posted August 30, 2023 I updated from 6.12.3 to 6.12.4-rc19 today and now I have the resource1_resize folder. Ran the script, but I still have "Unsupported GPU" in GPU-Z. A bonus is that I have seen a 13% increase for both 2D and 3D when running PassMark PerformanceTest after the update. 1 Quote Link to comment
opentux Posted December 5, 2023 Share Posted December 5, 2023 (edited) Hello, I have a question before I get started, in an unraid configuration to do gpu passthrough with rebar... under unraid 6.12.5 or 6.12.6. is it always necessary to patch the kernel? or is it supposed to work straight away. i see that some have had to patch on certain versions and others where it worked straight away. is this random? i currently have two nas under unraid without gpu. and i'm about to install my main machine (x570 ryzen 9 5950x 128gb ram and nvidia 3090) under unraid. i've seen some people do a conf with two gpu's and others only one. is there anything i should know before embarking on this idea? i've already tested unraid on my main machine and it works pretty well. but i've never tested gpu passthrough. and today i have the rebar active on windows... but i'd like to switch to full unraid to have several test operating systems (windows and linux) with access to the gpu with preferably the rebar working under windows for gaming... sorry in advance if my question seems silly. thank you in advance for your feedback or opinion on the matter. bye update * I've just come across a guide on the same subject. I'll try it out. but I'm still interested in your opinions ... thanks in advance Edited December 5, 2023 by opentux find a guide on unraid forum Quote Link to comment
reb0rn Posted December 12, 2023 Share Posted December 12, 2023 I have 5900x 64G, resbar enabled in bios, uefi boot and 3x 7900 xfx, this worked for me, but I had to adjust rebar to 4MB/32G or I get black screen, I have no idea why more can not work I am on acs 6.3 kernel as I need to have 3 GPU in 3 VM so their own group each maybe I need to adjust opt/ovmf/X-PciMmio64Mb,string=65536 for 24G GPU or I need to set more ram per VM (atm 14G) Quote Link to comment
opentux Posted December 20, 2023 Share Posted December 20, 2023 (edited) i will test with acs on 6.3 kernel have you a guide for this ? because i test this guide : https://gitlab.com/stefan99353/unraid-custom-kernel/-/blob/main/README.md but i have an error root@ryzen:/mnt/ssd/Data/unraid/linux# make oldconfig make: error while loading shared libraries: libffi.so.7: cannot open shared object file: No such file or directory i have install libfii and other with un-get, but i still this error. Do I have to do this from another operating system, vm or docker? thanks Edited December 20, 2023 by opentux modify precision Quote Link to comment
seyar Posted January 4 Share Posted January 4 (edited) Hi ! Stuck with this although I followed steps provided earlier : Config : Intel Arc 380 Unraid 6.12.3 here, with @thor2002ro's 6.2 kernel Machine: Q35-7.1 BIOS : OVMF-TPM Followed the following steps : - Host BIOS Enable ReBAR support - Host BIOS Enable 4G Decoding - Enable & Boot Custom Kernel syslinux configuration (near beginning of this thread) apparently not needed anymore - VM must have the top line of XML from <domain type='kvm'> to: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> - VM must have added the following (after the </device> line, before the </domain> line): <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> Only steps I'm unsure are the following : - Boot Unraid in UEFI Mode - VM must use UEFI BIOS Pretty sure it's a yes, but how can I be sure of it ? Attached some screenshots of config+ BIOS config + VM XML here : https://pastebin.com/6Re8fs6Z Any clue ? Thanks for your help! Edited January 4 by seyar Quote Link to comment
JesterEE Posted February 18 Share Posted February 18 With the Unraid 6.12 series on Linux kernel 6.1 natively, I decided to finally revisit this topic with my update to 6.12.8. After the OS update, I checked the lspci output to see if the OS was correctly assigning the correct memory size allocation for my ASUS KO GeForce RTX 3070 V2 OC Edition 8GB. I was pleasantly surprised that without doing anything, it was assigning the resource space to the maximum video memory allotment my card is able to provide (i.e. 8GB) (see full lspci output at bottom of this post). # lspci -vvvs 0c:00.0 0c:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070 Lite Hash Rate] (rev a1) (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. GA104 [GeForce RTX 3070 Lite Hash Rate] Capabilities: [bb0 v1] Physical Resizable BAR BAR 0: current size: 16MB, supported: 16MB BAR 1: current size: 8GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB BAR 3: current size: 32MB, supported: 32MB Note the BAR 1 size is set to 8GB. Before the kernel update (and with the kernel patch referenced in the earlier pages of this thread), it was set to a default of 256MB. All is looking good so far! I followed the these baseline steps ✅ Host BIOS UEFI w/o CSM ✅ Host BIOS Enable ReBAR support ✅ Host BIOS Enable 4G Decoding ⬛ Enable & Boot Custom Kernel syslinux configuration (near beginning of this thread) not needed anymore Before modifying my Windows 10 Pro VM configuration, I booted up the VM to see if anything was needed for the Guest OS to recognize ReBAR. I did make sure my guest bios was set to OVMF TPM (regular OVMF provided the same result as shown below though). Windows booted without issue and I ran both GPU-Z 2.57.0 and the NVIDIA Control Panel to check ReBAR support: This is what I saw: GPU-Z reported ReBAR as Enabled, but when I went into the Advanced settings, 4G Decode was shown as Disabled in BIOS. NVIDIA Control Panel shows ReBAR an Enabled and shows it's correctly allocating 8GB of dedicated video memory with an additional 16GB of shared memory for 24GB total. If I close the apps and relaunch them, GPU-Z reports differently, showing ReBAR as Disabled with the same advanced details (NVIDIA Control Panel stays reporting ReBAR Enabled with the same details). I shut down the VM and tried the XML edits noted in this thread and other online spaces talking about VFIO ReBAR: <domain type='kvm'> ➡️ <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> After relaunching the VM, I found the results to be the same. So, this is interesting in that the XML may not be required for ReBAR anymore either. However, since I'm getting inconsistent reporting using GPU-Z and the NVIDIA Control Panel, I can't be sure. I think I trust NVIDIA Control Panel more than GPU-Z on this one even though GPU-Z has never steered me wrong in the past. I figure the hardware vendors driver information software probably knows better and GPU-Z is looking at some inconsistent information and reporting incorrectly. But, I think putting a synthetic benchmark to test Host BIOS setting differences is probably called for in this scenario (ReBar and 4G Decoding On vs Off). I'll report some of that in a follow-up post. Anyone else see something similar to what I'm seeing and have verified ReBAR functional in their VM? -JesterEE # lspci -vvvs 0c:00.0 0c:00.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070 Lite Hash Rate] (rev a1) (prog-if 00 [VGA controller]) Subsystem: ASUSTeK Computer Inc. GA104 [GeForce RTX 3070 Lite Hash Rate] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 141 IOMMU group: 30 Region 0: Memory at fb000000 (32-bit, non-prefetchable) [size=16M] Region 1: Memory at 7c00000000 (64-bit, prefetchable) [size=8G] Region 3: Memory at 7e00000000 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at f000 [size=128] Expansion ROM at fc000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee00000 Data: 0000 Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s (downgraded), Width x8 (downgraded) TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range AB, TimeoutDis+ NROPrPrP- LTR- 10BitTagComp+ 10BitTagReq+ OBFF Via message, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, AtomicOpsCtl: ReqEn- LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS- LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+ EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [b4] Vendor Specific Information: Len=14 <?> Capabilities: [100 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 Status: NegoPending- InProgress- Capabilities: [258 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=255us PortTPowerOnTime=10us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=0ns L1SubCtl2: T_PwrOn=10us Capabilities: [128 v1] Power Budgeting <?> Capabilities: [420 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn- MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 00000000 00000000 00000000 00000000 Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Capabilities: [900 v1] Secondary PCI Express LnkCtl3: LnkEquIntrruptEn- PerformEqu- LaneErrStat: 0 Capabilities: [bb0 v1] Physical Resizable BAR BAR 0: current size: 16MB, supported: 16MB BAR 1: current size: 8GB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB BAR 3: current size: 32MB, supported: 32MB Capabilities: [c1c v1] Physical Layer 16.0 GT/s <?> Capabilities: [d00 v1] Lane Margining at the Receiver <?> Capabilities: [e00 v1] Data Link Feature <?> Kernel driver in use: vfio-pci Kernel modules: nvidia_drm, nvidia Quote Link to comment
mjorud Posted February 26 Share Posted February 26 (edited) I have see the same thing. GPU-Z shows that Resizable BAR is enabled, even though "Resizable BAR enabled in BIOS" shows Unsupported GPU. Nvidia shows that Resizable BAR is enabled with 8192 MB GDDR6 dedicated video memory and 24572 MB total available graphics memory. Does ReBAR work? Not sure, and I'm not sure how to confirm it in-game. Edited February 26 by mjorud Quote Link to comment
Noggers Posted March 16 Share Posted March 16 Hi Guys - I have been trying to enable resizable bar in a windows vm with and AMD 6900XT GPU. Has anyone actually managed to get resizable bar working for AMD GPUs in Unraid, or is it a case of it only works with nVidia GPUs? I have rebar enabled in bios as well as above 4g decoding. Have the array start script to set the resizable bar to 16GB which this card is capable of. Have also made the extra xml changes to the vm definition. I see no errors when the script runs, and the systems starts to boot but when it tries to load the video driver, the screen goes black and turns off. I haven't seen any posts so far with AMD card working - so was wondering if I am wasting my time or if indeed its possible to get an AMD card to work with REBAR enabled in unraid? if i turn rebar off in bios, vm boots normally and windows loads without issue. VM Def: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>wksRod</name> <uuid>xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx</uuid> <description>Rods Workstation</description> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/> </metadata> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='36'/> <vcpupin vcpu='2' cpuset='5'/> <vcpupin vcpu='3' cpuset='37'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='38'/> <vcpupin vcpu='6' cpuset='7'/> <vcpupin vcpu='7' cpuset='39'/> <emulatorpin cpuset='3,35'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-7.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader> <nvram>/etc/libvirt/qemu/nvram/d62a5da1-46ea-cbf7-dce7-a57944565dd7_VARS-pure-efi-tpm.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='4' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback' discard='unmap'/> <source file='/mnt/user/domains/wksRod/vdisk1.img'/> <target dev='hdc' bus='scsi'/> <serial>vdisk1</serial> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/en-us_windows_11_consumer_editions_version_22h2_updated_nov_2022_x64_dvd_c148a37b.iso'/> <target dev='hda' bus='ide'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.248-1.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:3d:14:8d'/> <source bridge='br0'/> <model type='virtio-net'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <tpm model='tpm-tis'> <backend type='emulator' version='2.0' persistent_state='yes'/> </tpm> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x23' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x23' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x04' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x25' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x46' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x47' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x47' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x47' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x4a' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> </domain> User Script: #!/bin/bash #echo -n "0000:23:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind #echo 14 > /sys/bus/pci/devices/0000\:23\:00.0/resource0_resize #echo -n "1002 73af" > /sys/bus/pci/drivers/vfio-pci/new_id || echo -n "0000:23:00.0" > /sys/bus/pci/drivers/vfio-pci/bind If I tried to use resource1_resize, i would get an error that file already exists. VGA PCI Any thoughts on what I might be doing wrong? Cheers Rod. Quote Link to comment
Noggers Posted March 16 Share Posted March 16 (edited) Hi Guys, quick update - Just for sh!ts and giggles, I turned off REBAR in BIOS, and I adjusted the REBAR size from 16GB to 8GB in the script and the VM now launches and shows the large memory addressing in the GPU. So even though the card is capable of 16GB REBAR size, the VM didn't like it. Moved it to the next one down, and it worked. Here is the updated user script. #!/bin/bash echo -n "0000:23:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind echo 13 > /sys/bus/pci/devices/0000\:23\:00.0/resource0_resize echo -n "1002 73af" > /sys/bus/pci/drivers/vfio-pci/new_id || echo -n "0000:23:00.0" > /sys/bus/pci/drivers/vfio-pci/bind Remember to adjust the id's and addresses based on your system configuration. Hope this helps some others - it absolutely works with AMD cards and works very well. Buttery smooth with REBAR enabled. I did some testing with Tiny Tinas Wonderlands on the Video benchmarking and that no longer stutters, whereas before it would stutter quite a bit. Cheers Rod. UPDATE - I just realised the script was commented out - updated accordingly, and gave myself an upper cut while at it. UPDATE 2 - I was able to get the full 16GB REBAR enabled and the machine is working fine. So using the script above, as well as in my case disabling REBAR In Bios solved the issue and the its working nicely. Your mileage may vary. Edited March 18 by Noggers Adding an update after more tinkering. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.