November 10, 20241 yr You Mileage will vary... Form EOL, Repos to what is doing what.. Its comes down to what you want it to do and how you want to interact with it... While the Dev don't want Unraid Virtualized as it can cause instability. I thought I would share my notes on how I'm running Unraid as a VM under Proxmox... YOU WILL NEED TO USE A HBA FOR UNRAID DISKs this is due to how unraid spins down and handles the array... while disk by ID can work you add extra disk thrashing to them... To accomplish this you will need to use a HBA... There are quite a few edits and changes that need to be done to the host for the VM and inside Unraid... Step 1 install Debian Install proxmox If you install proxmox via debian you get to keep debain security and release. Are able to use grub over system Linux / proxmox boot tool and get a better experience with how boot disk and other storage volumes are used. Debian install wiki instructions: https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm *Note that you can replace chrony with any other NTP daemon, but we recommend against using systemd-timesyncd on server systems, and the ntpsec-ntpdate option might conflict with bringing up networking on boot on some hardware. Configure packages which require user input on installation according to your needs. it is best to disable the systemd-timesyncd service to not brick the system on reboot... use chrony form proxmox... ??? systemctl disable systemd-timesyncd systemctl systemd-timesyncd disable Debian Proxmox Source list to install PVE https://pve.proxmox.com/wiki/Package_Repositories root@pve:~# cat /etc/apt/sources.list #deb cdrom:[Debian GNU/Linux 12.7.0 _Bookworm_ - Official amd64 NETINST with firmware 20240831-10:38]/ bookworm contrib main non-free-firmware deb http://deb.debian.org/debian/ bookworm main non-free-firmware deb-src http://deb.debian.org/debian/ bookworm main non-free-firmware deb http://security.debian.org/debian-security bookworm-security main non-free-firmware deb-src http://security.debian.org/debian-security bookworm-security main non-free-firmware # bookworm-updates, to get updates before a point release is made; # see https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_updates_and_backports deb http://deb.debian.org/debian/ bookworm-updates main non-free-firmware deb-src http://deb.debian.org/debian/ bookworm-updates main non-free-firmware # This system was installed using small removable media # (e.g. netinst, live or single CD). The matching "deb cdrom" # entries were disabled at the end of the installation process. # For information about how to configure apt package sources, # see the sources.list(5) manual. #Proxmox Sources #deb http://deb.debian.org/debian bookworm main contrib #deb http://deb.debian.org/debian bookworm-updates main contrib # security updates #deb http://security.debian.org/debian-security bookworm-security main contrib # Proxmox VE pve-no-subscription repository provided by proxmox.com, # NOT recommended for production use deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription deb http://download.proxmox.com/debian/ceph-reef bookworm no-subscription #proxmo7.2 #deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription *Proxmox Sub not required, but helpful in security and usability... With PVE installed, lets go through Proxmox VFIO / PCIE device pass through. https://pve.proxmox.com/wiki/PCI(e)_Passthrough https://pve.proxmox.com/wiki/PCI_Passthrough #Grub Stuff: https://wiki.ubuntu.com/Kernel/KernelBootParameters https://manpages.ubuntu.com/manpages/bionic/man7/kernel-command-line.7.html nano /etc/default/grub we look at line: GRUB_CMDLINE_LINUX_DEFAULT= *Order is everything! consoleblank=0 libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 kvm.ignore_msrs=1 intel_iommu=on pcie_acs_override=downstream,multifunction nvme_core.default_ps_max_latency_us=5500 default_hugepagesz=1G hugepagesz=1G transparent_hugepage=always rootflags=noatime pci=noaer pcie_aspm=off intremap=no_x2apic_optout Other noteworthy grub options... (these all worked under kernel 5.15 which can be installed on proxmox 8.4 pulled from bullseye repo... #we want to blacklist the disk by ID in the HBA to assist in less disk thrashing... modprobe.blacklist=ata-diskID,ata-disID #TO help force lspci -v devices such as the HBA or G-card into unraid... pci-stub.ids= vfio-pci.ids= #GPU Pass Through Remove Frame Buffers video=vesafb:off,efifb:off,simplefb:off,astdrmfb initcall_blacklist=sysfb_init -- In my case with the Nvidia opening nouveau for mdevctl and open source drivers, I'm using VGPU... Its similar but with system linux and proxmox boot tool refresh and the kernel command line: /etc/kernel/cmdline some grub options don't exist as system Linux options... Don't forget to update the kernel and grub: if you nano /etc/default/grub then update-grub *We want to make a grub boot option for IOMMU and vfio for out HBA. as we want driver in use to be vfio-pci for the VM... List VIFO PCIDs: lspci -v to list drive in use 06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller Kernel driver in use: vfio-pci Kernel modules: ahci lspci -n -s 01:00 to list vfio Hardware ID root@pve:~# lspci -n -s 06:00 06:00.0 0106: 1b4b:9215 (rev 11) My vfio pci ID is 1b4b:9215 Example VFIO option in config: /etc/modprobe.d/vfio.conf root@pve:~# cat /etc/modprobe.d/vfio.conf #HBA-Unraid options vfio-pci ids= disable_idle_d3=1 enable_sriov disable_denylist #USB pcie card passthorugh - Unriad options vfio-pci ids= disable_idle_d3=1 enable_sriov disable_denylist #Gcard Passthrough options- Nvdia - try VGPU first... #options vfio-pci ids= disable_vga=1 #VFIO Comand options: #parm: ids: Initial PCI IDs to add to the vfio driver, format is "vendor:device[:subvendor[:subdevice[:class[:class_mask]]]]" and multiple comma separated entries can be specified (string) #parm: nointxmask: Disable support for PCI 2.3 style INTx masking. If this resolves problems for specific devices, report lspci -vvvxxx to [email protected] so the device can be fixed automatically via the broken_intx_masking flag. (bool) #parm: disable_vga: Disable VGA resource access through vfio-pci (bool) #parm: disable_idle_d3: Disable using the PCI D3 low power state for idle, unused devices (bool) #parm: enable_sriov: Enable support for SR-IOV configuration. Enabling SR-IOV on a PF typically requires support of the userspace PF driver, enabling VFs without such support may result in non-functional VFs or PF. (bool) #parm: disable_denylist: Disable use of device denylist. Disabling the denylist allows binding to devices with known errata that may lead to exploitable stability or security issues when accessed by untrusted users. (bool) Don't forget to update the kernel and grub: If you nano /etc/modprobe.d/vfio.conf then: update-initramfs -u -k all Check IOMMU and remapping: dmesg | grep -e DMAR -e IOMMU dmesg | grep 'remapping' So we should now have IOMMU enabled... Next udev rule and grub black list for disk by id. https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM) YOU HAVE BEEN WARNED! Hot-Plug/Add physical device as new virtual SCSI/sata disk: qm set 592 -scsi2 /dev/disk/by-id/ata-DiskID I will still recommend using a HBA due to the 1 device removing host access to the VM. Doing this may cause other things which is why need to setup rules and other stuff on Proxmox to fix this... apt install lshw lsblk ls -l /dev/disk/by-id lshw -class disk -class storage Example: root@pve:~# ls -l /dev/disk/by-id total 0 lrwxrwxrwx 1 root root 9 Nov 9 13:00 ata-WD_easystore_240GB_####### -> ../../sdd /dev/disk/by-id/ata-WD_easystore_240GB_####### grub I blacklist ata-WD_easystore_240GB_####### Here we need a list of ata-WD_easystore_240GB_####### for udev rules... Create a Udev Rule sudo nano /etc/udev/rules.d/99-ignore-vm-disks.rules # Ignore disk /dev/sdd KERNEL=="sd*", ENV{ID_SERIAL}=="WD_easystore_240GB_#########", ENV{UDISKS_IGNORE}="1", ENV{UDISKS_PRESENTATION_HIDE}="1" ^add your disks... Restart udev and apply rule: sudo udevadm control --reload-rules sudo udevadm trigger System Reboot and test... After rebooting, verify that these disks are ignored by Proxmox and remain untouched by running: lsblk -f This configuration should ensure that the specified disks remain untouched by Proxmox and are available for pass-through to VMs. So Lets recap: We have IOMMU enabled We have an HBA ready to pass physical disk to unraid We have set Proxmox to ignore disk leaving the host to guest VM Ok we are now ready for a Unraid VM... as there are some vm ID### Conf Edits required... Lets begins create the vm set name and ID ### No Disk Drive, Linux 6.x It is recommended to use q35 and UEFI it will work for both ifx and sea bios... If you want a 5 GB cach disk... as I went beta and ditched the array for a zfs pool only, This is more for a btrfs partition for unraid swap plugin... Otherwise, hit the trashcan, no disk... as HBA will be added for them... Set # of cores you want it to run... (We will be changing some of the these setting in the vm###.conf) *Recommend to use type host and set flag based on host intel/amd for microcode... Give Unraid ram Min 8GB NOT BALLOONING RAM! unraid existing in the ram... It is recommended to use a virtual model nic such as the intel E1000 We should now have a VM to add devices too and then edit the conf manual to give it a better system run... Next let's add the USB Unraid boot disk: *Pass the USB via passing the entire port! Next lets add the HBA My HBA register as 2 separate devices under the same VFIO PCID. YOU MUST PASS BOTH IN IRQ ORDER! We are now ready to edit the config manually Locate the VM Configuration File VM exist here: /etc/pve/qemu-server/ cd /etc/pve/qemu-server/ ls to see all of your PVE VM data templates... in our case our ID is 105 so we will edit 105.conf nano /etc/pve/qemu-server/105.conf We need to fix some things... 1 add args. If your AMD: args: -device amd-iommu This allows Immu for VM in unraid... If intel use args: -device intel-iommu next we need to change the cpu: option... cpu: x86-64-v2-AES to: cpu: host,hidden=1,flags=+pcid *its recommend to run hidden like windows as this hides that kvm is running and can help with pcie device passthough and help stop code 43 errors... Save and exit. You now have a good to go Unraid VM: Last, we set unriad USB to boot first. *If unraid won't boot with UEFI edit and use sea bios... Edited December 4, 20241 yr by bmartino1 Topic Name change with Typo - Data fixes
November 10, 20241 yr Author *I have this working with stable 6.12.13 and beta 7 v4 Unraid v7 rc.1 With that, we should now see a console and the ability to boot into Unriad: Example of my experimental Test machine for unraid so lets load the Web UI and make some key changes to help fix some small things... Change 1 since this is beta 7... I use some user script to fix some things here and there... Some are nice, but all are optional... What's required... Well, in my case q35 sea bios has a known error noted in the bugs. So q35 uefi doesn't have/show this, and some older and operon had these issues. Kernel 6... so let's disable that feature... Web UI > Main > Syslinux Configuration: kernel /bzimage append initrd=/bzroot default_hugepagesz=1G hugepagesz=1G transparent_hugepage=always kernel.cprng-disable.jitterentropy=true *other syslinux command line posted later... to help vm guest os... This will fix a cngr rng at kernel start if any for errors. See bug report: If you want QEMU guest agent installed, we need to look at the extra folder and install some software. DO AT YOUR OWN RISK!!! Thanks SimonF! For the QEMU agent, assist and script! as the last prepackaged binary was for 14, one doesn't exist for 15 / current... we need the qemu-guest-agent package... Example 3rd party downloads: https://slackbuilds.org/repository/14.2/system/qemu-guest-agent/ https://slackware.pkgs.org/current/alienbob-x86_64/qemu-guest-agent-6.2.0-x86_64-4alien.txz.html https://packages.slackonly.com/pub/packages/14.2-x86_64/system/qemu-guest-agent/ We also need a missing lib...: https://slackware.pkgs.org/15.0/slackware-x86_64/liburing-2.1-x86_64-2.txz.html *Do at your Own RISK!!!! Better to use SimonF command in the go file! cd /boot/extra/ wget https://slackware.uk/slackware/slackware64-15.0/slackware64/l/liburing-2.1-x86_64-2.txz wget https://slackware.uk/people/alien/sbrepos/current/x86_64/qemu/qemu-guest-agent-6.2.0-x86_64-4alien.txz great. we now need to reboot to apply the grub setting and install the packages... we need to make a user script to start the qemu guest agent... old script save for code prosperity... #!/bin/bash sleep 10 chmod +x /etc/rc.d/rc.qemu-ga /etc/rc.d/rc.qemu-ga start *Again use SimonF info in post: So what does adding the qemu guest agent get you? Some improvements with proxmox qm commands especial for backups... and IP on dashboard: But wait there more. I'm using VGPU what about the Nvidia Driver to use the device? Let take a look: We will need to use the Nvdia plugin thanks to ICH777. Sadly, the plugin will work only for standard NVIDIA graphics cards and works very well! Step 1 lspci to see if you see your g card Install the open source Driver: update and download. Open Source Driver - This package includes the Open Source Kernel modules and the libraries/executables from the proprietary Nvidia driver package. Supported GPUs: This driver packages only supports Turing based cards and newer! ATTENTION: If you want to use the Open Source driver with GeForce and/or Workstation cards you have to create the file "/boot/config/modprobe.d/nvidia.conf" on your USB boot device with the contents, before you reboot your server: options nvidia NVreg_OpenRmEnableUnsupportedGpus=1 follow the instructions... we need that modprobe options... nano /boot/config/modprobe.d/nvidia.conf options nvidia NVreg_OpenRmEnableUnsupportedGpus=1 then reboot the unraid machine. This wil install the open Source driver. with unraid fully booted. go to the nvidia plugin (as we mainly wanted the modprobe....and some libs from open source...) click If using pascal vgpu like a tesla p4 this works with: and update & Download... Then reboot, you will now have a working NVIDIA VGPU driver running for bare min stuff like plex transcoding... Nvidia has removed and is removing support for pascal drivers... If doing VGPU its best to use a turning card... Edited December 4, 20241 yr by bmartino1 Unraid step guide- typo/data
November 10, 20241 yr Author So why won't the dev support this. Well that because Unraid and Slackware have additional requirements that a virtualized environment needs... everything above while it has been stable for me may not for you. Let's take a look at some things... Let's install the Dynamix System Info plugin and take a look around... Tools> System Profiler Bios: Motherboard: Processor Cache Memory Memory Summary Memory Devices Ethernet As you can see... The Virtualized instance that is this Unraid is missing data for what its Motherboard is...Ram shows up as 1 dim... and other data is missing... Some CPU data and information, so I'm processor locked in to using 2.0 GHZ (This is what proxmox reports to the VM as a static Value review Proxmox Summary of the VM and see the full threads/cores that are being used in load balancing...) So When I have a AMD 2.5 GHZ / overclocking... I'm still getting my performance (there is some loss being nested...) Its For This reason That YOU SHOULD NOT Virtualize UNRAID, and if it is virtualized it will be unsupported! Do so at your own risk! Then you w will then be playing the virtualized bare metal game and workaround game... That said there are some benefits to virtualization... Mainly in backups, depending on how you handle disk. Another example is to run truenas scale for nfs/samba and full zfs implementation, then run a VM virtualize a single cached disk and pass the unraid USB to make a single cache btrfs vdisk in truenas and use unraid for dockers only.... Edited December 4, 20241 yr by bmartino1 Unraid Disclaimer and warning - typo/data
November 10, 20241 yr 45 minutes ago, bmartino1 said: If you want QEMU guest agent installed we need to look at the extra folder and install some software. DO AT YOUR OWN RISK!!! It's already in Unraid. Just need to add this to go file. /usr/bin/qemu-ga -l /var/log/qemu-ga.log -d &
November 10, 20241 yr Author 4 minutes ago, SimonF said: It's already in Unraid. Just need to add this to go file. /usr/bin/qemu-ga -l /var/log/qemu-ga.log -d & Awesome, Thank you, good to know!
November 15, 20241 yr Awesome guide! I will review it and make changes to my installation. I migrated my unraid from ESXi to proxmox a few months ago and would like to run as stable as when it was on ESXi.
November 15, 20241 yr On 11/10/2024 at 8:43 PM, bmartino1 said: this is due to how unraid spins down and handles the array... First: great write-up! I have been passing disks with no problems, sure I have to deal with smartctl and spindown on the Proxmoxhost. I am now SSD-only, but when I used spinning rust, I simply ran a ”hdparm -S60 /dev/disk/by-id/YOUR-PASSEDTHROGH-DISK-HERE” on Proxmox startup to make the passed disk spin down after 5 min. Worked like a charm Edited November 15, 20241 yr by BarbaGrump
November 17, 20241 yr Author On 11/15/2024 at 11:09 AM, BarbaGrump said: First: great write-up! I have been passing disks with no problems, sure I have to deal with smartctl and spindown on the Proxmoxhost. I am now SSD-only, but when I used spinning rust, I simply ran a ”hdparm -S60 /dev/disk/by-id/YOUR-PASSEDTHROGH-DISK-HERE” on Proxmox startup to make the passed disk spin down after 5 min. Worked like a charm Glad this is helping people. I didn't realize there was a separate forum location when i originally made the post... whoops! The reason why I went proxmox is for VGPU... better VMS etc... Unraid is my go to for docker / lxc and with it already stable with my data and samba share i don't see why it shouldn't be a vm... if this is working in stable 7 release or if I can get this plugin to work with the patch / normal vgpu by running the nvidia.run file... https://github.com/stl88083365/unraid-nvidia-vgpu-driver I may go back to unraid... Cool, I was aware of hdparm and other things one could do... but even at start script with disk by id, there are other instances of similar sh corn tasks for hdparm and other commands with lshw and other tools to check proxmox hold on disk. There is more to udev rules and blacklisting... I do other things as well but as a quick baseline as a lot in on proxmox forum and their wiki... Mainly rant: *If you have the blacklist and udev from above for disk by id passing, that should be enough from older testing in proxmox 6/7 8.4 is new to me and I HATE kernel 6 for alot of things they broke... S*#T just worked in kernel 5 and still does... Proxmox and with EOL of kernel 5 this may not be viable for VGPU stuff latter... but hey it works now and currently is stable for me... (you should also install linux headers...) apt-get install pve-kernel-5.15 *Especial for VGPU with Pascal cards... I recommend turning card like the 2080 and polo guides to vgpu... https://gitlab.com/polloloco/vgpu-proxmox It more going through the kernel 6 10000 lines that were removed as a lot referenced other lines, and they are still working through some of the blowback... ALL for new ARM stuff but with open source and different types of arm... its a hole thing that i could rant on .... #Rant over... I Will still recommend a HBA... its easier to test and trouble shoot things later with the ability to boot bare-metal to the unraid flash for testing if bug stuff is needed for lime tech. yes, there are some side things you can do at proxmox host level. I will always recommend a HBA over disk by id passing. (not much changes at first boot to have unraid on bare metal with a HBA as well for support if needed...) For Me, Its has more to do with proxmox smart mon tools and auto running smart. since a bad experience in the past... when doing test with truenas with disk by id pass truenas and proxmox had more disk thrashing and problems. HBA fixed that. When testing in the past with unraid I found issues with how unraids array touches disk and found that proxmox would also try to stop unraid from spinning down the disk causing thrashing and other read writes. I rather have unraid control the disk, record smart etc. Hba fixes that... On another note... so when using proxmox installer it makes logical volumes which may also auto try and mount my zfs pool I used in unraid / truenas... and that why I will recommend installing debain first as it can be a nightmare to not have proxmox do thing stuff to other disk meant for VMs. HBA can fix that too but it about wht proxmox kernel boot sees and what fully booted Host see and when proxmox decides to do some disk actions... So, I recommend the blacklist and udev rule and to look into other promxox configs, crons, log rotate and how its default parameters are set and to adjust to your needs if doing disk by id. Edited December 4, 20241 yr by bmartino1 spelling
December 4, 20241 yr Author Posting more for share among friends for easier code grab and go... with version 7 rc1 released thought I would d add some extra info. I run AMD. i can almost clam this to be just as stable now... a side proxmox configuration Nested Virtualization... (turtles all the way down.) cat /sys/module/kvm_amd/parameters/nested ^- show "1" if nesting is enabled... AMD: echo "options kvm-amd nested=1" > /etc/modprobe.d/kvm-amd.conf modprobe -r kvm-amd modprobe kvm-amd Intel: echo "options kvm-intel nested=Y" > /etc/modprobe.d/kvm-intel.conf modprobe -r kvm-intel modprobe kvm-intel ^- to enable nested VM.... Also since microcode is applied at host level on Proxmox let make sure we have both the mitigation and microcodes for intel and AMD... apt update && apt install intel-microcode amd64-microcode swtpm Backup - This is my 100.conf I use on amd for my BMM-Unraid test Unraid instance root@pve:/etc/pve/qemu-server# cat 100.conf #10 cores 16 GB of Ram #PCIE Pass through%3A #HBA with 3x16TB disk #-PVE grub and mod probe blacklist for disk by id. if disk ever change, update there... #-USB pcie device for offsite backup and other docker devices. # #System is meant for Data storage and Dockers only! agent: 1 args: -device amd-iommu balloon: 0 boot: order=usb0 cores: 10 cpu: host,hidden=1,flags=+md-clear;+ibpb;+virt-ssbd;+amd-ssbd;+amd-no-ssb;+pdpe1gb;+aes efidisk0: data-pve:100/vm-100-disk-1.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K hostpci0: 0000:04:00,pcie=1 hostpci1: 0000:06:00,pcie=1 hostpci2: 0000:0e:00,pcie=1 hostpci3: 0000:01:00.0,mdev=nvidia-259,pcie=1 hotplug: network machine: q35 memory: 16384 meta: creation-qemu=9.0.2,ctime=1727936737 name: UnRaid net0: virtio=BC:24:11:A1:95:73,bridge=vmbr0,firewall=1 numa: 0 onboot: 1 ostype: l26 scsi0: data-pve:100/vm-100-disk-2.qcow2,backup=0,iothread=1,size=5G,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=51090d87-06eb-40b1-84c4-604c0bb7594b sockets: 1 tpmstate0: data-pve:100/vm-100-disk-0.raw,size=4M,version=v2.0 usb0: host=3-7 vmgenid: b5a6e175-eb0d-41ac-9180-232addccaad3 root@pve:/etc/pve/qemu-server# * Note the CPU Line. This is for AMD Proxmox fixed most of the setting flags in the *Depends on your CPU and unraid syslog... with v7rc1 i need to add flags for amd to report that mitigation are in affect... So I added extra flags depending on your host processor. Proxmox has introduced virtual iommu. I rather deal with bare-metal iommu... If you trust the host proxmox, you can use plugin on Unraid and disable mitigation as well... example in logs: Dec 3 21:48:36 BMM-Unraid kernel: Speculative Return Stack Overflow: IBPB-extending microcode not applied! Dec 3 21:48:36 BMM-Unraid kernel: Speculative Return Stack Overflow: WARNING: See https://kernel.org/doc/html/latest/admin-guide/hw-vuln/srso.html for mitigation options. Dec 3 21:48:36 BMM-Unraid kernel: Speculative Return Stack Overflow: Vulnerable: Safe RET, no microcode as a VM mitigation may not apply... also a updated unraid syslinux/grub command line: Command line: BOOT_IMAGE=/bzimage initrd=/bzroot default_hugepagesz=1G hugepagesz=1G transparent_hugepage=always kernel.cprng-disable.jitterentropy=true acpi=force pci=nocrs amd_iommu=on iommu=pt mitigations=auto noapic *To help with Host VM integrations. Explanation of Each Parameter default_hugepagesz=1G and hugepagesz=1G: Optimizes memory usage for high-performance workloads by enabling 1GB huge pages. transparent_hugepage=always: Ensures huge pages are used wherever possible to improve memory performance. kernel.cprng-disable.jitterentropy=true: Disables jitter entropy for better performance on virtualized environments. acpi=force: Forces ACPI to enable power and system management on virtualized environments. pci=nocrs: Avoids conflicts with PCI resource allocation reported by ACPI. amd_iommu=on: Enables AMD IOMMU for PCI passthrough and device isolation. iommu=pt: Sets IOMMU in passthrough mode for improved performance on virtualized environments. mitigations=auto: Automatically enables all available CPU mitigations for vulnerabilities like Spectre and Meltdown. noapic: Disables Advanced Programmable Interrupt Controller (APIC) for improved stability in certain virtualized setups. With testing this I have had a uptime of more then 2 month on v7 beta 4 now testing with v7 rc 1 as things seem to be working great with hardly any noticeable issues that I can find atm. Edited December 4, 20241 yr by bmartino1 7RC1 typo - data
January 13, 20251 yr Author with unriad 7 in final release... I recmoend installing the disable mitigation pluign. this is more to check that vm microcode is not vunlerale... as syslog noted a potential VM issue. this us usaly affect on how proxmox calls the cpu type. I prefer to call host and flags my CPU support... 8Noted above at "Backup - This is my 100.conf I use on amd for my BMM-Unraid test Unraid instance" Here is my stable release of v7 VM syslinux grub... My AMD current recomendation for unraid syslinux grub is: append initrd=/bzroot mitigations=auto noapic default_hugepagesz=1G hugepagesz=1G transparent_hugepage=always kernel.cprng-disable.jitterentropy=true acpi=force pci=nocrs spec_store_bypass_disable=on spectre_v2=on pti=on Other options explained above... spec_store_bypass_disable=on: Enables mitigation for speculative store bypass. spectre_v2=on: Forces Spectre v2 mitigations. pti=on: Ensures Kernel Page Table Isolation (PTI) is active. more to assist and help with nested virtualization sylog error: Jan 13 17:39:29 BMM-Unraid kernel: Speculative Return Stack Overflow: IBPB-extending microcode not applied! Jan 13 17:39:29 BMM-Unraid kernel: Speculative Return Stack Overflow: WARNING: See https://kernel.org/doc/html/latest/admin-guide/hw-vuln/srso.html for mitigation options. Jan 13 17:39:29 BMM-Unraid kernel: Speculative Return Stack Overflow: Vulnerable: Safe RET, no microcode earlier beta images loaded and protected against this. may be a kernel issue. as for AMD, The error messages you're seeing indicate that the Speculative Return Stack Overflow (SRSO) vulnerability cannot be fully mitigated without applying a microcode update from your CPU vendor. Unfortunately, without the necessary microcode update, you won't be able to fully address this vulnerability thats more a qemu/vm/promox thing to add, but you could refer and add adition microcodes... via: initrd=/bzroot,/boot/microcode/your_microcode_file ^ Advanced high level kernal and linux knowlodge needed... as you can break things doing this... install 3rdparty slackware microcode and use command at your own risk! Since Unraid doesn't natively support package managers or tools like intel-microcode or amd64-microcode, you'll need to manually apply the microcode. so agian grub commands: Mitigation Options Without Microcode mitigations=auto,nosmt spec_rstack_overflow=off mitigations=auto,nosmt: Enables most mitigations but disables simultaneous multithreading (SMT/hyper-threading), which can reduce exposure to speculative execution vulnerabilities. spec_rstack_overflow=off: Explicitly disables the SRSO mitigation check (this is a workaround, not a fix). you can always verify your mitigation is the systemlog: cat /sys/devices/system/cpu/vulnerabilities/* dmesg | grep -i spec More of a FYI... otherwise you will need to move off cpu host inthe vm config to a cpu that loads with unraids microcode.
February 17, 20251 yr Can't wait to try out some of these tweaks, been running unraid on proxmox for the last couple of years, it's been rock solid after inital config. Not sure if this has happened with 7.0 final but share to share copies are very slow, about 20mb/s. Disk to disk is fast. Doesn't affect copying anythings over the network to the share eg a with Windows copy i get over 200mb/s. Using Virtio for scsi and netw, if that makes a different. Anyway, I'm intrested to experiment, this weekend Intel based system.
March 3, 20251 yr Author with unraid v7.0.1 vgpu broke... to fix it I had to readd the unsupported to another modprobe module: and add an additional syslinux grub line. v7.0.1 I don't see any kernel level error... kernel /bzimage append initrd=/bzroot mitigations=auto,nosmt spec_rstack_overflow=off noapic default_hugepagesz=1G hugepagesz=1G transparent_hugepage=always kernel.cprng-disable.jitterentropy=true acpi=force acpi_osi=Linux pci=nocrs spec_store_bypass_disable=on spectre_v2=on pti=on irqpoll usbcore.autosuspend=-1 nvidia.NVreg_OpenRmEnableUnsupportedGpus=1 to make sure the vgpu grid driver loaded I added: nvidia.NVreg_OpenRmEnableUnsupportedGpus=1 I also had to drop form the production branch back to the v550: but proxmox vgpu is working as it should and is being detected... Using polo vgpu unlock. Thought I'd update as I again had to install the opensource driver first. add the additional gub command. and add the otehr nvida_modset option for unsupported GPU... Updating and sharing the test system and how as things progress.
April 13, 20251 yr PVE updated to 8.4 Unraid updated to 7.1.0RC1 There is a problem that the nvidia driver cannot be installed and the nvidia driver plugin does not have the 550 version anymore The open source driver of the RC version does not seem to support VGPU
April 13, 20251 yr Author 7 hours ago, 121644 said: PVE updated to 8.4 Unraid updated to 7.1.0RC1 There is a problem that the nvidia driver cannot be installed and the nvidia driver plugin does not have the 550 version anymore The open source driver of the RC version does not seem to support VGPU I have not tested and moved off 8.3 .. I also have the older kernel 5.15 something... I still on debian bookworm for security updates and see no reason to move off 8.3 if things are stable and working... no Known CVEs... Unriad Nvidia plugin via ICH777 will have a deprecated driver: Ask nvidia plugin support for a older version... I'd recommend sting on v7.0.1 Stable release, the beta has alot going on with it and I don't feel like trying to fix beta nonsense that may or may not be there in the end for stable release. As I experience with running the v7 beta to current stable release... Its also why the topic is "experimental" as I use advance linux stuff and knowledge to accomplish things. Sometimes outside the norm. Theses are more guidelines on how one can make it stable. your issue appears to be more related to unriad, VGPU or ich777 nviida driver plugin... Edited April 14, 20251 yr by bmartino1 typo
April 13, 20251 yr Author Besides. As I main tain this for some others with unraid stable 7... https://forum.proxmox.com/threads/updates-failing-after-8-3-upgrade.157884/#post-761047 nothing chagned in the guide... Quote the correct way to upgrade a Proxmox VE system is apt full-upgrade or apt-get dist-upgrade, not apt upgrade! See: https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#system_software_updates apt-get update apt-get upgrade -y reboot then Proxmox webUI updates refresh and web terminal run apt full-upgrade and i'm on the next release... if asked on cofigs such as sshd and package maintainers, keep local installed... But I see no reason to move builds...
April 14, 20251 yr Author General nvdia gpu things to try and test with unraid v7 Depending on your CPU QEMU model (this is why I orginaly used host above...) https://qemu-project.gitlab.io/qemu/system/qemu-cpu-models.html Also as said earlier as well. ICH777 has done a great job with the nvidia plugin. Many Thank to ICH777 You may be experiencing a nvidia out of support error with the nvidia driver install plugin (I did at one time that why I posted about it here earlier in the forum...) Please note that post pictures... note the system drivers... I bet if you try to search nvidia it will not show up... and may appear blank... this is a sign that unraid slackware linux didn't install or detect the nvidia gpu correctly... To not break the driver install and to keep unraid running the pluigin din't move forward with installing... so I decided to update to the next build in proxmox... i'm onw on pve 8.4.1.... update went without a htich... no issues with any previous configurations.... so, @121644 you are most likely seeing something like this: 7 hours ago, 121644 said: PVE updated to 8.4 Unraid updated to 7.1.0RC1 There is a problem that the nvidia driver cannot be installed and the nvidia driver plugin does not have the 550 version anymore The open source driver of the RC version does not seem to support VGPU as an example in system log: (as I changed my cpu to fix other things on my test server....) Quote Apr 13 20:26:01 Docker kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 240 Apr 13 20:26:01 Docker kernel: Apr 13 20:26:01 Docker kernel: nvidia 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=none Apr 13 20:26:01 Docker kernel: NVRM: The NVIDIA GPU 0000:03:00.0 (PCI ID: 10de:1e30) Apr 13 20:26:01 Docker kernel: NVRM: installed in this system is not supported by the Apr 13 20:26:01 Docker kernel: NVRM: NVIDIA 550.144.03 driver release. Apr 13 20:26:01 Docker kernel: NVRM: Please see 'Appendix A - Supported NVIDIA GPU Products' Apr 13 20:26:01 Docker kernel: NVRM: in this release's README, available on the operating system Apr 13 20:26:01 Docker kernel: NVRM: specific graphics driver download page at www.nvidia.com. Apr 13 20:26:01 Docker kernel: nvidia: probe of 0000:03:00.0 failed with error -1 Apr 13 20:26:01 Docker kernel: NVRM: The NVIDIA probe routine failed for 1 device(s). Apr 13 20:26:01 Docker kernel: NVRM: None of the NVIDIA devices were initialized. Apr 13 20:26:01 Docker kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 240 Apr 13 20:26:01 Docker kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 240 Apr 13 20:26:01 Docker kernel: Apr 13 20:26:01 Docker kernel: nvidia 0000:03:00.0: vgaarb: VGA decodes changed: olddecodes=none,decodes=none:owns=none Apr 13 20:26:01 Docker kernel: NVRM: The NVIDIA GPU 0000:03:00.0 (PCI ID: 10de:1e30) Apr 13 20:26:01 Docker kernel: NVRM: installed in this system is not supported by the Apr 13 20:26:01 Docker kernel: NVRM: NVIDIA 550.144.03 driver release. all I can say is you need to work with it, play around and test and edit things and test stuff... example of my sytem drivers with it working... So In unriad i use the following syslinux grub boot options: My bare min atm is kernel /bzimage append initrd=/bzroot mitigations=auto default_hugepagesz=1G hugepagesz=1G transparent_hugepage=always acpi=force acpi_osi=Linux usbcore.autosuspend=-1 nvidia.NVreg_OpenRmEnableUnsupportedGpus=1 nvidia_drm.NVreg_OpenRmEnableUnsupportedGpus=1 nvidia_modeset.NVreg_OpenRmEnableUnsupportedGpus=1 as I had issues at one time with the modprobe being gone and unraid not reading the modprobe configs in /boot/config/modprobe.d as sometimes they would not move into the /etc/modprobe.d folder earlier enough in the boot process... Whilte using the nvdia vgpu (unlock and offical) it does requires the nvidia license servers aswell... this unlocks the core componet behind vgpu and its use as a 3d acelerated device in linux and its use simalr to a real physical G-card. So, after putzing aroundwith it: (as I was more focused on fixing other things...) I decide to use proxmox webui and changed off host and move to a stable qemu cpu (x86-64v2-aes) You may need to run this command on proxmox and verify your cpu chipset instruction sets and pick a compatible CPU grep -m1 flags /proc/cpuinfo as seen in my proxmox configurations... This way I can get out of unraids way of patching and applying the mitigations to the kernel at boot. So to achieve vgpu in unraid: *its one of the reason why I have you install the opensource driver first to get the modprobe stuff and make edits above... test... I was able to get unriad to see and read my nvidia gpu device once more... This is also why this is experimental.... You will have mixed results... there is no one size fits all shoe option here. since i'm running a ryzen 5700 g I decided to leave host to re-enable the mitigation for the kernel... Even with some of the option here, some are ignored don't mean anything... but this unriad VM is also my test server. you can try and force a modpre and unriad system driver install for nano /boot/config/modprobe.d/nvidia.conf options nvidia NVreg_OpenRmEnableUnsupportedGpus=1 options nvidia_drm NVreg_OpenRmEnableUnsupportedGpus=1 options nvidia_modeset NVreg_OpenRmEnableUnsupportedGpus=1 as its more of getting the driver plugin to install on a unsupported device... I use a grouping of grub boot options, system drivers, modprobe etc... and play around... Besides this is more prof of concept... (especial when using a older EOL kernel...) Edited April 14, 20251 yr by bmartino1 Data - typo
April 14, 20251 yr Thanks for this guide! About to upgrade my Ryzen 5600 to a 5900XT. Usually I just run Debian VMs on UnRaid (prefer docker via compose/ansible rather than UnRaids docker apps). With 16 cores/32 threads, it's time for me to go the Proxmox - UnRaid in a VM, route. I'm wondering how few cores I'll need for the UnRaid VM if I'm only using it for NAS purposes..
April 14, 20251 yr Author 13 hours ago, Lebowski89 said: Thanks for this guide! About to upgrade my Ryzen 5600 to a 5900XT. Usually I just run Debian VMs on UnRaid (prefer docker via compose/ansible rather than UnRaids docker apps). With 16 cores/32 threads, it's time for me to go the Proxmox - UnRaid in a VM, route. I'm wondering how few cores I'll need for the UnRaid VM if I'm only using it for NAS purposes.. depends on general rule of thumb and services.... Your mileage will vary. Unraid is More Ram Hungry. As it should, given that the Entire OS runs in Ram... While I provisioned 10 threads. Proxmox load balances them fairy well. Its ok to over provision a little bit along as not all 100% of the VM are running at full load... I rather give something to much then deal with not enough latter. I usually go by 1 core = 2 threads, each service requires a min of 1 core and 1 GB of ram my amd ryzen 5700G runs 16 threads at: 3.8 GHZ and can hit 4.6 https://www.techpowerup.com/cpu-specs/ryzen-7-5700g.c2472 For me, My unraid is more for nas storage, plex and immich (medai/photo access). Thus, I have to provision resources from proxmox into unraid and have unraid load balance for the 2 additional servers. My needs required in the end (due to other testings) is 10 threads of my 16 threads total.... (as more then 4 plex users I start to see some lag and sever IO upticks...) This is mainly Due to other factors and silicon lottery.... FBS, Memory and other factors. as you are using a 5900XT, compared to the GHZ of the processor and the speed of the cores and its access for I/O. AS The goal is to keep IO as low as possible. https://www.techpowerup.com/cpu-specs/ryzen-9-5900xt.c3653 If 0 running dockers(and using Proxmox for VMs). I would say a min of (4-6 threads) 2 cores is more than enough for general Unraid nas duties (samba, nfs array disk management). With a ram min 16 GB in a VM. If using zfs like I am in the VM you may need additional ram and threads/cores to assist zfs and additional services... Per the docs: https://docs.unraid.net/unraid-os/getting-started/ A 64-bit capable processor, that runs at 1 GHz or higher. A minimum of 4 GB of RAM for basic NAS functionality. Linux hardware driver support for storage, Ethernet, and USB controllers. so it more of noticeable slowness as you will see that from time to time you may need to increase resources as time goes on when required... Giving proxmox cpu threads to a machine can also hit the issue of single threaded task vs muti-core task. Just because you throw more threads to the VM doesn't mean its IO and processing is lessened. There some finessing you can do both within the Unriad and on Proxmox side. Newer processor may be able to get away with fewer cores assigned. This is due to the FBS(front bus speed) on the Processor chip and is ability to move things into memory. you could also go the other way and assign actual cores: Similar to my windows VM: compared to my unraid docker VM: more info in the forum and proxmox wiki / docs as this more goes towards how you want to handle resources and load balancing... https://forum.proxmox.com/threads/sockets-vs-cores-vs-vcpus.56339/ Edited April 14, 20251 yr by bmartino1 Data - typo
April 14, 20251 yr 1 hour ago, bmartino1 said: If 0 running dockers(and using Proxmox for VMs). I would say a min of (4-6 threads) 2 cores is more than enough for general Unraid nas duties (samba, nfs array disk management). With a ram min 16 GB in a VM. If using zfs like I am in the VM you may need additional ram and threads/cores to assist zfs and additional services... Sounds good + I offloaded Plex and its friends (tautulli, etc) to a N100 Mini-PC (Beelink BQ-12) a while back, so that my AMD processor isn't having to worry about media transcoding. Cheers
April 17, 20251 yr Got around to setting this up - worked pefect on UnRaid 7.0.1 (after changing to SeaBios). Ended up with 4 cores / 8GB ram will change if required. Removed cache drive as well, since all UnRaid will be doing is storing data - will remove all docker containers and VMs from UnRaid itself. Thanks again for the guide.
April 19, 20251 yr Author On 4/17/2025 at 9:17 AM, Lebowski89 said: Got around to setting this up - worked pefect on UnRaid 7.0.1 (after changing to SeaBios). Ended up with 4 cores / 8GB ram will change if required. Removed cache drive as well, since all UnRaid will be doing is storing data - will remove all docker containers and VMs from UnRaid itself. Thanks again for the guide. unraid usb by default is not setup to use efi / uefi... in unraid If you want to use uefi. Main > Flash http://192.168.2.251/Main/Flash?name=flash scroll to the bottom: this is due to the original machines using bios and not uefi. uefi has some befits with pcie passthrough. there no reason to use one over the other...
May 4, 20251 yr On 11/11/2024 at 6:43 AM, bmartino1 said: If you want a 5 GB cach disk... as I went beta and ditched the array for a zfs pool only, This is more for a btrfs partition for unraid swap plugin... Have also ditched XFS for a full ZFS-pool setup. Is there any real downside to just making the unraid cache disk a virtual disk using space from the SSD hosting your Proxmox VMs? I could pass the whole SSD through, but if I could just give UnRaid some of the space and make the rest available for other VMs, that would be better.
May 4, 20251 yr Author 9 hours ago, Lebowski89 said: Have also ditched XFS for a full ZFS-pool setup. Is there any real downside to just making the unraid cache disk a virtual disk using space from the SSD hosting your Proxmox VMs? I could pass the whole SSD through, but if I could just give UnRaid some of the space and make the rest available for other VMs, that would be better. Only a few things of note, but no, no issue that I can find... It why I saw one could be virtual just the cache disk and use unraid for docker only on truenas using an expandable cache singe file vdisk image to hold the docker xfs img file and app data... Some things to not though... 1 proxmox supports command line adding more data to a vdisk. unraid needs a restart to see the data expansion and potential use as a new format, the existing extension... --Use proxmox disk commands sparingly let unriad control them... even a proxmox controled vdisk has some weired io and performance issues... Also Unriad smart may flag it as old or broken... (as that what's being reported by the proxmox system to the vm. I've been running this since the v6 to v7 beta to current v7 stable release.. I've not see or had issues using a qemu vdisk as a pool device. -I have see issues using them as the array devices (more to do with how parity works and the writing of the 1 and 0 to a virtual disk. Especial if the disk is set to be ballooning storage... I recommend enabling the proxmox discard option and have it emulate as a ssd I backup my vdisk thus the backup options... I used a qcow as i migrated, but its better to use raw here... If I were to recreate the disk, these are the proxmox vdisk add Hard drive options I would choose... For me this is following the as a general rule of thumb for unraids docker system... a 50 GB vdisk, 5 gb swap plugin, and 120 GB for general docker appdata and scratch space for unraid... side note ... you can even use qemu disk as metadata special disks if using unraids zfs... If you note my signature spoiler under my post on desktop view and logged in to forum... you can see my hardware setup, and note that my proxmox is using 2x nvme as storage and the nvem are where the vdisk sit. I can partition a part of the nvme and use them for other zfs related tasks. --Just something to note... my disk when the hba passes the others to unraid.. as the 120 GB is my debain proxmoxx boot partition. and the 2 NVME are setup as a zfs mirror / stripe I don't remember need to go double check...(mirror) been 24/7 a good 8 ish month, but having ran simlar on v6 this ha been known test good for a while... before I found the virtualized forum post for unriad... zfs mirror to keep vms happy and redundant... ( I could got 1 nvme for vdisk storage on proxmox and pcie passed the nvme direct to unraid...) as the goal is about low IO... You could look more into the async IO that the vm does and how it reads and writes to the virtual disk view info and benchmarking here all depends on hardware your running... https://forum.proxmox.com/threads/proxmox-ve-7-2-benchmark-aio-native-io_uring-and-iothreads.116755/ BUT The Main thing is to understand the the reason for raw over qcow: as one should use qcow when possible... as Qcow2 (QEMU Copy-On-Write version 2) is a versatile disk image format that supports features like compression, snapshots, and thin provisioning. IN my testing, I found Raw handles the 0/1 writes if used in the unriad array and guarantees the storage... as a continued 1 file vdisk, especial when taking in the account of its physical location... Here's a more detailed comparison: Raw: Simplicity: Raw files are straightforward, representing the entire virtual disk as a single, contiguous file on the host system. Performance: Typically faster than QCOW2 because it bypasses the indirection layers of QCOW2. Storage: Requires the entire virtual disk space to be pre-allocated on the host, even if not all of it is used. Snapshots: Does not inherently support snapshots. Thin provisioning: Does not support thin provisioning. QCOW2: Copy-on-write (COW): Only writes changes to the disk image, not the entire underlying data, improving storage efficiency. Storage Efficiency: Uses sparse allocation, meaning the file only grows as the virtual machine's data fills the disk. Snapshots: Supports creating and managing snapshots of the virtual disk, allowing you to revert to previous states. Thin Provisioning: Enables thin provisioning, where storage space is allocated on demand as the virtual machine uses it. Performance: While generally faster than raw, it can be slightly slower due to the indirection required by the COW mechanism. When to choose which: Raw: Choose Raw when performance is critical, and you don't need snapshots or thin provisioning. QCOW2: Choose QCOW2 when you need snapshots, thin provisioning, or when storage efficiency is a primary concern. In summary: Raw is faster, while QCOW2 offers more features and is better for resource-constrained environments or when snapshots are needed. Some other side notes while talking about vdisks... (as once could go 3 v disk of the same size and zfs raidz1 them and have 4 disk as a full virtualized instance....) -- While possible, this add to much disk IO and disk thrashing... even with zfs and pool only... Possible yes, if your ok with the performance hit... (thus machines running proxmox string heading into server and workstation hardware...) Also something of note... There's a way to make a vdisk of the usb drive as well but that still more a beta things As Most who use proxmox go cepth and run 3 of the same machine for clustering... This is where having the unraid vdisk usb with the same GUIDs makes sense to keep a unraid vm up and running... just another example as a proof of concept: https://forum.proxmox.com/threads/attach-a-raw-image-as-a-usb-disk-to-the-vm.27306/post-137470 https://forum.proxmox.com/threads/virtual-pendrive-on-vm.114143/ as yes you can but really should be using physical media passed via pcie lanes. All I can say is that your experience may different silicon lottery are you play a workaround game. Is it possible, yes. Its what I'm running. I can't endorse nor recommend doing so though. I don't see a reason why one should. For me its more a issue of space (atm)... As i rather pcie pass a nvme/sata via the already passed hba or neve x1-x4 nvme slot... as pcie cards can have addition chips that virtualizing doesn't do to help with performance, read write and how one interacts with the chips interfaces...
May 7, 20251 yr Author It was fun while it lasted Decided It's not worth trying to keep using proxmox vgpu on lattest stable... well, a end of an era.... So some last notes, rants and info as I no longe see a need to run unraid if basic functionally of services are dead. Truenas going true docker not helm so i hear... and there are other free and open-source alternative... That exist to meet my needs... I will miss the unraid docker webui... unraid is no longe maintain features and systems that fit my needs and keeps breaking stuff while not being very secure... So its time to move off and away from bad systems and bad practices... As with unraid stable release of v7.1 and a move from v7.01 to v7.1 Looks like vgpu is dead... ish.... someone else may be able to fix or get it working with some extra notes and data below... As this is more a driver kernel issues dealing with kernel v6.6 and kernel v6.12 have some vast code differences and my workaround is no longer working... --I really do hate kernel 6... with limetech finally adding the novuea driver with the kerneal for altease NVIDIA g cards and general use... This breaks other kernel level edits and workarounds... as the nova open source driver will not support vgpu... since bzfirmware and root come with the latest slackware updates and kernel its not a easy drop or use xyz kernel so go v7.0.1 and proxmox kenrneal 5.15 to still use NVIDIA GPU after playing around with installing the drivers and mod probe edits. I ended on this: and a Hard blocks with no known resolve as the latest nvidia driver no longer has the options I was using... As I tried to explain to ICH777 before and was dismissed as I told him to go do some dam research!... WELL fine.. decded not to even respond and left for a while after that... AS Unfortunately, NVIDIA silently removed NVreg_OpenRmEnableUnsupportedGpus from the proprietary branch. But here’s what can help instead... nvidia.NVreg_EnableGpuFirmware=1 nvidia.NVreg_EnableStreamMemOPs=1 as Nvidia slowing closing gaps even the toolkit and removing support for the older cards... having read through the nvidia driver and plugin ich777 has made on tehr github and the current release. The ich77 nvdia plugin is very well-made... we were forcing a unsupported feature and using the nvdia driver plugin as a base to get some NVIDIA functionality... Not sure if he added a v550 if that would fix this or not... As I'm sure there was something script and call wise that was happening before that I no longer have working and causing a hard stop error to use the nvdia gpu with v565 and higher... Quote NVRM: Xid (PCI:0000:03:00): 78, Guest driver is incompatible with host driver NVRM: rm_init_adapter failed Also tried to explain in driver forum with a no it does work ... But as this is in the grey and proff of concept only... Its time to close this proxmox vgpu sytem and build fresh... notes for code prosperity... as I can drop to v7.01 and use kernel v6.6 to continue using my vgpu... but with proxmox kenral at v5.15 its questionable to continue playing the work around game... My unraid grub syslinux kernel /bzimage append initrd=/bzroot mitigations=auto,pt spectre_v2=on spectre_v2_user=on spec_store_bypass_disable=on default_hugepagesz=1G hugepagesz=1G transparent_hugepage=always acpi=force acpi_osi=Linux usbcore.autosuspend=-1 trace_clock=local nvidia.NVreg_OpenRmEnableUnsupportedGpus=1 nvidia_drm.NVreg_OpenRmEnableUnsupportedGpus=1 nvidia_modeset.NVreg_OpenRmEnableUnsupportedGpus=1 nvidia.NVreg_EnableGpuFirmware=1 nvidia.NVreg_ResmanDebugLevel=255 modprobe.blacklist=nouveau fo thoese who want to try get it working... In escense I ended up using the Production driver (Which nvida release open-source support) and split the drive into choice installing open vs prioripertary.... so I grabbed the nvdia driver that was compiled and running on kernel so I had the libs from v6.6 added but used production driver libs for the extra install of last known working driver v550... as noted in picture above... as I tried open source, coping it in as v565 which worked till the end due to missing kernel libs as a copy of v550 put the libs in folder kernel v6.6 and not the path others things were looking for v6.12 ... Also heavily edited system drivers with modprobe configs... options nvidia NVreg_OpenRmEnableUnsupportedGpus=1 options nvidia NVreg_EnableGpuFirmware=1 options nvidia NVreg_EnableStreamMemOPs=1 options nvidia NVreg_ResmanDebugLevel=255 options nvidia NVreg_UseVBios=0 options nvidia_drm NVreg_OpenRmEnableUnsupportedGpus=1 options nvidia_modeset NVreg_OpenRmEnableUnsupportedGpus=1 which are also set at the grub but with other script driver calls you need both sometimes... I can only theorize Why It Used to Work on kernel 6.6: Unraid 6.6 + v550 let you: Spoof the hardware ID Inject modprobe parameters Bypass Nouveau (which wasn't bundled) Load driver even for vGPU-assigned devices But now, in 6.12: The newer kernel and firmware behavior is stricter. vGPU guest enforcement is tighter in both open and closed drivers. You're still running a driver (v570.144) even though you said you're loading v550 — logs show otherwise: https://github.com/ich777/unraid-nvidia-driver/blob/d70d97fdb41de53e096e1fa9d62ff223595b275a/source/compile.sh So don't go bothering asking on the forum about NVIDIA VGPU drivers and support.. as its get gray and borders at violates the mit NVIDIA license and limetech rules... as we move on past unraid v71 I hope open-source access and use of vgpu get better... I guess its too small a niche case to properly use and run on unraid... As the VGPU system was working in v6.9 to use vgpu nvidia install on unraid... as we lost AMD mxgpu variant which is the better course and gpu partition way. (the only 2 cards taht support it are EOL and aren't worth the cost...) Nvidia still trying to cleanup and remove consumer vgpu and a open-source community working between multiple and nvida both helps and gets in the way... So I think vgpu for homelabs consumer on the cheap will die off.. Especial since we are moving into the 5000 series cards and issues with vgpu has still not come out from the community on the 3000 cards... Since its offical dead with the next release. Dont go asking for Unraid assistance with using VGPU past v7.0.1 Even this reliese on some legacy code that may no exisit in the furture... Its easeier to just pass a nvdia gpu into the vm for unriad then tryign to split it... It was fun while it lasted. time to refactor and move on.
November 29, 2025Nov 29 Author May be time to make a new post delete this or move to the proxmox forum as its more on how to edit and install proxmox then virtualize unraid on proxmox...
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.