March 23, 20251 yr Hi everyone, I’m in the process of upgrading my Unraid server setup and would appreciate some advice from the community. Currently, I’m running Unraid on an old Dell Optiplex 7070, which has served me well, but I want to enhance my capabilities for local LLM inference. I have committed to a new build with the following key specifications: Relevant Info on Server Build CPU + Motherboard: AMD 9 Ryzen 7900 + ProArt Creator X870E WiFi Memory: Corsair Vengeance RGB 64 GB (2 x 32 GB) DDR5-6000 CL30 Video Cards: 1. MSI RTX 3090 Ti Suprim X 24G 2. ROG-STRIX-RTX3090-O24G-WHITE Power Supply: Corsair HX1500i Fully Modular Ultra My initial plan was to run LLMs using Ollama and Open WebUI in Docker containers. However, since I won't be utilizing all of the GPU resources at all times, I’m considering setting up a VM that can pass through one of the GPUs for gaming (primarily World of Warcraft), as well as photo and video editing. I have a couple of specific concerns: GPU Binding: I understand that when a GPU is passed through to a VM, it can be bound to that VM. Is it possible to unbind and free up the GPU effectively when the VM is shut down? Idle GPU Power Draw: If the VM is halted and the GPU is unbound, will I still face issues with idle power draw from the GPU? Anti-Cheat Issues with VMs: I read about there being anti-cheat issues when running games in VMs. Additionally, I’ve read that Proxmox may offer better VM management capabilities. Would it be more beneficial to run Proxmox with an Unraid VM alongside a Windows VM, rather than using Unraid directly with a Windows VM? I would greatly appreciate any insights, advice, or recommendations based on your experiences. Anyone else in a similar situation and what approach have you taken? Thank you!
March 23, 20251 yr Community Expert proxmox is the better hyper V. Unraid is a Nas first, Unraid excels with dockers.... So some info dump first: There are ways to even use VGPU on consumer Hardware, splitting a gpu into portions and using it as 3d acetated across multiple host lxc/VM for additional graphical compute. (you lose physical display out.... I will always recommend proxmox for anyone using VMs. Unraids HyperV and virtual machine has gotten better with v7 it is still a mess using libvrit and xml domain configs... https://libvirt.org/formatdomain.html Unriad has a better single device pasthrough and vfio binding in the web ui. There's quite a bit you would need to do in proxmox to config edit and confirm. Unriad is get the iommu group separated, check a box and it will bind the device... Proxmox Guide for Virtualizing UnRaid 7: View Promxox VM Unriad Guide: To virtualize unriad if you go that route... if the gpu device is vfio bound, it is hidden from the host system and can't be used on the host. I would have you look into the proxmox user scripts for LLM and run the LLM in a LXC / VM on proxmox to get better granular control on the LLM and running it as a system.. https://forum.proxmox.com/threads/ollama-ai-llm-in-lxc.156346/ https://community-scripts.github.io/ProxmoxVE/scripts?id=openwebui https://www.bujarra.com/ollama-empezando-con-la-ia-local/?lang=en *As usually its starts with a open web ui LXC and ollama in the same container... This said. I'm partial to local AI docker on unriad. In terms of a 15 GB image and quick and fast implementation of AI Models you could run with local ai: https://localai.io/gallery.html Now on to Q/A ################### Quote GPU Binding: I understand that when a GPU is passed through to a VM, it can be bound to that VM. Is it possible to unbind and free up the GPU effectively when the VM is shut down? With Unraid binding the GPU and giving it to a VM requires the host to be rebooted. The webUI is easy to uncheck and check a device to be bound... You have to watch VMs autostart if you do this. Sadly, the same can be said in proxmox. This is why I mentioned VGPU. as you could allocate and bind a allocated gpu to a VM.... The proper way is to go to the sys setting and unbind the device and reboot the host. If you are ok with reboots, this is fine. Quote Idle GPU Power Draw: If the VM is halted and the GPU is unbound, will I still face issues with idle power draw from the GPU? I understand you concerns for power and the need to lower it when using a LLM and in general. I'm not the right person to ask. as I disable and break this as much as possible. The power and sleep states break VFIO and binding for me every time I try, so I actively go out of my way to disable them... can it be done yes, but its recommend to disable sleep and some power options to assist devices being transferred between the host and virtual host. Sadly this scenario is dictated by the first a bit. To properly unbind and remove a vfio bound device, the Host needs to be rebooted. This more is affected by PCI IRQ memory mappings. There are some advance Linux commands to target dev devices and unbind them and bind them again but nothing that a reboot will properly detect run and bind... Quote Anti-Cheat Issues with VMs: I read about there being anti-cheat issues when running games in VMs. Even proxmox struggles with this sometimes... Roblox as example... its game code is actively looking for a VM environment and will die if it detected wine... You have to start playing the hidden and pass real xyz to fix these... Any anti cheat game is proof of the game developer's inability to program and control the memory they use in a game and instead of programming and fixing the major bug for the game they rather monitor your system resources and use rootkits to assist there games and there game play. VMs have the ability to read the guest running game system and read memory to cheat at areas... while not imposable it has been done to cheat... this is usually why "anti cheat" software flags the game running in a VM... There are many ways to spoof, but that unfortunately is a trail and error with your hardware, how you run that hardware and how you run the game. In general, I would advise you to not support or play games that require anti cheat... ###################### Given the hardware you have and the steps you need or want to take. You will need a GPU. TO properly virtualize unraid for samba nas and dockers. I advise you to use an HBA. -This may not work with your current hardware Mother board and space/power limitations... Example: With Unraid Virtualized, running the docker local AI with a VGPU partitioned of 2-3 GB ram card using VGPU unlock on a NVIDIA 2080 as example will run 2B models on unraid without issues. Unraid has weird vgpu capabilities but all would require some TLC. but unraid is capable of using vgpu and branching its use between dockers and VMs... Language barrier and other prevented me. But I did VFIO with debian and proxmox before using unraid so i understand the gist and what they are trying to do... IT claimed to stillb e working with unriad v7 but the unraid VGPU is more a v6.9 thing... github README: Latest version currently supported: 6.12.9 as VGPU fits the niche you are tying to do... I Currently run a RDP parsec windows gaming VM... along with my unraid system under proxmox. However, the unraid system I have spun up is more for dev and testing. Unraid has a nice web UI and decent docker support. the forum is active and people are available here to assist... that said. Everything you do in unraid can be accomplished by other means... What function does unraid server you? other OS and system exist such as truenas scale, open media vault, protainer... etc In the end its what you want it to do, and how you want to interact with it. Proxmox has a forum as well. https://forum.proxmox.com/ So I would recommend Proxmox even if you do unraid v7 with a vdisk and no array cache pool only for dockers... This said. I would have you pass 1 NVIDIA 3090 to your gaming VM in unraid, and use the other 3090 with local AI docker. Edited March 23, 20251 yr by bmartino1 Typo
March 23, 20251 yr Author Hi @bmartino1, First off, thank you so much for your very detailed and helpful response! It is very kind of you to spend time explaining how this works and sharing your views this to me which is helpful especially since I'm new to all of this. I stumbled upon Unraid last year when I was looking for a NAS solution with old hardware. Then I got into experimenting with LLM interfaces this year and various hosted LLMs and now I want to experiment with local LLMs, which is why I have a setup that is overkill for NAS and docker use... The two GPUs are for running larger local LLMs giving me a total of 48 GB VRAM - this is my priority. The reason for my post was to see if there is a way for me to have both GPUs assigned to LLM use for most of the time and occasionally temporary unbind a GPU and passthrough a GPU to the gaming VM. I would then reassigned the GPU back to the host for use by the LLM inferencing engine when I shut down the VM. Please correct me if I've misunderstood, your reply suggests that a GPU passthrough is not helpful in my situation because I will need to reboot the host to unbind the GPU - this applies to both Proxmox and Unraid baremetal. 3 hours ago, bmartino1 said: To properly unbind and remove a vfio bound device, the Host needs to be rebooted. This more is affected by PCI IRQ memory mappings. There are some advance Linux commands to target dev devices and unbind them and bind them again but nothing that a reboot will properly detect run and bind... You mentioned exploring vGPU and when I looked it up, it seems exactly what I hope to do since I don't want to connect a monitor directly and was planning to use dummy HDMI plugs - I will explore this further although I note that the RTX 3090 does not support vGPU. I appreciate that vGPU is better supported on proxmox than on Unraid. I will think about it but I'm leaning towards sticking with Unraid bare metal because I'm comfortable with it plus it is super flexible hardware wise for a NAS - Proxmox scares me a little given my non-IT background. Lastly, I appreciate the recommendation to run Promox with one GPU passed to the gaming VM, the other GPU passed to the local AI docker with Unraid. Unfortunately, as explained above, this does not suit my use case because my main goal is to have both GPUs for LLM use most of the time and occasionally temporarily allocate one GPU to VM upon start up for gaming & photo/video editing. Once again, thank you @bmartino1! Edited March 23, 20251 yr by Michael_Nat Adding a bit of detail.
March 23, 20251 yr Community Expert Solution Quote Please correct me if I've misunderstood, your reply suggests that a GPU passthrough is not helpful in my situation because I will need to reboot the host to unbind the GPU - this applies to both Proxmox and Unraid baremetal. You have explained it correctly. To use gpu pasthrough when binding a device and unbinding the device properly, you will need to reboot the host machine each time. While doable there are other ways to go about it to get to the end goal. For more examples, Review unraid vifo bind and passthorugh: (Spaceinvaders videos) To further explain, Which is why I recommended proxmox. But after re reviewing... i'm not sure of proxmox vgpu setups for you due to both being 3090s. Due to access to hardware I have, I followed: https://gitlab.com/polloloco/vgpu-proxmox *This guide and all my tests were done on a RTX 2080 Ti which is based on the Turing architechture. and there are china blogs to grab the enterprise grid nvdia downloads installers to patch to use VGPU. as nvida "let the genie out of the bottle" and has been trying to put it back for a long while now... folowing a proxmox vgpu consumer unlock: The following consumer/not-vGPU-qualified NVIDIA GPUs can be used with vGPU: Most GPUs from the Maxwell 2.0 generation (GTX 9xx, Quadro Mxxxx, Tesla Mxx) EXCEPT the GTX 970 All GPUs from the Pascal generation (GTX 10xx, Quadro Pxxxx, Tesla Pxx) All GPUs from the Turing generation (GTX 16xx, RTX 20xx, Txxxx) !!! THIS MEANS THAT YOUR RTX 30XX or 40XX WILL NOT WORK !!! *However, There have been reports and attempts to use the 3000 series cards(due to ther large video ram). some had success other not so much. To use vgpu technology the only way I know how is to use windows hyper v and the gpup partition scripts in windows power shell to make Virtual machines. *Played with this and ran this a few times. widows is not that good hyper v system for device passthrough... However, you could also run both 3090 under windows and use hyper V for NAS vdisk and others partition the GPU. Review: https://learn.microsoft.com/en-us/windows-server/virtualization/hyper-v/partition-assign-vm-gpu?tabs=windows-admin-center https://github.com/jamesstringerparsec/Easy-GPU-PV https://forum.level1techs.com/t/2-gamers-1-gpu-with-hyper-v-gpu-p-gpu-partitioning-finally-made-possible-with-hyperv/172234 There have also been pushes to use widnwos and use wsl for LLM... And wsl / direct widnows docker installs: https://docs.docker.com/desktop/setup/install/windows-install/ *Not all dockers will run on windows... As I have experimented with a lot of systems hardware and setups... I may actualy recommend the use of windows pro or enterprise edition. (do not use widnows Server, while server has better pcie devce passthorugh for hyperv it loses the gpup partitions abiltiy) Likewise, Nvidia has alos worked with windows on improving there devices with wsl and hyper v... Regardless of the options you chose. it comes down to what you want it to do and how you want to interact with it.
March 24, 20251 yr Author Once again, thank you so much @bmartino1! I appreciate the various options that you have shared - it is good to know what are the alternatives available so that I can weigh my options and decide based on my needs. I think for now, I want to keep running Unraid because I want the NAS functionality with the option to use whatever hardware I like. While there is a greater possibility of getting the vGPU option on Proxmox to work, I think it is a bit risky to rely on unofficial GPU drivers. My worry is that the GPUs or the system might get damaged due to the drivers. That's why I don't think I will go with this. However, I might set up a separate machine for playing around in Proxmox to learn about it cause it seems quite interesting. The Windows / WSL seems to be a decent option for Nvidia vGPU functionality. However, after going through all of the pros and cons, I am leaning towards just staying with Unraid and running a VM with GPU passthrough and rebooting when I shutdown the VM. I will probably only use the VMs once in a while so it shouldn't be too disruptive plus the Unraid server really only serves me personally so a reboot isn't too disruptive. I saw online that it may be possible to run commands to unbind the GPU from the VM's drivers without a reboot so I might explore that - though I note that such options may have stability issues. All in all, thanks so much for the advice and comments. I've learnt a lot and I may revisit this as my needs change!
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.