PikkonMG

:HOLDER: For furture post if needed.

This is the support thread for the NVIDIA NIM Single Unraid template. Made this for my personal use so I could use NVIDIA-NIM with Unraid and my AnythingLLM container. GitHub Repo Template Readme This guide explains how to run NVIDIA NIM containers on Unraid using a consumer NVIDIA GPU. NIM provides optimized inference servers with an OpenAI-compatible API, making it easy to connect tools like AnythingLLM or Open WebUI. Tested environment: RTX 3060 12 GB Unraid 6.12+ NIM 1.10.1 PrerequisitesYou will need the following: Unraid 6.12 or later NVIDIA GPU (Turing architecture or newer) Examples: RTX 20xx, RTX 30xx, RTX 40xx NVIDIA drivers installed in Unraid Community Applications → NerdTools or GPU Statistics plugin Free NVIDIA NGC account https://build.nvidia.com NGC API key generated from your NGC dashboard Model SelectionNIM uses pre-optimized engine profiles, which are primarily designed for data center GPUs. Consumer GPUs require smaller models and reduced context windows. Example modelsModel VRAM Required Fits 12 GB GPU meta/llama-3.2-3b-instruct ~6 GB ✅ Recommended microsoft/phi-3-mini-4k-instruct ~8 GB ✅ Yes nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 ~10 GB ✅ Yes mistralai/mistral-7b-instruct-v0.3 ~14 GB fp16 ❌ OOM meta/llama-3.1-8b-instruct ~22 GB bf16 ❌ OOM meta/llama-3.1-70b-instruct ~80 GB ❌ Multi-GPU If you want to run 7B+ models on a 12GB GPU, consider Ollama, which supports quantized weights. NGC Registry Login⚠️ This must be done before Unraid can pull NIM images. NIM images are hosted on NVIDIA's private registry (nvcr.io), not Docker Hub. Run this one time in the Unraid terminal: docker login nvcr.ioLogin credentials: Username: $oauthtoken Password: YOUR_NGC_API_KEY⚠️ Important: docker login → allows pulling the container image NGC_API_KEY → allows downloading model weights at runtime Both are required. Cache Directory PermissionsBefore starting the container, create the cache directory with the correct permissions. NIM runs inside the container as UID/GID 1000. If the cache directory is owned by root, the container will fail to start. Run: chown -R 1000:1000 /mnt/user/appdata/nvidia-nim/cache chmod 775 /mnt/user/appdata/nvidia-nim/cacheEnvironment VariablesVariable Value Notes NGC_API_KEY your_ngc_api_key Required. Used to download model weights NIM_MODEL_NAME meta/llama-3.2-3b-instruct Must match the image tag NIM_MAX_MODEL_LEN 16384 Required for consumer GPUs NIM_CACHE_PATH /opt/nim/.cache Cache directory CUDA_VISIBLE_DEVICES 0 Use 0 for single GPU PYTORCH_CUDA_ALLOC_CONF expandable_segments:True Reduces memory fragmentation First RunOn the first startup, NIM downloads the model weights to the cache directory. Example size: ~6 GB for llama-3.2-3b This can take several minutes depending on your internet connection. You can verify the container is running with: curl http://localhost:8000/v1/modelsConnecting ClientsNIM exposes an OpenAI-compatible API, so most AI clients work out of the box. Connection settingsSetting Value Docs http://[unraid-ip]:8000/docs Base URL http://[unraid-ip]:8000/v1 API Key Any non-empty string (e.g. nim) Model meta/llama-3.2-3b-instruct Compatible clientsAnythingLLM Open WebUI LangChain LlamaIndex Cursor (custom OpenAI base URL) Any application with configurable OpenAI endpoints Switching ModelsCurrently the template uses model-specific container images. To switch models: Stop the container Change the Repository field Example: nvcr.io/nim/microsoft/phi-3-mini-4k-instruct:latestUpdate the model variable: NIM_MODEL_NAME=microsoft/phi-3-mini-4k-instructStart the container To run multiple models, create additional containers on different ports: 8000 8001 8002They can share the same cache directory — weights will not be duplicated. TroubleshootingCache Permission ErrorIf the container fails with: PermissionError: [Errno 13] Permission denied: '/opt/nim/.cache/local_cache'Run: chown -R 1000:1000 /mnt/user/appdata/nvidia-nim/cache chmod 775 /mnt/user/appdata/nvidia-nim/cacheKV Cache Size ErrorExample error: ValueError: The model's max seq len (131072) is larger than the maximum number of tokens that can be stored in KV cacheFix: NIM_MAX_MODEL_LEN=16384If needed, try: 8192Consumer GPUs cannot handle the full context window used by data center profiles. Common ErrorsError Cause Fix 401 Unauthorized Not logged into nvcr.io Run docker login nvcr.io ValueError: invalid literal 'all' CUDA_VISIBLE_DEVICES=all Change to 0 PermissionError on .cache Wrong permissions Fix cache directory permissions max seq len > KV cache Context window too large Set NIM_MAX_MODEL_LEN=16384 CUDA out of memory Model too large Use a smaller model No compatible profiles GPU too old Requires RTX 20xx or newer nvfp4 unsupported warning Consumer GPU limitation Safe to ignore XML TemplateThis repository includes an Unraid Community Applications-compatible template: nvidia-nim-single.xmlTo install manually: /boot/config/plugins/dockerMan/templates-user/After copying the file there, it will appear in the Unraid Docker template list. Uploading Attachment...Uploading Attachment...

eth1 is only port I can somewhat get to work. I've also taken and deleted the network-rules.cfg and reset the network.cfg were eth1 in logs above is eth0 and it gets stuck in this bonded mode and still have same overall issue. Booted safe mode here is logs from a backup I just tried to restore from Generate Flash Backup dated 12/05/2022 reset the network.cfg ezra-diagnostics-20221218-1005.zip

Having issues with network. Unable to connect to server. I have 4 ethernet ports and none of them are wanting to work. I don't know if an update broke it or if network card is going out. Was able to connect and get logs by changing port and plugin direct into my modem. ezra-diagnostics-20221218-0120.zip

Safe mode no change but im starting to learn towards this might be a Windows issue.

I don't understand? You want same info from safe mode boot?

Scroll up a little! lol

Sorry was in maintenance mode. drwxrwxrwx 1 nobody users 131 Sep 22 10:46 ./ drwxr-xr-x 13 root root 260 Sep 24 10:19 ../ drwxrwxrwx 1 nobody users 31 Aug 21 18:10 .Trash-99/ drwxrwxrwx 1 nobody users 6 Apr 25 2017 Backup_Settings/ drwxrwxrwx 1 nobody users 66 Sep 2 01:31 ESG_Misc/ drwxrwxrwx 1 nobody users 17 Aug 14 14:43 Education/ drwxrwxrwx 1 nobody users 23 Aug 21 11:57 Gaming/ drwxrwxrwx 1 nobody users 21 Sep 8 02:21 Machine\ Backups/ drwxrwxrwx 1 nobody users 59 Sep 19 20:48 MyMedia/ drwxrwxrwx 1 nobody users 230 Mar 4 2022 Pepto/ drwxrwxrwx 1 nobody users 191 Feb 23 2022 Pursuit/ drwxrwxrwx 1 nobody users 4096 Feb 17 2021 SavedApps/ drwxrwxrwx 1 nobody users 186 Sep 19 22:14 appdata/ drwxrwxrwx 1 nobody users 24 Nov 9 2018 domains/ drwxrwxrwx 1 nobody users 4096 Sep 8 02:17 isos/ drwxrwxrwx 1 nobody users 26 Nov 9 2018 system/

/bin/ls: cannot access '/mnt/user': No such file or directory ezra-diagnostics-20220924-0111.zip

Have an issues with my share. Started when I upgraded a few of my drive on my Unraid server. If I reboot the server the login info for my Shares Access users fails. I have to go to just one share any one doesn't matter and go to SMB User Access switch user from Read/Write to Read Only hit apply then switch it back to Read/Write. After this login will work with network shares till next reboot. Sounds like some kind of permission issues but no idea how to mess with it.

Have a 120 GB SSD that holds nothing important, most just a cache drive for video files from the encoder. In Attributes tab it giving 187 Reported uncorrect with a count of 1 been this way for a few weeks, Every SMART test I have ran comes back with no issues.

Trying to move a file to my server that is 476GB. I have 240GB cache drives. Since the file is larger than any free space I have on my cache drives should it just skip them and add file straight to my pool? Im unable to move file because it keeps trying to send it to cache then move it over but file is to large for that. How can I work around this?

Question on disk limit for Unraid Basic. I know it's set to 6 but is that system limited or array limited? Mean could I have 6 disk in my array and 1 set to Unassigned Devices plugin for a total of 7 disk in the system but only 6 to the array?

It wont download the game after the docker is installed unless my network type is set to bridged. How do I post log?

Trying to setup the v rising container but im having few issues mainly with network. My Unraid server has 2 network cards in it. eth0-br0 is primary and eth1-br1 is what I run dockers ect on and is on a totally different network. If I select custom-br1 as my network type it wont run, goes for 17-18 sec and crashes. Never even downloads. Now I can go in and download say Barotrauma or about any other server files and select my network type as custom br1 and they will run fine.