msalad

Hi @ich777, Final update I was able to fix the issue 🙂 I used a new flash drive and fresh Unraid 6.12.15 install to rebuild my server from scratch. When I booted up the array for the first time, all of my data and dockers came back. I then did a fresh install of the Nvidia driver plugin, and to my amazement I still had the nvidia-container-cli permissions issue when starting Plex w/ my gpu passed through. To fix, I found that an specific order of operations key: My first attempt: 1) delete docker.img, 2) reinstall Plex docker and configure gpu passthrough, 3) install Nvidia driver, 4) disable + enable docker, 5) restart. This failed - I encountered the nvidia-container-cli permissions issue again on my next boot. Attempt two: 1) disable docker, delete docker.img, enable docker, 2) install Nvidia driver, 3) disable + enable docker, 4) install Plex docker and configure gpu passthrough, 5) restart. WORKS!! For some reason, it was crucial that I a) delete my docker.img and then b) install the Nvidia driver first before any of my old dockers. I do wonder if deleting my docker.img and the other steps I took would have fixed the issue in my original Unraid install but I'm not going to get hung up on that. This saga is finally at an end. Thanks for all of your help!

After looking at your plugin's changelogs, this was likely the issue. I was using v2025.03.18 at the latest at the time

Hey @ich777, I did the steps you indicated. Plex booted with my nvidia gpu passed through without issue. I installed the gpu statistics plugin and it worked as expected. I also have an intel A310 gpu in my server and was able to install intel_gpu_top, which worked as expected as well. Interestingly, in my original server, the gpu statistics plugin only works with my nvidia gpu, not my intel gpu. Not sure if this is relevant but just wanted to give you as much info as I can. This reminded me that I also installed intel_gpu_top while still on unraid 6.12.14, immediately before upgrading to unraid 7.0.0. Maybe that caused a conflict? It seems like something is broken with my original server and is not related to my hardware. I think it's time to format my original usb, install fresh unraid 7.0.1, and rebuild my server piece by piece. Would you agree?

Hi @ich777, Back in Feb2025 we discussed me running into a strange "nvidia-container-cli permission denied" error, seemingly out of nowhere, after I upgraded from Unraid 6.12.14--> 7.0. For reference, our discussion is here Unfortunately, this issue hasn't gone away but I'm desperate for a solution. I downgraded from Unraid 7.0.1 --> 6.12.15 but still encounter the issue. I have removed the Nvidia driver plugin, turned docker on/off, rebooted, and reinstalled the Nvidia driver plugin again but still encounter the issue. My question is, how can I "start fresh"? I need some help getting back to my previous state where this permissions issue doesn't happen anymore Currently, my 2 main headaches that this permissions issue causes are: 1) I do a weekly, overnight appdata backup w/ the associated plugin. This plugin stops each docker, backs up it's appdata folder, and then restarts the container. So my Plex container (w/ Nvidia GPU) is stopped, backed up, but then can't restart again because of this nvidia-container-cli permissions error. Recall that my only workaround for this error is to install and then uninstall any plugin (I like using the CA GUI Links plugin for this but any seem to work), and then my Plex (or other docker w/ Nvidia GPU passed through) will start back up. So I end up with many hours of downtime until I'm able to manually install/uninstall a plugin the following morning and restart my Plex docker. Idea - do you know if there is a way to install a plugin that's in the app store using the command line or using a script? If I could automate installing and then uninstalling a plugin using say a userscript, I could just configure that to run after my backup process. That would effectively work around this issue for me 2) 2nd major headache - I have my array set to auto start on boot, but my dockers w/nvidia gpu passed through always fail to start on first system boot until I install and then uninstall any plugin. This is a significant issue since I now have to manually monitor each boot. As above, if I could automate installing and uninstalling a plugin at first array start (like with a userscript), that would be a good workaround for this issue. Final question - Alternatively, I'm willing to do the extreme - can I format my Unraid USB drive, install a fresh copy of Unraid 7.0.1 to it, and essentially remake my server from scratch? I've never done that before so I'm just not sure how it would work regarding not losing my 400 TB of data. If I reassign my hard drives to the array (I don't use parity) + assign my nvme cache drives to their respective, correctly named pools, would I be able to start my array without formatting my disks and access my data? I appreciate any final advice you can give on this issue

I don't think it's well knwon but you can already do this. Signup for a free account at docker.com, then go to your profile --> personal access tokens to create a personal access token. Then in unraid, make a userscript scheduled to run at startup of array as follows: #!/bin/bash docker login -u <username> -p <access-token>

@JorgeBThank you for this info. I have since downgraded back to Unraid 6.12.15 and no longer experience this issue with nemory. I'm not sure what is going on with the Unraid 7.0/7.0.1 kernel but I no longer experience this call trace after downgrading and my server stopped hard crashing. Unfortunately the fixes didn't help for me - my server kept crashing and that call trace kept occurring. I didn't disable hugepages like you did though. I have since downgraded to Unraid 6.12.15 and no longer have any issues. No call traces and no server crashes. Not sure what is going on with the Unraid 7.0/7.0.1 kernel but it was very unstable for me.

Any new findings on your end@therapist? With my swapfile set to 32GB and swapiness=10, 15GB was filled up in 48 hrs. Even though I have 200 GB of free ram right now. Makes no sense. Still no help from my ticket that's open with Unraid Support - they told me to "run some tests and let us know how it goes"... Not super helpful. I reconfigured the swapfile to 256 GB and swappiness=1. Gonna see how fast it fills up now

Hi, I've successfully passed through my Arc310 gpu and it works great. I've found that I don't need to use --device=/dev/dri/:/dev/dri/ in extra parameters too, just use the "add path, variable, ..." button and choose Device. But the assignment for my Arc310 keeps changing on reboot (eg. /dev/dry/card1 and /dev/dri/renderD128 to /dev/dri/card2 and /dev/dri/renderD129). So I created 2 udev rules to make symbolic links tied to the vendor ID + device ID of the GPU. The gpu symbolic links are /dev/arc310-card and /dev/arc310-render. I've verified that both symbolic links do in fact refer back to the gpu (ls -l /dev/arc310-card --> /dev/dri/card1) But when I pass these 2 symbolic links through in the docker, Plex doesnt recognize that I passed through a GPU at all. Is there something additional I need to do to get these symbolic links to work in the docker run command for this container?

Hey @therapist Thanks for the details. So I added the boot modifications iommu=pt and mitigations=off, I set the file descriptors to 813384, and I installed the swap file plugin. I didn't disable hugepages. For my swapfile, I put it on a nvme drive as an unassigned device, set the swapfile size to 32 GB, and swapiness=10. I have 215 GB of ram free right now but even with swapiness=10, I am seeing the swapfile being used - about 110 MB in the first ~30 min of enabling it. I do not understand what's going on. The swapfile shouldn't be seeing use until my free memory is < 10%... I reached out to unraid support before doing all of this but they simply said to try your fixes and didn't offer any additional insight. Not sure how to diagnose further but this is definitely concerning. I'll report back to unraid support with these results and see what they say

I'm experiencing this vmalloc issues as well ever since updating to unraid 7 from v6.12.14. I have 256 GB of ram and am using less than half of it. I will implement a swap file first and then the boot modifications. @therapistwhich swap file plugin did you use? The only one I see in the appstore is called "Swapfile for Unraid 6.9". Also, what does command "added shareNOFILE="813384" to /boot/config/share.cfg" do? Since upgrading to Unraid 7, this call trace started and my server has been hard crashing. I'm wondering if the two are related.

Oh very interesting. Thanks for the details and for helping me out!

That curl command worked! Now the settings I enter for "custom scripts" and "some extra options" are saved correctly. Can you explain a little bit about what the issue I had was? Is it because the # of containers I had made some kind of command too long?

what the heck. sorry! try these? new window - https://pastebin.com/0EeFFArT behind window - https://pastebin.com/D79zmuhQ The pastebin site says my posts are "Pending moderation". Not sure what that means? If those don't work, after a cursory look, the paths I enter are in the new window that pops up, but aren't reflected in the page behind the new window

fixed 😃

I used the debugForm to add `/mnt/user/Media` to the Global exclusions list and `/mnt/user/appdata/misc/appdata_backup_to_gdrive.sh` to the post-run script section This is the new window that pops up This is the page behind the new window that pops up

Are you on unraid 7.0.1? This only started happening to me after I upgraded from 6.12.14. I've tried on both Chrome and Firefox - same result, these settings are discarded. I attached my diagnostics file. It is strange that this isn't getting written to my flash drive. Could it be a weird permissions issue or something? I wonder if I can manually add these settings in the .conf file as a workaround? I'm just not sure what the structure would be and of course would like to fix the issue instead if possible msaladunraid-diagnostics-20250304-1213.zip

The plugin settings at the top - "main settings" and "notifications and scheduling" - are all saved correctly. I'm backing up 50+ containers - no real change in # count from when I was on unraid 6.12.x. For the entire "custom scripts" and "some extra settings" sections, anything I put into those sections is discarded after hitting save. I've also confirmed that after hitting save, they aren't being saved to the .conf file for the plugin either - meaning it's not just a visual bug on my end

Hi, Since updating to Unraid 7, I can't set a post-run script path or set any paths in the Global Exclusion box. Once I do and then click save, whatever I have put into those boxes disappears. I've uninstalled and reinstalled the plugin but this didn't help. Any help would be greatly appreciated.

Hey @ich777, do you have anymore insight into my nvidia-container-cli issue above? It now does seem unrelated to Nerd Tools plugin since I found that I can install a different plugin to fix the issue too. I'm just not sure where to go from here. Having to install/uninstall a plugin after every reboot and then manually start the affected containers isn't ideal for long term

Thanks for taking a look at my post. I looked at the github issue but it seems like for that user, the problem went away by itself? I wish I was that lucky! I removed that script in my go file and downgraded to nvidia driver 565.77 but the same issue occurs. I currently use my rtx 3060 ti in the following docker containers: whisper-asr-service, immich, plex. Surprisingly, installing a different plugin (I randomly chose "AppData Cleanup") also fixes this issue - my containers (plex, immich, and whisper-asr-service) can start again as normal. I booted up Unraid, the containers failed to start with the same nvidia-container-cli error message, and then I installed the AppData Cleanup plugin (note I did not uninstall Nerd Tools this time, it is still currently installed). I then tried to start the containers again and they started up no problem. What the heck?

Hi @ich777, I'm experiencing a bug after updating from Unraid 6.12.14 to Unraid 7 involving an interaction between the Nerd Tools plugin and the Nvidia driver plugin where Nerd Tools interacts somehow with `nvidia-container-cli` resulting in it giving a permission denied error when it accesses `/proc/sys/kernel/overflowuid` while trying to start dockers that use nvidia gpus. The end result is that docker containers with passed-through nvidia gpus fail to start with the system (not container) logs showing: `rc.docker: nvidia-container-cli: initialization error: open failed: /proc/sys/kernel/overflowuid: permission denied: unknown` error message. # Steps to reproduce: 1. Be in Unraid 6.12.x with NerdTools plugin and Nvidia driver plugin installed. Have dockers with an nvidia you passed through. 2. Upgrade from Unraid 6.12.x to Unraid 7. 3. Restart to complete the upgrade. 4. On restart, Unraid notifies you that the NerdTools plugin is not supported in Unraid 7 and auto-disables it. Remove the plugin using the webgui. At this point, docker containers with nvidia gpus passed through still work as expected. 5. Restart again. 6. On restart, docker containers with passed-through nvidia gpus fail to start with error message `rc.docker: nvidia-container-cli: initialization error: open failed: /proc/sys/kernel/overflowuid: permission denied: unknown` # Attempts to fix docker containers that fail to start: 1. Uninstall Nvidia drivers, restart, reinstall Nvidia drivers, disable and enable docker service, and finally restart again. This does not fix the issue. 2. Remove the affected docker container and reinstall it. This does not fix the issue. 3. Run Docker Safe New Perms. This does not fix the issue. 4. Panic. This does not fix the issue. # Notes Running `nvidia-smi` in the Unraid terminal still works as expected, showing a detected gpu. My gpu also still shows in the GPU Statistics plugin dashboard. # Workaround: 1. Enable "show incompatible apps" in the appstore and reinstall the Nerd Tools plugin. The affected docker containers now start as expected. The interesting part is that my Nerd Tools plugin is not doing anything - no packages are enabled, it is simply actively installed (screenshot) This same issue was briefly mentioned by ich777 on the Unraid forums (here) but the cause was unclear. # Reoccurring Issue: On reboot, the issue occurs again and the same dockers fail to start with the same error message. In order to fix, I needed to uninstall and reinstall the Nerd Tools plugin again. Any insight into a permanent fix for this would be greatly appreciated. msaladunraid-diagnostics-20250214-1748.zip

I also can't get this to work @Draco. Did you ever figure this out @frollard?

Thanks for the quick reply. The messages have stopped after I closed the tab and opened a new one. I'll keep an eye on my logs to see if it pops up again

Hey, I'm running version 2024.03.04b on unraid 6.11.5 and I just started getting the below error message in my unraid's log: nginx: 2024/03/05 00:14:31 [error] 78533#78533: *9345199 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 192.168.1.22, server: , request: "GET /plugins/nut/include/nut_status.php?all=false HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.1.235", referrer: "http://192.168.1.235/Dashboard" where I'm accessing my unraid dashboard from 192.168.1.22 and my unraid server is located at 192.168.1.235. Any ideas on what this means?

Hey, I just installed the plugin - I was able to access the webgui of my dockers via tailscale on my phone no problem but I couldn't access Unraid's webgui at all. I restarted my server and that fixed it. I read a page or two earlier that you were trying to chase down a bug related to not being able to access the Unraid webgui after first installing the tailscale plugin. I wonder if i just experienced that bug?