pureelectricity

I just updated to RC v7.3 and I ran into what looks like a similar issue @ich777 found here https://github.com/NVIDIA/nvidia-container-toolkit/issues/1385#issuecomment-3497279007 All my GPU containers with --gpus=all were failing before startup with errors like: error running prestart hook #0: exit status 1 Auto-detected mode as 'legacy' nvidia-container-cli: ldcache error: process /sbin/ldconfig failed with error code: 1 With toolkit debug enabled, the specific error was: could not start /sbin/ldconfig: process confinement failed: invalid argument In my case, a fix was to add: [nvidia-container-cli] no-pivot = true to /etc/nvidia-container-runtime/config.toml, then restart Docker. Specifically: no-pivot = true needed to be under [nvidia-container-cli] . After that and restarting docker, my GPU containers appear to be working again (until reboot since that's not persisted) For context, I had originally hit this after updating to RV 7.3, then even after rolling Unraid back and changing driver versions the issue was still there until I applied the no-pivot fix. Originally posted here: https://discord.com/channels/216281096667529216/1440856243156680757/1496671614107127879 Versions: Driver: 595.58.03 (Also tried 590) nvidia-container-cli: 1.19.0~1.19.0 nvidia-container-runtime: 1.19.0 EDIT: It seems this is the related commit https://github.com/unraid/nvidia-container-toolkit/commit/5085b229cfc356d01eca4823feb6dae7e9afbf49 I believe no-pivot = true should be under [nvidia-container-cli] not [nvidia-container-runtime]? Looking at @zoggy 's docs post it seems: --gpus all alone on Docker → Docker directly triggers the hook path → that is the legacy-compatible path. There was a bug in the config for this path, but it works after fixing no-pivot = true --gpus all --runtime=nvidia → Docker explicitly uses the NVIDIA runtime, which is NVIDIA’s recommended Docker invocation. Avoids the bug above gilfoyle-diagnostics-20260422_1709.zip

I don't use LibreChat anymore, but I updated the CA template to use release instead of dev. Wasn't aware updates were daily now!

I'm curious what filesystem is your cache drive and what docker implementation are you using? Mine is ZFS and I've tried all the docker implementations: XFS image, BTRFS image, and filesystem with both drivers. I get different errors depending on the implementation but it's always related to docker. I am not sure, but I think I actually started getting this issue since 6.12.14, because I remember hoping the kernel upgrade in 7.0 would fix it. I will be trying with different RAM eventually. It could also be a failing SSD (despite me not getting any SMART issues)? Both my RAM and cache disk are ~5 years old.

Ran for about 12 hours, all 6 passed. Same result when I was using the memtest plugin. Not sure where else to look or what to try.

@JorgeB Unfortunately I noticed some weirdness where my container upon reboot would not properly read any updates I made to its appdata config file. This caused me to reboot and upon reboot, the Docker server failed start and was corrupted again, this time with the docker image. I've attached the diagnostics after a reboot where the image started corrupt... gilfoyle-diagnostics-20250127-2054.zip

After switching back to the docker image, everything seems to be stable now for 42 hours. This is odd, because the docker image previously kept getting corrupt as well. I'll miss the overlay2 speeds, but glad that things do appear stable. I'm not sure why it's working now, but the only thing that's changed is after boot issues, I plugged my unRAID USB drive into another USB 2.0 slot and it booted and has been smooth since then. This is a common problem I experience across multiple USB Drive attempts, so I didn't suspect it was related. I'll mark the solution after a full week of testing

True, I've been running memtest using Live Memory Tester plugin and it succeeded with 10 rounds without issue. I'll do a full memtest later on... I woke up this morning and noticed it happened again, it seems not all containers are effected. Some can be force updated just fine and some result in the error. It seems it's not realizing the image is there because it tries pulling it again even if there are no updates (I guess that's why it's also "not available". I've attached the correct diagnostics this time without a reboot (sorry ), including a scrub on the ZFS cache. I will switch back to docker image and see if it helps, and provide those diagnostics as well if it doesn't. gilfoyle-diagnostics-20250118-1141.zip

I ran a memtest a few months ago for a different issue and things were fine. I'll add it to the list! Should be faster on 7?

Sorry, just realized these diagnostics were from after a reboot & repair (following the steps you've just linked). I'll share diags again when it happens again (probably a few hours). After I send those diags, I'll test docker image again and post diags if that image also gets corrupted (which happened 3 times before, which is why I switched to docker folder, hoping it would fix it lol)

gilfoyle-diagnostics-20250117-1147.zip Attached are the diagnostics

Hi, I'm experiencing persistent issues with Docker containers. Here's my situation: System Details: - unRAID Version: 7.0 - Docker Storage: Directory-based (previously BTRFS) - Storage Driver: overlay2 - Cache Drive: ZFS Issue Description: 1. After a fresh setup (clearing Docker folder and redownloading all containers), everything works normally for a bit (I'm not sure when it starts to break) 2. After a couple of hours I notice some containers become "not available" (orange highlight in UI), despite definitely being available on dockerhub or github 3. Once this happens, I can't update ANY containers (even available ones) and receive the following error: Pulling image: esphome/esphome:latest [Various layer downloads...] Error: failed to register layer: open /var/lib/docker/overlay2/c9d51d50aaab263982141b1cdadc2f28757be5c6be53001702e0d92f705e4bac/link: no such file or directory What I've Tried: - Cleared Docker folder and redownloaded all images (4 times) - Switched from BTRFS to directory-based storage due to previous corruption issues - System reboots (results in "No Docker containers installed" message despite images being present) - Scrubbing my ZFS cache drive (says NO errors, even after the issue) Additional Observations: - No other issues on the cache drive, only docker. - System reboot without fixing the issue first results in Docker UI showing no containers (despite the images existing in the docker folder) - Previous BTRFS setup had similar corruption issues, which led to switching to directory-based storage. When the BTRFS docker image would be scrubbed, it would have unfixable errors Has anyone experienced similar issues? Any suggestions for troubleshooting or potential fixes would be appreciated. Given that I was having similar issues on 6.12.14 with the BTRFS image, perhaps it's not related to unRAID 7?

It's a bit confusing right now! Configuration for LibreChat is currently split between both docker variables and a yaml file. The docker image currently doesn't create the yaml file if it doesn't exist, so I'm sticking to the docker variables only for now. For more advanced configuration and options please see the documentation and create your own librechat.yaml file in a appdata/LibreChat folder and manually add it as a directory in your docker configuration.

I had trouble with the OpenID social login. The button would appear but wouldn't do anything and I'm certain my variables were right. The docs seem straightforward (https://docs.librechat.ai/install/configuration/dotenv.html#social-logins) but I couldn't get it to work either. For now I'm just using email.

Hello! If you weren't able to get it working, I've made a CA app. It's now on the CA app store!

Note: The developers are eventually changing from many docker variables to a librechat.yaml in the container at /app/librechat.yaml. See documentation here: https://docs.librechat.ai/install/configuration/custom_config.html