32M /run keeps filling up.

[xtrap225]

i keep needing to reboot to release 32M /run

 

do you have any suggestions on how to mitigate this issue?

 

tmpfs            32M   32M     0 100% /run

[xtrap225]

stopped the array (i don't know if i had to do that or not, kinda wish i tried before doing that).

 

then ran 'mount -o remount,size=10G /run'

 

it seemed to work. but i doubt that will survive a reboot.  anyone know how to make that change permanent?

 

can i put it in my /boot/config/go or /boot/config/extra.cfg (i don't know how to use that one, tried to look it up in the manual but couldn't find it).

 

** so i didn't have to stop my array to do the change cause i. did it again and made it 256M after thinking for more than a second and realizing i was potentially wasting a ton of RAM **

[itimpi]

The /run location should never run out of space.    You need to look to see what is taking up that space.

[xtrap225]

thanks for the response, i am running the following for as long as it takes to hopefully figure this out.

 

nohup watch -n600 '(df -h |grep /run; echo; echo) | tee -a /boot/run.filling_up.txt; (du -h --max-depth=1 /run; echo; echo) | tee -a /boot/run.filling_up.txt' &

 

tail -F nohup.out

Edited May 27, 2023 by xtrap225

[xtrap225]

so its would seem that my plex container is filling up a log.json file with 'stuff' from the nvidia i have passed through to it.  looks like the snippet below. While it is just a snippet, it does seem to just repeat over and over, so far its up to 9.8MB.  I checked plex and don't have any debug or verbose enabled.  I am running Nvidia Driver Package on Production Branch which is currently  v525.116.04.

 

anyone recognize this issue? something about "NVIDIAContainerRuntimeConfig" and "MountSpecPath\": \"/etc/nvidia-container-runtime/host-files-for-container.d\" and "Path\": \"nvidia-ctk\"

 

i should probably also note that transcoding and what not seems to work fine when i tested using 'watch nvidia-smi' while purposefully forcing a transcode.

 

i mean to hit submit on this last night and in the meantime its gone from just under 10MB to 16MB

 

tail -F /run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/501f72c7fc3a92557935aab9479c1fb048e40ac95c9833c44efb4ee18e671884/log.json
{"level":"info","msg":"Running with config:\n{\n  \"AcceptEnvvarUnprivileged\": true,\n  \"NVIDIAContainerCLIConfig\": {\n    \"Root\": \"\"\n  },\n  \"NVIDIACTKConfig\": {\n    \"Path\": \"nvidia-ctk\"\n  },\n  \"NVIDIAContainerRuntimeConfig\": {\n    \"DebugFilePath\": \"/dev/null\",\n    \"LogLevel\": \"info\",\n    \"Runtimes\": [\n      \"docker-runc\",\n      \"runc\"\n    ],\n    \"Mode\": \"auto\",\n    \"Modes\": {\n      \"CSV\": {\n        \"MountSpecPath\": \"/etc/nvidia-container-runtime/host-files-for-container.d\"\n      },\n      \"CDI\": {\n        \"SpecDirs\": null,\n        \"DefaultKind\": \"nvidia.com/gpu\",\n        \"AnnotationPrefixes\": [\n          \"cdi.k8s.io/\"\n        ]\n      }\n    }\n  },\n  \"NVIDIAContainerRuntimeHookConfig\": {\n    \"SkipModeDetection\": false\n  }\n}","time":"2023-05-28T01:03:23-04:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-05-28T01:03:23-04:00"}
{"level":"info","msg":"Running with config:\n{\n  \"AcceptEnvvarUnprivileged\": true,\n  \"NVIDIAContainerCLIConfig\": {\n    \"Root\": \"\"\n  },\n  \"NVIDIACTKConfig\": {\n    \"Path\": \"nvidia-ctk\"\n  },\n  \"NVIDIAContainerRuntimeConfig\": {\n    \"DebugFilePath\": \"/dev/null\",\n    \"LogLevel\": \"info\",\n    \"Runtimes\": [\n      \"docker-runc\",\n      \"runc\"\n    ],\n    \"Mode\": \"auto\",\n    \"Modes\": {\n      \"CSV\": {\n        \"MountSpecPath\": \"/etc/nvidia-container-runtime/host-files-for-container.d\"\n      },\n      \"CDI\": {\n        \"SpecDirs\": null,\n        \"DefaultKind\": \"nvidia.com/gpu\",\n        \"AnnotationPrefixes\": [\n          \"cdi.k8s.io/\"\n        ]\n      }\n    }\n  },\n  \"NVIDIAContainerRuntimeHookConfig\": {\n    \"SkipModeDetection\": false\n  }\n}","time":"2023-05-28T01:03:28-04:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-05-28T01:03:28-04:00"}

[xtrap225]

so /etc/nvidia-container-runtime/host-files-for-container.d doesn't exist.

 

the only thing in that folder is .. /etc/nvidia-container-runtime/config.toml

 

also tried running 'runc list' and there was nothing.  probably doing something wrong though..

 

/usr/bin/runc list
ID          PID         STATUS      BUNDLE      CREATED     OWNER

 

i might have to just clear the log on cron for a while until a new update comes  for the nvidia driver and fixes the problem (hopefully)...

 

not sure if anyone has any better thoughts/ideas?

[xtrap225]

i don't know what is going on but i am convinced it has to. do with the nvidia drivers. i updated to the 'New Feature Branch'.

 

Just to change it up, and it is no longer the same errors in the log. just the same line over and over

 

 tail -F /run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/501f72c7fc3a92557935aab9479c1fb048e40ac95c9833c44efb4ee18e671884/log.json
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-05-29T22:57:18-04:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-05-29T22:57:23-04:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-05-29T22:57:28-04:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-05-29T22:57:33-04:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-05-29T22:57:38-04:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-05-29T22:57:43-04:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-05-29T22:57:48-04:00"}
{"level":"info","msg":"Using low-level runtime /usr/bin/runc","time":"2023-05-29T22:57:53-04:00"}

 

seems the nvidia is only working in my one plex docker, tdar and handbrake for example don't seem to work . (i was wrong plex and tdar are working; it was just handbrake that wasn't but i don't know if or what that is, maybe its a totally differnet issue.)

 

does anyone know how to fix this. properly rip out the nvidia drivers and start from scratch maybe?

Edited June 1, 2023 by xtrap225

[Polar]

I am currently seeing this same behavior unraid 6.11.5 and nvidia 525.116.04.  I do not think I was ever seeing this before doing nvidia driver update a few days ago

[Polar]

another thread said this may be the plex healthcheck that docker is running all the time.  You can disable the healthcheck in advanced settings for plex container, or userscript schedule deletion of the health logs hourly/daily.  Would prefer for the log to work without this workaround but this does seem to work for now...

 

find /run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/ -maxdepth 999999 -noleaf -type f -name "log.json" -exec rm -v "{}" \;