Hey all,

I recently started having this issue after installing and setting up some docker containers for media managers where after about an hour of running the Unraid system goes completely unresponsive via webgui and none of the individually running docker containers seem to actually be running, even though they are accessible at their assigned ip/ports. Trying to do anything on the pages yields zero response. I can get the login page for the webgui, but on the tower.local domain it goes to a white page after "login", and using the direct IP address it seemingly refuses the login.

 

I have tried turning off all docker containers and disabling autostarts in any of the suspect containers but it seems like maybe I misconfigured something somewhere and now its going to give me the issue whether or not the containers are running. Restarting the system temporarily clears up the issue, but its a crapshoot if it does a graceful shutdown or if it comes out unclean.

 

I've attached logs and diags, from what I can see maybe I have something saving to /boot that shouldn't be, and possibly a network issue? I'm not sure how to parse these exactly though so any help would be appreciated.

tower-diagnostics-20250119-1854.zip syslog-20250119-185428.txt

Edited January 23, 20251 yr by igabugalo

The diagnostics is full of empty files.

 

I suspect you have filled rootfs (/) somehow.

 

Reboot and try to get new diagnostics.

Alright, I've rebooted and this is the file I was given. So far everything is up and looking good.

E: Looks like I had another unclean shutdown though, but I imagine that's to be expected at the moment.

tower-diagnostics-20250119-1944.zip

Edited January 20, 20251 yr by igabugalo
added info

I've noticed that I've got more plugin updates and a few other obvious issues that I'm going to get cleaned up here tonight. I can't imagine that would actually solve this problem though, so if you see anything that I should probably change I'd certainly appreciate it. With any luck I'll catch it crash early into the evening here and I can post a new set of logs/diags for you guys.

You might want to watch the usage of the rootfs (i.e. /) as in your diagnostics it already looks as if more is used than might be expected (certainly more than on my main Unraid server).  You can check using the 'df' command from a console session.

Unrelated, your system share has files on the array. Possibly you enabled Docker and/or VM Manager before you had cache.

 

Why is your docker.img 512G? Often, default 20G is more than enough, sometimes a little more if you have a lot of containers.

 

Have you had problems filling docker.img? If docker.img usage is growing, making it larger won't fix the problem. It will only make it take longer to fill.

 

The usual cause of filling docker.img is an application writing to a path that isn't mapped.

 

And, possibly related to your other problem, one of the causes of filling rootfs is a container with a host path that isn't actually storage.

 

After fixing the couple of obvious things last night it hasn't done it again yet, so I'm going to start one by one starting and letting run these other docker containers to see if I can catch which isn't running right. Based on all of what I can see my mappings are correct and I have nothing being saved to (/) or (/boot) so I'm not exactly sure if that's the primary cause, but it sure seems like it would be the case.

 

I removed an incorrect mapping to an unassigned drive towards a download client docker. I fixed the share files on cache even though cache is not enabled issue, uninstalled an incompatible plugin that I noticed I had failed to do prior to this. I also applied any updates that were showing since the last time I had been in to do that.

 

12 hours ago, trurl said:

Unrelated, your system share has files on the array. Possibly you enabled Docker and/or VM Manager before you had cache.

 

Why is your docker.img 512G? Often, default 20G is more than enough, sometimes a little more if you have a lot of containers.

 

Have you had problems filling docker.img? If docker.img usage is growing, making it larger won't fix the problem. It will only make it take longer to fill.

 

The usual cause of filling docker.img is an application writing to a path that isn't mapped.

 

I am currently sitting at about 40G usage on my docker image with all that I usually like to have going on this system. (I've got a lot of stuff still installed that I dont regularly use though.) There was one point in time that I was playing around with AI stuff that really wound up liking having my docker image a whole heck of a lot larger for whatever reason, so I increased it to where its at and I eventually forgot about it. I've got enough Cache space that I wasn't overly worried about it. You're right though, I should probably trim that down a bit.

 

I suppose I'll keep you guys updated if something happens. With any luck it really was just something stupid that I've been able to resolve here. Sorry for bothering you if it was, otherwise thanks for the help. I'm going to give this a couple of days to see if it starts doing this again here then I'll mark the post solved.

 

I was originally thinking that this was going to be a case of an improperly configured network stack for my media managers, and that by having them configured as I did they were spamming the router and having it be blocked somehow. When the issue still happened going on a week later after turning it all off and not having anything but my media server running that started to not sound right. I still will probably look into changing how those are all connected in that manner to make sure they're not trying to hit the router rather than the internal network, but I'm no pro with this stuff so I'll need to do some more reading before I completely kill this thing.

 

Hey folks, an update on this.

 

I left it running without anything but my media center docker for an extra day here and when I got back from home the system was down. Absolutely could not connect into anything via web, when I flicked my monitor over to it the terminal was showing a kernel panic error and the entire system was utterly unresponsive. I can reboot into the system ok, but it didnt save any syslogs so all I have is a fragment I was able to find, which looks like it ends right around 17:00 so it looks like it might be the right log here. New diagnostics for you too after I rebooted.

 

What I'm getting out of this is that the gpu I have installed is giving problems? I do have the bios set to be able to make use of both iGPU and the Nvidia card I have, could that possibly be throwing an issue? I must admit I'm not sure what the correct steps for fixing this is, and I'd really rather not screw this up worse. Any help would be appreciated.

 

I'll try to see if I can get a better syslog whenever it happens again here.

tower-diagnostics-20250121-1822.zip syslog 01212025 1705pmish.txt

There does appear to be a GPU related issue:

 

Jan 20 09:57:25 Tower kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0xffff:2482)
Jan 20 09:57:25 Tower kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
Jan 20 09:57:29 Tower kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x65:2482)
Jan 20 09:57:29 Tower kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
Jan 20 09:57:31 Tower kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x65:2482)
Jan 20 09:57:31 Tower kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
Jan 20 09:57:33 Tower kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x65:2482)
Jan 20 09:57:33 Tower kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0