Nuuki

Probably me being dense, but how are you handling authentication? Hyperbackup requires it but your image doesn't reference setting a password.

Its now been running for about 17 hours without issues - plugged in at the wall rather than to a smart plug and UPS. I'll probably try moving it back to the UPS to see if the issue comes back, as if so I think that's then conclusive. Either way this doesn't seem to be Unraid related, so I'll close this out. Thanks for the support, as ever!

OK - so I powered the server back on after the above outage, but I then started seeing Emergency logs from nginx, indicating "Address already in use". I issued a reboot over SSH and its now come back up without those same issues - I've attached the logs from when it came up following the loss of power. Maybe its a result of the sudden power failure, and not part of the root cause? Anyway the server is up again, this time running from a standard output (rather than the UPS). I'll see if it stays up for longer than 15 minutes... new logs.txt

OK, so I had the server running for about an hour with no issues. I removed the out of date mover plug-in, checked nothing unexpected in the logs and fired up Docker. I'd moved it so I could connect a monitor and keyboard, so once I was happy that everything seemed fine, I moved it back to my equipment cupboard. After about 15 minutes it powered off again. No beep to indicate it was going to shut down, and when I check the remote syslog I see it was in the middle of bringing up containers, with nothing indicating any shut doen had been initiated. I just looked over and realised the power light was off. I'm now noticing that my UPS is clicking about once per minute. However its not beeping to indicate its on battery, and none of the other equipment is having any issues. Its on Low sensitivity for input voltage. So overall I'd be a bit surprised if that was causing the server to completely power off, but its clearly possible that its related. I guess my next step will be to leave it where it is, but run a separate power cable to it, bypassing the UPS. Anything else I should be doing in Unraid?

Yeah sorry for not being fully clear. I'm just running a check on 1 of the 2 drives in the cache pool. Once that's done I'll power up in safe mode and see what it does.

Thanks. I don't actually run an array - I'm just using Unraid for Docker. I'll remove that plug-in though, once I get things back up and stable.

The server initially powered down shortly after 1am. I turned it back on manually, but then initiated a standard reboot. So that reboot was likely from me.

Last night my server unexpectedly shut down shortly after 1am - no issues with the UPS and everythng else was running fine. I powered it up but after a short while it shut off again. I did grab a diag dump before that happened, though I asume it will be missing any relevant info from the over night crash. It now seems to be powering off every time - the UI is useable but after a few minutes or so it powers off. I do also ship SNMP events to a remote server though, so I've also attached logs from just before the point where it originally shut down. Any help is much appreciated. server-diagnostics-20250306-0751.zip syslog.csv

Thanks! I've added a memory limit for now, but I guess I'll need to do more digging if it starts running out of control again. As noted above its a shared postgres container, so maybe its time to run 1 per application...

It looks like the postgres container was to blame. I use that for about a dozen application - the one exception being Immich, which I have moved to its own postgres instance. I've bounced back and forth on whether to have a dbms per app stack or one shared one - up to now I've gone the shared route as I'm not using compose and it seemed more elegant, but maybe its time to split postgres out, to make troubleshooting and resource management easier. Good to know. I've rebooted since then so I can't check which container that was I don't think, but I'll keep an eye on the logs to see if it crops up again.

Been running my Unraid server for a few years without issue. In the last couple of weeks I've had a few occurances of it grinding to a halt - some apps still run, but the Unraid UI is inaccessible, and even SSH is inconsistent. Apps are typically not accessible. A reboot clears it for a while but then I can't grab the diags. Today it happened again, but I was able to use the GUI. Fix Common Problems indicated "Out Of Memory errors detected on your server". I was also able to grab a diag download. I assume one of my containers if probably to blaim, but the system is so unresponsive that I'm struggling to troubleshoot it at the time. server-diagnostics-20241130-1548.zip

In the end I switched to a Docker image - its clearly not necessarily anything to do with the root cause, but its resolved things for me for now.

I noticed that a couple of containers were Stopped. When I try to start then I get the following: docker: Error response from daemon: Conflict. The container name "/Rallly" is already in use by container "fc8e9affcd6dc1be9853b7aec51893d1af5a5fd8f6dcdd8fd4f557d71c5e2144". You have to remove (or rename) that container to be able to reuse that name. Tried deleting and recreating the image but no luck. A quick search on them forums indicated a possible corrupted image. However in my case I'm using a docker directory (zfs). Is there a way I should be removing just this image? server-diagnostics-20231031-1400.zip

Ah right - I switched to a (btrfs) folder about a year ago so as not to worry about sizing, but if performance takes a hit its not a major issue to switch back to an image and just increase the size if needed.

Understood. I'm using a ZFS pool. and for Docker I'm using a folder with ZFS. Its running OK now - the real test will be to re-enable auto-start and bounce docker. I may try that later but I'll see if it runs stably for a day or so before I try that. Thanks for your help.

Well, I've been able to bring all the containers online. However, whenever I started one the CPU would spike much more than I would expect - over 60%, for around 30 seconds, before dropping back down. If I started 2 or 3 at once it would go much higher. Maybe it always does that, but given that there'd usually be 50 or 60 starting in quick succession, it stood out a bit. I also noted that the ZFS resource meter was usually at 85%+ I'd note that I did migrate my cache pool from btrfs to zfs at the weekend, though it went find and not had any issues until today - thought it was worth nothing anyway. So right now its all back up and none of the containers seemed individually problematic, but if I turned container auto-start back on and bounced docker, I suspect I might well see the same symptoms again.

OK I've done that and the CPU is looking normal. Shall I enable containers one by one and see if/when it goes crazy? I did change some container volume mounts earlier, but all seemed fine at the time. It seems feasible that that's caused a problem somewhere I guess.

@JorgeB I rebooted in the end as I couldn't stop the array. After it came back up I was able to stop all running containers, but dockerd is still pinned at 100% CPU.

I previously tried stopping the array (which is a USB drive) and though CPU dropped, it looks like its been stuck trying to actuallyt stop the array for some time. When I check the logs I see a bunch of these log blocks: Oct 18 17:46:19 Server emhttpd: Unmounting disks... Oct 18 17:46:19 Server emhttpd: shcmd (2345): /usr/sbin/zpool export cache Oct 18 17:46:19 Server root: cannot unmount '/var/lib/docker/zfs/graph/f9d3a0840bae10b5cb2ff414bab9a4d047e4acd9101f69ba1566b21406728928-init': unmount failed Oct 18 17:46:19 Server emhttpd: shcmd (2345): exit status: 1 Oct 18 17:46:19 Server emhttpd: Retry unmounting disk share(s)...

Good question, I could shut each container down in turn I guess. Let me try to do that and see what it does.

I was doing some server admin earlier when my CPU spiked to 100% and sat there. It looks like dockerd is the cause, but when I run docker stats nothing stands out - I did kill any containers using any level of substantial CPU, but no change. I've rebooted, but as soon as docker fires up it again nails at 100%. It has dropped a few times, but the GUI isn't responsive enough for me to shut docker down, get to "Fix Common Problems", logs etc. I was able to generate a diag file though, eventually. server-diagnostics-20231018-1654.zip

For sure. I realise I'm pushing things a bit, but there's no data on the drive itself, and as I understand it its the only viable way to ru Unraid as a container / VM server. Maybe another hypervisor solution would be better, but I really like Unraid. So if it gives me the occasional headache its not a huge issue, so long as the core is stable and reliable (which it is).

"Interestingly" I was unable to Stop the array. Even after I've removed the array USB drive, its still showing as Online in the GUI, Docker is still running etc. I'm slightly nervous about rebooting, but I'm guessing whatever state its gotten into is the cause of the error. I'll probably leave it as is for now, and once I have a new USB 2.0 drive I'll reboot and see what happens.

I'm using a Dell Optiplex 7060 Micro, which sadly doesn't seem to have any 2.0 USB ports. I can try a different flash drive though - should I be using a USB 2 one?

I've started getting the error "Unable to write to disk1" with the suggested fix "Drive mounted read-only or completely full." Disk 1 is a USB drive that I only have connected in order to run an array - I don't store anything on it, but its needed to run the docker and VM engines, which is what I use the server for. Anyway its only about 5% full, so hard to see why that would be the issue. I saw some comments in the linked FAQ that its often linked to docker. In my case I'm using a Docker directory rather than an image, and its on the cache pool (which again has plenty of space). So I'm slightly unsure what to check for next. I've attached diags - any help much appreciated. server-diagnostics-20230911-1451.zip