Nuuki

In the end I switched to a Docker image - its clearly not necessarily anything to do with the root cause, but its resolved things for me for now.

I noticed that a couple of containers were Stopped. When I try to start then I get the following: docker: Error response from daemon: Conflict. The container name "/Rallly" is already in use by container "fc8e9affcd6dc1be9853b7aec51893d1af5a5fd8f6dcdd8fd4f557d71c5e2144". You have to remove (or rename) that container to be able to reuse that name. Tried deleting and recreating the image but no luck. A quick search on them forums indicated a possible corrupted image. However in my case I'm using a docker directory (zfs). Is there a way I should be removing just this image? server-diagnostics-20231031-1400.zip

Ah right - I switched to a (btrfs) folder about a year ago so as not to worry about sizing, but if performance takes a hit its not a major issue to switch back to an image and just increase the size if needed.

Understood. I'm using a ZFS pool. and for Docker I'm using a folder with ZFS. Its running OK now - the real test will be to re-enable auto-start and bounce docker. I may try that later but I'll see if it runs stably for a day or so before I try that. Thanks for your help.

Well, I've been able to bring all the containers online. However, whenever I started one the CPU would spike much more than I would expect - over 60%, for around 30 seconds, before dropping back down. If I started 2 or 3 at once it would go much higher. Maybe it always does that, but given that there'd usually be 50 or 60 starting in quick succession, it stood out a bit. I also noted that the ZFS resource meter was usually at 85%+ I'd note that I did migrate my cache pool from btrfs to zfs at the weekend, though it went find and not had any issues until today - thought it was worth nothing anyway. So right now its all back up and none of the containers seemed individually problematic, but if I turned container auto-start back on and bounced docker, I suspect I might well see the same symptoms again.

OK I've done that and the CPU is looking normal. Shall I enable containers one by one and see if/when it goes crazy? I did change some container volume mounts earlier, but all seemed fine at the time. It seems feasible that that's caused a problem somewhere I guess.

@JorgeB I rebooted in the end as I couldn't stop the array. After it came back up I was able to stop all running containers, but dockerd is still pinned at 100% CPU.

I previously tried stopping the array (which is a USB drive) and though CPU dropped, it looks like its been stuck trying to actuallyt stop the array for some time. When I check the logs I see a bunch of these log blocks: Oct 18 17:46:19 Server emhttpd: Unmounting disks... Oct 18 17:46:19 Server emhttpd: shcmd (2345): /usr/sbin/zpool export cache Oct 18 17:46:19 Server root: cannot unmount '/var/lib/docker/zfs/graph/f9d3a0840bae10b5cb2ff414bab9a4d047e4acd9101f69ba1566b21406728928-init': unmount failed Oct 18 17:46:19 Server emhttpd: shcmd (2345): exit status: 1 Oct 18 17:46:19 Server emhttpd: Retry unmounting disk share(s)...

Good question, I could shut each container down in turn I guess. Let me try to do that and see what it does.

I was doing some server admin earlier when my CPU spiked to 100% and sat there. It looks like dockerd is the cause, but when I run docker stats nothing stands out - I did kill any containers using any level of substantial CPU, but no change. I've rebooted, but as soon as docker fires up it again nails at 100%. It has dropped a few times, but the GUI isn't responsive enough for me to shut docker down, get to "Fix Common Problems", logs etc. I was able to generate a diag file though, eventually. server-diagnostics-20231018-1654.zip

For sure. I realise I'm pushing things a bit, but there's no data on the drive itself, and as I understand it its the only viable way to ru Unraid as a container / VM server. Maybe another hypervisor solution would be better, but I really like Unraid. So if it gives me the occasional headache its not a huge issue, so long as the core is stable and reliable (which it is).

"Interestingly" I was unable to Stop the array. Even after I've removed the array USB drive, its still showing as Online in the GUI, Docker is still running etc. I'm slightly nervous about rebooting, but I'm guessing whatever state its gotten into is the cause of the error. I'll probably leave it as is for now, and once I have a new USB 2.0 drive I'll reboot and see what happens.

I'm using a Dell Optiplex 7060 Micro, which sadly doesn't seem to have any 2.0 USB ports. I can try a different flash drive though - should I be using a USB 2 one?

I've started getting the error "Unable to write to disk1" with the suggested fix "Drive mounted read-only or completely full." Disk 1 is a USB drive that I only have connected in order to run an array - I don't store anything on it, but its needed to run the docker and VM engines, which is what I use the server for. Anyway its only about 5% full, so hard to see why that would be the issue. I saw some comments in the linked FAQ that its often linked to docker. In my case I'm using a Docker directory rather than an image, and its on the cache pool (which again has plenty of space). So I'm slightly unsure what to check for next. I've attached diags - any help much appreciated. server-diagnostics-20230911-1451.zip

I've swapped it out so I'll see if that makes any difference. As you say even if its not the root cause its clearly a problem.