CorserMoon Posted October 30, 2021 Share Posted October 30, 2021 (edited) A few days ago I woke up to Plex not working and the container refusing to start. Quickly learned that the docker log was full. I rebooted the system and all was fine. I monitored the log size for the next couple days and it stayed at 1%. This morning however, back to 100%. I need help trying to figure out what is causing this issue. The docker.log is full of the errors that you see in the below screenshot. I've also attached my diagnostic zip. I've since rebooted the server to get everything working again. executor-diagnostics-20211030-0853.zip Edited October 31, 2021 by CorserMoon Quote Link to comment
Squid Posted October 30, 2021 Share Posted October 30, 2021 It's most likely not your log actually filling up, but rather the image itself due to a misconfigure of the various paths? What does Container Size show on the Docker tab? A number of entries in the Docker FAQ here about the image filling up. Quote Link to comment
CorserMoon Posted October 30, 2021 Author Share Posted October 30, 2021 1 hour ago, Squid said: It's most likely not your log actually filling up, but rather the image itself due to a misconfigure of the various paths? What does Container Size show on the Docker tab? A number of entries in the Docker FAQ here about the image filling up. I actually checked the docker img, it's sitting at 50% and was when the log filled. Here is a screenshot I grabbed: And in the screenshot in the previous post, you can see that docker.log.1 is 127mb. I grabbed it off the server to check and it is in fact 127mb and full almost entirely of those "...read only file system" errors . Quote Link to comment
trurl Posted October 30, 2021 Share Posted October 30, 2021 Even though it isn't full, it is corrupt. You have to recreate and reinstall your dockers. https://wiki.unraid.net/Manual/Docker_Management#Re-Create_the_Docker_image_file Why have you given it 50G? 20G is often more than enough. Maybe you did fill it earlier. The usual reason for filling is an application writing to a path that isn't mapped. Linux is case-sensitive. Quote Link to comment
CorserMoon Posted October 30, 2021 Author Share Posted October 30, 2021 (edited) 1 hour ago, trurl said: Even though it isn't full, it is corrupt. You have to recreate and reinstall your dockers. https://wiki.unraid.net/Manual/Docker_Management#Re-Create_the_Docker_image_file Why have you given it 50G? 20G is often more than enough. Maybe you did fill it earlier. The usual reason for filling is an application writing to a path that isn't mapped. Linux is case-sensitive. How do you know the img is corrupt? I've checked the paths and they are working correctly. I increased to 50gb because I was about 80% full when using the default size. I run a lot of containers and several are ~3gb each freshly installed. I think I figured out the culprit though. I translated the container IDs in the log output to the container name. It turns out that the container(s) in question were trying to write to a secondary cache pool I created which was full. This also explains the space unavailable errors I was seeing in the syslog. I cleared some space in this cache pool and will see if that does the trick. Edited October 30, 2021 by CorserMoon Quote Link to comment
Squid Posted October 30, 2021 Share Posted October 30, 2021 2 hours ago, CorserMoon said: How do you know the img is corrupt? Oct 30 00:03:11 Executor kernel: verify_parent_transid: 7967 callbacks suppressed Oct 30 00:03:11 Executor kernel: BTRFS error (device loop2): parent transid verify failed on 17584734208 wanted 1640023 found 1639989 By far the easiest way for this to happen is when the image fills up. Either way, the solution is the same. Delete the image, then reinstall from Apps - Previous Apps. 5 minutes and you're done 1 Quote Link to comment
CorserMoon Posted October 30, 2021 Author Share Posted October 30, 2021 (edited) Thanks for the help. I followed the guide to delete and reinstall the containers and everything worked like a charm except for swag and a couple other containers that were using a custom docker network that wasn't preserved. I created the customer network via the terminal command docker network create swagnet Is there a better way to create a custom docker network that will remain if docker.img is deleted? Edited October 31, 2021 by CorserMoon Quote Link to comment
CorserMoon Posted October 31, 2021 Author Share Posted October 31, 2021 (edited) OK, log file filled back up again over night, docker.img still at only 57% utilization. The docker.log.1 file is filled with this: Attached latest diagnostic file. I deleted and rebuilt the docker.img so that shouldn't be the issue. executor-diagnostics-20211031-1258.zip Edited October 31, 2021 by CorserMoon Quote Link to comment
Squid Posted October 31, 2021 Share Posted October 31, 2021 Have you run a memtest lately? Same errors within the docker.img Quote Link to comment
CorserMoon Posted October 31, 2021 Author Share Posted October 31, 2021 9 minutes ago, Squid said: Have you run a memtest lately? Same errors within the docker.img I'm trying to get better at investigating so forgive my questions. But how do you know that the log errors like: Oct 30 00:03:11 Executor kernel: verify_parent_transid: 7967 callbacks suppressed Oct 30 00:03:11 Executor kernel: BTRFS error (device loop2): parent transid verify failed on 17584734208 wanted 1640023 found 1639989 are from docker.img? I also see that those errors started after the log space errors, so maybe the docker.log is filling first and then throwing those errors that look like docker.img corruption? In the meantime, I'll run a memtest and see if that's ok. Quote Link to comment
Squid Posted October 31, 2021 Share Posted October 31, 2021 /dev/loop2 40G 23G 18G 57% /var/lib/docker If you keep having trouble, I'd suggest using a folder instead of an image for the docker (Settings - Docker and switch to folder) 1 Quote Link to comment
CorserMoon Posted October 31, 2021 Author Share Posted October 31, 2021 1 hour ago, Squid said: /dev/loop2 40G 23G 18G 57% /var/lib/docker If you keep having trouble, I'd suggest using a folder instead of an image for the docker (Settings - Docker and switch to folder) Thanks for the tip. I'll try that after memtest finishes a pass. 45% and no errors so far. Aside from the obvious, whats the difference between having docker as a folder vs img? Quote Link to comment
Squid Posted October 31, 2021 Share Posted October 31, 2021 Usual reason for the image getting corrupted is the image filling up. Setting it as a folder allows it to expand up to the entire size of the drive its on without actually reserving any space in advance for it. From what I'm seeing in the logs though there's no obvious reason for what's going on (hence the memtest). So, my next shot is try a folder 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.