Something is filling up my log and I don't know what?!

[Tinker]

Disclaimer: I am a complete Newbie with Unraid/Linux/Servers in General. 

 

I have a typical Unraid setup with a few Dockers running. I left it last night with my LOG at 1% full, this morning its at 80% full. I had this happen once before where something was filling up my LOG but I have no idea what. Can someone help me make sense of this? 

 

I tried to stop the two Dockers I was most concerned with (Sabnzdb and Plex), Sabnzdb stopped but Plex wont stop and it is currently giving me a write error 403 onto the Docker Image and I'm running a balance. 

 

Thank you!

 

tower-syslog-20191231-2108.zip tower-diagnostics-20191231-1308.zip

[Squid]

Looks like you're running a balance on the cache pool which is logging everything it's doing, and then some issues with the docker.img.  Not sure if that's related to the ongoing balance or not though.  If not, then you'd simply delete it and recreate it.  https://forums.unraid.net/topic/57181-docker-faq/#comment-564309

 

[trurl]

Your syslog isn't very large, but a lot of your log space is used. And I see you have Nerd Pack installed, atop from that is known to fill log space. Are you using atop?

 

Also your docker image has 40G allocated, twice what I usually recommend.

 

Have you had problems filling docker image? Making it larger won't help that problem.

[Tinker]

Hey guys, 

Thank you for suggestions. Currently my LOG is at 93% so it's still creeping up. 

 

1. I am running balance because it had previously helped when my Docker image had become read only for some reason. 12 hours later it still has 18% to go. When balance is done, I'm going to stop all dockers, delete the Docker image, run mover to clear the cache and reboot. Then just let the system sit there for a day to see if the LOG moves without any Dockers or Balance running. 

 

2. I did adjust the Docker image size up thinking that would help with the space constraint in LOG but it did not. 

 

3. I haven't been running ATOP at all, I did run TOP a few times because I was trying to figure out if I had a rogue process causing trouble. 

 

Thanks for the suggestions,

 

 

[JorgeB]

Because of this bug your cache pool is not redundant, see there to correct.

[trurl]

3 hours ago, Tinker said:

I haven't been running ATOP at all

Do you have it installed?

 

What do you get at the command line with this?

ls -lah /var/log

 

[Tinker]

@trurl Here is the output of that command. It's clearly docker.log.1 that's eating all the space. I have no idea how to access that file to figure out what is causing the mess. 

 

 

@johnnie.black Thanks for pointing that out. I will make that fix now. 

[Tinker]

@johnnie.black Thanks, I ran the commands you suggested and made the fix so the cache pool is redundant. I actually had no intention of running the cache as a Raid1 but I can see the point. I need to swap out the 500gb drive in my pool for another 1tb SSD so the cache is more useful. 

 

root@Tower:~# btrfs fi usage -T /mnt/cache
Overall:
    Device size:                   1.35TiB
    Device allocated:            706.06GiB
    Device unallocated:          672.58GiB
    Device missing:                  0.00B
    Used:                         44.50GiB
    Free (estimated):            665.22GiB      (min: 665.22GiB)
    Data ratio:                       2.00
    Metadata ratio:                   2.00
    Global reserve:               51.52MiB      (used: 0.00B)

             Data      Metadata  System              
Id Path      RAID1     RAID1     RAID1    Unallocated
-- --------- --------- --------- -------- -----------
 1 /dev/sdf1 351.00GiB   2.00GiB 32.00MiB    94.10GiB
 2 /dev/sdg1 351.00GiB   2.00GiB 32.00MiB   578.48GiB
-- --------- --------- --------- -------- -----------
   Total     351.00GiB   2.00GiB 32.00MiB   672.58GiB
   Used       22.07GiB 181.95MiB 96.00KiB    

Edited January 2, 2020 by Tinker

[trurl]

docker.log is all that gets into diagnostics so you would have to post that older docker.log.1

[Tinker]

Sorry, newb question, how do I download the docker.log.1 file from the Unraid GUI or from terminal? 

[trurl]

That's really too large to post even zipped I guess. Probably a lot of the same stuff over and over anyway. Might be better to start over by rebooting, and then later get diagnostics so we can get the new stuff from docker.log the usual way.

[Tinker]

@trurlI did reboot to clear the log. I deleted the Docker Image and relocated it to the Cache. I've re-installed exactly 1 Docker item and I'm watching the LOG % for 1 hour. If no change, I'll add 1 more, hopefully narrowing down the culprit.