Log File almost full after a few days

[CrazyBoyS]

So something is eating up my log space. I have it at 385MB but its still filling up within 48-72h/

 

I have a few dockers running.

 

I have included the Diagnostic file and some more info as the syslog.1 and syslog.2 are the biggest used files.

 

ls -ail /var/log
total 119780
 1 drwxr-xr-x  9 root   root      660 Apr  2 04:40 ./
10 drwxr-xr-x 15 root   root      340 Mar 31 06:22 ../
 2 -rw-------  1 root   root        0 Mar  8 17:57 btmp
 3 -rw-r--r--  1 root   root        0 Apr  9  2020 cron
 4 -rw-r--r--  1 root   root        0 Apr  9  2020 debug
32 -rw-rw-rw-  1 root   root    75894 Mar 31 06:22 dmesg
81 -rw-rw-rw-  1 root   root        0 Mar 31 06:23 docker.log
 5 -rw-r--r--  1 root   root        0 Jun 16  2020 faillog
83 -rw-rw-rw-  1 root   root     1866 Mar 31 06:24 gitflash
 6 -rw-r--r--  1 root   root        0 Apr  7  2000 lastlog
 7 drwxr-xr-x  3 root   root       60 Jul 16  2020 libvirt/
 9 -rw-r--r--  1 root   root        0 Apr  9  2020 maillog
10 -rw-r--r--  1 root   root        0 Apr  9  2020 messages
11 drwxr-xr-x  2 root   root       40 May 15  2001 nfsd/
12 drwxr-x---  2 nobody root       60 Mar 31 06:23 nginx/
13 lrwxrwxrwx  1 root   root       24 Mar  8 17:56 packages -> ../lib/pkgtools/packages/
14 drwxr-xr-x  5 root   root      100 Mar 31 06:22 pkgtools/
17 drwxr-xr-x  2 root   root      760 Apr  1 05:53 plugins/
67 -rw-rw-rw-  1 root   root        0 Mar 31 06:22 preclear.disk.log
19 lrwxrwxrwx  1 root   root       25 Mar  8 17:58 removed_packages -> pkgtools/removed_packages/
20 lrwxrwxrwx  1 root   root       24 Mar  8 17:58 removed_scripts -> pkgtools/removed_scripts/
74 lrwxrwxrwx  1 root   root       34 Mar 31 06:22 removed_uninstall_scripts -> pkgtools/removed_uninstall_scripts/
21 drwxr-xr-x  3 root   root      180 Mar 31 06:26 samba/
22 lrwxrwxrwx  1 root   root       23 Mar  8 17:56 scripts -> ../lib/pkgtools/scripts/
23 -rw-r--r--  1 root   root        0 Apr  9  2020 secure
24 lrwxrwxrwx  1 root   root       21 Mar  8 17:56 setup -> ../lib/pkgtools/setup/
25 -rw-r--r--  1 root   root        0 Apr  9  2020 spooler
26 drwxr-xr-x  3 root   root       60 Jul 16  2020 swtpm/
89 -rw-r--r--  1 root   root  7055152 Apr  2 07:24 syslog
87 -rw-r--r--  1 root   root 59168374 Apr  2 04:40 syslog.1
29 -rw-r--r--  1 root   root 56336064 Apr  1 04:40 syslog.2
31 -rw-rw-rw-  1 root   root        0 Mar 31 06:21 vfio-pci
30 -rw-rw-r--  1 root   utmp     6912 Mar 31 06:23 wtmp

 

du -s /var/log/*
0       /var/log/btmp
0       /var/log/cron
0       /var/log/debug
76      /var/log/dmesg
0       /var/log/docker.log
0       /var/log/faillog
4       /var/log/gitflash
0       /var/log/lastlog
0       /var/log/libvirt
0       /var/log/maillog
0       /var/log/messages
0       /var/log/nfsd
4       /var/log/nginx
0       /var/log/packages
4       /var/log/pkgtools
0       /var/log/plugins
0       /var/log/preclear.disk.log
0       /var/log/removed_packages
0       /var/log/removed_scripts
0       /var/log/removed_uninstall_scripts
24      /var/log/samba
0       /var/log/scripts
0       /var/log/secure
0       /var/log/setup
0       /var/log/spooler
0       /var/log/swtpm
6972    /var/log/syslog
57784   /var/log/syslog.1
55016   /var/log/syslog.2
0       /var/log/vfio-pci
8       /var/log/wtmp

king-koopa-diagnostics-20210402-0726.zip

[ChatNoir]

Hello,

Your log is spammed by lines such as those :

Mar 31 08:11:09 KiNG-KOOPA kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window]
Mar 31 08:11:09 KiNG-KOOPA kernel: caller _nv000708rm+0x1af/0x200 [nvidia] mapping multiple BARs
Mar 31 08:11:09 KiNG-KOOPA shfs: share cache full
### [PREVIOUS LINE REPEATED 24 TIMES] ###

 

I see two different errors:

1. issues with your graphics card driver I think (first two lines)
2. you cache is full (third line)

For your cache, check if all your shares are properly set and run the mover.

 

to transfer from cache to Array = cache : Yes

to transfer from Array to cache = cache : Prefer

 

You share M---a (probably Media) is set to cache prefer and has files all over the place

shareUseCache="prefer"
shareCachePool="cache"
# Share exists on cache,disk1,disk2,disk3,disk4,disk5,disk6,disk7,disk8,disk9,disk10

 

You might want to start there and check and adjust all shares as needed.

 

For the graphics card, I am not sure but hopefully someone can chip in.

[CrazyBoyS]

Might be a driver that is stuck for my P400.

 

I am going to update it to the newest version and see what happens.

 

As for the shares just made the change.

[John_M]

4 hours ago, CrazyBoyS said:

Might be a driver that is stuck for my P400.

 

A lot of people with Quadro cards are reporting the same problem at the moment. Maybe a newer driver from Nvidia will fix it.

 

[trurl]

10 hours ago, CrazyBoyS said:

As for the shares just made the change.

@CrazyBoyS

Post new diagnostics. There were a number of things you need to fix about your shares.

 

Typically, you want appdata,domains,system shares all on cache and set to stay on cache so your array disks won't stay spun up due to docker / VM open files, and so dockers / VMs won't have performance impacted by slower array.

 

Also, why do you have 50G for docker.img? Have you had problems filling it? 20G is usually more than enough. Making it larger won't fix the problem of filling it, it will only make it take longer to fill.

[CrazyBoyS]

I read somewhere in the forums some dockers cause it to get large in the docker image. Mine in the past has filled and crashed all my dockers but also could be related to the shares.

 

Since I fixed the shares and updated the nvidia driver and then rebooted my log has not moved past 1%.

 

I also found when my last docker crashed it held everything in my cache and wouldent let it go so that was not helping with the logs. Moved everything off my cache to the array with unbalance so that's fixed too.

[trurl]

 

43 minutes ago, trurl said:

Also, why do you have 50G for docker.img? Have you had problems filling it? 20G is usually more than enough. Making it larger won't fix the problem of filling it, it will only make it take longer to fill.

The usual reason for filling docker.img is an application writing to a path that isn't mapped. Linux is case-sensitive.

 

44 minutes ago, trurl said:

Post new diagnostics. There were a number of things you need to fix about your shares.