Jump to content
theone

Out of space for log file?

9 posts in this topic Last Reply

Recommended Posts

I am getting the following email from my unraid server every minute:

Subject: cron for user root /usr/lib/sa/sa1 2 1 1> /dev/null

Cannot write data to system activity file: No space left on device
[code]

I ran "df" and got:
[code]
root@Tower:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
tmpfs                   131072    131072         0 100% /var/log
/dev/sda1              7819328    575232   7244096   8% /boot
/dev/md1             1953454928 1878535224  74919704  97% /mnt/disk1
/dev/md2             976732736 959285160  17447576  99% /mnt/disk2
/dev/md4             1953454928 1914671804  38783124  99% /mnt/disk4
/dev/md5             2930177100 1494300852 1435876248  51% /mnt/disk5
/dev/sdd1            244191092 128747676 115443416  53% /mnt/cache
shfs                 7813819692 6246793040 1567026652  80% /mnt/user

 

It seems that my tmpfs is 100% FULL !!!

 

This is my syslog file:

348 -rw-r--r--  1 root root 356352 2014-02-05 17:46 syslog

 

Why is this happening? This is not the first time. It usually happens ~40 days after server is up. currently ~40 days.

 

 

edit:

After deleting the syslog:

root@Tower:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
tmpfs                   131072    130752       320 100% /var/log
/dev/sda1              7819328    575232   7244096   8% /boot
/dev/md1             1953454928 1878535224  74919704  97% /mnt/disk1
/dev/md2             976732736 959285160  17447576  99% /mnt/disk2
/dev/md4             1953454928 1914671804  38783124  99% /mnt/disk4
/dev/md5             2930177100 1494300852 1435876248  51% /mnt/disk5
/dev/sdd1            244191092 128747696 115443396  53% /mnt/cache
shfs                 7813819692 6246793040 1567026652  80% /mnt/user

 

How can I see what is taking all the space in tmpfs?

 

Share this post


Link to post

It's the System Stats plugin for SF and Dynamix.  I've never had issues with /var/log filling up until enabling that plugin.

 

I imagine you can direct logging for sa to go somewhere else or set it to roll over but I just haven't looked into it.

 

ls -l /var/log/sa

 

hmmm just checked mine again and it looks like it does roll over after sa31.  Still might be worth looking into moving the logging to a different place...

 

Share this post


Link to post

I ran:

root@Tower:~# du -a /var | sort -n -r | head -n 10
143116  /var
130756  /var/log
128952  /var/log/sa
11912   /var/cache
11892   /var/cache/samba
11360   /var/cache/samba/locking.tdb
6312    /var/log/sa/sa31
6312    /var/log/sa/sa30
6312    /var/log/sa/sa29
6312    /var/log/sa/sa28

 

and indeed it is SA.

 

It seems that everyone should have ha this issue, I haven't come across it on the forums before.

 

Share this post


Link to post

I think given a certain combination of plugins and activity for your particular setup it may affect you.

Share this post


Link to post

I am having this same issue since upgrading to v6.  It is indeed SA filling tempfs and then logging stops.  I have been manually deleting the saXX files once tempfs fills up so that logging can resume. 

 

Wondering if this is on LimeTech's radar as an issue?

Share this post


Link to post

Try this. It will increase your tmpfs. I have it in my go file but you can just execute it and put it in the go file for reboot. There a thread about it somewhere.

# resize tmpfs
mount -o remount,size=256m /var/log

Share this post


Link to post

Try this. It will increase your tmpfs. I have it in my go file but you can just execute it and put it in the go file for reboot. There a thread about it somewhere.

# resize tmpfs
mount -o remount,size=256m /var/log

Been in my go file since before the last upgrade.

Share this post


Link to post

Good to know, I had missed that.  I'll do it now, thanks.

 

I assume 256MB for tempfs provides enough space for the SA logs to rotate through sa31 without filling up tempfs, and therefore resolves the issue?

Share this post


Link to post

Good to know, I had missed that.  I'll do it now, thanks.

 

I assume 256MB for tempfs provides enough space for the SA logs to rotate through sa31 without filling up tempfs, and therefore resolves the issue?

For me it did. I think mine peaked at 140-150. That was on v6 and last beta had a libvirt debug flag set or something that cause the libvirt log to get too big. If not just raise it more.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now