[SOLVED] syslog tmpfs full, syslog stops logging.

[madburg]

@weebotech

cat /etc/fstab

/dev/disk/by-label/UNRAID  /boot    vfat  auto,rw,exec,noatime,nodiratime,umask=0,shortname=mixed  0  0

 

exactly as in the fstab source file

 

I ran mount -t tmpfs tmpfs /var/log -o size=256m,remount

I didn't bother with the 'touch' as the syslog file (0 kb) was there ready and waiting (see screenshot before I ran a remount)

just ran /etc/rc.d/rc.syslog restart

 

Syslog back and running. Thank you both.

 

 

[WeeboTech]

@weebotech

cat /etc/fstab

/dev/disk/by-label/UNRAID  /boot    vfat  auto,rw,exec,noatime,nodiratime,umask=0,shortname=mixed  0  0

 

exactly as in the fstab source file

 

I ran mount -t tmpfs tmpfs /var/log -o size=256m,remount

I didn't bother with the 'touch' as the syslog file (0 kb) was there ready and waiting (see screenshot before I ran a remount)

just ran /etc/rc.d/rc.syslog restart

 

Syslog back and running. Thank you both.

 

Thanks. I was wondering for a moment if you had a different version then I.

I just looked at rc.S and saw how tmpfs is being mounted.

 

# tmm - Move /var/log to a tmpfs
mv /var/log/* /var/empty
mount -t tmpfs -o size=128m tmpfs /var/log
mv /var/empty/* /var/log

 

At the very least we know it can be adjusted manually or through some plugin.

[mrow]

I understand what your saying but at the same time I see this

"Tom: can you try this for me, create an extra.cfg, enter the following into it..., stop and start the array, do xyz and report back your finding"

"User: Ah Tom, I did exactly what you said, but my syslog after a while just stopped? I don't know what happened, what do I do?"

"Tom: Arghhh bleep bleep bleep, you probably ran out of tmpfs space"

"User: What? what's tmpfs space? what do I do?"

 

Not far fetched, I have run tests in the past for Tom (with no problem), and again just recently and hit the problem due to this change. Luck or not, I happen to have followed and remembered about a change to syslog via tmpfs I didn't really pull my hair out and was pretty sure I knew the culprit but NOT how to fix it. Tom was not able to help me fix it right there and then either, so I attempted to delete the old syslog (.1 and .2) on my own that didn't help, so an unwanted reboot in RC15a I had to do, and once again it happened. Only now 6 days later with no syslog all this time, posting here and the guys jumping in did I find a solution for now.

 

I don't believe leaving it as is helps anyone, when sooner or later additional logging will be required to be turned on to figure something out and this will need to be remembered...

 

Then maybe a good option would be a debug logging upon array start check box so the novice users out there would be able to handle it. Have it write to, say, their cache drive or the root of disk 1 of their array if no cache drive is present. Either that or the debug check box could enable an increased tmpfs size limit just for that array start/stop cycle and it would go back to 128MB upon next stop/start unless the option is checked again. Then all Tom would have to do is modify his instructions to the user to:

 

"Tom: can you try this for me, create an extra.cfg, enter the following into it..., check debug logging and stop and start the array, do xyz and report back your finding"

[madburg]

I just looked at rc.S and saw how tmpfs is being mounted.

I did the same

 

At the very least we know it can be adjusted manually or through some plugin.

Yes this worked very well, easily, painlessly for me thankfully (I don't know if a carpenter over 40 is comfortable with commandline/putty/telnet/Winscp, etc...). Again I am trying with 256mb for now (I know I could easily do 1GB) just to see and can adjust if it doesn't work out but at least its a gauge for Tom, and i am not saying 256mb would cover shfs & afp and/or NFS logging or any others I don't even know about.

[WeeboTech]

I just looked at rc.S and saw how tmpfs is being mounted.

I did the same

 

At the very least we know it can be adjusted manually or through some plugin.

Yes this worked very well, easily, painlessly for me thankfully (I don't know if a carpenter over 40 is comfortable with commandline/putty/telnet/Winscp, etc...). Again I am trying with 256mb for now (I know I could easily do 1GB) just to see and can adjust if it doesn't work out but at least its a gauge for Tom, and i am not saying 256mb would cover shfs & afp and/or NFS logging or any others I don't even know about.

 

Here's the thing, you don't really need to limit it that way.

You could go with 1GB or more.

If it's not being used, the kernel, buffer cache and root fs can still use it. (it's all shared space)

The amount is just an upper limit.

Before you had NO upper limit.

I would suggest you pick a higher value and check it daily to see how high it goes.

If you exceed 256m you're only going to expand it more and possibly be frustrated you ran out of space.

 

you might as well do the 1GB (you loose nothing by doing this now) and prune it down later when you find you are only using XXXm. i.e. You can resize down too.

 

It's like having a 20 gallon gas tank and only putting in 5 gallons to see how far you can go before you run out of gas.

[madburg]

I understand what your saying but at the same time I see this

"Tom: can you try this for me, create an extra.cfg, enter the following into it..., stop and start the array, do xyz and report back your finding"

"User: Ah Tom, I did exactly what you said, but my syslog after a while just stopped? I don't know what happened, what do I do?"

"Tom: Arghhh bleep bleep bleep, you probably ran out of tmpfs space"

"User: What? what's tmpfs space? what do I do?"

 

Not far fetched, I have run tests in the past for Tom (with no problem), and again just recently and hit the problem due to this change. Luck or not, I happen to have followed and remembered about a change to syslog via tmpfs I didn't really pull my hair out and was pretty sure I knew the culprit but NOT how to fix it. Tom was not able to help me fix it right there and then either, so I attempted to delete the old syslog (.1 and .2) on my own that didn't help, so an unwanted reboot in RC15a I had to do, and once again it happened. Only now 6 days later with no syslog all this time, posting here and the guys jumping in did I find a solution for now.

 

I don't believe leaving it as is helps anyone, when sooner or later additional logging will be required to be turned on to figure something out and this will need to be remembered...

 

Then maybe a good option would be a debug logging upon array start check box so the novice users out there would be able to handle it. Have it write to, say, their cache drive or the root of disk 1 of their array if no cache drive is present. Either that or the debug check box could enable an increased tmpfs size limit just for that array start/stop cycle and it would go back to 128MB upon next stop/start unless the option is checked again. Then all Tom would have to do is modify his instructions to the user to:

 

"Tom: can you try this for me, create an extra.cfg, enter the following into it..., check debug logging and stop and start the array, do xyz and report back your finding"

A check box is another option, true to remount it larger (but, I will get to this shortly). I can say I am not to crazy about re-directing it else where. As Tom pointed out and if I was in his shoes I would not move it to the flash, a disk is not ideal because you maybe logging to figure out something based on a disk, no guarantee there is is a disk#x (or cache for that matter), I decide to start with disk#24 and work my way down ,so I dont have disk#1-15 (see my point).

 

Back to the first point with the check box, from history Tom doesn't like (for a lack of better words) to make changes to emhttp all that much...

I going to guess he's not too in love with emhttp himself and from past experiences making changes in it resulted in headaches. He's ironed out a lot with it so at least things like new permission, etc.. come back and don't hang the gui or not display results (as one example). So I don't blame him if he feels like not making an emhttp change, and would wait on a replacement GUI in 5.x/6.x instead.

 

What he does in scripts, work, this worked, its just the 128mb thing... he got some details/idea's here, he'll make a decision what to do, if anything. I just ask its noted, if nothing will change then state it in the thread and close it. Someone can wiki it or something. If a change is made, it would at the very least make it into the release notes.

 

[WeeboTech]

Here's another way to resize it

This would be the normal size if there were no size= specification in rc.S

 

mount -t tmpfs -o remount,size=50% tmpfs /var/log

 

This will keep it at a maximum of 50% of available ram.

[madburg]

I just looked at rc.S and saw how tmpfs is being mounted.

I did the same

 

At the very least we know it can be adjusted manually or through some plugin.

Yes this worked very well, easily, painlessly for me thankfully (I don't know if a carpenter over 40 is comfortable with commandline/putty/telnet/Winscp, etc...). Again I am trying with 256mb for now (I know I could easily do 1GB) just to see and can adjust if it doesn't work out but at least its a gauge for Tom, and i am not saying 256mb would cover shfs & afp and/or NFS logging or any others I don't even know about.

 

Here's the thing, you don't really need to limit it that way.

You could go with 1GB or more.

If it's not being used, the kernel, buffer cache and root fs can still use it. (it's all shared space)

The amount is just an upper limit.

Before you had NO upper limit.

I would suggest you pick a higher value and check it daily to see how high it goes.

If you exceed 256m you're only going to expand it more and possibly be frustrated you ran out of space.

 

you might as well do the 1GB (you loose nothing by doing this now) and prune it down later when you find you are only using XXXm. i.e. You can resize down too.

 

It's like having a 20 gallon gas tank and only putting in 5 gallons to see how far you can go before you run out of gas.

You're right I didn't think about it that way, to monitor what size (in total) gets stored and having a higher ceiling (and forgot u mentioned u tested lowering it via  a remount as well). I won't be frustrated though I feel comfortable with how to manage this up and down now, again thank you both for this.

 

[madburg]

Here's another way to resize it

This would be the normal size if there were no size= specification in rc.S

 

mount -t tmpfs -o remount,size=50% tmpfs /var/log

 

This will keep it at a maximum of 50% of available ram.

Sweet, good to know as well.

 

BAM done:

 

Filesystem    Type    Size  Used Avail Use% Mounted on

proc          proc      0    0    0  -  /proc

sysfs        sysfs      0    0    0  -  /sys

tmpfs        tmpfs    3.0G  89M  2.9G  3% /var/log

/dev/sdb      vfat    3.8G  595M  3.2G  16% /boot

/dev/sda1    vfat    4.0G  873M  3.2G  22% /mnt/vmdk

/dev/md1  reiserfs    2.8T  2.6T  158G  95% /mnt/disk1

/dev/md2  reiserfs    2.8T  2.6T  156G  95% /mnt/disk2

/dev/md3  reiserfs    2.8T  2.6T  151G  95% /mnt/disk3

/dev/md4  reiserfs    2.8T  2.6T  150G  95% /mnt/disk4

/dev/md5  reiserfs    2.8T  2.6T  151G  95% /mnt/disk5

/dev/md6  reiserfs    2.8T  2.6T  158G  95% /mnt/disk6

/dev/md7  reiserfs    2.8T  949G  1.9T  34% /mnt/disk7

/dev/md11 reiserfs    233G  213G  21G  92% /mnt/disk11

/dev/md12 reiserfs    1.9T  1.8T  97G  95% /mnt/disk12

/dev/md13 reiserfs    1.9T  1.8T  97G  95% /mnt/disk13

/dev/md14 reiserfs    1.9T  1.8T  110G  95% /mnt/disk14

/dev/md15 reiserfs    1.9T  1.8T  104G  95% /mnt/disk15

/dev/md16 reiserfs    1.9T  1.8T  96G  95% /mnt/disk16

/dev/md17 reiserfs    1.9T  1.7T  176G  91% /mnt/disk17

/dev/md18 reiserfs    1.9T  1.7T  197G  90% /mnt/disk18

/dev/sdf1 reiserfs    1.9T  1.2T  640G  66% /mnt/cache

shfs    fuse.shfs    33T  29T  3.6T  89% /mnt/user0

shfs    fuse.shfs    33T  29T  3.6T  89% /mnt/user

 

[WeeboTech]

I found this free ware program which can accept remote syslog messages.

It's kinda crude but it works. If anyone finds a nicer one let us know.

 

http://sourceforge.net/projects/syslog-server/?source=dlp

 

I added this line to my /etc/syslog.conf

A few tabs to align things between fields.

Note at least 1 tab is necessary. Alignment does not really matter but at least 1 tab needs to separate the two.

use your respective ip address with the syslog server after the @ sign.

This says, forward all these messages to the host ip after the @ sign.

 

*.debug  @192.168.1.10

 

did the

kill -1 $(</var/run/syslogd.pid)

 

and away we go

 

logger "testing"

sent a testing message and I received it on the windows client.

[madburg]

Oh man, a while back I was searching for something that would do this for devices other than unRAID and everything I found was too old and PITA.

This looks refreshing, I will give it a whirl in my spare time, thanks for sharing.

 

[dgaschk]

OS X has syslogd running and just needs to be configured to accept remote logging: http://stackoverflow.com/questions/5510563/how-to-start-syslogd-server-on-mac-to-accept-remote-logging-messages

 

There are a lot of Windows options...

[aim60]

Another syslog server program worth trying

 

  http://www.kiwisyslog.com/free-edition.aspx