June 26, 201313 yr Sorry in advanced if this is a dumb question, but how much space was available to syslog's prior to this change to move to "'tmpfs' to consume no more than 128MB"? Was it all the free space available on the USB stick or something else...? I ask because while troubleshoot and getting logs for Tom, I enabled shfs AND afp logging and it filled up that 128MB overnight on that tmpfs. Syslog stopped logging. Deleting a prior syslog to free up space did not kick start the syslog back up so I need to restart the unRAID server. Now that that particular test was completed, I turn off the shfs logging but still needed the AFP logging enabled, after 2.5-3 days, same thing tmpfs is full syslog is no longer logging. This is very annoying to say the least. I am not sure what the advantage to moving syslog to tmpfs was for, because these same tests never caused this prior to this change of syslog to tmpfs. So it would be appreciated if either the tmpfs setting of 128mb is increased or this change is completed backed out of.
June 26, 201313 yr Sorry in advanced if this is a dumb question, but how much space was available to syslog's prior to this change to move to "'tmpfs' to consume no more than 128MB"? Was it all the free space available on the USB stick or something else...?All available RAM. Which meant an out of control syslog (or just very long uptimes) would crash the server as the RAM was depleted.
June 26, 201313 yr here is how you can restart syslog. touch /var/log/syslog /etc/rc.d/rc.syslog restart Starting sysklogd daemons: /usr/sbin/syslogd -m0 /usr/sbin/klogd -c 3 -x This command should rotate a large log out. /etc/cron.daily/logrotate unfortunately it does not compress the old logs. *sigh* I bet there's a way to remount tmpfs with a new size. do a cat /etc/fstab so i can see what is in the fstab.
June 26, 201313 yr unfortunately it does not compress the old logs. *sigh* You can just uncomment the line in /etc/logrotate.conf before running to have them compressed.
June 26, 201313 yr FWIW, if you want to log to your boot flash, you can add an entry similiar to this line. *.debug -/var/log/syslog add a new line *.debug(press tab a few times)-/boot/syslog save the file do a kill -1 $(</var/run/syslogd.pid) and you should see lines being appended to your boot flash. to test do logger "syslog test `date`"
June 26, 201313 yr Author Thanks @jonathanm, I really never knew as I never ran out before this change. Thanks for the explanation. So I think its obvious this is to small (128mb) and allowing someone (not sure if any one has but still viable) run out of ram is not good either. So how would 500MB sounds? I can safely say that this change hurts more than it possible could help someone in its current setting. I need to continue testing with AFP logging enabled, and had AFP logging on for weeks/months and never had an issue until this change. I cannot carry extra logging to syslog more than a 2.5-3 day period (with shfs and afp only 1 day). @WeeboTech, based on the fact Tom (via email comummication we had going on) did not instruct me to run that command " /etc/rc.d/rc.syslog restart " I don't believe it will work. As for "/etc/cron.daily/logrotate " that did run on its own (as scheduled by unraid) and did rotate the last syslog out freeing up the space, but the syslog never started backup again. So I will try that restart command tonight, it wont hurt and post back the result. I will also provide the 'cat /etc/fstab' as well. Thanks.
June 26, 201313 yr I would not have used the 128MB limit. I would have left the limit to the system default which is 50% of available memory. I think limiting the /var/log directory to 50% of available memory is a good thing. I would rather syslog stop, then crash the system if over run. as far as syslog being restarted. the restart will work, however it depends on the output file existing first. so if /var/log/syslog was moved or removed and you restarted syslog, it would not write anything out. touch /var/log/syslog then issue the restart command. Another option is to forward all messages to another unix host or windows host that is running a syslog server. I used to forward the syslogs to all of my machines to an unRAID server that had the gigabyte 4GB PCI ramdisk. Worked great! if you have an SSD you can write it there too.
June 26, 201313 yr Author FWIW, if you want to log to your boot flash, you can add an entry similiar to this line. *.debug -/var/log/syslog add a new line *.debug(press tab a few times)-/boot/syslog save the file do a kill -1 $(</var/run/syslogd.pid) and you should see lines being appended to your boot flash. to test do logger "syslog test `date`" Thanks, but I don't want to start changing a bunch of sh^t, It should work like it has always. I am not feeling very warm and fussy right now with having to reboot RC15 (for the second time due to this issue) with a chance of losing some data. So preferable this can get fixed with the sync bug (or whatever the root cause of that other bug is). If you guys agree this needs a change (one way or the other) then please add it to the list. Thanks. The change (as I understand it now) was to put in a safety, but at it current setting is hampering what once worked. So I hold to my statement either the value needs to be increased or this change rolled back. Tom know's which would be best I am sure.
June 26, 201313 yr if you manually issue the sync command you should not loose data if you stop and start the unRAID array, you should not loose data. if you reboot/powerdown unRAID gracefully you should not loose data It's a sudden loss of power before a sync operation that puts the data in the buffer cache at risk.
June 26, 201313 yr I did googled a bit and tested on my tests VM and you can just remount tmpfs to a new size without losing files on it. I did just used: mount -t tmpfs tmpfs /var/log -o size=1G,remount Anyone running more extensive/debug logging may just increase it, no?
June 26, 201313 yr I did googled a bit and tested on my tests VM and you can just remount tmpfs to a new size without losing files on it. I did just used: mount -t tmpfs tmpfs /var/log -o size=1G,remount Anyone running more extensive/debug logging may just increase it, no? Correct, I wanted to see the /etc/fstab first, so I could test and formulate the proper command.
June 26, 201313 yr Author if you manually issue the sync command you should not loose data if you stop and start the unRAID array, you should not loose data. if you reboot/powerdown unRAID gracefully you should not loose data It's a sudden loss of power before a sync operation that puts the data in the buffer cache at risk. Good to know, as there has been way to much psoted over there and every time one feels they understand what's the bug, back to lost! again lol. Sorry I didn't notice you had already moved this to the Issue List, my bad. Thanks for that. I just want to have a syslog and remain up and running on RC15a until the next release that fixes that bug. So no testing for me with reboots involved. Hopefully you are correct and I can get my syslog turning once again without having to reboot. I will try all your suggestions above and let you know.
June 26, 201313 yr Author Sweet! thank you nars. So what I am imagining (to do once I get home) is increase the tmpfs first, and use that touch weebotech stated and a syslog restart, should get the syslog back up and running then. With the increase I wont have to do the touch and restart every few days. Thank you both.
June 26, 201313 yr FWIW, if you want to log to your boot flash, you can add an entry similiar to this line. Note that I do not recommend doing this except perhaps for a single-time special case. This is because a lot of people use cheap flash devices which will quickly crap out with syslog writing to them all the time. It's much better to just leave a window open on your PC with syslog logging to it. Another alternative is use above method but have it go to the cache drive. You could also have it send log messages to another host.
June 26, 201313 yr I would not have used the 128MB limit. I would have left the limit to the system default which is 50% of available memory. I think limiting the /var/log directory to 50% of available memory is a good thing. I would rather syslog stop, then crash the system if over run. I thought about doing it that way, but then someone with a lot of plugins and small memory would complain of still getting OOM conditions. It's easy to change it via re-mount though.
June 26, 201313 yr Author FWIW, if you want to log to your boot flash, you can add an entry similiar to this line. Note that I do not recommend doing this except perhaps for a single-time special case. This is because a lot of people use cheap flash devices which will quickly crap out with syslog writing to them all the time. It's much better to just leave a window open on your PC with syslog logging to it. Another alternative is use above method but have it go to the cache drive. You could also have it send log messages to another host. Agreed, I had no intention of doing that one or any other redirections for that matter.
June 26, 201313 yr Author I would not have used the 128MB limit. I would have left the limit to the system default which is 50% of available memory. I think limiting the /var/log directory to 50% of available memory is a good thing. I would rather syslog stop, then crash the system if over run. I thought about doing it that way, but then someone with a lot of plugins and small memory would complain of still getting OOM conditions. It's easy to change it via re-mount though. Didn't you put in the OOM command line fix JoeL. mentioned in already (forgot if you did or didn't, believe you did) that makes sure emhttp & SMB would not be the likely candidates to be killed by OOM in that scenario? What if I change via remount to only 256MB and see if it holds and report back, if its does at least its a gauge and a number you could possible work with for a change to the 128MB.
June 26, 201313 yr FWIW, if you want to log to your boot flash, you can add an entry similiar to this line. Note that I do not recommend doing this except perhaps for a single-time special case. This is because a lot of people use cheap flash devices which will quickly crap out with syslog writing to them all the time. It's much better to just leave a window open on your PC with syslog logging to it. Another alternative is use above method but have it go to the cache drive. You could also have it send log messages to another host. I agree wholeheartedly. Some people want to do that way. The option is there to capture a critical crash situation. This is why I either buy very large quality flash keys or a card reader with large quality SD cards. I used to have a script that would add /dev/tty12 and write all syslogs there. Then with chvt switch to /dev/tty12 so that it was always being printed to a visible screen. Usually even a kernel crash (other then a hard lock) would allow you to switch to tty12 and see it on the screen.
June 26, 201313 yr I would not have used the 128MB limit. I would have left the limit to the system default which is 50% of available memory. I think limiting the /var/log directory to 50% of available memory is a good thing. I would rather syslog stop, then crash the system if over run. I thought about doing it that way, but then someone with a lot of plugins and small memory would complain of still getting OOM conditions. It's easy to change it via re-mount though. The beauty is that if they have swap, it will be used to swap out parts of tmpfs that are not being used. I've tested this with success. I filled up tmpfs with a dd and watched it swap out.
June 26, 201313 yr Sweet! thank you nars. So what I am imagining (to do once I get home) is increase the tmpfs first, and use that touch weebotech stated and a syslog restart, should get the syslog back up and running then. With the increase I wont have to do the touch and restart every few days. Thank you both. How much memory is in your system? Do you have swap space configured on your system?
June 26, 201313 yr Author Sweet! thank you nars. So what I am imagining (to do once I get home) is increase the tmpfs first, and use that touch weebotech stated and a syslog restart, should get the syslog back up and running then. With the increase I wont have to do the touch and restart every few days. Thank you both. How much memory is in your system? Do you have swap space configured on your system? 1) 6GB 2) If unRAID doesn't configure it, then the answer is no I don't have swap space configure; and then I would have to ask should it be?
June 26, 201313 yr If unRAID doesn't configure it, then the answer is no I don't have swap space configure; and then I would have to ask should it be? Only if you are loading allot of plugins and you have a busy memory hungry system. I would not worry about it right now. with 6GB of ram you could easily resize the tmpfs to 1GB and not have any cause for alarm. tmpfs will only grow to the space it actually uses, then be limited by the upper number you provide. So if you were to resize to 1GB, then check every day, you would see the maximum needed without issue to the kernel or other apps.
June 26, 201313 yr The remount,size= works nicely. You can size up or down. You cannot resize below the value that is currently being utilized. I tried. I put a 100MB file on a test tmpfs and received root@unRAID2:/var/log.tmpfs# ls -l --si total 105M -rw-r--r-- 1 root root 128k 2013-06-26 19:03 syslog -rw-rw-rw- 1 root root 105M 2013-06-26 19:03 testlog tmpfs 269M 105M 164M 40% /var/log.tmpfs root@unRAID2:/var# mount -t tmpfs -o remount,size=64M tmpfs /var/log.tmpfs mount: /var/log.tmpfs not mounted already, or bad option root@unRAID2:/var# mount -t tmpfs -o remount,size=164M tmpfs /var/log.tmpfs tmpfs 172M 105M 67M 62% /var/log.tmpfs root@unRAID2:/var# mount -t tmpfs -o remount,size=128M tmpfs /var/log.tmpfs tmpfs 135M 105M 30M 79% /var/log.tmpfs
June 27, 201313 yr What if I change via remount to only 256MB and see if it holds and report back, if its does at least its a gauge and a number you could possible work with for a change to the 128MB. I don't think changing the tmpfs size limit for everyone is the right answer, even if it is only to 256MB. 99.9% of people will not have a log larger than 128MB, except for instances of out of control logs. For the people doing debug logging changing the limit manually seems like a better choice.
June 27, 201313 yr Author What if I change via remount to only 256MB and see if it holds and report back, if its does at least its a gauge and a number you could possible work with for a change to the 128MB. I don't think changing the tmpfs size limit for everyone is the right answer, even if it is only to 256MB. 99.9% of people will not have a log larger than 128MB, except for instances of out of control logs. For the people doing debug logging changing the limit manually seems like a better choice. I understand what your saying but at the same time I see this "Tom: can you try this for me, create an extra.cfg, enter the following into it..., stop and start the array, do xyz and report back your finding" "User: Ah Tom, I did exactly what you said, but my syslog after a while just stopped? I don't know what happened, what do I do?" "Tom: Arghhh bleep bleep bleep, you probably ran out of tmpfs space" "User: What? what's tmpfs space? what do I do?" Not far fetched, I have run tests in the past for Tom (with no problem), and again just recently and hit the problem due to this change. Luck or not, I happen to have followed and remembered about a change to syslog via tmpfs I didn't really pull my hair out and was pretty sure I knew the culprit but NOT how to fix it. Tom was not able to help me fix it right there and then either, so I attempted to delete the old syslog (.1 and .2) on my own that didn't help, so an unwanted reboot in RC15a I had to do, and once again it happened. Only now 6 days later with no syslog all this time, posting here and the guys jumping in did I find a solution for now. I don't believe leaving it as is helps anyone, when sooner or later additional logging will be required to be turned on to figure something out and this will need to be remembered...
Archived
This topic is now archived and is closed to further replies.