February 23, 200917 yr On a low memory requirement distribution with no pagefile it is vital that the logs are managed properly. unRAID does not have any means to manage log explosion and will blindly allow any size of log until the system either crashes or becomes unusable. Many things can cause a log explosion, such as the bug/feature " duplicate entrys of shfs: duplicate object: in /var/log/syslog", however the actual mechanism that generates the logs is irrelevant. Please implement a means to rotate logs should they grow to large. An added "nice to have" would be to allow up to xx% of RAM to be allocated to logs to cater for different unRAID memory installation however this is not a requirement.
February 23, 200917 yr If memory serves, this bug was seen with pre-4.4 versions if the parity check found so many sync errors that the log file got huge and consumed all of available RAM. (Since the ramdisk gets bigger and bigger as more and more space is needed, eventually the ramdisk starves the OS for memory and the system crashes.) The 4.4.2 version seems to refrain from logging sync errors. This eliminates the possibility of this "log explosion" (catchy name) from occurring in this situation (which is the only situation it ever happened I think). To me, the cure is worse than the cause, though. The logged sync errors give some insight into whether the sync errors all occurred at the beginning of the disk or elsewhere. They might also give enough information for a future tool to traverse the RFS structures and say what files are on the affected sector on each of the data disks. If you've only got a few, this is great information. If you've got enough sync errors to cause syslog explosion, you've got big problems. Perhaps unRAID could keep track of the number of sync errors and stop logging each one after a few thousand.
February 23, 200917 yr Author I disagree on both counts. Today my syslog is 1.2GB in 4.4.2 proving it is not fixed. However as i said the specifics of what causes the explosion is completely irrelevant. It is naive to assume this cannot happen again with another type of log explosion. The current mechanism is fundamentally flawed as it assumes the log cannot grow past ram which is not a correct assumption without a mechanism to control it.
February 23, 200917 yr I disagree on both counts. Today my syslog is 1.2GB in 4.4.2 proving it is not fixed. However as i said the specifics of what causes the explosion is completely irrelevant. It is naive to assume this cannot happen again with another type of log explosion. The current mechanism is fundamentally flawed as it assumes the log cannot grow past ram which is not a correct assumption without a mechanism to control it. It is a byproduct of running unRAID from the ramdisk. The impact of having the server crash after so many errors is near zero. In fact you could argue that if you had millions and millions of disk errors, having the server go down is preferable to having it continue to have error after error. Those errors could be causing physical damage or overheating. If you need help figuring out what went wrong, post the first couple hundred K of your syslog to pastebin. You likely have a bad cable or failing/failed disk.
February 24, 200917 yr Author Squashing bugs that cause log explosion is obviously important but completely irrelevant to this post. Specifically I do not have any issues i need help with. This post is about OS architecture and it incorrectly assuming that a large log space is available and blindly allowing gigantic logs. As for... In fact you could argue that if you had millions and millions of disk errors, having the server go down is preferable to having it continue to have error after error. Those errors could be causing physical damage or overheating. It is NEVER preferable to have an OS crash under any circumstances.
February 24, 200917 yr Let me second this request!!! One of my systems crashed twice in the last month. On my cache drive of Tower1 I have a mount point to the second machine (Tower2). This mount point is called /mnt/cache/.Tower2. The dot should refuse unRAID to populate this folder - BUT IT DOES! My syslogs grow so fast you can't imagine. This is a bug and no side effect!!! Thanks for listening. Harald
February 24, 200917 yr Let me second this request!!! One of my systems crashed twice in the last month. On my cache drive of Tower1 I have a mount point to the second machine (Tower2). This mount point is called /mnt/cache/.Tower2. The dot should refuse unRAID to populate this folder - BUT IT DOES! My syslogs grow so fast you can't imagine. This is a bug and no side effect!!! Thanks for listening. Harald Why don't you post a syslog (at least the first 200K of it) using pastebin.com. It sounds like a problem in the mover script. I have seem some posted mover script corrections (which I don't think are in the official 4.4.2 release) that you might want to research. This post is about OS architecture ... Good luck with this.
February 24, 200917 yr Let me second this request!!! One of my systems crashed twice in the last month. On my cache drive of Tower1 I have a mount point to the second machine (Tower2). This mount point is called /mnt/cache/.Tower2. The dot should refuse unRAID to populate this folder - BUT IT DOES! My syslogs grow so fast you can't imagine. This is a bug and no side effect!!! Thanks for listening. Harald Tom posted the fix for the "mover" script in this post He said it would be in the next release, but he has not made any releases since. It does not solve the exploding syslog issue, as it can fill from any source, but it does take care of the issue you are seeing with it trying to move files the cache drive where it should not. Joe L.
February 24, 200917 yr Author Why don't you post a syslog (at least the first 200K of it) using pastebin.com. It sounds like a problem in the mover script. I have seem some posted mover script corrections (which I don't think are in the official 4.4.2 release) that you might want to research. and again.... Squashing bugs that cause log explosion is obviously important but completely irrelevant to this post. Specifically I do not have any issues i need help with. I want to stay focused on what this thread is about and not a random bug that caused the syslog explotion. The log entries themselves are completely irrelevant to this post. So to get back on topic my original post. On a low memory requirement distribution with no pagefile it is vital that the logs are managed properly. unRAID does not have any means to manage log explosion and will blindly allow any size of log until the system either crashes or becomes unusable. Many things can cause a log explosion, such as the bug/feature " duplicate entrys of shfs: duplicate object: in /var/log/syslog", however the actual mechanism that generates the logs is irrelevant. Please implement a means to rotate logs should they grow to large. An added "nice to have" would be to allow up to xx% of RAM to be allocated to logs to cater for different unRAID memory installation however this is not a requirement. Limetech can you comment on viable options?
February 24, 200917 yr A viable option is to create a tmpfs file system, limit it to a specific size (50% of ram?), and then use mount --bind to overlay /var/log, or as an alternative, modify the syslogd to write its log file in the tmpfs file system and create a symbolic link back to the /var/log/syslog file. I tried the following commands and they seemed to do the trick on my system: /etc/rc.d/rc.syslog stop mkdir /tmp/log mount tmpfs /tmp/log -t tmpfs -o size=100m cp /var/log/syslog /tmp/log/ ln -s /tmp/log/syslog /var/log/syslog /etc/rc.d/rc.syslog start The commands stop the syslog daemon process, create a mount point /tmp/log, create a tmpfs file system on it limited to 100 Meg, then copy the existing syslog to the tmpfs file system, rempve the original syslog file and replace it with a symbolic link to the size limited one. Lastly, I re-start the syslog daemon process. Tom could do something similar. It would not stop a process from filling the syslog, but it would stop the syslog from using all available memory since it exists in a file-system that can only grow to a specified size. Joe L. Exit: fixed typo.
February 25, 200917 yr Author A viable option is to create a tmpfs file system, ... I tried the following commands and they seemed to do the trick on my system: /etc/rc.d/rc.syslog stop mkdir /tmp/log mount tmpfs /tmp/log -t tmpfs -o size=100m cp /var/log/syslog /tmp/log/ ln -s /tmp/log/syslog /var/log/syslog /etc/rc.d/rc.syslog start ... this looks like an excellent solution. It also could easily be dimensioned based on actual installed RAM and not a generic figure (allowing for diverse hardware). Joe L. as always excellent support. Tom/Limetech please could you implement this or similar to fix this bug or at least comment on its viability.
February 25, 200917 yr I tried the following commands and they seemed to do the trick on my system: /etc/rc.d/rc.syslog stop mkdir /tmp/log mount tmpfs /tmp/log -t tmpfs -o size=100m cp /var/log/syslog /tmp/log/ ln -s /tmp/log/syslog /var/log/syslog /etc/rc.d/rc.syslog start Curiosity question.... What will happen when the syslog fills the new tmpfs file system? I assume you would like to move the file out of the tempfs (delete? or maybe just get rid of the first 95%?) so that the syslog can continue to be outputted to? Cheers, Matt
February 25, 200917 yr I tried the following commands and they seemed to do the trick on my system: /etc/rc.d/rc.syslog stop mkdir /tmp/log mount tmpfs /tmp/log -t tmpfs -o size=100m cp /var/log/syslog /tmp/log/ ln -s /tmp/log/syslog /var/log/syslog /etc/rc.d/rc.syslog start Curiosity question.... What will happen when the syslog fills the new tmpfs file system? I assume you would like to move the file out of the tempfs (delete? or maybe just get rid of the first 95%?) so that the syslog can continue to be outputted to? Cheers, Matt A process named "logrotate" is normally configured to keep a set of logs so no single file gets too big. On my 4.4.2 version of unRAID it seems to be configured to create a new log when the existing log grows to more then 1 Meg in size. The logrotate process is invoked once daily by cron. Try this and see what the debug output says on your system. On mine it simply says the log file does not need rotation. logrotate -d /etc/logrotate.conf Joe L.
February 25, 200917 yr I modified my logrotate.conf file to point to /tmp/log/syslog and to rotate when the size was over 100k I re-ran logrotate and it shows: rotating pattern: /tmp/log/syslog 1048576 bytes (2 rotations) empty log files are rotated, only log files >= 102400 bytes are rotated, old logs are removed considering log /tmp/log/syslog log does not need rotating not running postrotate script, since no logs were rotated My /tmp/log/syslog file is bigger than 100k, it should have been rotated. root@Tower# ls -l /tmp/log total 772 -rw-r--r-- 1 root root 778980 Feb 25 01:45 syslog Tom, is there a logrotate bug you need to investigate? It might help some to prevent a system crash if it was working, as it is, everybody will eventually run out of syslog space unless they reboot. (Might take years, but the log does not seem to rotate on release 4.4.2) Joe L.
February 25, 200917 yr Author logrotate -d /etc/logrotate.conf reading config file /etc/logrotate.conf reading config info for /var/log/wtmp error: /etc/logrotate.conf:25 unknown unit 'K' I thought that was fixed. md5sum /etc/logrotate.conf 782a30177575845b3072a79ac895208e /etc/logrotate.conf does that match anyone elses?
February 25, 200917 yr logrotate -d /etc/logrotate.conf reading config file /etc/logrotate.conf reading config info for /var/log/wtmp error: /etc/logrotate.conf:25 unknown unit 'K' I thought that was fixed. md5sum /etc/logrotate.conf 782a30177575845b3072a79ac895208e /etc/logrotate.conf does that match anyone elses? Not until the next beta... which has not been released yet to my knowledge.. you still need to use the fix.sh script found here: http://lime-technology.com/forum/index.php?topic=2549.0 Cheers, Matt
February 25, 200917 yr I modified my logrotate.conf file to point to /tmp/log/syslog and to rotate when the size was over 100k I re-ran logrotate and it shows: rotating pattern: /tmp/log/syslog 1048576 bytes (2 rotations) empty log files are rotated, only log files >= 102400 bytes are rotated, old logs are removed considering log /tmp/log/syslog log does not need rotating not running postrotate script, since no logs were rotated My /tmp/log/syslog file is bigger than 100k, it should have been rotated. root@Tower# ls -l /tmp/log total 772 -rw-r--r-- 1 root root 778980 Feb 25 01:45 syslog Tom, is there a logrotate bug you need to investigate? It might help some to prevent a system crash if it was working, as it is, everybody will eventually run out of syslog space unless they reboot. (Might take years, but the log does not seem to rotate on release 4.4.2) Joe L. I'm on 4.3.3 does not rotate here either.... reset from 10M to 100k in /etc/logrotate.conf ran logrotate -d /etc/logrotate.conf got the following: rotating pattern: /var/log/syslog 1048576 bytes (2 rotations) empty log files are rotated, only log files >= 102400 bytes are rotated, old logs are removed considering log /var/log/syslog log does not need rotating not running postrotate script, since no logs were rotated then tried: logrotate /etc/logrotate.conf my syslog is unchanged after both attempts... and is larger than 100k -rw-r--r-- 1 root root 124476 Feb 25 01:02 syslog (before, in between and after running logrotate) Cheers, Matt
February 25, 200917 yr I read the logrotate manual pagte... The -d option does not seem to actually rotate anything, it just tells you the rules that are present. When I invoked with the -f option it did rotate the log, but it ignored the size. (It forced the rotation) Without the -f option, it apparently is not doing any rotation. It is complaining that /var/log/wtmp and /var/log/btmp do not exist. It still looks like some kind of bug to me. At least one outstanding bug with logrotate exists where sizes less than 1 meg are ignored. I suspect this is the case with our version. See here Another bug report here Joe L.
March 3, 200917 yr Author Limetech replied to another thread so Im bumping this one to get his attention.
March 13, 200917 yr Author I believe this is fixed in 4.5-beta2 can anyone provide an independent confirmation. ta
March 13, 200917 yr I wouldn't mind seeing an option where you have the logs rotated to the flash drive (I am guessing here, but I bet most people use a 1GB+ USB drive, so should have plenty of space, especially if the logs are compressed when rotated). A possibly even better solution would be to email the rotated log. just my 2c.
March 13, 200917 yr Since my original post describing the use of a tmpfs as a method of limiting a log explosion's effect I've learned that the default "size" of a tmpfs file-system is 50% of ram. In other words, I do not need to supply an "-o size=nnn" option at all when creating the tmpfs file-system. Now, the tmpfs will only use what is really needed, so this will only use a meg or so for most people. The rest of the ram will still be available for normal buffering of disk access and running processes. In the event of a "log" explosion between this and logrotate odds are we will have the data we need for analysis and not crash the server. Even with a small amount of RAM it will still provide plenty of clues in the syslog. It might even make sense to put the entire /var/log in a tmpfs file system. That way all the logs are prevented from crashing the server. This would all need to be set up in the init process, before programs start their logging. /etc/rc.d/rc.syslog stop mkdir /tmp/log mount tmpfs /tmp/log -t tmpfs cp /var/log/syslog /tmp/log/ ln -s /tmp/log/syslog /var/log/syslog /etc/rc.d/rc.syslog start Joe L.
March 13, 200917 yr Since my original post describing the use of a tmpfs as a method of limiting a log explosion's effect I've learned that the default "size" of a tmpfs file-system is 50% of ram. In other words, I do not need to supply an "-o size=nnn" option at all when creating the tmpfs file-system. Now, the tmpfs will only use what is really needed, so this will only use a meg or so for most people. The rest of the ram will still be available for normal buffering of disk access and running processes. In the event of a "log" explosion between this and logrotate odds are we will have the data we need for analysis and not crash the server. Even with a small amount of RAM it will still provide plenty of clues in the syslog. It might even make sense to put the entire /var/log in a tmpfs file system. That way all the logs are prevented from crashing the server. This would all need to be set up in the init process, before programs start their logging. /etc/rc.d/rc.syslog stop mkdir /tmp/log mount tmpfs /tmp/log -t tmpfs cp /var/log/syslog /tmp/log/ ln -s /tmp/log/syslog /var/log/syslog /etc/rc.d/rc.syslog start Joe L. I have a script for this which I already wrote for use with the new boot script manager... when i get a chance to finish it.... I will edit it and share with others, once I get an opportunity to fix it (currently it accepts as an argument the size of the tmpfs... but I guess that needs to be changed now) Cheers, Matt
March 14, 200917 yr Author I believe this is fixed in 4.5-beta2 can anyone provide an independent confirmation. ta I think i spoke too soon. After the first rotation it seems like nothing gets added to syslog ever again. Normally when i move files around using mc syslog goes mental. Today not a single entry
March 18, 200917 yr Author Can someone else running the beta confirm this so we can lodge it as a bug.
Archived
This topic is now archived and is closed to further replies.