July 23, 20187 yr I need to restart my two servers every three weeks because the dashboard shows the log (out of 'flash log docker') is running full (100%). There seems to be no autorotate for logs or something like that. How can I omit the required reboots? And the last question is, how can I find out what's eating up the log space? Many thanks in advance. Edited July 24, 20187 yr by hawihoney
July 23, 20187 yr For a start, go to Tools >>> Diagnostics and then upload the resulting Diagnostics file with your next post.
July 23, 20187 yr Author Here's the diagnostics. I can't see what causes 73%. Yesterday it was at 51%. Any ideas? Thanks in advance. tower-diagnostics-20180723-1343.zip
July 23, 20187 yr You got hundreds of entries like this: Jul 8 04:30:02 Tower sshd[20587]: Accepted publickey for root from 192.168.178.34 port 58842 ssh2: RSA SHA256:NlDaBhofPoWmFOvqcBxxPRT3HmcRZFEqIdAma/CLoBQ Jul 8 04:32:43 Tower sshd[20587]: Received disconnect from 192.168.178.34 port 58842:11: disconnected by user Jul 8 04:32:43 Tower sshd[20587]: Disconnected from user root 192.168.178.34 port 58842 Jul 8 04:32:43 Tower sshd[1812]: Accepted publickey for root from 192.168.178.34 port 58844 ssh2: RSA SHA256:NlDaBhofPoWmFOvqcBxxPRT3HmcRZFEqIdAma/CLoBQ Jul 8 04:32:59 Tower sshd[1812]: Received disconnect from 192.168.178.34 port 58844:11: disconnected by user Jul 8 04:32:59 Tower sshd[1812]: Disconnected from user root 192.168.178.34 port 58844 Each time it seems like the port number increases. I am not expert but it looks like you are being probed for a way into your system...
July 23, 20187 yr Author Thanks for your answer. Ah, I use User Scripts plugin to rsync backups from one server to a second unRAID server in our house. Can I omit these messages or rotate the logs on my own. syslog is several 100K currently. That's not much but leads to over 70% fullness. Can I increase the syslog threshold? How? Thanks in advance.
July 23, 20187 yr 19 minutes ago, hawihoney said: Thanks for your answer. Ah, I use User Scripts plugin to rsync backups from one server to a second unRAID server in our house. Can I omit these messages or rotate the logs on my own. syslog is several 100K currently. That's not much but leads to over 70% fullness. Can I increase the syslog threshold? How? Thanks in advance. Why do you run rsync so often?
July 23, 20187 yr Author Oh, rsync runs every night - but I call it for every share I take a backup from. Pictures, Projects, Programs, Documents, ... last time I did count 12 shares. Roundabout one million files of a long lasting digital Life. Some shares will be backuped once per night, some once per week and some once per month. Is there a better way? TIA Edited July 23, 20187 yr by hawihoney
July 23, 20187 yr Author Here's an example of one of my three user scripts. This one runs every night. Didn't know that SSH (or whatever) fills the syslog with Disconnect messages. Both server are standing in my house and are members of the same network (192.168.178.0/24). If there is a better way I would liekt to learn. rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Pictures Sorted/" /mnt/user/Pictures Sorted/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Pictures Unsorted/" /mnt/user/Pictures Unsorted/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Pictures Favorites/" /mnt/user/Pictures Favorites/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Data/" /mnt/user/Data/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Documents/" /mnt/user/Documents/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Movies Private/" /mnt/user/Movies Private/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Notes/" /mnt/user/Notes/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Programs/" /mnt/user/Programs/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Projects/" /mnt/user/Projects/ ... Many thanks in advance.
July 24, 20187 yr Author Don't use rsync with SSH to backup files between two unRAID servers. Use the Unassigned Devices plugin, mount the backup server as SMB and go that route.
July 24, 20187 yr 21 minutes ago, hawihoney said: Don't use rsync with SSH to backup files between two unRAID servers. Use the Unassigned Devices plugin, mount the backup server as SMB and go that route. The advantage with rsync using SSH is that the main server does not have access to the bacckup server, giving additional protection to the backup. There aren't really that many SSH lines in the log. CA Backup/Restore produces more log lines when it stops/starts all the Dockers.
July 24, 20187 yr Author This leads to the questions: Why is there no log rotate? And why is the log nearly full with 800KB syslog on a 16GB flash with 15.5GB free space. Is something wrong with dashboard or my system?
July 24, 20187 yr 1 hour ago, hawihoney said: This leads to the questions: Why is there no log rotate? And why is the log nearly full with 800KB syslog on a 16GB flash with 15.5GB free space. The log does rotate, but the old logs are still stored. And logs as well as all other OS files are in RAM not on flash. Flash only contains an archive of the OS which is unpacked into RAMfs at boot. Flash also contains settings from the webUI. That is all flash contains unless you have intentionally used it for some other purpose such as scripts.
July 24, 20187 yr 1 hour ago, hawihoney said: why is the log nearly full with 800KB syslog You must have other things in /var/log besides that. Normally 128MB is reserved. Some users have recently reported atop taking a lot of space there but I don't remember why they even had that running. I don't think it is builtin.
July 24, 20187 yr There's more in /var/log than just the syslog, and your syslog is nowhere near large enough to account for the folder getting full What is the output of ls -ail /var/log
July 24, 20187 yr 48 minutes ago, trurl said: You must have other things in /var/log besides that. Normally 128MB is reserved. Some users have recently reported atop taking a lot of space there but I don't remember why they even had that running. I don't think it is builtin. A running atop requests the Linux kernel to accumulate statistics until atop is correctly exited - so it is only when atop is kept running indefinitely or when it is killed with signal 9 that Linux will continue to fill /var/log/ with accounting for all running processes.
July 24, 20187 yr Author Here's the output. log shows 82% currently. That's to much for under 1MB log currently. root@Tower:~# ls -ail /var/log total 980 1455 drwxr-xr-x 13 root root 560 Jul 7 07:55 ./ 1044 drwxr-xr-x 13 root root 300 Jul 25 2016 ../ 14251 drwxr-xr-x 2 root root 160 Jul 24 04:40 atop/ 1458 -rw------- 1 root root 384 Jul 17 08:44 btmp 1459 -rw-r--r-- 1 root root 0 Mar 10 00:53 cron 1460 -rw-r--r-- 1 root root 0 Mar 10 00:53 debug 28016 -rw-rw-rw- 1 root root 59786 Jul 24 10:24 diskinfo.log 3761 -rw-rw-rw- 1 root root 72513 Jul 7 07:55 dmesg 35210 -rw-rw-rw- 1 root root 19654 Jul 24 07:37 docker.log 1461 -rw-r--r-- 1 root root 1440 Jul 7 07:55 faillog 1462 -rw-r--r-- 1 root root 13140 Jul 24 18:01 lastlog 1463 drwxr-xr-x 5 root root 100 Feb 21 14:29 libvirt/ 1467 -rw-r--r-- 1 root root 0 Mar 10 00:53 maillog 1468 -rw-r--r-- 1 root root 0 Mar 10 00:53 messages 1469 drwxr-xr-x 2 root root 40 May 16 2001 nfsd/ 1470 drwxr-x--- 2 nobody root 60 Jul 7 07:55 nginx/ 1471 drwxr-xr-x 2 root root 4380 Jul 23 07:06 packages/ 1656 drwxr-xr-x 2 root root 260 Jul 23 07:06 plugins/ 23078 -rw-rw-rw- 1 root root 0 Jul 11 07:20 preclear.disk.log 1658 drwxr-xr-x 2 root root 80 Jul 22 07:29 removed_packages/ 1659 drwxr-xr-x 2 root root 40 Oct 27 1998 removed_scripts/ 1660 drwxr-xr-x 3 root root 160 Jul 7 09:00 samba/ 1661 drwxr-xr-x 2 root root 500 Jul 7 07:55 scripts/ 1662 -rw-r--r-- 1 root root 0 Mar 10 00:53 secure 1663 drwxr-xr-x 3 root root 60 Jun 12 19:25 setup/ 1665 -rw-r--r-- 1 root root 0 Mar 10 00:53 spooler 1666 -rw-r--r-- 1 root root 783796 Jul 24 18:01 syslog 1667 -rw-rw-r-- 1 root utmp 43392 Jul 24 18:01 wtmp
July 24, 20187 yr 4 minutes ago, hawihoney said: Here's the output. log shows 82% currently. That's to much for under 1MB log currently. root@Tower:~# ls -ail /var/log total 980 1455 drwxr-xr-x 13 root root 560 Jul 7 07:55 ./ 1044 drwxr-xr-x 13 root root 300 Jul 25 2016 ../ 14251 drwxr-xr-x 2 root root 160 Jul 24 04:40 atop/ 1458 -rw------- 1 root root 384 Jul 17 08:44 btmp 1459 -rw-r--r-- 1 root root 0 Mar 10 00:53 cron 1460 -rw-r--r-- 1 root root 0 Mar 10 00:53 debug 28016 -rw-rw-rw- 1 root root 59786 Jul 24 10:24 diskinfo.log 3761 -rw-rw-rw- 1 root root 72513 Jul 7 07:55 dmesg 35210 -rw-rw-rw- 1 root root 19654 Jul 24 07:37 docker.log 1461 -rw-r--r-- 1 root root 1440 Jul 7 07:55 faillog 1462 -rw-r--r-- 1 root root 13140 Jul 24 18:01 lastlog 1463 drwxr-xr-x 5 root root 100 Feb 21 14:29 libvirt/ 1467 -rw-r--r-- 1 root root 0 Mar 10 00:53 maillog 1468 -rw-r--r-- 1 root root 0 Mar 10 00:53 messages 1469 drwxr-xr-x 2 root root 40 May 16 2001 nfsd/ 1470 drwxr-x--- 2 nobody root 60 Jul 7 07:55 nginx/ 1471 drwxr-xr-x 2 root root 4380 Jul 23 07:06 packages/ 1656 drwxr-xr-x 2 root root 260 Jul 23 07:06 plugins/ 23078 -rw-rw-rw- 1 root root 0 Jul 11 07:20 preclear.disk.log 1658 drwxr-xr-x 2 root root 80 Jul 22 07:29 removed_packages/ 1659 drwxr-xr-x 2 root root 40 Oct 27 1998 removed_scripts/ 1660 drwxr-xr-x 3 root root 160 Jul 7 09:00 samba/ 1661 drwxr-xr-x 2 root root 500 Jul 7 07:55 scripts/ 1662 -rw-r--r-- 1 root root 0 Mar 10 00:53 secure 1663 drwxr-xr-x 3 root root 60 Jun 12 19:25 setup/ 1665 -rw-r--r-- 1 root root 0 Mar 10 00:53 spooler 1666 -rw-r--r-- 1 root root 783796 Jul 24 18:01 syslog 1667 -rw-rw-r-- 1 root utmp 43392 Jul 24 18:01 wtmp It's not a file directly in /var/log/ that is eating the space. It's in one of the subdirectories. On the command line do: du -s /var/log/* In your previous info you sent in, your machine consumed over 93 MB of the log file system: tmpfs 128M 93M 36M 73% /var/log Edited July 24, 20187 yr by pwm Added du info
July 24, 20187 yr Author atop uses 107MB currently. I did never use it. After deleting it's log log shows 2% now. Thanks for the hint.
July 24, 20187 yr 6 minutes ago, hawihoney said: atop uses 107MB currently. I did never use it. After deleting it's log log shows 2% now. Thanks for the hint. Sounds like something installs an atop that is run as a daemon - in which case you most probably had one atop file / day in /var/log/atop. In that case, you either need to uninstall whatever runs atop as daemon. Or find and modify the configuration that specifies the number of days of log files to keep.
July 24, 20187 yr You probably somewhere have a script that contains find $LOGPATH -name 'atop_*' -mtime +28 -exec rm {} \; where the number 28 specifies number of days of atop accounting information to keep. But in reality, most people never need an atop run as daemon with daily accounting constantly collected. atop manages quite well to just start and run exactly when needed and then displaying information just based on new accounting retrieved as atop is run.
July 24, 20187 yr Author Don't know why but it seems that I did install atop with the Nerd pack on my two servers. Never used it. Now I need to figure out how to uninstall a single package on the Nerd pack page. Think that this is the reason that a lot of stuff is still installed whilst not needed. Did not understand the logic on the Dev pack and Nerd pack pages. Stupid me. Thanks for all your help to find that log filler. Edited July 24, 20187 yr by hawihoney
July 24, 20187 yr 10 minutes ago, hawihoney said: Don't know why but it seems that I did install atop with the Nerd pack on my two servers. To uninstall: go to Settings >>> NerdPack Then look for atop in the list of apps. Then determine if you installed. IF you did, you should be able to uninstall it on that same line with the toggle slide to 'off'. If some other app installed it, you may (should?) be able to see which one it did it. Edited July 24, 20187 yr by Frank1940
July 24, 20187 yr Author Thanks, this will ignore a package after next reboot I think. What's the usage of the uninstall slider in the top right of the Nerd pack?
September 11, 20205 yr My logs didnt disappear after reboot, but seems like its something wrong with spindown. So had to turn it off. Ref several thousand log entries and solution in this article https://forums.unraid.net/topic/79414-disk-settings-spin-down/
September 11, 20205 yr 2 hours ago, ajugland said: My logs didnt disappear after reboot You must be misinterpreting this. Take a look at the first timestamp in syslogs and you will see it corresponds to reboot time. They definitely do reset on reboot but that won't fix filling them up again of course.
Archived
This topic is now archived and is closed to further replies.