hawihoney Posted July 23, 2018 Share Posted July 23, 2018 (edited) I need to restart my two servers every three weeks because the dashboard shows the log (out of 'flash log docker') is running full (100%). There seems to be no autorotate for logs or something like that. How can I omit the required reboots? And the last question is, how can I find out what's eating up the log space? Many thanks in advance. Edited July 24, 2018 by hawihoney Quote Link to comment
Frank1940 Posted July 23, 2018 Share Posted July 23, 2018 For a start, go to Tools >>> Diagnostics and then upload the resulting Diagnostics file with your next post. Quote Link to comment
hawihoney Posted July 23, 2018 Author Share Posted July 23, 2018 Here's the diagnostics. I can't see what causes 73%. Yesterday it was at 51%. Any ideas? Thanks in advance. tower-diagnostics-20180723-1343.zip Quote Link to comment
Frank1940 Posted July 23, 2018 Share Posted July 23, 2018 You got hundreds of entries like this: Jul 8 04:30:02 Tower sshd[20587]: Accepted publickey for root from 192.168.178.34 port 58842 ssh2: RSA SHA256:NlDaBhofPoWmFOvqcBxxPRT3HmcRZFEqIdAma/CLoBQ Jul 8 04:32:43 Tower sshd[20587]: Received disconnect from 192.168.178.34 port 58842:11: disconnected by user Jul 8 04:32:43 Tower sshd[20587]: Disconnected from user root 192.168.178.34 port 58842 Jul 8 04:32:43 Tower sshd[1812]: Accepted publickey for root from 192.168.178.34 port 58844 ssh2: RSA SHA256:NlDaBhofPoWmFOvqcBxxPRT3HmcRZFEqIdAma/CLoBQ Jul 8 04:32:59 Tower sshd[1812]: Received disconnect from 192.168.178.34 port 58844:11: disconnected by user Jul 8 04:32:59 Tower sshd[1812]: Disconnected from user root 192.168.178.34 port 58844 Each time it seems like the port number increases. I am not expert but it looks like you are being probed for a way into your system... Quote Link to comment
hawihoney Posted July 23, 2018 Author Share Posted July 23, 2018 Thanks for your answer. Ah, I use User Scripts plugin to rsync backups from one server to a second unRAID server in our house. Can I omit these messages or rotate the logs on my own. syslog is several 100K currently. That's not much but leads to over 70% fullness. Can I increase the syslog threshold? How? Thanks in advance. Quote Link to comment
pwm Posted July 23, 2018 Share Posted July 23, 2018 19 minutes ago, hawihoney said: Thanks for your answer. Ah, I use User Scripts plugin to rsync backups from one server to a second unRAID server in our house. Can I omit these messages or rotate the logs on my own. syslog is several 100K currently. That's not much but leads to over 70% fullness. Can I increase the syslog threshold? How? Thanks in advance. Why do you run rsync so often? Quote Link to comment
hawihoney Posted July 23, 2018 Author Share Posted July 23, 2018 (edited) Oh, rsync runs every night - but I call it for every share I take a backup from. Pictures, Projects, Programs, Documents, ... last time I did count 12 shares. Roundabout one million files of a long lasting digital Life. Some shares will be backuped once per night, some once per week and some once per month. Is there a better way? TIA Edited July 23, 2018 by hawihoney Quote Link to comment
hawihoney Posted July 23, 2018 Author Share Posted July 23, 2018 Here's an example of one of my three user scripts. This one runs every night. Didn't know that SSH (or whatever) fills the syslog with Disconnect messages. Both server are standing in my house and are members of the same network (192.168.178.0/24). If there is a better way I would liekt to learn. rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Pictures Sorted/" /mnt/user/Pictures Sorted/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Pictures Unsorted/" /mnt/user/Pictures Unsorted/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Pictures Favorites/" /mnt/user/Pictures Favorites/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Data/" /mnt/user/Data/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Documents/" /mnt/user/Documents/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Movies Private/" /mnt/user/Movies Private/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Notes/" /mnt/user/Notes/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Programs/" /mnt/user/Programs/ rsync -avPX --delete-during --protect-args -e ssh "root@tower:/mnt/user/Projects/" /mnt/user/Projects/ ... Many thanks in advance. Quote Link to comment
hawihoney Posted July 24, 2018 Author Share Posted July 24, 2018 Don't use rsync with SSH to backup files between two unRAID servers. Use the Unassigned Devices plugin, mount the backup server as SMB and go that route. Quote Link to comment
pwm Posted July 24, 2018 Share Posted July 24, 2018 21 minutes ago, hawihoney said: Don't use rsync with SSH to backup files between two unRAID servers. Use the Unassigned Devices plugin, mount the backup server as SMB and go that route. The advantage with rsync using SSH is that the main server does not have access to the bacckup server, giving additional protection to the backup. There aren't really that many SSH lines in the log. CA Backup/Restore produces more log lines when it stops/starts all the Dockers. Quote Link to comment
hawihoney Posted July 24, 2018 Author Share Posted July 24, 2018 This leads to the questions: Why is there no log rotate? And why is the log nearly full with 800KB syslog on a 16GB flash with 15.5GB free space. Is something wrong with dashboard or my system? Quote Link to comment
trurl Posted July 24, 2018 Share Posted July 24, 2018 1 hour ago, hawihoney said: This leads to the questions: Why is there no log rotate? And why is the log nearly full with 800KB syslog on a 16GB flash with 15.5GB free space. The log does rotate, but the old logs are still stored. And logs as well as all other OS files are in RAM not on flash. Flash only contains an archive of the OS which is unpacked into RAMfs at boot. Flash also contains settings from the webUI. That is all flash contains unless you have intentionally used it for some other purpose such as scripts. Quote Link to comment
trurl Posted July 24, 2018 Share Posted July 24, 2018 1 hour ago, hawihoney said: why is the log nearly full with 800KB syslog You must have other things in /var/log besides that. Normally 128MB is reserved. Some users have recently reported atop taking a lot of space there but I don't remember why they even had that running. I don't think it is builtin. Quote Link to comment
Squid Posted July 24, 2018 Share Posted July 24, 2018 There's more in /var/log than just the syslog, and your syslog is nowhere near large enough to account for the folder getting full What is the output of ls -ail /var/log Quote Link to comment
pwm Posted July 24, 2018 Share Posted July 24, 2018 48 minutes ago, trurl said: You must have other things in /var/log besides that. Normally 128MB is reserved. Some users have recently reported atop taking a lot of space there but I don't remember why they even had that running. I don't think it is builtin. A running atop requests the Linux kernel to accumulate statistics until atop is correctly exited - so it is only when atop is kept running indefinitely or when it is killed with signal 9 that Linux will continue to fill /var/log/ with accounting for all running processes. Quote Link to comment
hawihoney Posted July 24, 2018 Author Share Posted July 24, 2018 Here's the output. log shows 82% currently. That's to much for under 1MB log currently. root@Tower:~# ls -ail /var/log total 980 1455 drwxr-xr-x 13 root root 560 Jul 7 07:55 ./ 1044 drwxr-xr-x 13 root root 300 Jul 25 2016 ../ 14251 drwxr-xr-x 2 root root 160 Jul 24 04:40 atop/ 1458 -rw------- 1 root root 384 Jul 17 08:44 btmp 1459 -rw-r--r-- 1 root root 0 Mar 10 00:53 cron 1460 -rw-r--r-- 1 root root 0 Mar 10 00:53 debug 28016 -rw-rw-rw- 1 root root 59786 Jul 24 10:24 diskinfo.log 3761 -rw-rw-rw- 1 root root 72513 Jul 7 07:55 dmesg 35210 -rw-rw-rw- 1 root root 19654 Jul 24 07:37 docker.log 1461 -rw-r--r-- 1 root root 1440 Jul 7 07:55 faillog 1462 -rw-r--r-- 1 root root 13140 Jul 24 18:01 lastlog 1463 drwxr-xr-x 5 root root 100 Feb 21 14:29 libvirt/ 1467 -rw-r--r-- 1 root root 0 Mar 10 00:53 maillog 1468 -rw-r--r-- 1 root root 0 Mar 10 00:53 messages 1469 drwxr-xr-x 2 root root 40 May 16 2001 nfsd/ 1470 drwxr-x--- 2 nobody root 60 Jul 7 07:55 nginx/ 1471 drwxr-xr-x 2 root root 4380 Jul 23 07:06 packages/ 1656 drwxr-xr-x 2 root root 260 Jul 23 07:06 plugins/ 23078 -rw-rw-rw- 1 root root 0 Jul 11 07:20 preclear.disk.log 1658 drwxr-xr-x 2 root root 80 Jul 22 07:29 removed_packages/ 1659 drwxr-xr-x 2 root root 40 Oct 27 1998 removed_scripts/ 1660 drwxr-xr-x 3 root root 160 Jul 7 09:00 samba/ 1661 drwxr-xr-x 2 root root 500 Jul 7 07:55 scripts/ 1662 -rw-r--r-- 1 root root 0 Mar 10 00:53 secure 1663 drwxr-xr-x 3 root root 60 Jun 12 19:25 setup/ 1665 -rw-r--r-- 1 root root 0 Mar 10 00:53 spooler 1666 -rw-r--r-- 1 root root 783796 Jul 24 18:01 syslog 1667 -rw-rw-r-- 1 root utmp 43392 Jul 24 18:01 wtmp Quote Link to comment
pwm Posted July 24, 2018 Share Posted July 24, 2018 (edited) 4 minutes ago, hawihoney said: Here's the output. log shows 82% currently. That's to much for under 1MB log currently. root@Tower:~# ls -ail /var/log total 980 1455 drwxr-xr-x 13 root root 560 Jul 7 07:55 ./ 1044 drwxr-xr-x 13 root root 300 Jul 25 2016 ../ 14251 drwxr-xr-x 2 root root 160 Jul 24 04:40 atop/ 1458 -rw------- 1 root root 384 Jul 17 08:44 btmp 1459 -rw-r--r-- 1 root root 0 Mar 10 00:53 cron 1460 -rw-r--r-- 1 root root 0 Mar 10 00:53 debug 28016 -rw-rw-rw- 1 root root 59786 Jul 24 10:24 diskinfo.log 3761 -rw-rw-rw- 1 root root 72513 Jul 7 07:55 dmesg 35210 -rw-rw-rw- 1 root root 19654 Jul 24 07:37 docker.log 1461 -rw-r--r-- 1 root root 1440 Jul 7 07:55 faillog 1462 -rw-r--r-- 1 root root 13140 Jul 24 18:01 lastlog 1463 drwxr-xr-x 5 root root 100 Feb 21 14:29 libvirt/ 1467 -rw-r--r-- 1 root root 0 Mar 10 00:53 maillog 1468 -rw-r--r-- 1 root root 0 Mar 10 00:53 messages 1469 drwxr-xr-x 2 root root 40 May 16 2001 nfsd/ 1470 drwxr-x--- 2 nobody root 60 Jul 7 07:55 nginx/ 1471 drwxr-xr-x 2 root root 4380 Jul 23 07:06 packages/ 1656 drwxr-xr-x 2 root root 260 Jul 23 07:06 plugins/ 23078 -rw-rw-rw- 1 root root 0 Jul 11 07:20 preclear.disk.log 1658 drwxr-xr-x 2 root root 80 Jul 22 07:29 removed_packages/ 1659 drwxr-xr-x 2 root root 40 Oct 27 1998 removed_scripts/ 1660 drwxr-xr-x 3 root root 160 Jul 7 09:00 samba/ 1661 drwxr-xr-x 2 root root 500 Jul 7 07:55 scripts/ 1662 -rw-r--r-- 1 root root 0 Mar 10 00:53 secure 1663 drwxr-xr-x 3 root root 60 Jun 12 19:25 setup/ 1665 -rw-r--r-- 1 root root 0 Mar 10 00:53 spooler 1666 -rw-r--r-- 1 root root 783796 Jul 24 18:01 syslog 1667 -rw-rw-r-- 1 root utmp 43392 Jul 24 18:01 wtmp It's not a file directly in /var/log/ that is eating the space. It's in one of the subdirectories. On the command line do: du -s /var/log/* In your previous info you sent in, your machine consumed over 93 MB of the log file system: tmpfs 128M 93M 36M 73% /var/log Edited July 24, 2018 by pwm Added du info Quote Link to comment
hawihoney Posted July 24, 2018 Author Share Posted July 24, 2018 atop uses 107MB currently. I did never use it. After deleting it's log log shows 2% now. Thanks for the hint. Quote Link to comment
pwm Posted July 24, 2018 Share Posted July 24, 2018 6 minutes ago, hawihoney said: atop uses 107MB currently. I did never use it. After deleting it's log log shows 2% now. Thanks for the hint. Sounds like something installs an atop that is run as a daemon - in which case you most probably had one atop file / day in /var/log/atop. In that case, you either need to uninstall whatever runs atop as daemon. Or find and modify the configuration that specifies the number of days of log files to keep. Quote Link to comment
pwm Posted July 24, 2018 Share Posted July 24, 2018 You probably somewhere have a script that contains find $LOGPATH -name 'atop_*' -mtime +28 -exec rm {} \; where the number 28 specifies number of days of atop accounting information to keep. But in reality, most people never need an atop run as daemon with daily accounting constantly collected. atop manages quite well to just start and run exactly when needed and then displaying information just based on new accounting retrieved as atop is run. Quote Link to comment
hawihoney Posted July 24, 2018 Author Share Posted July 24, 2018 (edited) Don't know why but it seems that I did install atop with the Nerd pack on my two servers. Never used it. Now I need to figure out how to uninstall a single package on the Nerd pack page. Think that this is the reason that a lot of stuff is still installed whilst not needed. Did not understand the logic on the Dev pack and Nerd pack pages. Stupid me. Thanks for all your help to find that log filler. Edited July 24, 2018 by hawihoney Quote Link to comment
Frank1940 Posted July 24, 2018 Share Posted July 24, 2018 (edited) 10 minutes ago, hawihoney said: Don't know why but it seems that I did install atop with the Nerd pack on my two servers. To uninstall: go to Settings >>> NerdPack Then look for atop in the list of apps. Then determine if you installed. IF you did, you should be able to uninstall it on that same line with the toggle slide to 'off'. If some other app installed it, you may (should?) be able to see which one it did it. Edited July 24, 2018 by Frank1940 Quote Link to comment
hawihoney Posted July 24, 2018 Author Share Posted July 24, 2018 Thanks, this will ignore a package after next reboot I think. What's the usage of the uninstall slider in the top right of the Nerd pack? Quote Link to comment
ajugland Posted September 11, 2020 Share Posted September 11, 2020 My logs didnt disappear after reboot, but seems like its something wrong with spindown. So had to turn it off. Ref several thousand log entries and solution in this article https://forums.unraid.net/topic/79414-disk-settings-spin-down/ Quote Link to comment
trurl Posted September 11, 2020 Share Posted September 11, 2020 2 hours ago, ajugland said: My logs didnt disappear after reboot You must be misinterpreting this. Take a look at the first timestamp in syslogs and you will see it corresponds to reboot time. They definitely do reset on reboot but that won't fix filling them up again of course. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.