December 25, 201510 yr Ahh... thanks. Got it. Here it is. All the files are 0 bytes !?!? Try just the full syslog
December 25, 201510 yr Author I didn't know another way of getting the diagnostics file off of the flash drive at the moment so I had to shutdown the server so I could put the flash drive in another computer. So these are the results with everything back up and running normally. Filesystem Size Used Avail Use% Mounted on tmpfs 128M 3.0M 126M 3% /var/log /dev/sda1 15G 175M 15G 2% /boot /dev/md1 466G 442G 25G 95% /mnt/disk1 /dev/md2 466G 132G 335G 29% /mnt/disk2 /dev/md3 466G 33M 466G 1% /mnt/disk3 /dev/md4 5.5T 3.4T 2.1T 62% /mnt/disk4 /dev/sde1 168G 12G 155G 8% /mnt/cache shfs 6.9T 4.0T 2.9T 58% /mnt/user0 shfs 7.0T 4.0T 3.1T 57% /mnt/user /dev/loop0 10G 3.8G 4.9G 44% /var/lib/docker
December 25, 201510 yr ok... I *think* your tmpfs was full based upon this line: Dec 25 04:40:01 Turtle logrotate: ALERT - exited abnormally. Dynamix Stats was (is? - not sure if its been fixed) a real pig on being a contributor to that issue. With the reset, probably everything is good to go, but I would leave stats uninstalled and see how it goes, or look at this thread: https://lime-technology.com/forum/index.php?topic=39687.msg371434#msg371434 (add the line to your config/go file) If this happens again with stats uninstalled, what I would do is restart the system again, then locally at the console type tail -f /var/log/syslog > /boot/syslog.txt Nothing will happen on the screen, but every entry to syslog will wind up getting copied to your flash drive. Then on another crash we'll have something to look at.
December 25, 201510 yr Author Thanks Squid. I don't need to run that plugin so I'll just leave it uninstalled. In the meantime, I guess it's just a waiting game to see what happens.
December 31, 201510 yr Author Well it happened again. For the following, when you say locally, do you mean I have to hook up a monitor and keyboard, or can this be done with SSH? If this happens again with stats uninstalled, what I would do is restart the system again, then locally at the console type tail -f /var/log/syslog > /boot/syslog.txt Nothing will happen on the screen, but every entry to syslog will wind up getting copied to your flash drive. Then on another crash we'll have something to look at. Also, I am not sure if it's relevant, but when using powerdown -r in SSH there was this message. /usr/local/sbin/powerdown: line 41: echo: write error: No space left on device
January 1, 201610 yr Community Expert Well it happened again. For the following, when you say locally, do you mean I have to hook up a monitor and keyboard, or can this be done with SSH? If this happens again with stats uninstalled, what I would do is restart the system again, then locally at the console type tail -f /var/log/syslog > /boot/syslog.txt Nothing will happen on the screen, but every entry to syslog will wind up getting copied to your flash drive. Then on another crash we'll have something to look at. Also, I am not sure if it's relevant, but when using powerdown -r in SSH there was this message. /usr/local/sbin/powerdown: line 41: echo: write error: No space left on device If ssh/telnet doesn't crash you can do it that way. The powerdown message is probably another symptom of tmpfs full.
January 1, 201610 yr You want to do it locally. (the keyboard and monitor don't have to stay there. Just long enough to enter in the command) If you do it through ssh and you close putty or don't use screen then this exercise is moot
January 1, 201610 yr Author It's running. If it follows the pattern, it should happen again within a week.
January 1, 201610 yr ... Dynamix Stats was (is? - not sure if its been fixed) a real pig on being a contributor to that issue. In the latest version of Dynamix System Stats the location of the daily log files is moved from /var/log/sa to /var/sa. This solves the issue that the log filesystem gets full overtime. If you haven't done so, please update to the latest version.
January 7, 201610 yr Author Ok, it just did it again. Looks like it happens about every 5-6 days. So what's the next step with that command still running?
January 7, 201610 yr Author total 1000 -rw-r--r-- 1 root root 90 Jan 3 04:40 apcupsd.events -rw-r--r-- 1 root root 198 Jan 3 04:40 apcupsd.events.1 -rw------- 1 root root 0 Dec 1 17:18 btmp -rw-r--r-- 1 root root 0 Dec 11 2014 cron -rw-r--r-- 1 root root 0 Dec 11 2014 debug -rw-rw-rw- 1 root root 39011 Jan 1 10:24 dmesg -rw-rw-rw- 1 root root 0 Jan 1 10:25 docker.log -rw-r--r-- 1 root root 0 Feb 14 2014 faillog -rw-r--r-- 1 root root 292 Jan 7 11:24 lastlog drwxr-xr-x 5 root root 100 Aug 3 19:29 libvirt/ -rw-r--r-- 1 root root 0 Dec 11 2014 maillog -rw-r--r-- 1 root root 0 Dec 11 2014 messages drwxr-xr-x 2 root root 40 May 15 2001 nfsd/ drwxr-xr-x 2 root root 3820 Jan 1 10:25 packages/ drwxr-xr-x 2 root root 340 Jan 1 10:25 plugins/ drwxr-xr-x 2 root root 40 Oct 27 1998 removed_packages/ drwxr-xr-x 2 root root 40 Oct 27 1998 removed_scripts/ drwxr-xr-x 3 root root 100 Jan 1 10:24 samba/ drwxr-xr-x 2 root root 620 Jan 1 10:25 scripts/ -rw-r--r-- 1 root root 0 Dec 11 2014 secure drwxr-xr-x 3 root root 80 Aug 21 2012 setup/ -rw-r--r-- 1 root root 0 Dec 11 2014 spooler -rw-r--r-- 1 root root 214087 Jan 7 11:24 syslog -rw-rw-rw- 1 root root 743278 Jan 7 10:44 unassigned.devices.log -rw-rw-r-- 1 root utmp 7680 Jan 7 11:24 wtmp
January 7, 201610 yr There are no more dynamix stats files in here anymore (folder /var/log/sa doesn't exist), so it can be ruled out. At this moment there isn't (too) much usage, you can confirm that by running the command: df -h /var/log Do the same commands again once the issue happens.
January 7, 201610 yr Author There are no more dynamix stats files in here anymore (folder /var/log/sa doesn't exist), so it can be ruled out. At this moment there isn't (too) much usage, you can confirm that by running the command: df -h /var/log Do the same commands again once the issue happens. Dynamix Stats was ruled out the last time this happened because I had uninstalled it the time before. Do what command when what issue happens? It's in the middle of happening right now. I have not restarted the server. The only difference between this time and the other times is that my shares are still accessible at the moment despite my array saying offline. (Previously if they've been accessible, it's been for only a short time.)
January 7, 201610 yr I meant the commands: ls -l /var/log df -h /var/log Something must be filling the file system...
January 8, 201610 yr I didn't follow the complete thread, but in situations as these it is often a matter of elimination. First thing you can do is disable Docker and VM (see the settings page). Next reboot into safe mode and let it run to verify that the issues are gone. Once confirmed it will be a matter of selectively adding plugins, dockers, VMs to find out where it goes wrong.
January 8, 201610 yr Author I am not going to disable everything for a week or possibly even longer. Things like Emby server get used everyday. I've uninstalled all the plugins and dockers I can do without for awhile and that will have to do. This leaves Emby server and Delugevpn. Interesting thing to note. When uninstalling plugins, I noticed that my disks reappeared again, just like when I uninstalled system stats a few weeks back. I'm not sure which one did it because I had uninstalled a few before I noticed. Useful Link, Unassigned Devices, Speedtest, Preclear Disks, Powerdown Package.
January 13, 201610 yr Author 5 days in and no problems yet. tmpfs is sitting at 1.9M. I was reading through http://lime-technology.com/forum/index.php?topic=45361.0 and I am wondering if my problem could be related because I also had Checksum installed but hadn't gotten around to setting it up yet?
January 14, 201610 yr Is it me or people in this forum immediately suggest to replace power supply to fix a problem with the unraid server before actually trying anything else? I had low writing speeds on my unraid and it turned out to be a problem with the network switch, which once rebooted sorted itself out. Needless to say, someone suggested I needed a new power supply. It looks like in your case too the power supply was not the cause of your problems...
Archived
This topic is now archived and is closed to further replies.