May 9, 201610 yr I'm having an issue with 6.1.9 on my server. I'm unable to attach diagnostics because I get this error when I try to run diagnostics from command line (GUI gives me a 404). Warning: file_put_contents(): Only 0 of 34 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/dynamix/scripts/diagnostics on line 82 done. The error happens after the machine has been up for about 2 weeks, and my first thought was that the log filesystem is out of space, but I don't seen anything full when I do a df (I have also increased the tmpfs size, and it doesn't seem to change anything. A reboot fixes the issue for about 2 weeks. When this happens the shares are still accessible, the GUI just has varying issues (right now it shows the array as offline, but I can still stop/start dockers for example, other times, the GUI has a bunch of the out of space errors). Here is a the output of a df Filesystem 1K-blocks Used Available Use% Mounted on tmpfs 524288 35596 488692 7% /var/log /dev/sdb1 3906544 454336 3452208 12% /boot /dev/md1 2928835740 2402477096 526358644 83% /mnt/disk1 /dev/md2 1952560688 1776952304 175608384 92% /mnt/disk2 /dev/md3 1952560688 1720042324 232518364 89% /mnt/disk3 /dev/md4 2928835740 2563215852 365619888 88% /mnt/disk4 /dev/md5 1952560688 1620659252 331901436 84% /mnt/disk5 /dev/md6 1952560688 1724716320 227844368 89% /mnt/disk6 /dev/md7 2928835740 2649971288 278864452 91% /mnt/disk7 /dev/md8 1464423156 775481576 688941580 53% /mnt/disk8 /dev/md9 1464423156 731048424 733374732 50% /mnt/disk9 /dev/sdf1 488385560 36553476 450372044 8% /mnt/cache shfs 19525596284 15964564436 3561031848 82% /mnt/user0 shfs 20013981844 16001117912 4011403892 80% /mnt/user /dev/loop0 10485760 3517472 5254496 41% /var/lib/docker /dev/loop1 1843 86 1547 6% /etc/libvirt Here is what I currently see on a tail of syslog: May 9 09:02:01 unraid-server1 crond[1703]: exit status 2 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Any suggestions of what to look at to see what's filling up and causing this issue?
May 22, 201610 yr Author So here we are 2 weeks later, and the GUI lockup happened again. This time it was able to write at least a partial diagnostics. I noticed that diagnostics listed space errors for lines 67, 80, and 82 if that helps. The other thing that I was able to capture is on the console, there is this line, which seems to point to an issue with the mounting of /var/log, and mentions a plugin (unbalance), which I have uninstalled through. /bin/bash: line 3: /usr/local/emhttp./plugins/unbalance/scripts/stop: No such file or directory mount: /var/log not mounted or bad option I'm really beginning to think it's something with Dynamix, since the server itself seems to be fine, it's just the GUI that has issue (shows the array off line, no CPU % on dashboard, doesn't update up time, etc). I have left the server up since it's still working in case there are things anyone can think of that I should try try. Here is the output of df -h showing no file systems are full: Filesystem Size Used Avail Use% Mounted on tmpfs 128M 35M 94M 27% /var/log /dev/sdd1 3.8G 412M 3.4G 11% /boot /dev/md1 2.8T 2.4T 352G 88% /mnt/disk1 /dev/md2 1.9T 1.7T 190G 90% /mnt/disk2 /dev/md3 1.9T 1.6T 231G 88% /mnt/disk3 /dev/md4 2.8T 2.4T 349G 88% /mnt/disk4 /dev/md5 1.9T 1.5T 332G 83% /mnt/disk5 /dev/md6 1.9T 1.7T 218G 89% /mnt/disk6 /dev/md7 2.8T 2.5T 307G 90% /mnt/disk7 /dev/md8 1.4T 787G 611G 57% /mnt/disk8 /dev/md9 1.4T 696G 702G 50% /mnt/disk9 /dev/sde1 466G 34G 431G 8% /mnt/cache shfs 19T 15T 3.3T 83% /mnt/user0 shfs 19T 16T 3.7T 81% /mnt/user /dev/loop0 10G 3.6G 4.9G 42% /var/lib/docker /dev/loop1 1.8M 86K 1.6M 6% /etc/libvirt tower-diagnostics-20160522-0825.zip
May 22, 201610 yr Community Expert OK, I am assuming that you are doing this from the GUI. When I click on the Diagnostics tab, it first pops up the following (in red): Please wait, collecting diagnostic information... Then Firefox pops up a window and asks me if I want to open the file with my default zip file application or save the file. I always save the file and it goes to the folder location on my Windows 7 computer where Firefox normally stores downloaded files. Now when you run this same command from the Command Line, it saves the file to the logs folder on your Flash Drive. You could possibly have an issue with your Flash Drive so you might have a good look at it. By the way, do you have an Ad-blocker installed in your browser. If you do, you will have to 'whitelist' your unRAID server!
May 22, 201610 yr You might consider installing Fix Common Problems, and tossing it into Troubleshooting Mode It'll grab a diagnostics every 30 minutes, and do some additional tests every 10 minutes, along with doing a tail on the syslog so that something can hopefully be discerned as to what's going on with your system. Since your uptime seems to be around 2 weeks before the crash, if you do this, you might want to go and delete the excess diagnostics (/logs folder) from the flash drive every day or so. Net result is that your diagnostics as posted are functionally unusable due to those errors (ie: they are blank). FCP will at least always give you a valid syslog to post, and within that syslog there are some basic checks of the memory usage, processes, etc.
May 22, 201610 yr Author OK, I am assuming that you are doing this from the GUI. When I click on the Diagnostics tab, it first pops up the following (in red): Please wait, collecting diagnostic information... Then Firefox pops up a window and asks me if I want to open the file with my default zip file application or save the file. I always save the file and it goes to the folder location on my Windows 7 computer where Firefox normally stores downloaded files. Now when you run this same command from the Command Line, it saves the file to the logs folder on your Flash Drive. You could possibly have an issue with your Flash Drive so you might have a good look at it. By the way, do you have an Ad-blocker installed in your browser. If you do, you will have to 'whitelist' your unRAID server! Nope, have to do diagnostics from command line since GUI doesn't allow it. The flash drive is clean, verified with check on another pc, and has plenty of space. Sent from my SM-T700 using Tapatalk
May 22, 201610 yr Author You might consider installing Fix Common Problems, and tossing it into Troubleshooting Mode It'll grab a diagnostics every 30 minutes, and do some additional tests every 10 minutes, along with doing a tail on the syslog so that something can hopefully be discerned as to what's going on with your system. Since your uptime seems to be around 2 weeks before the crash, if you do this, you might want to go and delete the excess diagnostics (/logs folder) from the flash drive every day or so. Net result is that your diagnostics as posted are functionally unusable due to those errors (ie: they are blank). FCP will at least always give you a valid syslog to post, and within that syslog there are some basic checks of the memory usage, processes, etc. I'll give it a shot. Installed that plugin before and had it do a check which didn't find any issues, but didn't try troubleshooting mode Sent from my SM-T700 using Tapatalk
May 30, 201610 yr Author So I did notice something after looking at the log output from fix common problems. It appears the fs filling up is root, I've noticed it is steadily growing, and doesn't show up in a normal df. It is at 63% after one week which would explain the issue happening at 2 weeks. I had to do a df -h . to display it. Any ideas how to see what is filling the fs? I tried find . -size +10000000c -exec ls -larth {} \, and it lists files, mostly in /var/lib/docker and are btrfs sub volumes, and containers. Nothing showed up over 100 megs when I ran this - find / xdev -type f -size +100M.
May 30, 201610 yr So I did notice something after looking at the log output from fix common problems. It appears the fs filling up is root, I've noticed it is steadily growing, and doesn't show up in a normal df. It is at 63% after one week which would explain the issue happening at 2 weeks. I had to do a df -h . to display it. Any ideas how to see what is filling the fs? I tried find . -size +10000000c -exec ls -larth {} \, and it lists files, mostly in /var/lib/docker and are btrfs sub volumes, and containers. Nothing showed up over 100 megs when I ran this - find / xdev -type f -size +100M. We dont know unless you provide actual details. Otherwise the only thing anyone can do is guess.
May 30, 201610 yr Author I guess what I'm asking is how to give the details. I don't think diagnostics will she'd any light, but I can attach if that would be helpful. The reason I don't think it will be helpful is right now there aren't errors, and when there finally are errors, diagnostics can't write the data to the drive. I'm hoping there is something else I can provide that will help narrow down the search, but I'm not sure what that is. Sent from my SM-T700 using Tapatalk
May 30, 201610 yr Try using the command: du -s -x /* to list the usage of each directory on the rootfs (you will get a few error lines which can be ignored). Then repeat it some time later (6, 12, 24 hours) and compare the results. If one directory shows noticeable increase start running the command on that directory. Repeat until you have narrowed down where the space is being consumed.o You don't happen to have the Checksum plugin installed (and probably not configured) do you? This plugin has previously been reported as consuming space on the rootfs when it is not configured for use and which unfortunately cannot be detected by the above method. If you do have it installed, try uninstalling it.
May 30, 201610 yr Author I do have checksum, so I'll try removing it to see if that fixes the issue Sent from my SM-T700 using Tapatalk
May 30, 201610 yr Author That looks promising. When I removed checksum usage from df - h . Went from 65% to 5%. I will update later to solved if this does solve it. Sent from my SM-T700 using Tapatalk
June 1, 201610 yr Author That looks promising. When I removed checksum usage from df - h . Went from 65% to 5%. I will update later to solved if this does solve it. Sent from my SM-T700 using Tapatalk I would say this is solved, over 2.5 days and root is still sitting at 5%. Thanks for the help getting to the bottom of this Sent from my SM-T700 using Tapatalk
Archived
This topic is now archived and is closed to further replies.