January 9, 201610 yr I have two unraid VM's and my backup unraid VM keeps running out of space. I get device out of space errors in the console and my raid array shows as down or my user shares are down when this happens. For what ever reason the root file system, which should exist in memory, doesn't show in the df -h output however df -h / will show root out of space. The size of the root file system is 2.4 GB, My primary VM does have more memory assigned, and of the 15 GB (I used this for plex transcoding caching in memory at one point if you're wonder why so much is assigned) of memory. It so happens to show 2.4GB used out of 7.9 GB (File system is over 5GB larger then my backup VM). That would suggest getting up to 2.4GB of file system usage can be normal. How is the size of the file system being determined? A percentage of the assigned memory? I bumped my backup VM's memory a bit and the size of the root file system didn't budge. root@unRAIDprime:/# df -h ./ Filesystem Size Used Avail Use% Mounted on - 7.9G 2.4G 5.5G 30% / root@unRAIDprime:~# free -m total used free shared buffers cached Mem: 15044 13008 2036 0 63 11045 -/+ buffers/cache: 1900 13144 Swap: 7095 0 7095 Was just rebooted and the root filesystem usage dropped considerably. root@unRAIDbackup:~# df -h ./ Filesystem Size Used Avail Use% Mounted on - 2.4G 566M 1.9G 24% / root@unRAIDbackup:~# free -m total used free shared buffers cached Mem: 4965 785 4179 0 6 628 -/+ buffers/cache: 150 4814 Swap: 0
January 10, 201610 yr Author Happened again, taks less then 2 days to fill up. My other VM clears the usage on reboot too but then seems to stop at 2.4GB usage which is fine since 7GB is mounted. root@unRAIDbackup:/# df -h / Filesystem Size Used Avail Use% Mounted on - 2.4G 2.4G 0 100% /
January 10, 201610 yr You probably have something writing/saving data to the FS that should be saving to the array or cache drive instead.
January 10, 201610 yr Author That's what I was thinking but my du results say otherwise, I tried to do a high level find with the -xdev switch to stay in the root file system and it eventually quits with a process term error when the file system is full. I started going into the individual OS related folders excluding /boot and /mnt one by one with a du -sm /folder | sort -n command and not a single one comes close to adding up to 2.4 GB. /usr is under 500 mb, rest are all 20MB or less. On a side note I think I found a ramdisk resize option in the sysconfig.conf that is working, I was able to bump the ram file system from 2.4GB to 2.9GB. Considering my primary VM is at 2.4GB usage, I'm hoping that might be enough and 2.4 GB usage might actually be normal. This is where I am at after making that change last night to add .5 GB to the ramdisk. root@unRAIDbackup:~# uptime 07:49:56 up 14:49, 2 users, load average: 1.16, 1.17, 1.15 root@unRAIDbackup:~# df -h / Filesystem Size Used Avail Use% Mounted on - 2.9G 2.0G 960M 68% / total used free shared buffers cached Mem: 5973 5887 85 0 0 5597 -/+ buffers/cache: 290 5682 Swap: 0 0 0 root@unRAIDbackup:/# ls bin/ dev/ home/ lib/ mnt/ root/ sbin/ tmp/ var/ boot/ etc/ init@ lib64/ proc/ run/ sys/ usr/ root@unRAIDbackup:/# du -sm /bin 10 /bin root@unRAIDbackup:/# du -sm /lib 15 /lib root@unRAIDbackup:/# du -sm /lib64 20 /lib64 root@unRAIDbackup:/# du -sm /root 1 /root root@unRAIDbackup:/# du -sm /sbin 15 /sbin root@unRAIDbackup:/# du -sm /tmp 1 /tmp root@unRAIDbackup:/# du -sm /var 16 /var root@unRAIDbackup:/# du -sm /etc 6 /etc root@unRAIDbackup:/# du -sm /run 1 /run root@unRAIDbackup:/# du -sm /usr 473 /usr
January 12, 201610 yr Author so it filled up again after bumping it. This is the standby VM with no dockers, and minimal plugins that just runs the stoarage. The only recent change before this happened was an OS upgrade. I can't find the file usage with the du command against any of the ram based directories. Kinda feels like a memory leak.
January 12, 201610 yr Do you happen to have the checksum plugin installed? I had exactly the same problem and I stopped the filling up of the root file system when after much investigations I uninstalled the Checksum plugin. I had installed this plugin but had not got around to configuring it to actually do any checksums .... Which maybe was the problem. I couldn't actually find what was actually causing the file system to fill up. I suspected that it was a file that was open in some program which was writing to it, but also the file was deleted by another program so the directory entry for the file had been removed and therefore did show up in any directory listing but still existed and continuing to grow (but would be freed up when the program finally stopped).
January 13, 201610 yr Author Actually..... I do. Never used it yet so away it goes. I was at the point I was going to start removing plugins one at a time anyway.
January 13, 201610 yr Author Holy crud.... that may have been it. My file usage was 2.7GB before the uninstall. Dropped like a rock. root@unRAIDbackup:~# df -h / Filesystem Size Used Avail Use% Mounted on - 2.9G 544M 2.4G 19% / root@unRAIDbackup:~#
January 13, 201610 yr Author Ha my primary VM was impacted too, didn't realize it was growing the last few days since it has a much larger memory file system. Before root@unRAIDprime:~# df -h / Filesystem Size Used Avail Use% Mounted on - 7.4G 7.3G 57M 100% / After the uninstall root@unRAIDprime:~# df -h / Filesystem Size Used Avail Use% Mounted on - 7.4G 582M 6.8G 8% / Good deal, where's your tip jar? You deserve a dollar and a coke
January 13, 201610 yr Both of my systems running checksum are showing ~500meg used. If someone does actually find what the issue is and if it's checksum feel free to issue a pull request on github and I'll release an update
January 13, 201610 yr The only thing people have had in common Squid, including myself, was that the plugin was installed but not running/being used... (Sorry I never got around to setting it up...)
January 13, 201610 yr Both of my systems running checksum are showing ~500meg used. If someone does actually find what the issue is and if it's checksum feel free to issue a pull request on github and I'll release an update Upon studying a little bit your code (though I am not using the plugin) there are two potential areas which may need further examination: 1. disks_mounted starts a number of scripts/services unconditionally. This means that even when no user configuration is done, as soon as the array starts these scripts/services start to run too. 2. The function "logger" in checksum.php appears to do log rotation on the wrong file. In theory the file checksumLog.txt can grow unrestricted. Hope this may help you in your quest!
January 13, 201610 yr Try running just df -h and see what's full. This can happen too depending on different plugins logging and filling up tempfs (/var/log) which is why I have in my go file this to increase the size of tempfs # resize tmpfs mount -o remount,size=256m /var/log
January 13, 201610 yr Both of my systems running checksum are showing ~500meg used. If someone does actually find what the issue is and if it's checksum feel free to issue a pull request on github and I'll release an update Upon studying a little bit your code (though I am not using the plugin) there are two potential areas which may need further examination: 1. disks_mounted starts a number of scripts/services unconditionally. This means that even when no user configuration is done, as soon as the array starts these scripts/services start to run too. 2. The function "logger" in checksum.php appears to do log rotation on the wrong file. In theory the file checksumLog.txt can grow unrestricted. Hope this may help you in your quest! As I announced in the checksum thread, and via Moderator Comments within CA on this plugin, development on checksum has ceased in favor of Dynamix File Integrity Plugin. TBH, if checksum is giving you issues, uninstall it.
January 13, 201610 yr ]As I announced in the checksum thread, and via Moderator Comments within CA on this plugin, development on checksum has ceased in favor of Dynamix File Integrity Plugin. TBH, if checksum is giving you issues, uninstall it. Sorry Squid, didn't realise you'd ceased dev on the checksum plugin, last time we spoke I was under the impression you were continuing with it as it has some key differences with FIP...
January 13, 201610 yr Sorry Squid, didn't realise you'd ceased dev on the checksum plugin, last time we spoke I was under the impression you were continuing with it as it has some key differences with FIP... It does have some key differences with FIP... so this is pretty bad news to some of us. I do understand why Squid decided to cease devlopment... hope someone else can pick it up or fix some things.
January 14, 201610 yr Author I was tracking why my unraid VM's both had constant 20-30% CPU showing up as kernel time and no active visible PID's driving it. This appears to have resolved that problem as well. I started using the integrity plugin instead I been meaning to setup anyway. I'm rather happy I knocked out both problems.
January 14, 201610 yr Community Expert I have been using Checksum since the beginning and have no issues.
January 14, 201610 yr I have noted that the checksum plugin can generate numerous messages if you have files on the system whose owner has no entry in the passwd file.
November 10, 20169 yr This same exact thing just happen to me. I had it installed but never set it up and then all of a sudden I started getting errors that I was out of disk space. As soon as I uninstalled it, boom, back to normal. Was there ever a fix for this? Sorry for bringing up a dead thread but it just saved me :-) With Plugin rootfs 8094684 8094684 0 100% / Plugin Uninstalled rootfs 8094684 917628 7177056 12% /
November 10, 20169 yr This same exact thing just happen to me. I had it installed but never set it up and then all of a sudden I started getting errors that I was out of disk space. As soon as I uninstalled it, boom, back to normal. Was there ever a fix for this? Sorry for bringing up a dead thread but it just saved me :-) With Plugin rootfs 8094684 8094684 0 100% / Plugin Uninstalled rootfs 8094684 917628 7177056 12% / Checksum is a pig, and while myself (and trurl and others) continue to use it and not have any issues, other people have nothing but a ton of problems. Major reason why I abandoned it (unless a bug affects myself) - Too much of a pita to figure out what was going on for those people.
Archived
This topic is now archived and is closed to further replies.