Last Thursday, I saved a DVD iso file (generated by ImgBurn from a DVD video folder on a PC and saved directly to my Media Server (spec below)).  There was no indication of a problem when doing this.  We then watched the movie and half-way through, it just stopped playing.  After a bit of testing to see if the problem was in the viewing setup and determining that it was not, I went and checked the server logs and found that there had been issues starting with a OOM error. 

 

I have attached the diagnostics file with many, many error messages near the end of the syslog file and those lines begin with this:

 

Feb  2 13:32:07 Elsie1 kernel: cpuload invoked oom-killer: gfp_mask=0x24200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0

 

Now, a bit of background.  The MB, CPU and memory are all new with the upgrade being done about January 10th.  This new hardware has performed with any issues except for this one incident.  The server is currently running version 6.3.0rc6 and the server had been up and running for over eleven days when this issue occurred.  (I did not upgrade because I waiting to see if there is an under-laying issue which should be addressed first--  The old change only one-thing-at-a-time principal!)  I checked all of the SMART reports and the only thing I saw  was the *54 *  UMDA CRC errors (#199) on Disk 5 which have been there since before this incident happened.  I have most of the Dynamix plugins, Community Apps, Fix Common Problem, Tips and Tweaks, and NerdPack plugins installed.  No Dockers or VM's. 

 

I have tried to generate the problem again and have not been successful.  I have copied several more movies (again using ImgBurn to generate iso's) to the server and all of the transfers were successful.  The syslog has stayed 'clean'. 

 

Does anyone have any thoughts on what might be the problem or this one of those gremlins that is a one of a kind thing? 

elsie1-ERRORSdiagnostics-20170202-1503.zip

What exactly is the cpuload process? I can't find any reference to it on the Internet. Is it something written by LT? Ah, wait - it lives in /usr/local/sbin and it's a bash script and it says it's used by the Dynamix Dashboard page.

 

I'm afraid I don't have the experience to be able to interpret the error log as well as I would like but with a configuration as straightforward as yours and 8 GB it shouldn't be running out of memory.

What exactly is the cpuload process? I can't find any reference to it on the Internet. Is it something written by LT? Ah, wait - it lives in /usr/local/sbin and it's a bash script and it says it's used by the Dynamix Dashboard page.

 

I'm afraid I don't have the experience to be able to interpret the error log as well as I would like but with a configuration as straightforward as yours and 8 GB it shouldn't be running out of memory.

 

I agree with your conclusion about it being a simple straightforward NAS box and 8GB is more than enough memory for this configuration.  I was hoping that someone with a whole more knowledge than myself might have some insight into what happened...

 

A bit more that I forgot to include in the original post.  First, the time in the syslog when all of these errors messages are occurring is the approximate time (only because I didn't actually note the exact time) when the DVD iso is being written.  And I was not accessing the GUI at that point as this was part of a simple routine to store yet another movie on the server which is something that I have done a minimum of 800 times in the past without a problem!

 

The write method used for both of may servers is the 'Reconstruct Write' method to get high write speeds without requiring a cache drive.  (Only cost is a few pennies a year for a (perhaps) a KWH for the extra power of spinning up all of the drives during the actual write.)

I've recently also been getting some unexplained OOM errors, and after some research it appears that the problem is more a case of fragmented RAM than not enough memory, the problem happens when the kernel tries allocate RAM to cache writes, I've since changed the vm.dirty_background_ratio and vm.dirty_ratio from default to very low values, 1 and 2 respectively, and no more issues as of yet, although it's still early to be positive this is helping.

I've recently also been getting some unexplained OOM errors, and after some research it appears that the problem is more a case of fragmented RAM than not enough memory, the problem happens when the kernel tries allocate RAM to cache writes, I've since changed the vm.dirty_background_ratio and vm.dirty_ratio from default to very low values, 1 and 2 respectively, and no more issues as of yet, although it's still early to be positive this is helping.

 

Interesting!  (Note for those wondering where these settings are:  You have to install the Tips and Tweaks plugins and you will find them under the 'Tweaks' tab.)  My settings are the defaults for both of these.  Do you know of any disadvantages to setting them as low as 1% and 2%?  The 'Help' seems to indicate that there is none..

 

How many times did has this happened to you? And I wonder from your description if it is something that is happening when a server has been up for a very long time and whatever process that is use by the kernel to clean up memory isn't doing its job properly.  What versions did you see it with?  (Perhaps, the issue has already been addressed...)

I notice that you have Dynamix cache-dirs plugin installed, Frank. I wonder if that's significant.

 

IIRC it happened 4 or 5 times, all recently, I suspect it's kernel related, I only write to the array once a week, by using the mover, I believe that it mainly happens if there are no writes for a long time and the memory gets fragmented and when the kernel can't allocate it for cache there is an OOM, but just a theory for now.

 

The disadvantage of lowering the dirty ratio is lech RAM used for cache, practically none with my settings, but since all my array writes are done by the mover and I don't need RAM cache when writing to the cache pool I don't really care, and if this is the issue it may still be OK with higher values, like 5/10.

I read that Guide that one is to read before changing any of 'dirty' parameters and here is the output of the command to see what all of the settings are:

 

Command

sysctl -a | grep dirty

 

Output

vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200

 

In doing a (tiny) bit of research, it appears that a ratio of 10% may be more valid setting when one has a small amount of RAM (say 2GB) then when one has a large amount (say 32GB).  What having the smaller value does is simply forces the write to begin before the vm.dirty_expire_centisecs variable forces the write to start anyway.  What is the problem with this?  (By the way, these defaults appear to the defaults from the Linux developers-- not the unrAID developers.)  I am wondering with moderns fast processors, fast hard disks, and copious amounts of RAM, why have such high values.  I also found that apparently that are some other folks (non-unRAID users) who are also having OOM issues!  (The only rational that I saw for delaying the write is for an application that continually rewrites the same block of storage with different information.  That way there may only be one actual disk write for five changes in the actual data-- the latest one.)

 

About Cache Directories.  I have it installed because it seems to make my Media players seem more responsive to when opening up a new folder and having the file names come up quicker.  Perhaps, I will need to revisit that again... 

 

Here are my values. I'm not running Cache Directories:

 

root@Lapulapu:~# sysctl -a | grep dirty
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 2
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 5
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
root@Lapulapu:~#

 

EDIT: Sorry, I should add that these are stock values, untweaked.

 

Here are my values. I'm not running Cache Directories:

 

root@Lapulapu:~# sysctl -a | grep dirty
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 2
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 5
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
root@Lapulapu:~#

 

EDIT: Sorry, I should add that these are stock values, untweaked.

 

My 'Disk Cache 'vm.dirty_background_ratio' (%):' is set to 10%

      and

My 'Disk Cache 'vm.dirty_ratio (%):' is set to 20%  on both of my servers. 

 

When I clicked on the 'Default' button, these values remained unchanged.  The Media server is running 6.3.0rc6 and the Test Bed server is running 6.3.0.  If you did not tweak them, I wonder what changed from the values that Tips and Tweaks says should be the defaults? 

 

Perhaps, the unRAID developers saw fit to adjust them from the Linux (and the Tips and Tweaks') defaults for some reason.  I know that loading Tips and Tweaks at startup is going to set them to whatever its defaults are.  Do you have a setup without Tips and Tweaks that you could double check against?

 

 

Well, I've checked my /boot/config/plugins/tips.and.tweaks/tips.and.tweaks.cfg file and it reads:

 

FLOW_CONTROL="no"

TSO_OFFLOAD="no"

ETHERNET_NICS="eth0"

RX_BUFFER="256"

TX_BUFFER="256"

VM_BACKGROUND="2"

VM_DIRTY="5"

CPU_GOVERNOR="ondemand"

START_TIME=""

STOP_TIME=""

FTP_TELNET="no"

ARCHIVE_LOGS="no"

LOGSAVE=""

 

so maybe I did change it from the defaults, after all. Someone must have made a convincing argument! I haven't got a server without Tips & Tweaks to compare it with.