Moving Large Amount Files Locks Unraid Up

[Wavey]

Hello All

 

I was originally running Unraid on consumer hardware with 6 sata pcie cards so i assumed it was the cards themselves slowing everything down. Now Ive switched to enterprise hardware, SC846, and it’s still a problem. I’ve tried using the cache and not using the cache, tried looking up others problems but it seems to be multiple reasons. So I added my diagnostics and a netdata image to show IOWait. Is there anything i can do to fix this freezing when I’m moving 50+ GB around?

 

Netdata IOwait

https://imgur.com/a/jJ3Qs

diagnostics-20180216-0827.zip

[Frank1940]

9 minutes ago, Wavey said:

Is there anything i can do to fix this freezing when I’m moving 50+ GB around?

 First thing, I noticed in your syslog is that you are running out of memory and the OS is apparently having problems reallocating memory.  You might try the suggestion and see if it helps.

 

    https://lime-technology.com/forums/topic/58855-regular-out-of-memory-problems/#comment-577424

 

 

It helps when you have a a fair amount of RAM installed as 20% of 32GB is 6.4GB!   Especially since what it is used for is a non-issue with unRAID.  

 

If that does not fix it, tell us more about what you are doing, i.e., the what, why and how...

[Wavey]

20 minutes ago, Frank1940 said:

 First thing, I noticed in your syslog is that you are running out of memory and the OS is apparently having problems reallocating memory.  You might try the suggestion and see if it helps.

 

    https://lime-technology.com/forums/topic/58855-regular-out-of-memory-problems/#comment-577424

 

 

It helps when you have a a fair amount of RAM installed as 20% of 32GB is 6.4GB!   Especially since what it is used for is a non-issue with unRAID.  

 

If that does not fix it, tell us more about what you are doing, i.e., the what, why and how...

 

 

Giving this one a try now, thanks!

[Wavey]

39 minutes ago, Frank1940 said:

 First thing, I noticed in your syslog is that you are running out of memory and the OS is apparently having problems reallocating memory.  You might try the suggestion and see if it helps.

 

    https://lime-technology.com/forums/topic/58855-regular-out-of-memory-problems/#comment-577424

 

 

It helps when you have a a fair amount of RAM installed as 20% of 32GB is 6.4GB!   Especially since what it is used for is a non-issue with unRAID.  

 

If that does not fix it, tell us more about what you are doing, i.e., the what, why and how...

Transferring a large file now and its looking great, no spikes.

 

This is so strange though because i only run one big program, Plex, the rest are for monitoring plex I didnt think memory would ever be a problem.

 

anywho ill try moving a folder thats 500GB and see how it goes.

[Frank1940]

I seem to vaguely recall seeing that Plex can  store some of its database in memory and this can be a big memory hog.  Perhaps, someone else will jump in with a bit more information.  (even telling me I am getting senile....    )

[Wavey]

3 hours ago, Frank1940 said:

I seem to vaguely recall seeing that Plex can  store some of its database in memory and this can be a big memory hog.  Perhaps, someone else will jump in with a bit more information.  (even telling me I am getting senile....    )

Yeah Probably right. Looks like this seemed to work so far, thanks.

I have a question though. Will the settings i changed affect Plex negatively?

[Frank1940]

This memory reserved for use by the OS so that writes to disks may be delayed until the CPU has time to write them without causing a delay in other user input processes.  It was added to Windows back in the early-to-mid 1990's when the 80386 was the king of the hill.   What it did was basically is not to allow disk writes to occur while the user was typing to the screen.  Without this buffering of the write, the typed characters would stop appearing on the screen until the disk write was finished.  Needless to say, this was most disconcerting to the touch typist!  It has been around ever since without a lot of thought by the guys who maintain the OS's.  (The old 'if it ain't broke...'- syndrome.)    PS--- That is also why you have to eject USB drives before removal to guarantee that this memory is flushed to the drive before its removal.  

 

When you only had a 1MB of memory, this is 200KB and that was big enough for most user files in those days and there was no RAM cache on the disk drives.  Today, I doubt if it hardly ever gets used unless it is on transactional servers with thousands of data request per seconds.  I am also pretty sure that it a single continuous black which is why you run into problems after the server has been in use of a while.  The free space memory becomes fragmented into small segments and the system has to do housecleaning to assemble a block big to meet the request of a new block of memory.