I know this sometimes is caused by a corrupted docker image but I am not sure what to look for to determine that. Additionally, I have had issues in the past related to BTRFS (from bad cables I replaced) but I don't see the usual mass amount of errors that I've seen when that happened previously and a scrub came back with no errors.

 

Any help in troubleshooting (and guidance on what to look for to self-diagnose in the future) would be greatly appreciated!

starkillerbase-diagnostics-20171102-0709.zip

Call traces are related to out of memory errors, these are very common on v6.3.5 and I was hoping the newer kernel on v6.4 would fix them, but it appears not since you're not the first on v6.4, @limetech maybe swap file is needed?

I am assuming that johnnie.black is correct about this being an issue on your server, I would suggest that you read the following post and the remainder of the thread:

 

  https://forums.lime-technology.com/topic/58855-regular-out-of-memory-problems/#comment-577424

 

 

I would suggest that you read the following post and the remainder of the thread:

 

Yeah, unfortunately those setting help on v6.3.x but appear to be less effective on v6.4.

 

An easy wat to create these OOM situations on v6.3.5 is using rsync, a large rsync transfer will sooner or later cause OOM errors, sometimes it needs to be really large, like several hours, most users get them after running the mover, I noticed them on my backup servers, since they are synced with rsync.

 

Mover has stopped using rsync on v6.4, so I now believe this is why we've seen less OOM issues there, but they still happen, seen a couple of instances that happens just after running CA backup, which you guessed it, uses rsync.

 

I'm not saying rsync is the culprit, but looks like it's a good way to provoke the problem, we may need to start using swap file or any other memory management improvements.

 

 

 

 

2 hours ago, johnnie.black said:

Yeah, unfortunately those setting help on v6.3.x but appear to be less effective on v6.4.

 

But I believe that there is no downside to doing this to reduce the RAM footprint of the Disk Cache function.  I really don't see where/how LimeTech can create a swap file setup that won't be 'stepping on someone's toes' at this point with its design philosophy. (i.e., that it runs from memory off a UBS flash drive with minimum writes to that boot disk.)  

 

What may have to happen is that the offending application(s) may have to be rewritten to use RAM in a more efficient manner.  And I would not be holding my breath on that happening any time soon!  

Edited November 2, 20178 yr by Frank1940

Thanks,

 

I'll give that a go since there seems to be no major downside.

13 minutes ago, Frank1940 said:

But I believe that there is no downside to doing this to reduce the RAM footprint of the Disk Cache function. 

 

That's true, no harm in trying, it may still help.