What's filling up my syslog? Help


Recommended Posts

Unraid 6.8.0

 

Hi guys,

 

This past saturday, out of the blue, i noticed the dashboard on my server wasn't showing any cpu stats... then i noticed none of my docker stats were working, then realized my syslog was at 100%. The whole system seemed very slow and unresponsive. Fix Common Problems showed nothing out of sorts, and the dockers themselves (plex for example) all seemed to be working fine. I tried to use the diagnostics zip tool and even that wouldn't work - unresponsive. I used terminal to capture the syslog to flash manually - the wiki said it would be ready to post to forums using this method, but if it's not anonymized let me know and i'll take it down straight away. this log file is below, titled "syslog-2020-01-11.zip". Clean reboot, everything seemed to return to normal.

 

Fast forward a couple days to just a few short moments ago, i logged in to find my syslog back up to 77%. FCP still not showing any problems. diagnostic zip tool worked this time, so i snagged the logs and have attached them below as well. file is "kal-el-diagnostics-20200113-1320.zip". I glanced at the logs from today and saw a few red lines, but unfortunately the rest is greek to me. 

 

I have a couple of suspicions... i installed folding@home last week, even with cpu and memory parameters, it caused problems so i uninstalled (no good deed... lol). other possibility is i have a semi permanent wireguard connection now with other remote unraid server. but that's been setup for weeks now with no issues (until now i guess). other than that, I can't think of any changes that might be the culprit, really don't know where to start. Thanks in advance

 

-Chaz

 

kal-el-diagnostics-20200113-1320.zipsyslog-2020-01-11.zip

 

Link to comment

Thanks guys, i'll give those a shot and report back.

 

Off topic for a second, but, @johnnie.black i feel compelled to tell you how much i like your handle and avatar. not sure if you're familiar with Christopher Hitchens or not, but you might find this short youtube clip of general interest. "The best blended scotch in the history of the world..."

Edited by Cpt. Chaz
typo
Link to comment

ok, nope, that wasn't the problem after all.

 

server locked up again. wasn't able to use the diagnostic tool to capture the syslog, used terminal again to capture "syslog.txt" and i also ran a short tail and exported it "tail syslog.txt". Attached below.

 

The tail output was running with docker service disabled, so this should disable nginx, right? I could try restarting in safe mode, but i don't know how to reproduce this problem. It just happened out of the blue again. i'm seeing

# Jan 17 16:27:00 Kal-El nginx: 2020/01/17 16:27:00 [crit] 4850#4850: ngx_slab_alloc() failed: no memory
# Jan 17 16:27:00 Kal-El nginx: 2020/01/17 16:27:00 [error] 4850#4850: shpool alloc failed
# Jan 17 16:27:00 Kal-El nginx: 2020/01/17 16:27:00 [error] 4850#4850: nchan: Out of shared memory while allocating channel /var. Increase nchan_max_reserved_memory.

over and over. nginx should be disabled with letsencrypt docker closed, right? and out of shared memory? getting out of my depth here, but with 192gb of ram, only ever getting up to 7-8% utilization with no more than i have running, seems strange. it says to "increase nchan_max_reserved_memory". should i do that? if so, how?

 

It also seems i'm not alone, but they had no resolution it looks like here.

 

edit: i also increased vm dirty ratios a few days ago per this and it obviously made no difference.

tail syslog.txt.zip syslog.txt.zip

Edited by Cpt. Chaz
addition
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.