joedotmac Posted April 8, 2017 Share Posted April 8, 2017 (edited) Description: Dashboard CPU percentage values remain static. How to reproduce: Normal boot and start of services, start docker instances, start VM instances. Expected results: Individual and aggregate total CPU percentage use values are representative of load and vary as CPU load changes. Actual results: Shortly after all services and instances startup and systems stabilize at running state and load, the values for CPU aggregate and individual CPU cores do not change. Potentially the memory values are affected as well. The fan RPM values do change. Other information: Using Netdata plugin, can monitor core and aggregate CPU utilization. Results here indicate expected representation of load fluctuating as expected. In screenshot for example the aggregate CPU load will remain at 19% with no change. Same with cores, the represented values after some period of startup, don't change. unraid-diagnostics-20170408-1248.zip Edited April 12, 2017 by joedotmac Quote Link to comment
Frank1940 Posted April 8, 2017 Share Posted April 8, 2017 First thing to check is your 'Setting's >>> 'Display Settings' and look at the ' Page update frequency: " setting. Is it on 'Real-time'? You can also Turn on 'Help' to explain what they various settings do. Quote Link to comment
joedotmac Posted April 10, 2017 Author Share Posted April 10, 2017 I've validated the setting is on real-time. I even toggled it to regular, then back to real-time and the CPU utilization percentages still remain where they were snapshot'd in the original post. It's like whatever process that publishes these metrics as abended and no longer making updates. Quote Link to comment
joedotmac Posted April 26, 2017 Author Share Posted April 26, 2017 Tracked down the CPU utilization percentage counters stop after this in syslog. Apr 26 14:26:53 unraid kernel: notify[29245]: segfault at 0 ip 00000000005f42ad sp 00007ffcf2a4f050 error 4 in php[400000+724000] Apr 26 14:27:30 unraid kernel: php[29631]: segfault at 0 ip 00000000005f42ad sp 00007fffe09d0560 error 4 in php[4 Quote Link to comment
joedotmac Posted April 27, 2017 Author Share Posted April 27, 2017 (edited) Caught the suspect in the act. Monitor script isn't happy. Monitor seems responsible for displaying the percentages back into the GUI Apr 27 12:27:53 unraid kernel: notify[3958]: segfault at 0 ip 00000000005f42ad sp 00007ffe8846fb10 error 4 in php[400000+724000] Apr 27 12:27:57 unraid emhttp: err: sendOutput: fork failed Apr 27 12:28:01 unraid kernel: monitor[4007]: segfault at 0 ip 00000000005f42ad sp 00007ffef8d2e710 error 4 in php[400000+724000] Apr 27 12:28:01 unraid crond[1656]: exit status 139 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null Apr 27 12:28:01 unraid crond[1656]: unable to fork: cron output for user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null to /dev/null Apr 27 12:28:14 unraid kernel: php[4083]: segfault at 0 ip 00000000005f42ad sp 00007ffc292442e0 error 4 in php[400000+724000] Apr 27 12:28:17 unraid kernel: php[4090]: segfault at 0 ip 00000000005f42ad sp 00007ffe0f7ab4b0 error 4 in php[400000+724000] Edited April 27, 2017 by joedotmac Quote Link to comment
joedotmac Posted May 3, 2017 Author Share Posted May 3, 2017 Made a change the other day to disable the CPU frequency scaling step down capability as there's some issue intel_pstate driver that affects Haswell and Ivory Bridge procs. Initially looked like I had some success. Not so fast. CPU counters on the dashboard still stop working, Dynamix Auto Fan Control plugin stops, VM's stop on their own. All kinds of errors unable to trace after weeks of trying. Removing all plugins, installing Tips and Tweaks, Fix Common Problems. May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: emhttp_event: popen /usr/local/sbin/emhttp_event svcs_restarting: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: shcmd (52641): /etc/rc.d/rc.nfsd start |& logger May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: shcmd: shcmd (52641): exit status: -1 May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: shcmd (52643): /etc/rc.d/rc.atalk start |& logger May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: shcmd: shcmd (52643): exit status: -1 May 2 20:04:07 unraid emhttp: shcmd (52644): cp /tmp/emhttp/afp.service /etc/avahi/services/afp.service May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: shcmd: shcmd (52644): exit status: -1 May 2 20:04:07 unraid emhttp: err: emhttp_event: popen /usr/local/sbin/emhttp_event svcs_restarted: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:07 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:04:08 unraid kernel: php[7300]: segfault at 0 ip 00000000005f42ad sp 00007ffe9fc79f20 error 4 in php[400000+724000] May 2 20:38:50 unraid kernel: br0: port 2(vnet0) entered blocking state May 2 20:38:50 unraid kernel: br0: port 2(vnet0) entered disabled state May 2 20:38:50 unraid kernel: device vnet0 entered promiscuous mode May 2 20:38:50 unraid kernel: br0: port 2(vnet0) entered blocking state May 2 20:38:50 unraid kernel: br0: port 2(vnet0) entered forwarding state May 2 20:38:50 unraid kernel: br0: port 3(vnet1) entered blocking state May 2 20:38:50 unraid kernel: br0: port 3(vnet1) entered disabled state May 2 20:38:50 unraid kernel: device vnet1 entered promiscuous mode May 2 20:38:50 unraid kernel: br0: port 3(vnet1) entered blocking state May 2 20:38:50 unraid kernel: br0: port 3(vnet1) entered forwarding state May 2 20:38:51 unraid kernel: kvm: zapping shadow pages for mmio generation wraparound May 2 20:38:51 unraid kernel: kvm: zapping shadow pages for mmio generation wraparound May 2 20:41:42 unraid kernel: php[17524]: segfault at 0 ip 00000000005f42ad sp 00007fff6e999240 error 4 in php[400000+724000] May 2 20:41:51 unraid emhttp: err: sendOutput: fork failed May 2 20:41:51 unraid emhttp: err: sendOutput: fork failed May 2 20:41:51 unraid emhttp: err: sendOutput: fork failed May 2 20:41:57 unraid emhttp: err: sendOutput: fork failed May 2 20:41:59 unraid kernel: notify[17653]: segfault at 0 ip 00000000005f42ad sp 00007ffd3cb91810 error 4 in php[400000+724000] May 2 20:42:00 unraid kernel: php[17654]: segfault at 0 ip 00000000005f42ad sp 00007ffd1cee9f20 error 4 in php[400000+724000] May 2 20:42:00 unraid kernel: php[17655]: segfault at 0 ip 00000000005f42ad sp 00007ffc318e4cc0 error 4 in php[400000+724000] May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: emhttp_event: popen /usr/local/sbin/emhttp_event svcs_restarting: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: shcmd (53245): /etc/rc.d/rc.nfsd start |& logger May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: shcmd: shcmd (53245): exit status: -1 May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: shcmd (53247): /etc/rc.d/rc.atalk start |& logger May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: shcmd: shcmd (53247): exit status: -1 May 2 20:42:02 unraid emhttp: shcmd (53248): cp /tmp/emhttp/afp.service /etc/avahi/services/afp.service May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: shcmd: shcmd (53248): exit status: -1 May 2 20:42:02 unraid emhttp: err: emhttp_event: popen /usr/local/sbin/emhttp_event svcs_restarted: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid emhttp: err: _shcmd: system: Cannot allocate memory May 2 20:42:02 unraid kernel: php[17665]: segfault at 0 ip 00000000005f42ad sp 00007ffefbed60c0 error 4 in php[400000+724000] May 2 20:42:03 unraid emhttp: err: sendOutput: fork failed May 2 20:42:06 unraid kernel: php[17673]: segfault at 0 ip 00000000005f42ad sp 00007ffc4fec0b40 error 4 in php[400000+724000] May 2 20:42:06 unraid emhttp: err: sendOutput: fork failed May 2 20:42:09 unraid emhttp: err: sendOutput: fork failed May 2 20:42:09 unraid emhttp: err: sendOutput: fork failed May 2 20:42:15 unraid kernel: php[17689]: segfault at 0 ip 00000000005f42ad sp 00007fffd3e4e2b0 error 4 in php[400000+724000] May 2 20:42:15 unraid emhttp: err: sendOutput: fork failed May 2 20:42:18 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:18 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:21 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:21 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:22 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:22 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:23 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:24 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:24 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:27 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:27 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:30 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:30 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:34 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:34 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:37 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:37 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:38 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:40 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:40 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:43 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:42:43 unraid root: error: webGui/include/DashUpdate.php: uninitialized csrf_token May 2 20:44:11 unraid kernel: notify[18838]: segfault at 0 ip 00000000005f42ad sp 00007ffddc5ce7d0 error 4 in php[400000+724000] May 2 20:44:11 unraid emhttp: err: sendOutput: fork failed May 2 20:44:15 unraid emhttp: err: sendOutput: fork failed May 2 20:44:15 unraid emhttp: err: sendOutput: fork failed May 2 20:44:15 unraid emhttp: err: sendOutput: fork failed May 2 20:44:18 unraid emhttp: err: sendOutput: fork failed Quote Link to comment
joedotmac Posted May 3, 2017 Author Share Posted May 3, 2017 (edited) Mover finishes. Ten minutes later VM stops for reasons not established. Last error in syslog was six hours previous. May 3 03:41:43 unraid root: mover finished May 3 03:52:08 unraid kernel: br0: port 2(vnet0) entered disabled state May 3 03:52:08 unraid kernel: device vnet0 left promiscuous mode May 3 03:52:08 unraid kernel: br0: port 2(vnet0) entered disabled state May 3 03:52:08 unraid kernel: br0: port 3(vnet1) entered disabled state May 3 03:52:08 unraid kernel: device vnet1 left promiscuous mode May 3 03:52:08 unraid kernel: br0: port 3(vnet1) entered disabled state Edited May 3, 2017 by joedotmac Quote Link to comment
Frank1940 Posted May 3, 2017 Share Posted May 3, 2017 If no on else has a solution, I would suggest using the 'Tweaks' Tab of the 'Tips and Tweaks' plugin to set " Disk Cache 'vm.dirty_background_ratio' (%): " to 2% and "Disk Cache 'vm.dirty_ratio (%):" to 4%. The percentages are a default of Linux and geared to systems with a minimal amount of memory. You have 32GB and these two parameters are utilizing a huge chuck of it. (As I understand it, these parameters allocate memory so that disk writes can be queued so that disk access functions are efficient and user responsiveness effects of disk IO are minimized.) 1 Quote Link to comment
joedotmac Posted May 3, 2017 Author Share Posted May 3, 2017 Thanks Frank. Will this effectively make greater use of available RAM, lessening IO impact with the drives? I made the suggested change. Reloaded unraid for a fresh boot and initialization of all processes, docker, and VM's. Would anything here be indicative of the unraid USB flash drive integrity being questionable? Quote Link to comment
Frank1940 Posted May 3, 2017 Share Posted May 3, 2017 33 minutes ago, joedotmac said: Thanks Frank. Will this effectively make greater use of available RAM, lessening IO impact with the drives? Yes 34 minutes ago, joedotmac said: Reloaded unraid for a fresh boot and initialization of all processes, docker, and VM's. Would anything here be indicative of the unraid USB flash drive integrity being questionable? I was looking more at your memory allocation failures in your post when I made my suggestion. But with you reboot apparently without issue-- for example, no parity check on restart, I would assume that your Flash Drive is OK. You could look at your syslog ( 'Tools' >>> 'System Log' ) and then uncheck the "Text" box (gets rid of 90+% of the log entries) at the top and look for errors that might be related to your Flash Drive. 1 Quote Link to comment
joedotmac Posted May 5, 2017 Author Share Posted May 5, 2017 Thanks Frank. I made the suggested change to the syslog view to narrow down any indication of flash drive issues. I'm not seeing any thing that would indicate flash drive problem. Haven't had much of any change in mitigating of issues. VM's and plugins stop randomly, CPU percentages on dashboard still hang after 30 minutes or so of system up-time. Current snapshot of errors and warnings over the last 24 hours. I keep researching looking for a release note or some glimmer of hope to find any even small success that may lead to something more considerable. syslog050417-errors.txt Quote Link to comment
joedotmac Posted May 5, 2017 Author Share Posted May 5, 2017 Another interested error find is this. Seems whichever plugin is associated with Dynamix plugin "monitor" when it fails it seems to try and call out google page to possibly post event details to a developer support page? The creds per the error message connection attempt aren't valid so it barfs even on that. Apr 30 05:05:01 unraid kernel: monitor[17472]: segfault at 0 ip 00000000005f42ad sp 00007fff178379e0 error 4 in php[400000+724000] Apr 30 05:05:16 unraid crond[1737]: exit status 139 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null Apr 30 05:05:16 unraid sSMTP[17495]: Creating SSL connection to host Apr 30 05:05:16 unraid sSMTP[17495]: SSL connection using ECDHE-RSA-AES128-GCM-SHA256 Apr 30 05:05:16 unraid sSMTP[17495]: Authorization failed (535 5.7.8 https://support.google.com/mail/?p=BadCredentials l46sm5351894ota.0 - gsmtp) Apr 30 05:06:01 unraid crond[1737]: exit status 139 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null Apr 30 05:06:01 unraid sSMTP[17560]: Creating SSL connection to host Apr 30 05:06:01 unraid sSMTP[17560]: SSL connection using ECDHE-RSA-AES128-GCM-SHA256 Apr 30 05:06:01 unraid sSMTP[17560]: Authorization failed (535 5.7.8 https://support.google.com/mail/?p=BadCredentials w6sm5454754ota.19 - gsmtp) Apr 30 05:06:16 unraid crond[1737]: exit status 127 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &>/dev/null Quote Link to comment
Squid Posted May 5, 2017 Share Posted May 5, 2017 It would be trying to email yourself with the credentials you've set up in notification settings, but your password is incorrect. Do you receive any notifications? What happens when you test the notification settings? Quote Link to comment
joedotmac Posted May 5, 2017 Author Share Posted May 5, 2017 (edited) I had never setup the email notification settings. The default email notification settings appear to select gmail, but there was no address, creds of any type configured. The settings for notification only were selected for browser notification, email wasn't checked to engage on any of the message levels. Regardless I updated the settings to pop my email domain. Have to see if anything changes, didn't see a test button or the like. Thanks for the insight. Will be looking to see if there are any changes. Edited May 5, 2017 by joedotmac Quote Link to comment
Squid Posted May 5, 2017 Share Posted May 5, 2017 7 minutes ago, joedotmac said: I had never setup the email notification settings. The default email notification settings appear to select gmail, but there was no address, creds of any type configured. The settings for notification only were selected for browser notification, email wasn't checked to engage on any of the message levels. Regardless I updated the settings to pop my email domain. Have to see if anything changes, didn't see a test button or the like. Thanks for the insight. Will be looking to see if there are any changes. Downside of the "cron" system. Abnormal exits from a job (in this case, monitor) linux always attempts to email to let the system administrator (you) know. Overrides any setting in the GUI. Also, syslog snippets are very rarely useful, if only because there is no context to them without seeing anything else. Always post the full diagnostics. Quote Link to comment
joedotmac Posted May 5, 2017 Author Share Posted May 5, 2017 Full snapshot and syslog since May 3rd. Thanks. unraid-syslog-20170505-0955.zip Quote Link to comment
joedotmac Posted August 9, 2017 Author Share Posted August 9, 2017 Problem is now solved. Not certain what actually solved it. Removing, reinstalling all the dockers, upgrading to 6.3.5 are possiblities Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.