When I try to update Plugins or Update OS, nothing happens.

What ever is wrong also appears to effect the Disk log popup window.

 

 

Edited August 3, 20178 yr by csmccarron

Post your diagnostics after you attempt to do that again.

Whitelist and/or exclude the GUI from any ad-blocker or anti-virus you are running.

 

On the unraid server I get

 

502 Bad Gateway

nginx

 

So it appears to be broke on the server also.

 

GO file is as follows:

#!/bin/bash
# Start the Management Utility
echo TERM=xterm >>/root/.bash_profile
echo PATH=.:$PATH:/usr/local/sbin:/boot/bin >>/root/.bash_profile

cp /boot/config/group /etc/group
chmod 644 /etc/group

ln -s /mnt/user/Apps/OpenVPN /root/openvpn
ln -s /mnt/user/Games/Steam/Server /root/steam
ln -s /mnt/user/Apps/Unifi /root/unifi
ln -s /mnt/user/Apps/Raspberry_Pi3/rpitc /root/rpitc
ln -s /mnt/user/Apps/UnRaid/kernel /root/kernel
ln -s /mnt/user/Dockers/Ark /root/Ark

 

/usr/local/sbin/emhttp &
 

 

 

Edited August 3, 20178 yr by csmccarron

Diagnostics attached

 

mccserverur01-diagnostics-20170807-1013.zip

I have a feeling that its all related to this:

Aug  6 23:15:03 mccserverur01 kernel: CPU 4/KVM: page allocation failure: order:4, mode:0x160c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null)

 

The majority of your syslog is filled with call trace after call trace all dealing with page allocation failures.  Not quite sure about that.  First guess is that you're using memory ballooning on the VM

 

But the actual problem with the display being blank is this:

Aug  7 10:11:58 mccserverur01 nginx: 2017/08/07 10:11:58 [error] 5108#5108: *59953 connect() to unix:/var/run/emhttpd.socket failed (11: Resource temporarily unavailable) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /logging.htm?cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkos&csrf_token=7ECF364FCD02DFB8 HTTP/1.1", upstream: "http://unix:/var/run/emhttpd.socket:/logging.htm?cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkos&csrf_token=7ECF364FCD02DFB8", host: "localhost", referrer: "http://localhost/Tools/Update"

 

 

So how do I fix the nginx socket issue?  

 

Any suggestions on further reading on the page allocation failure issue?

Edited August 7, 20178 yr by csmccarron

22 minutes ago, csmccarron said:

Any suggestions on further reading on the page allocation failure issue?

 

Best guess off the top of my head is to disable memory ballooning on the vm (ie: minimum memory and maximum memory are set to be the same).  Beyond that since I've never had the problem, I can't particularly give much more advice.

 

This seems to explain what a page allocation failure actually is though (but much is over my head on a quick skim): https://utcc.utoronto.ca/~cks/space/blog/linux/DecodingPageAllocFailures

The VM's have been operating normally past 8 days.  As for the min/max memory both are set like this:

 

  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
 

Any idea on how to fix the nginx socket issue? 

1 hour ago, csmccarron said:

Any idea on how to fix the nginx socket issue? 

 

Try to restart fpm.

/etc/rc.d/rc.php-fpm restart

 

Same problem, on the unRaid console if it try to update plugins or update unraid OS I get 502 Bad Gateway nginx.

 

And this in the syslog:

Aug  7 13:50:51 mccserverur01 nginx: 2017/08/07 13:50:51 [error] 5108#5108: *60728 connect() to unix:/var/run/emhttpd.socket failed (11: Resource temporarily unavailable) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /logging.htm?cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkall&csrf_token=7ECF364FCD02DFB8 HTTP/1.1", upstream: "http://unix:/var/run/emhttpd.socket:/logging.htm?cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkall&csrf_token=7ECF364FCD02DFB8", host: "localhost", referrer: "http://localhost/Plugins"

I just rebooted the server, and everything is working again.   Can you also restart the nginx process?

 

I believe the issues was with the emhttpd process.  Prior to reboot I could not find the process.  After reboot the process exists again and I get entries in the syslog file.

 

Aug  7 14:19:01 mccserverur01 emhttpd: req (4): cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkos&csrf_token=2212687B42011F39
Aug  7 14:19:01 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin checkos
Aug  7 14:19:07 mccserverur01 emhttpd: req (5): cmd=/webGui/scripts/disk_log&arg1=sdo&csrf_token=2212687B42011F39
Aug  7 14:19:07 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/disk_log sdo
Aug  7 14:25:21 mccserverur01 emhttpd: req (6): cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkall&csrf_token=2212687B42011F39
Aug  7 14:25:21 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin checkall

 

If this happens again, I will try and find out why the emhttpd process stopped working.

Edited August 7, 20178 yr by csmccarron

The emhttpd process just crashed on my server, attached is my diagnostics file.

 

Aug  7 14:19:01 mccserverur01 emhttpd: req (4): cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkos&csrf_token=2212687B42011F39
Aug  7 14:19:01 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin checkos
Aug  7 14:19:07 mccserverur01 emhttpd: req (5): cmd=/webGui/scripts/disk_log&arg1=sdo&csrf_token=2212687B42011F39
Aug  7 14:19:07 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/disk_log sdo
Aug  7 14:25:21 mccserverur01 emhttpd: req (6): cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkall&csrf_token=2212687B42011F39
Aug  7 14:25:21 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin checkall
Aug  7 14:42:23 mccserverur01 emhttpd: req (7): cmd=/webGui/scripts/disk_log&arg1=sdi&csrf_token=2212687B42011F39
Aug  7 14:42:23 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/disk_log sdi
Aug  7 14:42:55 mccserverur01 sshd[24599]: Accepted password for root from 10.205.1.120 port 52909 ssh2
Aug  7 15:05:41 mccserverur01 emhttpd: req (8): cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkos&csrf_token=2212687B42011F39
Aug  7 15:05:41 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin checkos
Aug  7 15:20:25 mccserverur01 emhttpd: req (9): csrf_token=2212687B42011F39&title=System+Log&cmd=%2FwebGui%2Fscripts%2Ftail_log&arg1=syslog
Aug  7 15:20:25 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
Aug  7 16:03:59 mccserverur01 sshd[2358]: Accepted password for root from 10.205.1.120 port 56329 ssh2

Aug  7 16:14:19 mccserverur01 kernel: traps: emhttpd[7524] trap divide error ip:418e6a sp:2b2684beade0 error:0 in emhttpd[400000+24000]
 

Is there anyway to restart the emhttpd process without rebooting?

 

mccserverur01-diagnostics-20170807-1620.zip

Edited August 8, 20178 yr by csmccarron

It is not normal for emhttpd to crash like that. This process isn't restartable and a reboot is required.

 

Have you done a memory test to ensure memory is okay?

 

A month ago I had 2 sticks of my original gskill memory go bad.  When the original memory went bad my system would not even boot into bios.  Once the two bad modules were identified and were removed the system was able to boot.  As a result i replaced the memory with new ddr 3 ecc server rated memory.  I then performed a memory test and the new memory checked out.

 

So I should suspect my entire system is bad because 1 process crashes unexpectantly while everything else operates without issue?  

 

I can reproduce this crash at will, is there anything I can do to capture extra debug information to figure out why it is crashing?  When it crashes the entire system will freeze for a few seconds and then everything continues.  It looks like some sort of divide by zero error to me.

Edited August 8, 20178 yr by csmccarron

15 minutes ago, csmccarron said:

 It looks like some sort of divide by zero error to me.

What does a "divide by zero error" look like?

1 / 0 = division by zero

1 minute ago, csmccarron said:

1 / 0 = division by zero

So that's what it says in the logs? I'm not sure what you are trying to say here. How can you tell what kind of error it is?

Aug  7 16:14:19 mccserverur01 kernel: traps: emhttpd[7524] trap divide error ip:418e6a sp:2b2684beade0 error:0 in emhttpd[400000+24000]

Edited August 8, 20178 yr by csmccarron

I have this as well, at which point I can no longer open the syslog from the side bar

 

 

Sep  7 16:54:57 Tower kernel: traps: emhttpd[17085] trap divide error ip:418e6a sp:2af0d215ede0 error:0 in emhttpd[400000+24000]

 

13 minutes ago, clowrym said:

I have this as well, at which point I can no longer open the syslog from the side bar

 

 

Sep  7 16:54:57 Tower kernel: traps: emhttpd[17085] trap divide error ip:418e6a sp:2af0d215ede0 error:0 in emhttpd[400000+24000]

 

 

Can you test with your server started in safemode.

 

I can yes, This isn't something I see all the time, Might be a while in safemode waiting for the error.... Happened once last week & then again yesterday

My system finally stabilized but I am not sure what fixed the issue.  It has not crashed since my previous post.  I did remove a PCI-Express USB3 card that was not working right and I changed the following in my syslinux.cfg file.

 

remove/change the following in syslinux.cfg append line:

remove:  vfio_iommu_type1.allow_unsafe_interrupts=1

change:  pci-stub-ids to vfio-pci.ids

I actually have this same problem. Only fix is to restart. Noticed it started happening around RC5 but kind of thought it was just a fluke.