Jump to content
csmccarron

Update Plugins appears to be Broke

35 posts in this topic Last Reply

Recommended Posts

What ever is wrong also appears to effect the Disk log popup window.

 

 

Disk Log.JPG

Edited by csmccarron

Share this post


Link to post

Post your diagnostics after you attempt to do that again.

Share this post


Link to post

Whitelist and/or exclude the GUI from any ad-blocker or anti-virus you are running.

 

Share this post


Link to post

On the unraid server I get

 

502 Bad Gateway

nginx

 

So it appears to be broke on the server also.

 

GO file is as follows:

#!/bin/bash
# Start the Management Utility
echo TERM=xterm >>/root/.bash_profile
echo PATH=.:$PATH:/usr/local/sbin:/boot/bin >>/root/.bash_profile

cp /boot/config/group /etc/group
chmod 644 /etc/group

ln -s /mnt/user/Apps/OpenVPN /root/openvpn
ln -s /mnt/user/Games/Steam/Server /root/steam
ln -s /mnt/user/Apps/Unifi /root/unifi
ln -s /mnt/user/Apps/Raspberry_Pi3/rpitc /root/rpitc
ln -s /mnt/user/Apps/UnRaid/kernel /root/kernel
ln -s /mnt/user/Dockers/Ark /root/Ark

 

/usr/local/sbin/emhttp &
 

 

 

Edited by csmccarron

Share this post


Link to post

I have a feeling that its all related to this:

Aug  6 23:15:03 mccserverur01 kernel: CPU 4/KVM: page allocation failure: order:4, mode:0x160c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO|__GFP_NOTRACK), nodemask=(null)

 

The majority of your syslog is filled with call trace after call trace all dealing with page allocation failures.  Not quite sure about that.  First guess is that you're using memory ballooning on the VM

 

But the actual problem with the display being blank is this:

Aug  7 10:11:58 mccserverur01 nginx: 2017/08/07 10:11:58 [error] 5108#5108: *59953 connect() to unix:/var/run/emhttpd.socket failed (11: Resource temporarily unavailable) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /logging.htm?cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkos&csrf_token=7ECF364FCD02DFB8 HTTP/1.1", upstream: "http://unix:/var/run/emhttpd.socket:/logging.htm?cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkos&csrf_token=7ECF364FCD02DFB8", host: "localhost", referrer: "http://localhost/Tools/Update"

 

 

Share this post


Link to post

So how do I fix the nginx socket issue?  

 

Any suggestions on further reading on the page allocation failure issue?

Edited by csmccarron

Share this post


Link to post
22 minutes ago, csmccarron said:

Any suggestions on further reading on the page allocation failure issue?

 

Best guess off the top of my head is to disable memory ballooning on the vm (ie: minimum memory and maximum memory are set to be the same).  Beyond that since I've never had the problem, I can't particularly give much more advice.

 

This seems to explain what a page allocation failure actually is though (but much is over my head on a quick skim): https://utcc.utoronto.ca/~cks/space/blog/linux/DecodingPageAllocFailures

Share this post


Link to post

The VM's have been operating normally past 8 days.  As for the min/max memory both are set like this:

 

  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
 

Any idea on how to fix the nginx socket issue? 

Share this post


Link to post
1 hour ago, csmccarron said:

Any idea on how to fix the nginx socket issue? 

 

Try to restart fpm.

/etc/rc.d/rc.php-fpm restart

 

Share this post


Link to post

Same problem, on the unRaid console if it try to update plugins or update unraid OS I get 502 Bad Gateway nginx.

 

And this in the syslog:

Aug  7 13:50:51 mccserverur01 nginx: 2017/08/07 13:50:51 [error] 5108#5108: *60728 connect() to unix:/var/run/emhttpd.socket failed (11: Resource temporarily unavailable) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /logging.htm?cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkall&csrf_token=7ECF364FCD02DFB8 HTTP/1.1", upstream: "http://unix:/var/run/emhttpd.socket:/logging.htm?cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkall&csrf_token=7ECF364FCD02DFB8", host: "localhost", referrer: "http://localhost/Plugins"

Share this post


Link to post

I just rebooted the server, and everything is working again.   Can you also restart the nginx process?

 

I believe the issues was with the emhttpd process.  Prior to reboot I could not find the process.  After reboot the process exists again and I get entries in the syslog file.

 

Aug  7 14:19:01 mccserverur01 emhttpd: req (4): cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkos&csrf_token=2212687B42011F39
Aug  7 14:19:01 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin checkos
Aug  7 14:19:07 mccserverur01 emhttpd: req (5): cmd=/webGui/scripts/disk_log&arg1=sdo&csrf_token=2212687B42011F39
Aug  7 14:19:07 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/disk_log sdo
Aug  7 14:25:21 mccserverur01 emhttpd: req (6): cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkall&csrf_token=2212687B42011F39
Aug  7 14:25:21 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin checkall

 

If this happens again, I will try and find out why the emhttpd process stopped working.

Edited by csmccarron

Share this post


Link to post

The emhttpd process just crashed on my server, attached is my diagnostics file.

 

Aug  7 14:19:01 mccserverur01 emhttpd: req (4): cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkos&csrf_token=2212687B42011F39
Aug  7 14:19:01 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin checkos
Aug  7 14:19:07 mccserverur01 emhttpd: req (5): cmd=/webGui/scripts/disk_log&arg1=sdo&csrf_token=2212687B42011F39
Aug  7 14:19:07 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/disk_log sdo
Aug  7 14:25:21 mccserverur01 emhttpd: req (6): cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkall&csrf_token=2212687B42011F39
Aug  7 14:25:21 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin checkall
Aug  7 14:42:23 mccserverur01 emhttpd: req (7): cmd=/webGui/scripts/disk_log&arg1=sdi&csrf_token=2212687B42011F39
Aug  7 14:42:23 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/disk_log sdi
Aug  7 14:42:55 mccserverur01 sshd[24599]: Accepted password for root from 10.205.1.120 port 52909 ssh2
Aug  7 15:05:41 mccserverur01 emhttpd: req (8): cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=checkos&csrf_token=2212687B42011F39
Aug  7 15:05:41 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin checkos
Aug  7 15:20:25 mccserverur01 emhttpd: req (9): csrf_token=2212687B42011F39&title=System+Log&cmd=%2FwebGui%2Fscripts%2Ftail_log&arg1=syslog
Aug  7 15:20:25 mccserverur01 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
Aug  7 16:03:59 mccserverur01 sshd[2358]: Accepted password for root from 10.205.1.120 port 56329 ssh2


Aug  7 16:14:19 mccserverur01 kernel: traps: emhttpd[7524] trap divide error ip:418e6a sp:2b2684beade0 error:0 in emhttpd[400000+24000]
 

Is there anyway to restart the emhttpd process without rebooting?

 

mccserverur01-diagnostics-20170807-1620.zip

Edited by csmccarron

Share this post


Link to post

It is not normal for emhttpd to crash like that. This process isn't restartable and a reboot is required.

 

Have you done a memory test to ensure memory is okay?

 

Share this post


Link to post

A month ago I had 2 sticks of my original gskill memory go bad.  When the original memory went bad my system would not even boot into bios.  Once the two bad modules were identified and were removed the system was able to boot.  As a result i replaced the memory with new ddr 3 ecc server rated memory.  I then performed a memory test and the new memory checked out.

 

So I should suspect my entire system is bad because 1 process crashes unexpectantly while everything else operates without issue?  

 

I can reproduce this crash at will, is there anything I can do to capture extra debug information to figure out why it is crashing?  When it crashes the entire system will freeze for a few seconds and then everything continues.  It looks like some sort of divide by zero error to me.

Edited by csmccarron

Share this post


Link to post
15 minutes ago, csmccarron said:

 It looks like some sort of divide by zero error to me.

What does a "divide by zero error" look like?

Share this post


Link to post
1 minute ago, csmccarron said:

1 / 0 = division by zero

So that's what it says in the logs? I'm not sure what you are trying to say here. How can you tell what kind of error it is?

Share this post


Link to post

Aug  7 16:14:19 mccserverur01 kernel: traps: emhttpd[7524] trap divide error ip:418e6a sp:2b2684beade0 error:0 in emhttpd[400000+24000]

Edited by csmccarron

Share this post


Link to post

I have this as well, at which point I can no longer open the syslog from the side bar

 

 

Sep  7 16:54:57 Tower kernel: traps: emhttpd[17085] trap divide error ip:418e6a sp:2af0d215ede0 error:0 in emhttpd[400000+24000]

 

Share this post


Link to post
13 minutes ago, clowrym said:

I have this as well, at which point I can no longer open the syslog from the side bar

 

 


Sep  7 16:54:57 Tower kernel: traps: emhttpd[17085] trap divide error ip:418e6a sp:2af0d215ede0 error:0 in emhttpd[400000+24000]

 

 

Can you test with your server started in safemode.

 

Share this post


Link to post

I can yes, This isn't something I see all the time, Might be a while in safemode waiting for the error.... Happened once last week & then again yesterday

Share this post


Link to post

My system finally stabilized but I am not sure what fixed the issue.  It has not crashed since my previous post.  I did remove a PCI-Express USB3 card that was not working right and I changed the following in my syslinux.cfg file.

 

remove/change the following in syslinux.cfg append line:

remove:  vfio_iommu_type1.allow_unsafe_interrupts=1

change:  pci-stub-ids to vfio-pci.ids

Share this post


Link to post

I actually have this same problem. Only fix is to restart. Noticed it started happening around RC5 but kind of thought it was just a fluke.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.