6b12 Update - UI Becomes Inaccessible

[meep]

Updated to 6b12 today. Love the UI (though black is a bit hard to read?)

 

Anyway, after an hour or so, I tried to connect to the web UI (having successfully done so earlier) but failed to do so (browser reports 'cannot connect to server'). I cleared cache etc. but no joy.

 

I tried this but no luck;

 

killall emhttp
nohup /usr/local/sbin/emhttp &

 

seems the relevant part from log files is this (full log attached);

 

Dec  6 21:27:58 UnRaid sshd[16479]: error: key_load_public: invalid format
Dec  6 21:28:01 UnRaid sshd[16479]: Accepted password for root from 192.168.1.6 port 51851 ssh2
Dec  6 21:42:00 UnRaid emhttp: unRAID System Management Utility version 6.0-beta12
Dec  6 21:42:00 UnRaid emhttp: Copyright (C) 2005-2014, Lime Technology, LLC
Dec  6 21:42:00 UnRaid emhttp: shcmd (1): mkdir -p /boot/config/plugins
Dec  6 21:42:00 UnRaid emhttp: shcmd (2): mkdir -p /boot/config/shares
Dec  6 21:42:00 UnRaid emhttp: rdevSpundown.0 not found
Dec  6 21:42:00 UnRaid kernel: emhttp[18224]: segfault at 0 ip 00002b5c9eaf3527 sp 00007fff6d80c700 error 4 in libc-2.17.so[2b5c9eab8000+1bf000]

 

 

Any pointers appreciated.

 

Peter

syslog-2014-12-06.txt

[BRiT]

I tried this but no luck;

 

killall emhttp
nohup /usr/local/sbin/emhttp &

 

That hasn't worked since the unRAID 5.x series; emhttp is no longer restartable. Any time you have to do that, you're better off restarting the server.

[johnodon]

Not to highjack but the subject line of meep's thread was appropriate for my situation...

 

My unraid webui just became unresponsive while the webuis for nzbget and nzbdrone were fine.  It eventually came back but I see these in the syslog now:

 

Dec 6 13:29:34 unRAID kernel: device vethe73cee5 left promiscuous mode
Dec 6 13:29:34 unRAID kernel: docker0: port 4(vethe73cee5) entered disabled state
Dec 6 17:20:12 unRAID emhttp: read_line: client closed the connection
Dec 6 17:20:12 unRAID emhttp: read_line: client closed the connection
Dec 6 17:20:12 unRAID emhttp: read_line: client closed the connection
Dec 6 17:20:12 unRAID emhttp: read_line: client closed the connection

 

EDIT:  It just did it again...

 

Dec 6 17:24:54 unRAID emhttp: read_line: client closed the connection
Dec 6 17:24:54 unRAID emhttp: read_line: client closed the connection
Dec 6 17:24:54 unRAID emhttp: read_line: client closed the connection
Dec 6 17:24:54 unRAID emhttp: read_line: client closed the connection
Dec 6 17:24:54 unRAID emhttp: read_line: client closed the connection
Dec 6 17:24:54 unRAID emhttp: read_line: client closed the connection
Dec 6 17:24:54 unRAID emhttp: read_line: client closed the connection
Dec 6 17:24:54 unRAID emhttp: read_line: client closed the connection

[PeterB]

I experienced an unresponsive gui earlier.  When I looked at ps, I could see that there were three copies of emhttp running.  I killed the two 'rogue' ones, which had zero accumulated cpu time, and my browser window came back to life.

 

Should there be multiple copies of emhttp running?  What could give rise to that?

[jonp]

Ok, couple of things on this, because an inaccessible webGui can be the source of many different issues.

 

First and foremost, what plugins, if any, do each of you have installed and can you confirm that when you boot into safe mode, these issues still persist?

[meep]

I had no plug ins at all - plain vanilla beta12 in xen mode.

 

Thankfully, there has been no reoccurrence of this issue

[jonp]

I had no plug ins at all - plain vanilla beta12 in xen mode.

 

Thankfully, there has been no reoccurrence of this issue

 

That's good to hear.  Unless I see more activity in this thread from folks with no plugins installed showing the same issue, I think we'll close this one.

[PeterB]

This has just happened again - I found that two copies of emhttp were running.

 

Since this only happens sporadically, I cannot leave my server running without plugins (especially apcupsd, in view of the frequency of power cuts here) for any length of time.

 

Since meep has experienced the issue with no plugins, I think that this does warrant investigation.

 

I ask again - what could give rise to two copies of emhttp running and would that be a normal occurrence?  What makes you think that it could be related to plugins?

[BRiT]

This has just happened again - I found that two copies of emhttp were running.

 

Since this only happens sporadically, I cannot leave my server running without plugins (especially apcupsd, in view of the frequency of power cuts here) for any length of time.

 

Since meep has experienced the issue with no plugins, I think that this does warrant investigation.

 

I ask again - what could give rise to two copies of emhttp running and would that be a normal occurrence?  What makes you think that it could be related to plugins?

 

Do you have anything in your /boot/config/stop script?

 

The only possibly way I can think of is somehow /etc/rc.d/rc.local script (which then triggers your 'go' file which then triggers 'emhttp') is being invoked multiple times. Maybe the system was started and running fine, then a power blip triggered the shutdown process [rc.K] but before it was fully shut down the power was restored, and the system was then put through the startup process [rc.M].

 

Would a power blip trigger the /etc/rc.d/rc.local script indirectly?

 

[PeterB]

This has just happened again - I found that two copies of emhttp were running.

 

Since this only happens sporadically, I cannot leave my server running without plugins (especially apcupsd, in view of the frequency of power cuts here) for any length of time.

 

Since meep has experienced the issue with no plugins, I think that this does warrant investigation.

 

I ask again - what could give rise to two copies of emhttp running and would that be a normal occurrence?  What makes you think that it could be related to plugins?

 

Do you have anything in your /boot/config/stop script?

 

I wasn't even aware that there was an option of a 'stop' script.  I don't have one.

 

The only possibly way I can think of is somehow /etc/rc.d/rc.local script (which then triggers your 'go' file which then triggers 'emhttp') is being invoked multiple times. Maybe the system was started and running fine, then a power blip triggered the shutdown process [rc.K] but before it was fully shut down the power was restored, and the system was then put through the startup process [rc.M].

 

I guess that is a possibility.  As many are aware, I suffer frequent power cuts and frequently start the generator before apcupsd triggers a shutdown/powerdown.  However, when I do that, I do restore power well before the five minute runtime, configured in apcupsd, expires, so it would imply that apcupsd does something before the full powerdown is intiated.

 

Would a power blip trigger the /etc/rc.d/rc.local script indirectly?

 

That would suggest that the UPS, with built-in AVR, is not doing its job properly.

 

This morning, I encountered the problem after the system had only been up and running for 30 minutes, following a 2 hour power cut, so the syslog is relatively short.  Is there anything that I should look for in the syslog?

 

Edit to add:

'There is certainly only one emhttp 'announce' message in the syslog.

 

Even now, I find that there are two emhttp processes:

root@Tower:~# ps -eaf | grep emhttp
root      4099     1  0 11:00 ?        00:00:17 /usr/local/sbin/emhttp
root      9881     1  0 11:10 ?        00:00:00 /usr/local/sbin/emhttp
root     15095  6919  0 12:45 pts/0    00:00:00 grep emhttp

 

There is nothing in the syslog at the time the second process started:

Dec 13 11:01:15 Tower kernel: docker0: port 1(veth7a87cd3) entered forwarding state
Dec 13 11:01:15 Tower kernel: docker0: port 2(veth343dacd) entered forwarding state
Dec 13 11:01:16 Tower kernel: docker0: port 3(veth63527c5) entered forwarding state
Dec 13 11:01:17 Tower kernel: docker0: port 4(veth2a439b6) entered forwarding state
Dec 13 11:16:27 Tower kernel: mdcmd (23): spindown 4
Dec 13 11:16:27 Tower kernel: mdcmd (24): spindown 5

 

Is this normal???

[PeterB]

Okay, a little bit of experimenting has revealed that any action on the webgui which raises a pop-up window will spawn another emhttp process, but this normally vanishes when the pop-up is closed, and owned by the parent.  So, what circumstances might cause the new task to become persistent, disowned by the parent (owner = process 1) and block operations on the main window?  Perhaps a problem while installing a plugin???

[PeterB]

I've just had this happen.  I clicked on the 'Log' button on top menu bar, next to 'Info', and nothing happened - no new browser pop-up window.  However, I immediately did a ps:

root@Tower:~# ps -eaf | grep emhttp
root      4084     1  1 Dec14 ?        00:08:26 /usr/local/sbin/emhttp
root     15429     1  0 00:28 ?        00:00:00 /usr/local/sbin/emhttp
root     18972 17295  0 00:34 pts/0    00:00:00 grep emhttp
root@Tower:~# 

 

Note that I now have two emhttp processes owned by process 1.  I clicked on the 'Log' again and now the webgui fails to respond.  Starting a new browser window and entering 'tower' also results in an unresponsive, blank, browser window.

 

Powerdown time!

[Super_Chicken]

I'm having the same issue. A friend of mine gave me his unraid server and went and updated everything before giving me it. It was on v6 10 and moved up to 12. Now after running the go script the tower:80 will stop working after about 20 minutes. 8080 will continue to work no problem, along with shares and everything else.

[spants]

I have had a problem with the UI crashing and restarting has no effect.

The only way to cure it (for me) is to delete the dynamix.cfg file on the boot/config/plugins/dynamix directory

[anthonws]

Same issue also occurs sporadically (haven't determined why and when).

 

I'm using 6.0-beta12 with no plugins (free version) with only 2 disks, and Samba shares, nothing else.

 

Thanks,

anthonws

[HellDiverUK]

Same issue here, seems to happen when Mover is running.  WebGUI works perfectly when Mover isn't running, then Mover seems to hog CPU (why I don't know) which makes emhttp segfault.

[jonp]

The description for this issue is so vague that I think anyone could reply with, "yeah, that happens to me sometimes too."

 

The WebUI for unRAID in it's current state does have situations in which it will not respond to other HTTP requests.  When the array is being started / stopped is a good example of that.  If no such events are occurring and traversing to the WebUI doesn't seem to work, there are a number of things to try next:

 

• Can you still connect to user shares over the network?
  
• When attempting to login to the webUI using a browser, are you using the IP address of the unRAID server or just the hostname?  Try the IP and see if that works.
  
• Can you connect via SSH via hostname or IP address?
  
• How much memory do you have in your system?  If < 4GB and you're using plugins / containers, you might consider adding more.
  
• What plugins are you using?  Plugins that mess with core functionality (SNAP, Powerdown, etc.) are not supported and may cause issues.  Use at your own risk.
  

 

@ HellDiverUK, please share a syslog with the segfault in a new topic and we can investigate your issue.

[Squid]

The description for this issue is so vague that I think anyone could reply with, "yeah, that happens to me sometimes too."

 

The WebUI for unRAID in it's current state does have situations in which it will not respond to other HTTP requests.  When the array is being started / stopped is a good example of that.  If no such events are occurring and traversing to the WebUI doesn't seem to work, there are a number of things to try next:

 

• Can you still connect to user shares over the network?
  
• When attempting to login to the webUI using a browser, are you using the IP address of the unRAID server or just the hostname?  Try the IP and see if that works.
  
• Can you connect via SSH via hostname or IP address?
  
• How much memory do you have in your system?  If < 4GB and you're using plugins / containers, you might consider adding more.
  
• What plugins are you using?  Plugins that mess with core functionality (SNAP, Powerdown, etc.) are not supported and may cause issues.  Use at your own risk.
  

 

@ HellDiverUK, please share a syslog with the segfault in a new topic and we can investigate your issue.

 

Just had this happen to me, and I *might* be able to shed some light on this (at least as it pertains to my situation), and my backstory.

 

Firstly, to answer JonP's questions,

 

I can still connect to user shares.  I generally use the hostname to connect, but the same issue (sortof) happens if I connect via the IP address (more on this later)  I can putty into the system no problems via either the IP or hostname.  I'm have 8 Gig of memory, all the Dynamix plugins (except S3 sleep) and powerdown.  My dockers are listed in the sig.

 

I've attached a syslog.

 

These lines in the log tend to show up (but not always) when the UI is locked up.

 

Jan 20 23:05:28 Server_A emhttp: read_line: client closed the connection

 

 

My backstory

 

I had a hard drive redball on me.  I know exactly why it did, and I'm not concerned.  Its currently rebuilding onto itself (the segfaults in the syslog are when I hotplugged the drive back into the backplane with the array stopped - not sure why it segfaulted doing that, but since its the only time the system has ever segfaulted, I'm not concerned)

 

GUI Lockups

 

Right now I can consistently lockup the GUI.  All I have to do is go to Tools / System Log, and the UI is pooched.  Will not respond to anything.  Also, will not reload the tab, will not open in a new tab, etc.

 

My investigation

 

- I have never had a problem going into Tools / System Log prior to this.  It is only locking up the GUI while the drive is being rebuilt.  (Never tried it though during a parity check)

- Like I said earlier, everything else (shares, putty, etc) is working perfectly

- At no time have during the investigation is there another copy of emhttp running

- If I close the tab that the GUI is locked up on, and try and open another one it won't load.  BUT, if I open another tab and connect to the UI via it's IP address (assuming that the locked up tab was using the hostname) everything is good. I can also do this in the reverse order - lock it up via the IP and still be able to access via the hostname

- If I completely close my browser and then re-open it, I can access the UI via the hostname again, and all is good again until I try and access Tools / Syslog

- I've tried this with the Docker service running AND completely shutdown

 

Conclusion (at least as far as it pertains to me)

 

The UI is NOT locking up.  Something is going on with Chrome.  The messed up thing is that this is only happening to me during a drive rebuild.  And during that process, I can consistently lock it up.  If I'm not rebuilding, the system log works perfectly.

 

Should this be happening?  Of course not, and there is a defect in there somewhere, but the simple solution is to just close Chrome and reload it.  (But, at the end of the day, I still cannot grab the syslog through the UI during a rebuild)

 

N.B.  The "Log" button on the menu bar will not lock up the UI for me.  Only Tool/System Log

 

syslog.zip

[jonp]

The description for this issue is so vague that I think anyone could reply with, "yeah, that happens to me sometimes too."

 

The WebUI for unRAID in it's current state does have situations in which it will not respond to other HTTP requests.  When the array is being started / stopped is a good example of that.  If no such events are occurring and traversing to the WebUI doesn't seem to work, there are a number of things to try next:

 

• Can you still connect to user shares over the network?
  
• When attempting to login to the webUI using a browser, are you using the IP address of the unRAID server or just the hostname?  Try the IP and see if that works.
  
• Can you connect via SSH via hostname or IP address?
  
• How much memory do you have in your system?  If < 4GB and you're using plugins / containers, you might consider adding more.
  
• What plugins are you using?  Plugins that mess with core functionality (SNAP, Powerdown, etc.) are not supported and may cause issues.  Use at your own risk.
  

 

@ HellDiverUK, please share a syslog with the segfault in a new topic and we can investigate your issue.

 

Just had this happen to me, and I *might* be able to shed some light on this (at least as it pertains to my situation), and my backstory.

 

Firstly, to answer JonP's questions,

 

I can still connect to user shares.  I generally use the hostname to connect, but the same issue (sortof) happens if I connect via the IP address (more on this later)  I can putty into the system no problems via either the IP or hostname.  I'm have 8 Gig of memory, all the Dynamix plugins (except S3 sleep) and powerdown.  My dockers are listed in the sig.

 

I've attached a syslog.

 

These lines in the log tend to show up (but not always) when the UI is locked up.

 

Jan 20 23:05:28 Server_A emhttp: read_line: client closed the connection

 

 

My backstory

 

I had a hard drive redball on me.  I know exactly why it did, and I'm not concerned.  Its currently rebuilding onto it self (the segfaults in the syslog are when I hotplugged the drive back into the backplane with the array stopped - not sure why it segfaulted doing that, but since its the only time the system has ever segfaulted, I'm not concerned)

 

GUI Lockups

 

Right now I can consistently lockup the GUI.  All I have to do is go to Tools / System Log, and the UI is pooched.  Will not respond to anything.  Also, will not reload the tab, will not open in a new tab, etc.

 

My investigation

 

- I have never had a problem going into Tools / System Log prior to this.  It is only locking up the GUI while the drive is being rebuilt.  (Never tried it though during a parity check)

- Like I said earlier, everything else (shares, putty, etc) is working perfectly

- At no time have during the investigation is there another copy of emhttp running

- If I close the tab that the GUI is locked up on, and try and open another one it won't load.  BUT, if I open another tab and connect to the UI via it's IP address (assuming that I locked up tab was using the hostname) everything is good. I can also do this in the reverse order - lock it up via the IP and still be able to access via the hostname

- If I completely close my browser and then re-open it, I can access the UI via the hostname again, and all is good again until I try and access Tools / Syslog

- I've tried this with the Docker service running AND completely shutdown

 

Conclusion (at least as far as it pertains to me)

 

The UI is NOT locking up.  Something is going on with Chrome.  The messed up thing is that this is only happening to me during a drive rebuild.  And during that process, I can consistently lock it up.  If I'm not rebuilding, the system log works perfectly.

 

Should this be happening?  Of course not, and there is a defect in there somewhere, but the simple solution is to just close Chrome and reload it.  (But, at the end of the day, I still cannot grab the syslog through the UI during a rebuild)

 

N.B.  The "Log" button on the menu bar will not lock up the UI for me.  Only Tool/System Log

Awesome write up squid. I am adding this to the task list to try and recreate and diagnose. Thanks for the detail.

[Squid]

Just for kicks, I tried doing the same thing via Internet Explorer, and it does NOT lock up.  Definitely a chrome issue

[Squid]

Gets a hair further complicated.

 

So like I said during the rebuild, Chrome would not load Tools / Syslog.  Internet Explorer would.  After the rebuild was finished, Chrome would still not load the syslog.  A reset of my desktop didn't help.  I had to reset the server in order to be able to access the syslog with chrome.

 

And unfortunately now that the issue is fixed, my investigation is at an end   I'm not going to purposely redball a drive to figure out more of this.

[loady]

Not that anyone's interested.... I have been getting kicked out all the time.. I have dockers running pms,  deluge, couch potato, sabnzbd,  nzb drone. .plugins I have are snap for v6 although it's running I'm not using it... just completed yet another 9 hour parity check after last hard reset, deluge is downloading stuff, been doing other bits..bang,  an hour later it's bombed out again.. I was told in another thread of mine to connect up a monitor to get the sys log,  cant do anything...its blank..maybe it should have been connected prior to the lockup ??...its have no gui access.. no putty access no shares access in windows...i don't have most peoples level of expertise here so I'm wading out in deep water with this problem and slowly giving up my unraid experience..again..im having to hard shut down.

[DoeBoye]

Not that anyone's interested.... I have been getting kicked out all the time.. I have dockers running pms,  deluge, couch potato, sabnzbd,  nzb drone. .plugins I have are snap for v6 although it's running I'm not using it... just completed yet another 9 hour parity check after last hard reset, deluge is downloading stuff, been doing other bits..bang,  an hour later it's bombed out again.. I was told in another thread of mine to connect up a monitor to get the sys log,  cant do anything...its blank..maybe it should have been connected prior to the lockup ??...its have no gui access.. no putty access no shares access in windows...i don't have most peoples level of expertise here so I'm wading out in deep water with this problem and slowly giving up my unraid experience..again..im having to hard shut down.

Are you using Chrome browser like Squid  said? If so, did you try Internet Explorer?

[reluctantflux]

So, I was having a similar issue and this is what I found in my scenario.

 

Symptoms:

Gui would hang for 5-10 minutes, or indefinitely

Shares were accessible

VM's stayed up (on non-array snaps)

All dockers were still accessible through their webpages.

Basically, it was just the unraid GUI that was hanging, which proved to be very frustrating.

 

Well, as it turned out, it was because my syslog was huge!  18MB (almost 200,000 lines) .  How'd this happen?  Well, I wanted external access so I setup port forwards for both the web and ssh (with very secure passwords).  Well, that doesn't stop others from pounding on your door like maniacs, and getting every attempt logged in your syslog.

 

Even after I disabled the port forwarding, I had to go and clear my syslog in order for the webgui to return to its prior snappy performance.

 

So yeah, 2 lessons. 

First, If your webgui is unresponsive, but everything else is fine, check your syslog.  It could be huge.

And second, if you really want external access, don't be stupid like me and just take the time to setup a VPN connection. 

 

[JonathanM]

if you really want external access, don't be stupid like me and just take the time to setup a VPN connection.

This. Unraid is not meant to be exposed to the internet. It can be hardened to some extent, but it's much easier to set up a VPN than it is to worry about all the little details like this that crop up when you expose it to the world. Also, setting up a private VPN does NOT mean buying a VPN service. I've seen more than a few problems on this forum where someone thought VPN internet access from a privacy provider meant they were securing their server when they were actually doing the exact opposite by opening up a tunnel to the internet in general.