Terminal/Console clearing/disconnecting

[TurboStreetCar]

Hello, having an issue since updating to 6.10.3 where the terminal/console will blank and return to base directory randomly in a short period of time (30 seconds or so). Very briefly, like 1/100'th of a second it will flash disconnected. 

This wasn't the case before, I've had terminal windows open for hours and never disconnected. 

No errors in the log related to anything. Not sure what to start looking at to diagnose.

 

Thanks,

Dan

[TurboStreetCar]

Bump this to the top, Really annoying when trying to do work in the terminal. It resets back to base folder and loses all information currently displayed. 

What can i check to try to find the fix? No issues before updating, so i dont believe its a hardware issue.

 

[trurl]

Are you talking about the console window you get from the Unraid webUI, or are you using some telnet/ssh client such as puTTY?

[TurboStreetCar]

On 9/11/2022 at 3:51 PM, trurl said:

Are you talking about the console window you get from the Unraid webUI, or are you using some telnet/ssh client such as puTTY?

The terminal from the WebGui. 
 

I also noticed consoles opened from dockers in the docker tab also reset/reconnect often also. 
 

it appears my connection is being dropped for an instant. Never happened before update. 

[trurl]

attach diagnostics to your NEXT post in this thread

 

setup syslog server

[TurboStreetCar]

4 hours ago, trurl said:

attach diagnostics to your NEXT post in this thread

 

setup syslog server

Zip file of diagnostics attached. 

I have syslog server set up.  Dont see anything there concerning network connection issues.

I Appreciate the help!

backup-diagnostics-20220918-1438.zip

[trurl]

Since you have chosen to ignore this, tell me what it means

Sep 14 20:55:06 Backup root: Fix Common Problems: Warning: Wrong DNS entry for host ** Ignored

 

[TurboStreetCar]

@trurl I use PFSense (separate machine) with HAProxy to use wildcard SSL certs and custom domain to access the webGUI via unraid.my domain.com. 
 

I believe the error is because I access the webgui through a proxy instead of directly. 
 

I tried entering my domain in the host portion of the Unraid settings and it didn’t fix it. So I believe the proxy is the problem. 
 

Unraid documentation says it can be ignored if using a proxy. 
 

edit: I’ve been using this method of access for months and no issues. Console/terminal problem started immediately after update. 

Edited September 18, 2022 by TurboStreetCar

[Arbadacarba]

I've seen this and the solution was to clear the browser cache. Worth a try.

[TurboStreetCar]

9 hours ago, Arbadacarba said:

I've seen this and the solution was to clear the browser cache. Worth a try.

 

SO, i think ive made a discovery. I didnt clear cache yet, BUT after reading your post i did try something.

I tried accessing webGUI using a incognito chrome browser, and i cannot access it using custom domain. I can using the IP of the unraid box. 

IF i access the unraid GUI using IP number, instead of using custom domain through HAProxy, the problem is gone. I then tried it in a non-incognito browser, and also good to go. 

SO i think somehow accessing through HAproxy using custom domain is the issue for some reason. It must be HAproxy or the PFsense box, dropping connection for an instant or something.

Not sure what changed, ive been accessing it this way for months and never had this issue before. Im going to have to check PFsense log for and updates that could have been done. 

Is there any common issue accessing Unraid through proxy? 

****EDIT****: One thing i can add, is sometime when i use a bookmark to access the webGUI using my custom domain, it will redirect to a different domain at "servername.local" seemingly randomly. Not sure if that has something to do with it. Not sure why that happens?

 

Edited September 20, 2022 by TurboStreetCar

[DANgerous25]

FWIW, I also use HAproxy on PFSense to get to my Unraid GUI using myhost.mydomain.com. I get the same problem. If I connect directly using the server's IP, then no issue at all. Very strange (and frustrating).

[lostinspace]

Same issue, also using HAProxy on Pfsense. 

[TurboStreetCar]

@DANgerous25 @lostinspace Glad to hear im not alone. 

I feel like this isnt normal, but i dont see it mentioned by many people.

[apandey]

Is this on a Mac? I see this occasionally and it's something to do with dns handling. A good way to verify is by going to chrome settings and toggle private dns on then off. Then it starts resolving again. I've reproduced it many times but not figured out why

[Vodros]

I have the exact same setup and the exact same problem: HAProxy on pfSense and accessing the WebUI terminal when using the domain name makes it refresh every ~10s while accesing it using the IP solves the problem. Maybe someone from the dev team could look into this?

[Zabaron]

I'm having a similar issue.  Whether I connect via ssh or via the web terminal it always disconnects after about 30 seconds.  My setup is a bit different (not using HAProxy), but for me at least it seems related to IPv6.  If I connect via the IPv4 address the same behavior doesn't occur.  Only when connecting via IPv6 address or hostname (which presumably resolves preferentially to the IPv6 address).

[ZooMass]

I have the same issue. If I don't type anything for 30 seconds, or if I change the focused window, the Unraid web terminal (web path https://unraid.example.com/webterminal/ttyd/ ) for a split second says Reconnected and the whole terminal session resets as if in a new window.

 

I use Unraid 6.11.5, pfSense 2.7.0, and HAProxy package for pfSense 0.61_11.

I use the Unbound DNS Resolver to resolve unraid.example.com to my pfSense IP.

HAProxy frontend listens on pfSense port 80/443 on my LAN and serves a wildcard certificate for *.example.com (substituting my actual domain I own) that pfSense ACME signs with LetsEncrypt CA.

 

Here are relevant parts of my HAProxy config file on pfSense at /var/etc/haproxy/haproxy.cfg

 

frontend COM_EXAMPLE_LOCAL-merged
        bind                    192.168.1.1:443 name 192.168.1.1:443   ssl crt-list /var/etc/haproxy/CO_BEREZIN_LOCAL.crt_list
        mode                    http
        log                     global
        option                  socket-stats
        option                  http-keep-alive
        timeout client          30000
        acl                     aclcrt_COM_EXAMPLE_LOCAL var(txn.txnhost) -m reg -i ^([^\.]*)\.example\.com(:([0-9]){1,5})?$
        acl                     unraid.example.com       var(txn.txnhost) -m beg -i unraid.example.com
        use_backend Unraid_Web_UI_ipv4  if  unraid.example.com
        
backend Unraid_Web_UI_ipv4
        mode                    http
        id                      12345
        log                     global
        timeout connect         30000
        timeout server          30000
        retries                 3
        source ipv4@ usesrc clientip
        server                  Unraid_UI 192.168.1.100:80 id 10110 check inter 1000

 

It seems to me that the web terminal simply hits the HAProxy timeout limit. Does any HAProxy guru have a handy ACL or other kind of config to make an exception of timeout just for the web terminal route?

 

[Vynce]

I'm also running into this issue with HAProxy on pfSense forwarding to Unraid and I think I've found a solution. The weird thing is this issue seems to be intermittent for me.

 

I also sometimes get HTTP 503 errors when doing docker container updates. The live progress UI will stop updating and I can see 503 errors in the browser inspector.

 

TL;DR: set the HAProxy client and server timeouts to 60000. Seems to work for me so far.

 

The HAProxy logs show that the client connection (cD--) is timing out after 30s (0/0/24/1/30028). See this page for details on the HAProxy log format and the timers, and this page for the session termination state descriptions.

Internal~ Unraid_ipvANY/Unraid 0/0/24/1/30028 101 369 - - cD-- 4/4/1/1/0 0/0 "GET /sub/dockerload?last_event_id=1696795636%3A0 HTTP/1.1"

 

So I bumped up the client timeout in the HAProxy frontend config from 30s to 60s. But then I started getting server connection (sD--) timeouts after 30s:

Internal~ Unraid_ipvANY/Unraid 0/0/36/1/30039 101 369 - - sD-- 5/5/1/1/0 0/0 "GET /sub/dockerload?last_event_id=1696795636%3A0 HTTP/1.1"

 

So I bumped up the server timeout in the HAProxy Unraid backend config from 30s to 60s. That seems to work.

 

My guess is that the Unraid console sends a keepalive packet every 30s. With the HAProxy default timeout also at 30s, HAProxy will often end up killing these connections. I have no idea if a 60s timeout is optimal or if Unraid is doing something unusual that doesn't play well with HAProxy.

 

Client timeout: pfSense -> Services -> HAProxy -> Frontend -> Edit Unraid frontend -> Client timeout -> 60000

Server timeout: pfSense -> Services -> HAProxy -> Backend -> Edit Unraid backend -> Server timeout -> 60000 

 

https://serverfault.com/questions/504308/by-what-criteria-do-you-tune-timeouts-in-ha-proxy-config

https://delta.blue/blog/haproxy-timeouts/

https://www.papertrail.com/solution/tips/haproxy-logging-how-to-tune-timeouts-for-performance/

[DANgerous25]

On 10/4/2022 at 10:13 PM, DANgerous25 said:

FWIW, I also use HAproxy on PFSense to get to my Unraid GUI using myhost.mydomain.com. I get the same problem. If I connect directly using the server's IP, then no issue at all. Very strange (and frustrating).

 

I can't explain why, but I'm not seeing this issue any more. I've done a few upgrades but can't say which if any have fixed. Currently on Unraid 6.12.4, pfSense 23.05.1-RELEASE, haproxy-devel 0.62_13 (on pfSense), Brave browser 1.58.131, MacOS Sonoma 14.0.

 

Hope the rest of you can sort this out somehow!