December 30, 20178 yr Just noticed a ttyd process taking 100% of a thread. Assuming a hanged web terminal. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4798 root 20 0 113072 5272 3752 R 100.3 0.0 22:03.58 ttyd -d 0 -i /var/run/ttyd.sock bash -l
December 30, 20178 yr 3 hours ago, realies said: Just noticed a ttyd process taking 100% of a thread. Assuming a hanged web terminal. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4798 root 20 0 113072 5272 3752 R 100.3 0.0 22:03.58 ttyd -d 0 -i /var/run/ttyd.sock bash -l How'd you make it happen?
December 30, 20178 yr Don't want to hijack the thread but it's somewhat related, I made my terminal crash yesterday but trying all the different browsers, it was either edge or iexplorer that crashed it, most likely edge: Dec 29 20:59:57 Tower1 kernel: traps: ttyd[5450] general protection ip:4103f2 sp:7ffc997e39b8 error:0 in ttyd[400000+31000] Dec 29 20:59:57 Tower1 nginx: 2017/12/29 20:59:57 [error] 5065#5065: *180948 upstream prematurely closed connection while reading upstream, client: 192.168.1.130, server: , request: "GET /webterminal/ HTTP/1.1", upstream: "http://unix:/var/run/ttyd.sock:/", host: "tower1.local" Dec 29 20:59:57 Tower1 nginx: 2017/12/29 20:59:57 [alert] 5454#5454: *23055 open socket #27 left in connection 17 Dec 29 20:59:57 Tower1 nginx: 2017/12/29 20:59:57 [alert] 5454#5454: aborting Dec 29 21:00:07 Tower1 nginx: 2017/12/29 21:00:07 [error] 5065#5065: *181138 connect() to unix:/var/run/ttyd.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.1.9, server: , request: "GET /webterminal/ws HTTP/1.1", upstream: "http://unix:/var/run/ttyd.sock:/ws", host: "tower1" Dec 29 21:00:17 Tower1 nginx: 2017/12/29 21:00:17 [error] 5065#5065: *181233 connect() to unix:/var/run/ttyd.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.1.9, server: , request: "GET /webterminal/ws HTTP/1.1", upstream: "http://unix:/var/run/ttyd.sock:/ws", host: "tower1" Dec 29 21:00:18 Tower1 nginx: 2017/12/29 21:00:18 [error] 5065#5065: *181248 connect() to unix:/var/run/ttyd.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.1.130, server: , request: "GET /webterminal/ HTTP/1.1", upstream: "http://unix:/var/run/ttyd.sock:/", host: "tower1.local", referrer: "http://tower1.local/Main" Dec 29 21:00:25 Tower1 nginx: 2017/12/29 21:00:25 [error] 5065#5065: *181302 connect() to unix:/var/run/ttyd.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.1.130, server: , request: "GET /webterminal/ HTTP/1.1", upstream: "http://unix:/var/run/ttyd.sock:/", host: "tower1.local", referrer: "http://tower1.local/Main" Dec 29 21:00:27 Tower1 nginx: 2017/12/29 21:00:27 [error] 5065#5065: *181323 connect() to unix:/var/run/ttyd.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.1.9, server: , request: "GET /webterminal/ws HTTP/1.1", upstream: "http://unix:/var/run/ttyd.sock:/ws", host: "tower1" Dec 29 21:00:35 Tower1 nginx: 2017/12/29 21:00:35 [error] 5065#5065: *180795 connect() to unix:/var/run/ttyd.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.1.130, server: , request: "GET /webterminal/ HTTP/1.1", upstream: "http://unix:/var/run/ttyd.sock:/", host: "tower1.local", referrer: "http://tower1.local/Main" Dec 29 21:00:37 Tower1 nginx: 2017/12/29 21:00:37 [error] 5065#5065: *181398 connect() to unix:/var/run/ttyd.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.1.9, server: , request: "GET /webterminal/ws HTTP/1.1", upstream: "http://unix:/var/run/ttyd.sock:/ws", host: "tower1" Dec 29 21:00:47 Tower1 nginx: 2017/12/29 21:00:47 [error] 5065#5065: *181460 connect() to unix:/var/run/ttyd.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.1.9, server: , request: "GET /webterminal/ws HTTP/1.1", upstream: "http://unix:/var/run/ttyd.sock:/ws", host: "tower1" Dec 29 21:00:49 Tower1 emhttpd: req (7): csrf_token=****************&title=System+Log&cmd=%2FwebGui%2Fscripts%2Ftail_log&arg1=syslog Dec 29 21:00:49 Tower1 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog Is there a way to restart it without rebooting?
December 30, 20178 yr This should work to restart it (but has to be from console or actual telnet/ssh session): /etc/rc.d/rc.nginx restart Also noticed that if there is an active Terminal session, nginx will not stop until session is terminated. This also will be fixed in next release, though normally nginx is never stopped anyway.
December 30, 20178 yr 52 minutes ago, limetech said: This should work to restart it (but has to be from console or actual telnet/ssh session): Didn't work, command completed and ngix restarted but terminal still doesn't work, no big deal, I'll reboot when convenient. 502 Bad Gateway nginx
December 30, 20178 yr Oh yeah something else you have to do: /etc/rc.d/rc.nginx restart chmod 666 /var/run/ttyd.sock
December 30, 20178 yr Author 1 hour ago, limetech said: Oh yeah something else you have to do: /etc/rc.d/rc.nginx restart chmod 666 /var/run/ttyd.sock This results in a 502 Bad Gateway page for /webterminal. It seems ttyd is not spawned correctly as it is missing from ps -aux | grep ttyd even after calling it manually with `ttyd -d 0 -i /var/run/ttyd.sock bash -l &>/dev/null &`. No output when spawning without redirecting the output to /dev/null and sending the process in the background. Edited December 30, 20178 yr by realies
December 30, 20178 yr 20 minutes ago, realies said: This results in a 502 Bad Gateway page for /webterminal. It seems ttyd is not spawned correctly as it is missing from ps -aux | grep ttyd even after calling it manually with `ttyd -d 0 -i /var/run/ttyd.sock bash -l &>/dev/null &`. No output when spawning without redirecting the output to /dev/null and sending the process in the background. Works for me. Also the 'chmod' should not be necessary unless you changed 'umask'
December 30, 20178 yr 1 minute ago, realies said: Would appreciate if it works for me too. You can look inside /etc/rc.d/rc.nginx to see how the thing is invoked. If you do a killall ttyd /etc/rc.d/rc.nginx stop You should see there is no more ttyd process and also no /var/run/ttyd.sock socket. Now if you do a /etc/rc.d/rc.nginx start You should see both of them again. If not, maybe there are some syslog messages? Can also take a look at the /var/run/ttyd.sock socket and see what the permissions are. You should see '777'. The socket needs to have at least rw permission for owner/group/world. I have noticed if you try to stop nginx with /etc/rc.d/rc.nginx stop If there is an active ttyd window open, nginx worker will not actually close, which prevents main nginx task from closing. Hence why the 'killall ttyd' above.
December 31, 20178 yr Author When a process hangs it is expected for it to malfunction. In this case, the unexpected behaviour is achieved by opening a web terminal and closing it prior to it fully loading. Most of the times the ttyd process will either lock up, utilising its thread to 100% or exit with a segmentation fault error. This leaves a socket file in place, removing it prior to launching another instance of ttyd, similarly to how you've done with Nginx, resolves the issue with restarting the Nginx + extras service. --- rc.nginx 2017-12-28 18:52:13.000000000 +0200 +++ rc.nginx.patched 2017-12-31 02:21:17.248054062 +0200 @@ -305,6 +305,7 @@ # side-load ttyd killall -q ttyd &>/dev/null + rm -f /var/run/ttyd.sock exec ttyd -d 0 -i /var/run/ttyd.sock bash -l &>/dev/null & # nginx does not unlink stale unix sockets before rebinding PS: Please consider open-sourcing (and keeping up-to-date) some of the unRAID modules so the community could help with resolving RC bugs and introducing new functionality.
December 31, 20178 yr 16 minutes ago, realies said: When a process hangs it is expected for it to malfunction. In this case, the unexpected behaviour is achieved by opening a web terminal and closing it prior to it fully loading. Most of the times the ttyd process will either lock up, utilising its thread to 100% or exit with a segmentation fault error. This leaves a socket file in place, removing it prior to launching another instance of ttyd, similarly to how you've done with Nginx, resolves the issue with restarting the Nginx + extras service. Good idea to remove the socket.
December 31, 20178 yr Author 27 minutes ago, limetech said: Good idea to remove the socket. Hope this does not mean you are changing to port communication. The actual issue has been reported. There's a similar issue on the ttyd repository and stable alternatives. Edited December 31, 20178 yr by realies
December 31, 20178 yr 1 hour ago, realies said: Hope this does not mean you are changing to port communication. The actual issue has been reported. There's a similar issue on the ttyd repository and stable alternatives. No it means we'll add that same line you did in rc.nginx.
December 31, 20178 yr Author When restarting Nginx (and extras) service, ttyd picks up the current folder ($PWD) as the initial folder for web terminals. Not great for when you are restarting the service from outside the home folder. Spawning a subshell that changes the context to the home folder before initialising ttyd overcomes this. --- rc.nginx 2017-12-31 06:36:53.804058672 +0200 +++ rc.nginx.patched 2017-12-31 06:33:13.034019513 +0200 @@ -306,7 +306,7 @@ # side-load ttyd killall -q ttyd &>/dev/null rm -f /var/run/ttyd.sock - exec ttyd -d 0 -i /var/run/ttyd.sock bash -l &>/dev/null & + (cd ~ && exec ttyd -d 0 -i /var/run/ttyd.sock bash -l &>/dev/null &) # nginx does not unlink stale unix sockets before rebinding # see: https://trac.nginx.org/nginx/ticket/753 Edited December 31, 20178 yr by realies
December 31, 20178 yr We changed: exec ttyd -d 0 -i /var/run/ttyd.sock bash -l &>/dev/null & to exec ttyd -d 0 -i /var/run/ttyd.sock login -f root &>/dev/null & works better.
December 31, 20178 yr Author Not that it is currently possible to login to the webGui with other users, but login -f anotheruser doesn't work. Noticed the web terminal title fixed to `login -f root ($HOSTNAME)` despite `export TERM="linux"` and preferences in `.bash_profile`. It seems ttyd is capable of changing the title. If running ttyd as a service is unstable, it could be spawned from a php script with the --once flag (Accept only one client and exit on disconnection). If it decides to crash there would be no need to restart the whole web server stack (losing the point of a web terminal). PS: It would be good to be able to open more than one web terminals. <a href="#" onclick="TerminalButton();return false;" ... to <a href="/webterminal/" onclick="TerminalButton();return false;" ...
December 31, 20178 yr 3 hours ago, realies said: PS: It would be good to be able to open more than one web terminals. <a href="#" onclick="TerminalButton();return false;" ... to <a href="/webterminal/" onclick="TerminalButton();return false;" ... The page reference is auto-generated for each button in the GUI, it can't be changed. It is possible though to open multiple web terminals by generating unique names for each window. Made a PR for that.
December 31, 20178 yr 9 hours ago, realies said: Not that it is currently possible to login to the webGui with other users, but login -f anotheruser doesn't work. Seemed to work ok for me when we tested. Note that referenced site often has incorrect/outdated info.
January 1, 20188 yr Author @bonienl, I mean that it would be great for users to be able to scroll-click the link and open a new tab. If the point was to make something looking visually as close as possible to a terminal client, you might as well hide the address bar in the popup. Making the name unique would allow you to have a bunch of floating windows that can't be grouped in tabs. It's 2018. Looking through the code, I have noticed that the custom user interface rendering engine is already using the template key called "Link" in /usr/local/emhttp/plugins/dynamix.system.temp/TempButton.page:2:Link="nav-user" Despite used only once, it was difficult for me to understand its purpose, the engine consumes it within DefaultPageLayout.php#L318-L322 if (empty($page['Link'])) { $icon = substr($page['Icon'],-4)=='.png' ? "<img src='/{$page['root']}/icons/{$page['Icon']}' class='system'>" : "<i class='system fa fa-{$page['Icon']}'></i>"; echo "<div id='nav-item' class='{$page['name']}'><a href='#' onclick='{$page['name']}();return false;' title='{$page['Title']}'>$icon<span>{$page['Title']}</span></a></div>"; } else echo "<div id='{$page['Link']}'></div>"; Inspecting the generated page and its source, it seems it has been rendered as an invisible empty element between script injections: </script><div id='nav-user'></div><script> Looking in the stylesheet, it seems it might be a placeholder for a custom user element that never served its purpose? #nav-block #nav-user{display:block;float:left;text-align:center;border:none} Nevertheless, utilising a brand new template key called "Href" does the job and makes it possible to scroll-click the terminal icon, which does not trigger the popup terminal window! Here's the PR. @limetech, changing the default login shell for the user did the trick. In my understanding, this is the remaining work related to providing a better web terminal: Create web terminal on demand instead of providing it as a service Fix web terminal title as currently stuck to `login -f root ($HOSTNAME)` Including straight from #unraid: 02:14 <wishie_> i was really looking forward to the web terminal thing 02:14 <realies> was? 02:15 <wishie_> but i cant use it because Safari doesnt work with it 02:15 <wishie_> because they are using HTTP auth over the websocket which Safari doesnt support Let me know if you disagree, have a fix before I do, have additional comments or would like to discuss anything. And happy 2k18! Edited January 1, 20188 yr by realies no pasta
January 1, 20188 yr 6 hours ago, realies said: Making the name unique would allow you to have a bunch of floating windows that can't be grouped in tabs. It's 2018 Right, it is 2018 and nowadays individual windows can be easily positioned next to each other. Windows 10 makes that very easy and large resolution screens benefit from having multiple windows opened and viewed at the same time. Microsoft made the "tab" mistake in the past with their office suite, but luckily today they have seen the light. 6 hours ago, realies said: Looking in the stylesheet, it seems it might be a placeholder for a custom user element that never served its purpose? The "Link" key is not superfluous, it can be used by third party plugins (like Dynamix System Temp) to hook themselves into the default page without consuming GUI space. Menu="Buttons" Link="nav-user" --- <script> function systemTemp() { ... } systemTemp(); </script> 6 hours ago, realies said: Including straight from #unraid: It is a known issue with Safari and websockets. All other major browsers such as Chrome, Edge and Firefox work fine. I can only second guess why Apple doesn't want to follow.
January 5, 20188 yr just as note, since update rc19b i also have the 502 error on web terminal, using chrome x64 Jan 5 07:51:16 AlsServer nginx: 2018/01/05 07:51:16 [error] 4950#4950: *159182 connect() to unix:/var/run/ttyd.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.1.127, server: , request: "GET /webterminal/ HTTP/1.1", upstream: "http://unix:/var/run/ttyd.sock:/", host: "192.168.1.2:88", referrer: "http://192.168.1.2:88/Dashboard" Edited January 5, 20188 yr by alturismo
January 5, 20188 yr 7 hours ago, alturismo said: just as note, since update rc19b i also have the 502 error on web terminal, using chrome x64 Jan 5 07:51:16 AlsServer nginx: 2018/01/05 07:51:16 [error] 4950#4950: *159182 connect() to unix:/var/run/ttyd.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.1.127, server: , request: "GET /webterminal/ HTTP/1.1", upstream: "http://unix:/var/run/ttyd.sock:/", host: "192.168.1.2:88", referrer: "http://192.168.1.2:88/Dashboard" Mine works fine. Are you using a proxy of some kind? If so you need to proxy the web socket traffic. I'm using the Let's Encrypt Nginx docker and am proxying all web socket traffic if the connection has been upgraded.
January 5, 20188 yr no proxy here, but probably also some websocket error like i have on mkvtools vnc ... with hostname it works, with lan ip it doesnt ... really weird
Archived
This topic is now archived and is closed to further replies.