lsmith5

Not really unfortunately If I run multiple 4k transcodes locally or on my alternative server NPM, htop also reports between that 30-50%. Here is a screen-grab of the moment it crashed when hosting NPM via unraid's docker service, low CPU usage:

Unfortunately this did not solve the issue. I really do not have much CPU load. Maybe 30-50% max. Thank you for the suggestion

I have been experiencing an issue where accessing my Jellyfin & Home Assistant container/vm via NPM (in docker) can soft-lock my unraid server if I have a large amount of bandwidth traveling through it. Accessing the same data using my local IP address does not result in any crashing. I have way more information and details on another post: but it was recommended that I post here as well. I tried with Caddy and still received the soft-locking (so perhaps its more of a general reverse-proxy issue). But I am able to run NPM on an Ubuntu VM or my Win11 PC pointing to the same Jellyfin/HA instances and it works flawlessly. Was hoping maybe someone more adept in reverse-proxies/npm could take a look at my thread and see if anything sticks out. Thanks!

Decided to just run docker in an Ubuntu VM and host my reverse-proxy there (NPM). Stress-tested it last night with no issues. For reference, here is a capture of a "docker version" between both hosts: Unraid ( NOT Working): Client: Version: 20.10.21 API version: 1.41 Go version: go1.18.7 Git commit: baeda1f Built: Tue Oct 25 17:56:30 2022 OS/Arch: linux/amd64 Context: default Experimental: true Server: Docker Engine - Community Engine: Version: 20.10.21 API version: 1.41 (minimum version 1.12) Go version: go1.18.7 Git commit: 3056208 Built: Tue Oct 25 18:02:03 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.6.9 GitCommit: 1c90a442489720eec95342e1789ee8a5e1b9536f runc: Version: 1.1.4 GitCommit: v1.1.4-0-g5fd4c4d1 docker-init: Version: 0.19.0 GitCommit: de40ad0 Ubuntu Server (Working): Client: Version: 20.10.17 API version: 1.41 Go version: go1.18.5 Git commit: 100c70180f Built: Mon Oct 17 06:20:04 2022 OS/Arch: linux/amd64 Context: default Experimental: true Server: Engine: Version: 20.10.17 API version: 1.41 (minimum version 1.12) Go version: go1.18.5 Git commit: a89b842 Built: Mon Oct 17 06:20:42 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.6.6 GitCommit: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1 runc: Version: 1.1.2 GitCommit: docker-init: Version: 0.19.0 GitCommit: de40ad0 Everything is reporting as up-to-date on my ubuntu server, but there is quite a difference in versioning between that and unraid. I'm curious if there is some way to downgrade the docker server to match that of my ubuntu server? Then maybe I can narrow it down to being a specific issue in the version of docker that unraid is running. But at the same time, running docker desktop on my win11 machine is also v20.10.21 and doesn't have any issues. Maybe its a linux + that version issue?

Crazy experiment today, was planning on trying out Traefik for a reverse-proxy. Figured it would be easier to download and use Docker compose (off the app store) due to all the labels. Set up NPM and Jellyfin in that compose file, got it all running, and now I can transcode via NPM without any crashing. I switched back to my non-compose NPM and Jellyfin and it immediately crashed again as soon as 4k transcoding started. Literally copy/pasted all of the same config, using the same docker images, etc. I have literally no idea the difference between the compose app vs the built-in app, but I can only imagine something is different on the backend that is allowing it to work suddenly. Perhaps compose is using a different version of docker? Or has some backend flags set differently? Not really sure if this is a "Solution" but its at least a work-around for me. UPDATE: Nevermind? I guess? It 100% worked, now suddenly does not after an unraid restart.

Am I allowed to bump?

Tried switching to using a non-custom docker network for my containers, still had the issue. Been running on my home PC fine, but would really like to get this working back on unraid.

More testing and talking to myself today. Spun up a local NPM instance using Docker Desktop on my home PC (win11). Set up all the routes and everything to Jellyfin/HA. Have not had any issues with unraid becoming unresponsive. In an attempt to narrow it down further, I switched back to using NPM in unraid and it seems like I am able to direct stream a 1080p movie with no issues alongside frigate. I switched to transcoding a 1080p and it still worked. As soon as I switched to a 4k title and using transcoding, it seemed to break. Makes me think that there is some reverse-proxy issue in unraid where it cant handle a high amount of bandwidth perhaps? Then it soft-locks.

Been running my server without Reverse proxy for ~2 days and havent had any issues so far. So I think I am getting to the point where I'm comfortable saying it was the reverse proxy stuff causing the issue. I went back and looked through the logs for NPM and saw the following in fallback_error.log (redacted a bit as IP addresses are uses) 2023/01/05 16:08:36 [error] 284#284: *1 connect() failed (111: Connection refused) while connecting to upstream, client: <HOME PC>, server: nginxproxymanager, request: "GET /api/users/me?expand=permissions HTTP/1.1", upstream: "http://127.0.0.1:3000/users/me?expand=permissions", host: "<SERVER>:81", referrer: "http://<SERVER>:81/nginx/proxy" 2023/01/05 16:08:49 [error] 311#311: *6 open() "/var/www/html/boaform/admin/formLogin" failed (2: No such file or directory), client: <Random IP #1>, server: localhost-nginx-proxy-manager, request: "POST /boaform/admin/formLogin HTTP/1.1", host: "<Public IPv4>:80", referrer: "http://<Public IPv4>:80/admin/login.asp" 2023/01/05 16:25:06 [error] 311#311: *306 open() "/var/www/html/actuator/health" failed (2: No such file or directory), client: <Random IP #2), server: localhost-nginx-proxy-manager, request: "GET /actuator/health HTTP/1.1", host: "<Public IPv4>" 2023/01/05 16:36:46 [error] 284#284: *1 connect() failed (111: Connection refused) while connecting to upstream, client: <HOME PC>, server: nginxproxymanager, request: "GET /api/ HTTP/1.1", upstream: "http://127.0.0.1:3000/", host: "<SERVER>:81", referrer: "http://<SERVER>:81/nginx/proxy" 2023/01/05 16:45:16 [error] 329#329: *1525 open() "/var/www/html/.env" failed (2: No such file or directory), client: <Random IP #3>, server: localhost-nginx-proxy-manager, request: "GET /.env HTTP/1.1", host: "<Public IPv4>" Presumably the random IPs are just bots attempting to scrape the webpage (#2 & #3 were DigitalOcean, and #1 from a VPN). Doesnt seem to be be anything related to the crashing. Then I have a bunch of these in the proxt-host-1_error.log: 2023/01/05 16:36:01 [error] 355#355: *516 connect() failed (111: Connection refused) while connecting to upstream, client: <cloudflare IP>, server: <my jellyfin domain>, request: "GET / HTTP/2.0", upstream: "https://<server ip>:8089/", host: "<my jellyfin domain>", referrer: "http://<server ip>:81/" Again, not sure what to make of this info, but thought it might be helpful to share.

Just posting to mention the update to my original post at the bottom. I spent a number of hours rebuilding my entire docker environment, slowly adding pieces until I could get the issue to reproduce. I discovered this issue only occurs when I attempt to access my jellyfin server (presumably frigate would also cause the issue) via my reverse proxy (typically NPM; but also CaddyV2). Not quite sure how to diagnose from here, but figured the update may give someone enough information to help me.

Have had no issues over the past ~40hr since generating the previous diagnostic, but also have not been running transcodes. Does anyone have any insight that may help me narrow down my issue? Edit: LOL spoke too soon. Was working at around 06:30 today (Jan 5), but seems to have stopped responding between then and 08:55 when I noticed the web gui was unresponsive. Was able to capture a diag (again, two diags; one before shutdown and one as the shutdown took too long) which I have attached. Like 99% sure Jellyfin wasnt even running (syslog seems to confirm that). Used Frigate this morning without any noticeable issue, but might have happened as I stopped maybe. Might try rebuilding my Frigate container just to see if it helps. tower-diagnostics-20230105-0902.zip tower-diagnostics-20230105-0857.zip

See bottom for important update Original Post: Have been dealing with an intermittent issue where my unraid server would seemingly become inaccessible (no SSH and no web GUI). I shrugged it off the first two times. Then, I noticed as I was attempting to stream something w/ transcoding via my Jellyfin docker container (w/ Nvidia GPU HW Acceleration), it became unresponsive immediately. I thought it was potentially something incorrect with the container, so I rebuilt it from scratch, ultimately working up to having HW Acceleration again. It worked up until it was being transcoded by my GPU. I then went through and changed the drive the transcodes were being stored on and downgraded my GPU drivers. Neither of those things seemed to help the issue that as soon as I started transcoding with my GPU, it locks up my host. The other two major pieces I have running which seem to be causing an impact are Home Assistant in a VM (No GPU passthrough), and Frigate in a docker container (initial also being passed the GPU for stream acceleration). During some troubleshooting, I tried to stream with my GPU transcoding, but frigate was off, and it worked without issues. So I did some googling and found conflicting information as to whether Frigate AND Jellyfin could both be transcoding using the same single GPU simultaneously; so I ultimately removed the HW Acceleration from Frigate. Except it still crashed with Frigate running.... I finally attached a KB&M and monitor to the host and found that I can actually still log in and use the console while the web GUI and SSH are no longer accessible. I was then able to reproduce the error (and can multiple times in a row) using the following steps tied with the approximate timestamp I initiated each step in accordance with my attached logs: 18:45 - Boot Unraid 18:47 - Login via console & swap KB&M to ZWave/Zigbee stick (only have one USB avail) 18:50 - All containers are started 18:51 - HA is started 18:52 - Manually start Frigate container (needs to start after HA due to MQTT being hosted in HA) 18:53-18:54 - Manually start Jellyfin container and wait for it to start 18:55 - Start a transcoded stream on Jellyfin. Appears to have worked for ~2min, until the next step 18:57 - Opened Frigate Birdseye in a second tab, as soon as Birdseye was opened, the connection was lost and the Web GUI became unresponsive along with all of my docker containers 19:00 - Switch USB back to KB&M from ZWave/Zigbee stick, dump diag, shutdown server During the shutdown, I guess it took so long that it threw an error and caused another diag file to occur at 19:02. I have attached both for reference. Some other diagnostics I have done: - Upgrade motherboard firmware from 1.02->1.03. No fix from that. Mobo does have an FF code showing on it all the time after posting, which I've seen mixed reviews on if that means Fault Free or Fault Found and I can't find anything in my model's manual. - Ran Memcheck for a full pass, no errors. Also swapped all memory with another kit. No Fix - Ran CHKDSK on my unraid thumb-drive with no errors - Installed an made a number of recommendations from Fix Common Problems App It has seemed SUPER inconstant, sometimes it works in some configurations, other times it doesn't in the exact same configuration. The real wrench in the cogs is that I know for a fact that it went unresponsive earlier today even when there was no Jellyfin container running. Unfortunately I was unable to grab a diag of that occurrence those. Jellyfin and Frigate seemed to get alone fine with my config until this past month or two. Any suggestions would be greatly appreciated, I've really been wracking my brain on this. Update It appears that this is actually only happening when I access Jellyfin through my reverse proxy (Nginx Proxy Manager). I have been able to have Frigate & Jellyfin running and multiple GPU transcodes going at the same time, as long as I am accessing the services via their local IP. As soon as I try and access it via my reverse proxy, it makes the unraid GUI non-responsive and the other issues outlined above. Tried switching to CaddyV2 for a quick and dirty attempt at a different reverse proxy, but it still "crashes" or whatever. tower-diagnostics-20230103-1900.zip tower-diagnostics-20230103-1902.zip