Rockstar

So this started last week, this container is as up to date as it can be. Running on Unraid 7.0.1 Binhex-Plex is eating literally every bit of ram it possibly can which halts the entire server. I used DBRepair to automate the repair process of plex's DBs but that came back clean and successful. Logs show nothing from either the container or unraid, just the halting of services before the eventual death of it all. Anyone else have this issue? After either gaining control and rebooting the container (or rebooting the server as a whole) Plex runs super snappy and loads no problem. It takes roughly 1 to 3 hours before it dies again. Edit: I've updated the server to 7.2.0 and I removed/reloaded the plex container. So far it's been stable again for ~12 hours at least Edit 2: nope, cratered again about an hour ago. attached plex log. I have a feeling this is what is causing it. Unsure if Plex issue specifically or not. please let me know. Plex Media Server.log

Excellent this worked! Thank you very much for the quick link. Back up and running again!

What cert is this pulling from? the OpenVPN profile?

I went on Vacation 7th and came back on the 9th... noticed the container was dead when I got home. Checked the logs, and the vpn isn't validating. No biggie, I use NordVPN so I went to their site and grabbed a new openvpn profile. Still not working...hmmm okay try this 3 or so more times to no success. I left it for the night and trying to troubleshoot some more. anyone else having these issues specifically with Nord VPN? I'm getting an AUTH_FAILED message: 2023-07-10 12:25:58,565 DEBG 'start-script' stdout output: [info] Starting OpenVPN (non daemonised)... 2023-07-10 12:25:58,620 DEBG 'start-script' stdout output: 2023-07-10 12:25:58 DEPRECATED OPTION: --cipher set to 'AES-256-CBC' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305). OpenVPN ignores --cipher for cipher negotiations. 2023-07-10 12:25:58 Note: '--allow-compression' is not set to 'no', disabling data channel offload. 2023-07-10 12:25:58,621 DEBG 'start-script' stdout output: 2023-07-10 12:25:58 WARNING: file 'credentials.conf' is group or others accessible 2023-07-10 12:25:58 OpenVPN 2.6.5 [git:makepkg/cbc9e0ce412e7b42+] x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO] built on Jun 13 2023 2023-07-10 12:25:58 library versions: OpenSSL 3.1.1 30 May 2023, LZO 2.10 2023-07-10 12:25:58,621 DEBG 'start-script' stdout output: 2023-07-10 12:25:58 DCO version: N/A 2023-07-10 12:25:58,621 DEBG 'start-script' stdout output: 2023-07-10 12:25:58 WARNING: --ping should normally be used with --ping-restart or --ping-exit 2023-07-10 12:25:58 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts 2023-07-10 12:25:58,622 DEBG 'start-script' stdout output: 2023-07-10 12:25:58 TCP/UDP: Preserving recently used remote address: [AF_INET]154.47.17.25:1194 2023-07-10 12:25:58,622 DEBG 'start-script' stdout output: 2023-07-10 12:25:58 Socket Buffers: R=[212992->212992] S=[212992->212992] 2023-07-10 12:25:58 UDPv4 link local: (not bound) 2023-07-10 12:25:58 UDPv4 link remote: [AF_INET]154.47.17.25:1194 2023-07-10 12:25:58,699 DEBG 'start-script' stdout output: 2023-07-10 12:25:58 TLS: Initial packet from [AF_INET]154.47.17.25:1194, sid=7e8fc475 43b39994 2023-07-10 12:25:58,857 DEBG 'start-script' stdout output: 2023-07-10 12:25:58 VERIFY OK: depth=2, C=PA, O=NordVPN, CN=NordVPN Root CA 2023-07-10 12:25:58,857 DEBG 'start-script' stdout output: 2023-07-10 12:25:58 VERIFY OK: depth=1, O=NordVPN, CN=NordVPN CA8 2023-07-10 12:25:58,858 DEBG 'start-script' stdout output: 2023-07-10 12:25:58 VERIFY KU OK 2023-07-10 12:25:58 Validating certificate extended key usage 2023-07-10 12:25:58 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication 2023-07-10 12:25:58 VERIFY EKU OK 2023-07-10 12:25:58 VERIFY X509NAME OK: CN=ca1701.nordvpn.com 2023-07-10 12:25:58 VERIFY OK: depth=0, CN=ca1701.nordvpn.com 2023-07-10 12:25:58,935 DEBG 'start-script' stdout output: 2023-07-10 12:25:58 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 4096 bit RSA, signature: RSA-SHA512 2023-07-10 12:25:58 [ca1701.nordvpn.com] Peer Connection Initiated with [AF_INET]154.47.17.25:1194 2023-07-10 12:25:58,935 DEBG 'start-script' stdout output: 2023-07-10 12:25:58 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1 2023-07-10 12:25:58 TLS: tls_multi_process: initial untrusted session promoted to trusted 2023-07-10 12:25:59,947 DEBG 'start-script' stdout output: 2023-07-10 12:25:59 SENT CONTROL [ca1701.nordvpn.com]: 'PUSH_REQUEST' (status=1) 2023-07-10 12:26:04,040 DEBG 'start-script' stdout output: 2023-07-10 12:26:04 SENT CONTROL [ca1701.nordvpn.com]: 'PUSH_REQUEST' (status=1) 2023-07-10 12:26:04,116 DEBG 'start-script' stdout output: 2023-07-10 12:26:04 AUTH: Received control message: AUTH_FAILED 2023-07-10 12:26:04,116 DEBG 'start-script' stdout output: 2023-07-10 12:26:04 SIGTERM[soft,auth-failure] received, process exiting I assume this is on Nords end? but surely it's not just me having this issue then? I've reset my password as well just in case, something much shorter than the original (but working) 30 char pass. I should note I can access the profiles via the Nord App no problem.

Something seems to be broken with the latest update to the container... nothing works anymore with an influxDB error. (GUS/UUD). Logs show: [error] influxdb crashed! [info] loki PID: 60 [info] Skip hddtemp due to USE_HDDTEMP set to no [error] telegraf crashed! [info] promtail PID: 75 [info] grafana PID: 96 over and over again. Rolling back to docker pull testdasi/grafana-unraid-stack:s230122 returns to a working state for me.

sooo... this was fun! I just went through this process and realized that the Nextcloud version does not update with the container...so here I was running a 2 year old version and in the same boat as you! how to fix: Stop container, edit and pull an older tag "linuxserver/nextcloud:20.0.4" or something like that. This should then allow you to start the container and login... now you can manually update and webui update. For some reason I could not update via the webui so I did it manually Nextcloud docker console updater.phar Follow the prompts. It's very very straight forward. You'll want to leave maintenance mode on after each session. You will need to run this command after each successful update (18 -> 19 -> 20 etc.) you do need to do this for every version you're behind. you can turn off maintenance mode once you're at the latest version! after all that, you can edit the docker container back to ":latest" and test it out! all should be working now.

So here's a new one that I cannot explain at all... I was watching Htop via SSH when I noticed all the cores peaking and then a ton of freezing and delays. The webui went down as it normally would (along with the containers) but the SSH terminal would still update every 30s or so. I watched for around 10 mins and then hard reset it, I couldn't make sense of any processes that where causing an issue. It wasn't until the server came back online and I went to check the log file to see it was 38GB in file size!?!?! It was repeating the text in the logfile I've uploaded over and over and over... perhaps there was other stuff but scrolling though 38gb of text is not something I was willing to do. logfile20211222.txt

Interesting! Would just stopping the container be enough to test or should I remove the the whole thing?

As far as I know I do not. I have Host Access to Custom Networks defined as Disabled but I do have a custom ipv4 network for pihole.

So over the course of the weekend the following has been done: Memtest on memory - 10 hours - 0 errors Memory speeds have been set to the base limits of the board (2666) (in this case, verified this was true) C-states have all been disabled (verification) Having learned the relationship between memory and btrfs, I removed my raid1 btrfs cache pool and formatted it, then set it all back up again Rebuilt the docker file, and added all my containers back. I'm slowly losing my goddamn mind... The lockups happen sometimes twice a day now, though this last instance after I finished rebuilding all the docker stuff was just under 48 hours uptime (not going to lie, I was very hopeful). Although I don't think there's anything new in these diagnostics and logs...they're attached. log20211219.txt chunker-diagnostics-20211219-1533.zip

Another crash that happened today. Sys logs and diagnostics attached. chunker-diagnostics-20211018-1659.zip syslog-2021-10-18.log

syslog and new diagnostics attached. This is post crash. chunker-diagnostics-20211014-2341.zip syslog-2021-10-14.log

Regrettably I turned off flash mirroring however, I've setup a quick syslog server and I will post up a new set of diagnostics and logs when it crashes again.

Update: Uptime lasted 3 days, 23 hours when it went down into a hard lock again. I've attached another diagnostic but I was not able to get any logs this time. Same situation were everything went down with no SSH or any other access. chunker-diagnostics-20211014-1358.zip

Alright, I've set the memory to 2666 (based on the chart for 3rd Gen Ryzen 4 stick dual channel), and disabled C-states. Here's to hoping! Thanks for the info everyone.