Earendur

Just to answer a previous commenter's question about what people use this for, I only added it so I could get jq functionality for a user script. I use jq to extract the certs generated by Let's Encrypt through my Traefik proxy, which stores them in JSON format. My user script parses the JSON using jq, constructs a .pem file and drops it in the appropriate location for unraid to use. This way, my UI has a valid cert whether I access it directly or through the Traefik proxy. It was convenient. Setting up a docker container with jq and scheduling the script to run with it will be more work. Doable, but more work nonetheless. I haven't removed the plugin yet so I haven't confirmed if jq is available without in version 7.

Migrated this morning and the Windows VM instance has crashed twice with the same error, despite the VM having 12gbs of ram and nothing else running on it. The game was reporting 2.5gbs of usage during the one crash I was able to watch in real time. I think it's safe to say the issue is not with the container. I'm going to continue to monitor it. I will plan to start up a new instance with no mods, as I'm currently running Hosav's UI and the Pippi admin mod, just to rule those out as potential causes. I'll update with some more info when I've completed more testing. To mitigate this issue, I've written the attached script which uses RCON and checks to see if the server responds to RCON commands. When this memory crash occurs, the executable for the server stays alive but hangs. The container therefore won't notice the problem and restart itself. RCON will receive an "i/o timeout" response when this happens so I check for that and force a restart if the script gets that response. I set this script to run every 3 minutes in User Scripts. This script requires ich777's RCON plugin to work, and it could easily be used for any other game server containers. RCON Zombie Check Script.sh

Both of my Conan Exiles docker containers seem to be running out of memory. I originally suspected a game memory leak but I can see that the server is using between 4.5 and 6 gbs before it crashes. Both my Siptah and Exiled Lands servers do this. The error is always the same at the end of the log: [2024.06.23-14.58.17:337][ 79]LogMemory:Warning: Freeing 33554432 bytes from backup pool to handle out of memory. [2024.06.23-14.58.17:337][ 79]LogMemory:Warning: MemoryStats: AvailablePhysical 11342008320 AvailableVirtual 140731004248064 UsedPhysical 0 PeakUsedPhysical 0 UsedVirtual 0 PeakUsedVirtual 0 [2024.06.23-14.58.17:337][ 79]Allocator Stats for binned2 are not in this build set BINNED2_ALLOCATOR_STATS 1 in MallocBinned2.cpp The server has memory available - total of 32gbs with over 14 available even with both servers running. I tried running only one of each at a time, same issue. It happens within 20-30 minutes of game play, sometimes it happens immediately upon login attempt. Single player instance doesn't have this issue when I attempt it. And this only started after the last posts I made when the yearly base image patch was done and I had to delete the WINE64 directories to fix the issue. I found an older post (https://forum.winehq.org/viewtopic.php?t=34322) that seems to indicate an issue with how WINE allocates memory. Any ideas? Worst case I migrate the servers back to a Windows VM. Your work is much appreciated, thanks!

Looks good and is working great! I could've sworn I was seeing server logs appear in the docker log. Admittedly I know very little about WINE and I don't know how an image can be configured to show certain logs in the container log, but I do think the WINE logs make it a bit confusing. Usefiul if there is a problem, but I think the server logs being shown is a better default as they are used more often than the WINE log. Thanks again!

Interesting. The server does appear to be running now. I believe your suggestion to delete WINE64 did it, but the docker log shows all those errors still. The Conan log appears to show the server is up an running and I can now see it is online and connect. I restored the original game data directory and deleted the WINE64 directory and both servers are now back and connectable again. As you suggested, seemed like an issue with WINE. Worth noting that the docker logs used to show Conan server events and information, but now they do not and I can only find that info in the game server logs directly. Thanks for your help!

Seems to be giving me the same error. Here's the docker log from the shutdown onward:dockerlog.txt I'm going to disable the 3 mods I have and see if that has an effect.

I see that some people have been having issues since the recent update. I too am having an issue with the Conan Exiles container. I'm getting the following error: This just repeats indefinitely. Both my Conan Exiles containers are doing the same thing since the update. So I stopped the container, renamed the server data volume so the container starts as if it was new. It downloaded the game, but it never came online. I just continued to throw these same errors as it did before.

I've successfully done the exact same thing here except I'm using a traefik docker container for my automatic certificate provisioning for my services and not SWAG. I created a User Script that extracts the certs using jq, concats the cert and key into a .pem bundle, then it uses openssl verify to check that the cert is valid before issuing the command to reload the web ui. Here's the script: #!/bin/bash traefik_path=/mnt/user/appdata/traefik acme_json=$traefik_path/acme.json domain=mydomain.com domain_cert=$traefik_path/certs/$domain.crt domain_key=$traefik_path/certs/$domain.key unraid_cert=/boot/config/ssl/certs/tower_unraid_bundle.pem # Use jq to extract the cert and the key - decode them from base64 - store them in files for later use jq -r '.[].Certificates[] | select(.domain.main=="'${domain}'") | .certificate' $acme_json | base64 -d > $domain_cert jq -r '.[].Certificates[] | select(.domain.main=="'${domain}'") | .key' $acme_json | base64 -d > $domain_key # concatenate the certs and the key into a .pem file at the correct location for unraid to utilize cat $domain_cert $domain_key > $unraid_cert # if openssl can verify the cert as valid, recycle the webui openssl verify -untrusted $domain_cert $unraid_cert 2>/tmp/err if [ -s /tmp/err ] then echo Certificate Failed to verify. else echo Certificate verified Successfully - recycling Unraid Web UI... # reload the web UI to accept the new cert /etc/rc.d/rc.nginx reload fi I believe jq comes with the NerdPack plugin, so you'll need to install that first. I haven't found a lot of info for people who use Traefik proxy, so hopefully this helps others who do. A possible enhancement to this might be to check that the acme.json has been updated, or check if the specific domain cert has been updated before running the script. A watch might be able to be used on the acme.json file to do this, or inotify, but this version works for me and the reload of the web UI doesn't seem to cause any issues. I don't have to re-log in for my logged in session, I can run this via the User Scripts UI and it doesn't cause issues, etc. To be honest, I'm not entirely sure that the way I used OpenSSL to verify is the correct way to do it.

My file has now more than doubled. It's 977,920 lines long, all but 25 or so lines are the sessions. 35 mbs in size. Something is definitely not functioning correctly here. These sessions shouldn't be accumulating like this and they should get cleaned up.

Interesting. Mine is 12 megabytes. When you say you deleted the sessions, you went into the JSON and removed these records: Of course, there are hundreds/thousands of these in the file. I wonder if deleting the file itself would cause an issue, or if it would just recreate a new one? I may leave them because the web ui is running, and watch it to see if they get cleaned up at all by the application.

It appears I'm having this exact same issue. Hangs at the same point and the WebUI doesn't load. 2022-05-07 11:09:10,133 DEBG 'watchdog-script' stdout output: [info] Deluge key 'listen_interface' currently has a value of '10.67.228.49' [info] Deluge key 'listen_interface' will have a new value '10.67.228.49' [info] Writing changes to Deluge config file '/config/core.conf'... 2022-05-07 11:09:10,211 DEBG 'watchdog-script' stdout output: [info] Deluge key 'outgoing_interface' currently has a value of 'wg0' [info] Deluge key 'outgoing_interface' will have a new value 'wg0' [info] Writing changes to Deluge config file '/config/core.conf'... Sometimes, after leaving it for a few hours, it seems to move to the next step: 2022-05-07 11:09:10,133 DEBG 'watchdog-script' stdout output: [info] Deluge key 'listen_interface' currently has a value of '10.67.228.49' [info] Deluge key 'listen_interface' will have a new value '10.67.228.49' [info] Writing changes to Deluge config file '/config/core.conf'... 2022-05-07 11:09:10,211 DEBG 'watchdog-script' stdout output: [info] Deluge key 'outgoing_interface' currently has a value of 'wg0' [info] Deluge key 'outgoing_interface' will have a new value 'wg0' [info] Writing changes to Deluge config file '/config/core.conf'... 2022-05-07 14:06:11,257 DEBG 'watchdog-script' stdout output: [warn] Deluge config file /config/web.conf does not contain valid data, exiting Python script config_deluge.py... 2022-05-07 14:06:11,592 DEBG 'watchdog-script' stdout output: [info] Deluge process started [info] Waiting for Deluge process to start listening on port 58846... 2022-05-07 14:06:11,802 DEBG 'watchdog-script' stdout output: [info] Deluge process listening on port 58846 2022-05-07 14:06:13,029 DEBG 'watchdog-script' stdout output: [info] No torrents with state 'Error' found 2022-05-07 14:06:13,029 DEBG 'watchdog-script' stdout output: [info] Starting Deluge Web UI... [info] Deluge Web UI started However, the WebUI is not responsive and won't load. I've made no changes to my configuration and it was working for many months before yesterday when this issue popped up. Edit: I was planning to follow the steps listed here today, but it seems to have started on it's own over night and is running fine now. I'll update if that changes.

Dang, that looks to be the exact issue I'm experiencing. The transcoding wasn't always running when I was getting full system hangs, and the syslog absolutely did not have any details about the crash. I'll comment there and indicate I'm experiencing the same thing.

I can report that my unraid machine has now been online for 4 days with no crashes. The culprit seems to have definitely been the /dev/dri being added as a device to the Plex container. I can confirm that I have HDR tone mapping disabled: So I don't think HDR tone mapping was the cause of the crashes, at least not for me because it was not on when I had the /dev/dri device added to the container. I still suspect it has something to do with docker or the container moving threads across the efficiency cores and the performance cores. I'd like to re-enable the hardware transcoding and test it by pinning the container to only the performance cores of the CPU. But before I do, I want to ask if anyone else has had similar problems? It seems that others have had no crashes when they add the device to the container so long as they have HDR tone mapping disabled, but that was not my experience.

I can report that I have now been online for 1 day and 20 hours with no crashes since I removed the device /dev/dri from the Plex container. If it remains stable for a few more days, I may re-add the device and pin the Plex container to the efficiency cores of the CPU. It's possible threads moving across different types of cores is what causes the crash, but this is pure speculation on my part so I'll have to test it out. Thanks for this tip! I actually worked late last night so I never attempted the memtest. I'll have to do it this way to save me the trouble of switching to legacy boot. I've never even tried to do GUI mode. I've only ever connected remotely or through command line directly on the server.

Just want to share my experience so far. I've got an MSI PRO Z690-A DDR4 board, i5-12600k, 32gigs of corsair 3200 ram, running 6.10.0-RC2. I've been experiencing instability issues as well. Unraid will seemingly crash, and the syslog doesn't seem to record the issue. The Web GUI becomes unresponsive, the system won't respond to pings on the network (I'm using a dual NIC add in card as the onboard ethernet isn't working which has been mentioned in this thread already), and shares become unavailable. When this happens, I can log in directly via a monitor hooked up directly to the server. Attempting a "powerdown" causes it to wait 90 seconds for a clean shutdown, then it tries to force shutdown which stalls there. I'm forced to cut the power by holding the power button. I can see the docker processes using docker ps and every one of them lists 0.0.0.0 as the IP and status as "unhealthy" at this point. My attempts to isolate the issue so far haven't been successful. I've set the syslog to go to a QNAP NAS I have that is running a syslog server, and I've set it to mirror to the USB hoping to glean some info from it, but so far it doesn't seem to record the problem. I have updated the BIOS to the latest version and I attempted to run a MemTest last night but couldn't boot into it - turns out I need to switch from UEFI boot to BIO so I can run it. I'm going to do that tonight. I had set up iGPU transcoding as per the details in this thread. I set this up immediately upon building the machine so I'm not sure this issue would have occurred without it. I tried disabling it by turning it off in plex itself. Crashed. I tried disabling HDR mapping, crashed. I tried turning off XMP profiles. Crashed. The crashing is also inconsistent. It'll work for half a day, then crash 3 times in an hour. I tried starting a windows VM I installed, crashed when I started it. I have only three ideas as to what the issue could be. 1.) bad memory (going to run that memtest tonight to find out, 2.) the /dev/dri iGPU is somehow causing it to hang up, and 3.) I wonder if maybe the containers switching between the efficiency cores and the power cores are causing a problem - I may set the plex docker container to be pinned to either the efficiency cores or the power cores to see if that resolves the issue. So last night after it crashed and after I couldn't figure out the memtest thing (again, I need to switch to legacy boot to run it), I actually removed the device /dev/dri from the Plex container to see if that makes a difference instead of just shutting off hardware transcoding in Plex itself. It's been over 14 hours since I did that and it hasn't crashed.