xcsascii
-
Posts
15 -
Joined
-
Last visited
Content Type
Profiles
Forums
Downloads
Store
Gallery
Bug Reports
Documentation
Landing
Posts posted by xcsascii
-
-
Quite a few discoveries
1. I ran a full parity check which took a little over 2 days. During that time docker service was disabled.
2. I started containers one at a time and found the culprit. It was my Plex docker. At around the 1 hour mark, there would be a shfs process (?) would generate iowaits and the system would just lock up.
I am still trying to find out what is causing the issue but at least for the mean time I can migrate my Plex work load to my proxmox cluster.
-
I have been having issues with docker crashing my server and/or docker not being able to stop. Would this solve my issue?
-
The server has been running for around 4 hours. However I have disabled docker.
Clearly something within docker is causing the problem. If it is a misconfiguration, or not is unknown at this time.
I am running a parity check right now, I don't think I've had a successful one for around a month.
Anyone having similar issues? I really can't get to the bottom of this.
-
I noticed that if I disable my network bonding that the server will stay up for quite a bit longer. I am talking 1-2 hours.
I don't understand why removing the network bond would decrease the amount of crashes. Here are updated logs.
-
I have been running memtest now for a little over 3 hours without any errors.
Is there anything else I should be looking at?
A few things I noticed today.
I noticed that at around a hour into a boot, I was unable to access the web gui (would just constantly load)
However I was able to log into the console, trying to shutdown from the console would just hang. No errors. Just wouldn't shutdown.
Kind of a crazy issue.
-
I have run memtest for around 30 mins, and it has passed.
Should I keep it running? or is there anything else I should be checking?
-
Good evening, I have been plagued by an ongoing server freeze/crash issue for the last 2 weeks or so. I really can't seem to find out what is causing it.
I was running on 6.12.7 without issue for quite some time, then all of a sudden I would get a crash, the web gui would almost act as if it was stuck logging in, the console would act the same. I have updated to 6.12.8 and now 6.12.9 and the crashes seem to have gotten worse. My most recent crash was 30 minutes exactly after startup.
Any help would be greatly appreciated
Attached is my diagnostic output as well as syslog from flash.
-
Ok, I think I may have figured out my issue. I was looking at another vpn container I am running, and found that it did not have any of my local DNS servers in resolv.conf.
I modified resolv.conf on the jackettvpn container and removed my local DNS servers. Only keeping 1.1.1.1 and 1.0.0.1.
I am now able to use jackettvpn without any issues.
Thanks for helping me better understand containers @Dyon learned a lot!
EDIT: Just to give you some background on my configuration incase this helps someone else in the future.
I am running 2x PiHole, and using PIA for VPN. I was able to ping local DNS, but was unable to resolve any domain names. I am unsure if using local based DNS caused the issue, or if it was something specific to PiHole that was causing my issue.
-
10 hours ago, Dyon said:
Very odd. Nothing has changed to the container since February. Iy only auto-updates / rebuilds whenever a new Jackett version has been released (which fixes indexing trackers). Nevertheless, I will investigate and look into what could be wrong.
Which Unraid version are you running @Skyshroud and @xcsascii?
I am on Version: 6.11.1
One other thing I wanted to mention is that when I add the RESTART_CONTAINER variable, the logs show:
[ERROR] Network is possibly down.
[INFO] Network is up
Over and over again, I am assuming this is the health check.
-
10 minutes ago, Skyshroud said:
Good evening all, I am currently experiencing the same issue as @xcsascii. Been running strong for over a year and the most recent update I performed on the the container seems to have broken it. Starting to work through the troubleshooting steps now.
Edit: After changing the repo to 'dyonr/jackettvpn:dev' and creating the RESTART_CONTAINER variable, the issue is even worse. The port mapping doesn't complete and I can't access the WebUI or logs to view what's going on with the container.
On your edit, if you power down the container, then look at logs they wont just close.
-
1
-
-
Thanks for the help so far.
My routing table looks slightly different as I have shim-br0 due to enabling: allow access to host networks
I noticed that if I change my network type to Custom : br0, and connect to the console I can ping one.one.one.one, but I cannot access the webui.
I am also using PIA if that makes any difference.
-
4 minutes ago, Dyon said:
Alright, then it seems to be something related to the DNS.
Could you run "cat /etc/resolv.conf" in the terminal and share the output
nameserver 10.10.10.30
nameserver 10.10.10.31
nameserver 10.10.10.1
nameserver 1.1.1.1
nameserver 1.0.0.1The first 3 are my local DNS servers. I am assuming I need to remove them? I am not really sure how.
-
12 minutes ago, Dyon said:
Could you edit the container and change/add the following:
Change the "Repository" field to "dyonr/jackettvpn:dev"
Add another Path, Port, Variable, Label or Device
> Config Type: Variable
> Name: RESTART_CONTAINER
> Key: RESTART_CONTAINER
> Value: 0
> Default Value: 0
> Description: (empty)
> Display: Always
> Required: No
> Password Mask: no
This will add a new variable called RESTART_CONTAINER, if this is set to '1', 'yes' or 'true', it will exit the container when the network is down. When set to something else (like 0) it won't shut down.
After changing this, start the container, open the console and please check if you can run the command `ping one.one.one.one`.
After adding the new variable, I can get into the console, I am unable to ping one.one.one.one, or any other domain names. I am able to ping IP addresses.
I can get to the webui now also.
-
I have been running jackettvpn for the last year or so, and this last week I have run into an issue.
jackettvpn will restart every 10 seconds or so with the error: [ERROR] Network is down, exiting this Docker
I noticed that if I do not define LAN_Network with my actual lan network, I could get to webui but wasn't able to successfully test any indexers
This seems to have started happening after I changed my default nic in unraid.
Here are my logs
dos2unix: converting file /config/openvpn/us_las_vegas.ovpn to Unix format... 2022-10-07 15:03:42.246707 [INFO] VPN remote line defined as 'us-lasvegas.privacy.network 1198' 2022-10-07 15:03:42.283072 [INFO] VPN_REMOTE defined as 'us-lasvegas.privacy.network' 2022-10-07 15:03:42.318004 [INFO] VPN_PORT defined as '1198' 2022-10-07 15:03:42.353312 [INFO] VPN_PROTOCOL defined as 'udp' 2022-10-07 15:03:42.388035 [INFO] VPN_DEVICE_TYPE defined as 'tun0' 2022-10-07 15:03:42.425347 [INFO] LAN_NETWORK defined as '10.10.10.0/24' 2022-10-07 15:03:42.460919 [INFO] NAME_SERVERS defined as '1.1.1.1,1.0.0.1' 2022-10-07 15:03:42.495560 [INFO] VPN_OPTIONS not defined (via -e VPN_OPTIONS) 2022-10-07 15:03:42.530211 [INFO] Adding 1.1.1.1 to resolv.conf 2022-10-07 15:03:42.565020 [INFO] Adding 1.0.0.1 to resolv.conf 2022-10-07 15:03:42.595756 [INFO] Starting OpenVPN... 2022-10-07 15:03:42 DEPRECATED OPTION: --cipher set to 'aes-128-cbc' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM). Future OpenVPN version will ignore --cipher for cipher negotiations. Add 'aes-128-cbc' to --data-ciphers or change --cipher 'aes-128-cbc' to --data-ciphers-fallback 'aes-128-cbc' to silence this warning. 2022-10-07 15:03:42 WARNING: file 'credentials.conf' is group or others accessible 2022-10-07 15:03:42 OpenVPN 2.5.1 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on May 14 2021 2022-10-07 15:03:42 library versions: OpenSSL 1.1.1n 15 Mar 2022, LZO 2.10 2022-10-07 15:03:42 CRL: loaded 1 CRLs from file -----BEGIN X509 CRL----- [REMOVED for forum post] -----END X509 CRL----- 2022-10-07 15:03:42 TCP/UDP: Preserving recently used remote address: [AF_INET]143.244.48.191:1198 2022-10-07 15:03:42 UDP link local: (not bound) 2022-10-07 15:03:42 UDP link remote: [AF_INET]143.244.48.191:1198 2022-10-07 15:03:42 [losangeles408] Peer Connection Initiated with [AF_INET]143.244.48.191:1198 2022-10-07 15:03:42 TUN/TAP device tun0 opened 2022-10-07 15:03:42 net_iface_mtu_set: mtu 1500 for tun0 2022-10-07 15:03:42 net_iface_up: set tun0 up 2022-10-07 15:03:42 net_addr_v4_add: 10.3.112.183/24 dev tun0 2022-10-07 15:03:42 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this 2022-10-07 15:03:42 Initialization Sequence Completed 2022-10-07 15:03:43.964770 [INFO] Docker network defined as 172.17.0.0/16 2022-10-07 15:03:44.006199 [INFO] Adding 10.10.10.0/24 as route via docker eth0 2022-10-07 15:03:44.041705 [INFO] ip route defined as follows... -------------------- 0.0.0.0/1 via 10.3.112.1 dev tun0 default via 172.17.0.1 dev eth0 10.3.112.0/24 dev tun0 proto kernel scope link src 10.3.112.183 10.10.10.0/24 via 172.17.0.1 dev eth0 128.0.0.0/1 via 10.3.112.1 dev tun0 143.244.48.191 via 172.17.0.1 dev eth0 172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.2 -------------------- iptable_mangle 16384 1 ip_tables 28672 3 iptable_filter,iptable_nat,iptable_mangle x_tables 45056 17 ip6table_filter,xt_conntrack,iptable_filter,ip6table_nat,nft_compat,xt_tcpudp,xt_addrtype,xt_CHECKSUM,xt_nat,ip6_tables,ipt_REJECT,ip_tables,iptable_nat,ip6table_mangle,xt_MASQUERADE,iptable_mangle,xt_mark 2022-10-07 15:03:44.084100 [INFO] iptable_mangle support detected, adding fwmark for tables 2022-10-07 15:03:44.213293 [INFO] iptables defined as follows... -------------------- -P INPUT DROP -P FORWARD ACCEPT -P OUTPUT DROP -A INPUT -i tun0 -j ACCEPT -A INPUT -s 172.17.0.0/16 -d 172.17.0.0/16 -j ACCEPT -A INPUT -i eth0 -p udp -m udp --sport 1198 -j ACCEPT -A INPUT -i eth0 -p tcp -m tcp --dport 9117 -j ACCEPT -A INPUT -i eth0 -p tcp -m tcp --sport 9117 -j ACCEPT -A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT -A INPUT -i lo -j ACCEPT -A OUTPUT -o tun0 -j ACCEPT -A OUTPUT -s 172.17.0.0/16 -d 172.17.0.0/16 -j ACCEPT -A OUTPUT -o eth0 -p udp -m udp --dport 1198 -j ACCEPT -A OUTPUT -o eth0 -p tcp -m tcp --dport 9117 -j ACCEPT -A OUTPUT -o eth0 -p tcp -m tcp --sport 9117 -j ACCEPT -A OUTPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT -A OUTPUT -o lo -j ACCEPT -------------------- 2022-10-07 15:03:44.270441 [INFO] A group with PGID 100 already exists in /etc/group, nothing to do. 2022-10-07 15:03:44.305489 [INFO] An user with PUID 99 already exists in /etc/passwd, nothing to do. 2022-10-07 15:03:44.338014 [INFO] UMASK defined as '002' 2022-10-07 15:03:44.477298 [WARNING] There is no password set via Jackett's web interface or as an environment variable! 2022-10-07 15:03:44.508562 [WARNING] Anyone on your network could access Jackett without authentication! 2022-10-07 15:03:44.538196 [WARNING] Or even the whole world if you did port-fortwarding! 2022-10-07 15:03:44.568537 [WARNING] It's adviced to set one via either the web interface or as environment variable 2022-10-07 15:03:44.598812 [INFO] Starting Jackett daemon... Logging to /config/Jackett/Logs/log.txt. 2022-10-07 15:03:45.645209 [INFO] Started Jackett daemon successfully... 2022-10-07 15:03:45.648701 [INFO] Jackett PID: 204 2022-10-07 15:03:45.683864 [ERROR] Network is down, exiting this Docker
Server locking up in random intervals
in General Support
Posted
It has been 10 days and I have had 8 days of uptime. Here is what I have learned and what I am doing now. If someone is having the same issue as I am this may not be the solution you are looking for.
I noticed that regardless of IOWAIT, I am not seeing any real disk usage on the array. Maybe 10-20mb/s on one or two disks. the CPU on the homepage GUI shows that the CPU is running high (80%+) but htop and glances contradicts that information. Showing a single CPU core running at 100% while the remaining are much much lower.
As a temporary work around I moved the Plex workload over to a VM running on unraid, and it worked without issue. I then moved the workload over to Proxmox and copied the existing cache and metadata. I have since been running Plex in Proxmox pointed to unraid via NFS.
Based on this I think it is safe to assume that the file manipulation on media files themselves isn't the root issue. It must be the appdata share. However the disk usage is also pretty minimal. My appdata is hosted on a zfs pool with 10k sas drives. HOWEVER the mount is still /mnt/user/appdata. The appdata folder is then set to the cache as the primary folder.
Based on this information, I have begun the process of migrating all of the containers off unraid, and onto proxmox. I will probably use unraid as pure storage only from now on. Which is really a shame because the community apps is what pushed me to purchase a license. Oh well maybe I have just hit the magic number of files to hardware to warrant a different system for compute.
TLDR; Plex container was causing high IOWAIT which was causing my issues. Moving the workload off unraid solved* my issue.