David Grenon

2 years later ! Saved my ass too ! Thanks.

TLDR; Setting Credit detection to online only (no local detection) and make sure nothing is in temp folder assigned to Plex container (related to plex) solved the memory usage issue. Ok so problem solved here and a lot of people having issues with 6.12.X might have the same root cause. Writing this down here so it could help other people in the same boat. Downgrading my server to 6.11.5 was the right initial thing to do to actually get some logs and pinpoint the actual issues. UNRAID OS VERSION WAS NOT THE CAUSE. But it was way more difficult to troubleshoot with 6.12.x* By putting Plex TMP folder to /shm instead of /tmp prevent from it to take all the memory and make the system misbehave. I was then able to see with the container and server logs to see whats going on without being in the front of the server when the issue happen. A thing that was kinda persistent is that the issue was almost all the time happening around 5h30 - 7AM in the morning. Digging down process and ram usage I found that: HTOP would show that couple Plex transcoder process using a lot of CPU Temp folder (/SHM) would actually fill up with couple creditDetection-(UID), etc Digging more, there is a thing called Credit detection (skip outro) that is not the same as the intro detection: https://support.plex.tv/articles/credits-detection/ By default this thing is set to : both, try online first Which mean if the Outro metadata doesn't exist on Plex DB let your server transcode the files to search for pattern or w/e magic this does. This option by default start when media is added AND at maintenance hours: By default at 2:00 AM End of maintenance tasks are also by default at 5AM and maybe something at 5AM was triggering end of task but wouldn't work. Anyway, putting this option to : online only (no local detection) completely solved my issue. This issue happened out of nowhere. Why ? because at some point, a TV show was added to my server and that one had probably no entry in the Plex db and made the task goes. Its unfortunate that this kind of features just break the whole system. Moral of the story, never map a container path to Unraid /tmp. Rather use /Shm.

Seeing this: mapping /dev/shm/ instead of /TMP would limit to the half of ram (16go) in my case. If I have no more issues after this, I'll have to check how specificaly that container work (why 32gb is filling up and never released) But thats another story. My next question is.. Could it be possible that this kind of behavior is not handled the same in 6.12.x ? Never had OOM before in 6.12.x only freeze + hard reboot. (Thats why I'm back to 6.11.5 now with no app store) And also, since you're a Moderator, do you have access to know if the future 6.13 kernel will handle these a better way ? Thank you.

Ok but do you think other errors were also linked to that issue (white logs after + webui flashing not long after) ?

got this this morning and I was on my cellphone so I only have printscreen (see attachement) Later on a computer the UI was responsive, but started to do the issue (WebUI refreshing and got these logs (See attachement) Still can't generatediag file when that issue is happening.

Ok nvm the NTP errors... I guess it will solve by itself: "This is normal after rebooting the server, time will get synchronized after a while (assuming you have NTP configured)."

Feb 26 14:00:01 Tower crond[1172]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null Feb 26 14:11:30 Tower emhttpd: shcmd (211): ln -sf /usr/share/zoneinfo/America/New_York /etc/localtime Feb 26 14:11:30 Tower emhttpd: shcmd (212): /usr/local/emhttp/webGui/scripts/update_access Feb 26 14:11:30 Tower root: sshd: no process found Feb 26 14:11:31 Tower emhttpd: shcmd (213): /etc/rc.d/rc.ntpd stop Feb 26 14:11:31 Tower ntpd[1151]: ntpd exiting on signal 1 (Hangup) Feb 26 14:11:31 Tower ntpd[1151]: 127.127.1.0 local addr 127.0.0.1 -> <null> Feb 26 14:11:31 Tower ntpd[1151]: 216.239.35.0 local addr 192.168.86.248 -> <null> Feb 26 14:11:31 Tower ntpd[1151]: 216.239.35.4 local addr 192.168.86.248 -> <null> Feb 26 14:11:31 Tower ntpd[1151]: 216.239.35.8 local addr 192.168.86.248 -> <null> Feb 26 14:11:31 Tower ntpd[1151]: 216.239.35.12 local addr 192.168.86.248 -> <null> Feb 26 14:11:31 Tower root: Stopping NTP daemon... Feb 26 14:11:31 Tower emhttpd: shcmd (214): date -s '2024-02-26 14:11:19' Feb 26 14:11:19 Tower root: Mon Feb 26 14:11:19 EST 2024 Feb 26 14:11:19 Tower emhttpd: shcmd (215): hwclock --utc --systohc --noadjfile Feb 26 14:11:19 Tower nginx: 2024/02/26 14:11:19 [error] 1533#1533: nchan: A message from the past has just been published. Unless the system time has been adjusted, this should never happen. Feb 26 14:11:19 Tower nginx: 2024/02/26 14:11:19 [error] 1533#1533: nchan: A message from the past has just been published. Unless the system time has been adjusted, this should never happen. Feb 26 14:11:19 Tower nginx: 2024/02/26 14:11:19 [error] 1533#1533: nchan: A message from the past has just been published. Unless the system time has been adjusted, this should never happen. Feb 26 14:11:19 Tower nginx: 2024/02/26 14:11:19 [error] 1533#1533: nchan: A message from the past has just been published. Unless the system time has been adjusted, this should never happen. Feb 26 14:11:19 Tower nginx: 2024/02/26 14:11:19 [error] 1533#1533: nchan: A message from the past has just been published. Unless the system time has been adjusted, this should never happen. Feb 26 14:11:20 Tower nginx: 2024/02/26 14:11:20 [error] 1533#1533: nchan: A message from the past has just been published. Unless the system time has been adjusted, this should never happen. Feb 26 14:11:20 Tower nginx: 2024/02/26 14:11:20 [error] 1533#1533: nchan: A message from the past has just been published. Unless the system time has been adjusted, this should never happen. Feb 26 14:11:21 Tower nginx: 2024/02/26 14:11:21 [error] 1533#1533: nchan: A message from the past has just been published. Unless the system time has been adjusted, this should never happen. Feb 26 14:11:22 Tower emhttpd: shcmd (216): ln -sf /usr/share/zoneinfo/America/New_York /etc/localtime Feb 26 14:11:22 Tower emhttpd: shcmd (217): /usr/local/emhttp/webGui/scripts/update_access Feb 26 14:11:22 Tower root: sshd: no process found Feb 26 14:11:23 Tower emhttpd: shcmd (218): /etc/rc.d/rc.ntpd restart Feb 26 14:11:23 Tower root: Stopping NTP daemon... Feb 26 14:11:24 Tower ntpd[3774]: ntpd [email protected] Fri Jun 3 04:17:10 UTC 2022 (1): Starting Feb 26 14:11:24 Tower ntpd[3774]: Command line: /usr/sbin/ntpd -g -u ntp:ntp Feb 26 14:11:24 Tower ntpd[3774]: ---------------------------------------------------- Feb 26 14:11:24 Tower ntpd[3774]: ntp-4 is maintained by Network Time Foundation, Feb 26 14:11:24 Tower ntpd[3774]: Inc. (NTF), a non-profit 501(c)(3) public-benefit Feb 26 14:11:24 Tower ntpd[3774]: corporation. Support and training for ntp-4 are Feb 26 14:11:24 Tower ntpd[3774]: available at https://www.nwtime.org/support Feb 26 14:11:24 Tower ntpd[3774]: ---------------------------------------------------- Feb 26 14:11:24 Tower ntpd[3776]: proto: precision = 0.044 usec (-24) Feb 26 14:11:24 Tower ntpd[3776]: basedate set to 2022-05-22 Feb 26 14:11:24 Tower ntpd[3776]: gps base set to 2022-05-22 (week 2211) Feb 26 14:11:24 Tower ntpd[3776]: Listen normally on 0 lo 127.0.0.1:123 Feb 26 14:11:24 Tower ntpd[3776]: Listen normally on 1 br0 192.168.86.248:123 Feb 26 14:11:24 Tower ntpd[3776]: Listen normally on 2 lo [::1]:123 Feb 26 14:11:24 Tower ntpd[3776]: Listening on routing socket on fd #19 for interface updates Feb 26 14:11:24 Tower ntpd[3776]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized Feb 26 14:11:24 Tower ntpd[3776]: kernel reports TIME_ERROR: 0x2041: Clock Unsynchronized Feb 26 14:11:24 Tower root: Starting NTP daemon: /usr/sbin/ntpd -g -u ntp:ntp I also get a lot of these: Feb 26 14:00:01 Tower crond[1172]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null My college has different message when the mover run. Sorry, lol I'm just trying to get unraid run the correct way and be stable...

Now I get this, but the system seems stable: Feb 26 11:11:01 Tower avahi-daemon[2411]: Registering new address record for 172.18.0.1 on br-3968de4c6e52.IPv4. Feb 26 11:11:01 Tower avahi-daemon[2411]: Joining mDNS multicast group on interface docker0.IPv4 with address 172.17.0.1. Feb 26 11:11:01 Tower avahi-daemon[2411]: New relevant interface docker0.IPv4 for mDNS. Feb 26 11:11:01 Tower avahi-daemon[2411]: Registering new address record for 172.17.0.1 on docker0.IPv4. Feb 26 11:11:01 Tower avahi-daemon[2411]: Joining mDNS multicast group on interface br-a5d67fd16cf9.IPv4 with address 172.31.200.1. Feb 26 11:11:01 Tower avahi-daemon[2411]: New relevant interface br-a5d67fd16cf9.IPv4 for mDNS. Feb 26 11:11:01 Tower avahi-daemon[2411]: Registering new address record for 172.31.200.1 on br-a5d67fd16cf9.IPv4. Feb 26 11:11:02 Tower avahi-daemon[2411]: Server startup complete. Host name is Tower.local. Local service cookie is 2077987440. Feb 26 11:11:02 Tower rc.docker: created network br0 with subnets: 192.168.86.0/24; Feb 26 11:11:02 Tower emhttpd: shcmd (189): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 1 Feb 26 11:11:02 Tower kernel: loop3: detected capacity change from 0 to 2097152 Feb 26 11:11:02 Tower kernel: BTRFS: device fsid 5e7e3797-4829-4ac6-8f6e-e3558bb1e997 devid 1 transid 23 /dev/loop3 scanned by mount (3305) Feb 26 11:11:02 Tower kernel: BTRFS info (device loop3): using free space tree Feb 26 11:11:02 Tower kernel: BTRFS info (device loop3): has skinny extents Feb 26 11:11:02 Tower kernel: BTRFS info (device loop3): enabling ssd optimizations Feb 26 11:11:02 Tower root: Resize device id 1 (/dev/loop3) from 1.00GiB to max Feb 26 11:11:02 Tower emhttpd: shcmd (191): /etc/rc.d/rc.libvirt start Feb 26 11:11:02 Tower root: Starting virtlockd... Feb 26 11:11:02 Tower root: Starting virtlogd... Feb 26 11:11:02 Tower root: Starting libvirtd... Feb 26 11:11:02 Tower kernel: tun: Universal TUN/TAP device driver, 1.6 Feb 26 11:11:02 Tower emhttpd: nothing to sync Feb 26 11:11:02 Tower avahi-daemon[2411]: Joining mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1. Feb 26 11:11:02 Tower avahi-daemon[2411]: New relevant interface virbr0.IPv4 for mDNS. Feb 26 11:11:02 Tower avahi-daemon[2411]: Registering new address record for 192.168.122.1 on virbr0.IPv4. Feb 26 11:11:02 Tower dnsmasq[3454]: started, version 2.87 cachesize 150 Feb 26 11:11:02 Tower dnsmasq[3454]: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset no-nftset auth cryptohash DNSSEC loop-detect inotify dumpfile Feb 26 11:11:02 Tower dnsmasq-dhcp[3454]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h Feb 26 11:11:02 Tower dnsmasq-dhcp[3454]: DHCP, sockets bound exclusively to interface virbr0 Feb 26 11:11:02 Tower dnsmasq[3454]: reading /etc/resolv.conf Feb 26 11:11:02 Tower dnsmasq[3454]: using nameserver 192.168.86.1#53 Feb 26 11:11:02 Tower dnsmasq[3454]: read /etc/hosts - 2 addresses Feb 26 11:11:02 Tower dnsmasq[3454]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses Feb 26 11:11:02 Tower dnsmasq-dhcp[3454]: read /var/lib/libvirt/dnsmasq/default.hostsfile Feb 26 11:11:02 Tower kernel: mpt3sas 0000:01:00.0: invalid VPD tag 0x00 (size 0) at offset 0; assume missing optional EEPROM Feb 26 11:11:03 Tower avahi-daemon[2411]: Service "Tower" (/services/sftp-ssh.service) successfully established. Feb 26 11:11:03 Tower avahi-daemon[2411]: Service "Tower" (/services/smb.service) successfully established. Feb 26 11:11:03 Tower avahi-daemon[2411]: Service "Tower" (/services/ssh.service) successfully established. Feb 26 11:14:04 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe987374c.3fc9424c does not match aorg 0000000000.00000000 from [email protected] xmt 0xe987374c.4335bf67 Feb 26 11:14:34 Tower ntpd[1151]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Feb 26 11:21:29 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9873909.3fc9aded does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9873909.42d54b5e Feb 26 11:21:31 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe987390b.3fc90f48 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe987390b.43ac6ed7 Feb 26 11:31:36 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9873b68.3fc90bcc does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9873b68.43446a36 Feb 26 11:38:38 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9873d0e.3fc9049d does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9873d0e.42c720b4 Feb 26 11:38:44 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9873d14.3fc9173a does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9873d14.43af68b2 Feb 26 11:40:39 Tower kernel: docker0: port 1(vethac0555d) entered blocking state Feb 26 11:40:39 Tower kernel: docker0: port 1(vethac0555d) entered disabled state Feb 26 11:40:39 Tower kernel: device vethac0555d entered promiscuous mode Feb 26 11:40:39 Tower kernel: docker0: port 1(vethac0555d) entered blocking state Feb 26 11:40:39 Tower kernel: docker0: port 1(vethac0555d) entered forwarding state Feb 26 11:40:39 Tower kernel: docker0: port 1(vethac0555d) entered disabled state Feb 26 11:40:39 Tower kernel: eth0: renamed from veth4fe4954 Feb 26 11:40:39 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethac0555d: link becomes ready Feb 26 11:40:39 Tower kernel: docker0: port 1(vethac0555d) entered blocking state Feb 26 11:40:39 Tower kernel: docker0: port 1(vethac0555d) entered forwarding state Feb 26 11:40:39 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready Feb 26 11:40:41 Tower avahi-daemon[2411]: Joining mDNS multicast group on interface vethac0555d.IPv6 with address fe80::70f9:a2ff:fe24:c674. Feb 26 11:40:41 Tower avahi-daemon[2411]: New relevant interface vethac0555d.IPv6 for mDNS. Feb 26 11:40:41 Tower avahi-daemon[2411]: Registering new address record for fe80::70f9:a2ff:fe24:c674 on vethac0555d.*. Feb 26 11:40:41 Tower avahi-daemon[2411]: Joining mDNS multicast group on interface docker0.IPv6 with address fe80::42:2bff:fecb:f1eb. Feb 26 11:40:41 Tower avahi-daemon[2411]: New relevant interface docker0.IPv6 for mDNS. Feb 26 11:40:41 Tower avahi-daemon[2411]: Registering new address record for fe80::42:2bff:fecb:f1eb on docker0.*. Feb 26 11:44:32 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9873e70.3fc92d1f does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9873e70.43c2b3ac Feb 26 11:49:09 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9873f85.3fc92c10 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9873f85.45df56fb Feb 26 11:56:10 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe987412a.3fc90fb8 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe987412a.42d41a2c Feb 26 11:56:33 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9874141.3fc964e5 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9874141.439751d3 Feb 26 12:00:01 Tower crond[1172]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null Feb 26 12:01:39 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9874273.3fc92654 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9874273.439d2f70 Feb 26 12:06:14 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9874386.3fc9043b does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9874386.42fff60f Feb 26 12:13:17 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe987452d.3fc96d7d does not match aorg 0000000000.00000000 from [email protected] xmt 0xe987452d.42bdc2c5 Feb 26 12:14:22 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe987456e.3fc8ec63 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe987456e.43b7275c Feb 26 12:19:43 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe98746af.3fc91a06 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe98746af.43aca198 Feb 26 12:23:18 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9874786.3fc90b54 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9874786.43075872 Feb 26 12:31:16 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9874964.3fc8d598 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9874964.42ae4cfa Feb 26 12:31:50 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9874986.3fc9286b does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9874986.438930be Feb 26 12:37:14 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9874aca.3fc92abe does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9874aca.43c78221 Feb 26 12:41:09 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9874bb5.3fc91e78 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9874bb5.432b7c42 Feb 26 12:48:52 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9874d84.3fc943d8 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9874d84.42baef11 Feb 26 12:49:04 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9874d90.3fc95537 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9874d90.43746389 Feb 26 12:54:57 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9874ef1.3fc91716 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9874ef1.43b5e76b Feb 26 12:59:10 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9874fee.3fc90b40 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9874fee.43226013 Feb 26 13:00:02 Tower crond[1172]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null Feb 26 13:06:11 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9875193.3fc901c1 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9875193.428d7fe1 Feb 26 13:06:47 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe98751b7.3fc93d2c does not match aorg 0000000000.00000000 from [email protected] xmt 0xe98751b7.43b9ebf2 Feb 26 13:12:39 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9875317.42be92f2 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9875317.469be7b2 Feb 26 13:13:43 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9875357.3fc945c4 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9875357.43a82aa1 Feb 26 13:14:47 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9875397.3fc95b96 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9875397.4393ad88 Feb 26 13:16:39 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9875407.3fca48f4 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9875407.43270072 Feb 26 13:17:43 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe9875447.3fc97c82 does not match aorg 0000000000.00000000 from [email protected] xmt 0xe9875447.4310d89f Feb 26 13:18:50 Tower ntpd[1151]: receive: Unexpected origin timestamp 0xe987548a.3fc9571d does not match aorg 0000000000.00000000 from [email protected] xmt 0xe987548a.432c3111

Ok. Now running in safemode with only Plex docker running.

Will do. Anything from the logs seems suspect to you?

Hi, I'm having an issue where at some point (2-3 days uptime) my server webui is refreshing non-stop and flap between Started/Unassigned. I wasn't able to download the diag because I get this webpage when it try to download it: 404 Not Found nginx I can give you the logs I see from the webui before reboot (see attachement) and diag after reboot (see attachement) For reference I've been having issues with 6.12.x freezing every 1-2 days and reverted to 6.11.5 I deleted statistic plugin before the last reboot. To be clear, this issue happened to me once while I was on 6.12.8. Now I was up for 3 days but I get this issue. Reference to another thread for full history: https://forums.unraid.net/bug-reports/stable-releases/6124-server-hangs-once-a-day-since-updating-to-6124-r2698/page/3/?tab=comments#comment-27386 Thank you, terminal logs before reboot.txt tower-diagnostics-20240226-1024.zip

Update: Now I need to reboot my server on a daily basis (pull the cord) because I can't connect through ssh or web server.... If they dont fix it soon ill revert to earlier version...ths is ridiculous.

It seems that I have similar issue as you. And nothing in my logs seems to point something. Never had this issue before Version: 6.12.3. Parity check is running and during the day my dockers seems to have stopped working. Restarting the service hanged and by refreshing the page I now see the service is stopped. Now I can't even go back to docker section to restart the service. My parity check is still running and its at around 60% now.