February 9, 20224 yr Thanks in advance, I was following the IBRACORP guide to installing Authelia, struggling with the configuration.yaml as I could not get Authelia to launch. I had been through several rounds of edits on the confiugration.yaml and attempted to launch Authelia when my array seemed to get kicked offline and the GUI was available but very unstable. The Docker manager and VM manager reported that the array was offline, no dockers were visible or accessible from the GUI. I tried to use the GUI to perform a clean reboot, but it became unresponsive and required a hard cycle. The machine booted back into a parity check, but everything else seemed normal. I deleted the Authelia installation and appdata, figured I would chase that setup at another time. Unfortunately this crashing has continued about every 15-20 hours. I'll attach the diagnostics, I'm no expert but it seems like the Docker.img is corrupt? Is there something else going on? Here's a snippet of the syslog, I received an email around 12:19AM stating that the parity check had finished. In reality it looks like the array just went offline. Thanks again, diagnostics attached. Feb 8 14:55:06 Tower kernel: eth0: renamed from vethea5eb4e Feb 8 14:55:06 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethb9f82a8: link becomes ready Feb 8 14:55:06 Tower kernel: docker0: port 6(vethba926a5) entered disabled state Feb 8 14:55:06 Tower kernel: eth0: renamed from veth060af26 Feb 8 14:55:06 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethba926a5: link becomes ready Feb 8 14:55:06 Tower kernel: docker0: port 6(vethba926a5) entered blocking state Feb 8 14:55:06 Tower kernel: docker0: port 6(vethba926a5) entered forwarding state Feb 8 14:55:11 Tower kernel: vethea5eb4e: renamed from eth0 Feb 8 14:55:11 Tower kernel: docker0: port 4(vethb9f82a8) entered disabled state Feb 8 14:55:11 Tower kernel: docker0: port 4(vethb9f82a8) entered disabled state Feb 8 14:55:11 Tower kernel: device vethb9f82a8 left promiscuous mode Feb 8 14:55:11 Tower kernel: docker0: port 4(vethb9f82a8) entered disabled state Feb 8 14:55:11 Tower kernel: docker0: port 4(veth3365f3b) entered blocking state Feb 8 14:55:11 Tower kernel: docker0: port 4(veth3365f3b) entered disabled state Feb 8 14:55:11 Tower kernel: device veth3365f3b entered promiscuous mode Feb 8 14:55:11 Tower kernel: docker0: port 4(veth3365f3b) entered blocking state Feb 8 14:55:11 Tower kernel: docker0: port 4(veth3365f3b) entered forwarding state Feb 8 14:55:11 Tower kernel: eth0: renamed from veth14b8d0f Feb 8 14:55:11 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth3365f3b: link becomes ready Feb 8 14:58:03 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 15:00:16 Tower crond[3055]: exit status 1 from user **** /usr/local/sbin/mover &> /dev/null Feb 8 15:10:08 Tower dhcpcd[2958]: br1: dhcp_sendudp: Network is unreachable Feb 8 15:26:25 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 15:54:48 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 16:00:16 Tower crond[3055]: exit status 1 from user **** /usr/local/sbin/mover &> /dev/null Feb 8 16:10:08 Tower dhcpcd[2958]: br1: dhcp_sendudp: Network is unreachable Feb 8 16:23:09 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 16:25:16 Tower emhttpd: cmd: /usr/local/emhttp/plugins/community.applications/scripts/pluginInstall.php install https://raw.githubusercontent.com/Squidly271/ca.turbo/master/plugins/ca.turbo.plg Feb 8 16:25:16 Tower ****: plugin: running: anonymous Feb 8 16:25:16 Tower ****: plugin: running: anonymous Feb 8 16:25:16 Tower ****: plugin: creating: /boot/config/plugins/ca.turbo/ca.turbo-2021.03.10-x86_64-1.txz - downloading from URL https://raw.github.com/Squidly271/ca.turbo/master/archive/ca.turbo-2021.03.10-x86_64-1.txz Feb 8 16:25:17 Tower ****: plugin: checking: /boot/config/plugins/ca.turbo/ca.turbo-2021.03.10-x86_64-1.txz - MD5 Feb 8 16:25:17 Tower ****: plugin: running: /boot/config/plugins/ca.turbo/ca.turbo-2021.03.10-x86_64-1.txz Feb 8 16:25:17 Tower ****: plugin: running: anonymous Feb 8 16:25:17 Tower ****: plugin: running: anonymous Feb 8 16:25:38 Tower ****: Starting Auto Turbo Feb 8 16:25:38 Tower kernel: mdcmd (37): set md_write_method 1 Feb 8 16:25:38 Tower kernel: Feb 8 16:26:28 Tower ****: Stopping Auto Turbo Feb 8 16:26:28 Tower ****: Starting Auto Turbo Feb 8 16:26:29 Tower kernel: mdcmd (38): set md_write_method 1 Feb 8 16:26:29 Tower kernel: Feb 8 16:36:58 Tower emhttpd: spinning down /dev/sdh Feb 8 16:36:58 Tower emhttpd: spinning down /dev/sdi Feb 8 16:51:32 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 17:00:02 Tower crond[3055]: exit status 1 from user **** /usr/local/sbin/mover &> /dev/null Feb 8 17:10:08 Tower dhcpcd[2958]: br1: dhcp_sendudp: Network is unreachable Feb 8 17:19:53 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 17:46:32 Tower webGUI: Successful login user **** from ****217 Feb 8 17:48:14 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 18:00:16 Tower crond[3055]: exit status 1 from user **** /usr/local/sbin/mover &> /dev/null Feb 8 18:10:08 Tower dhcpcd[2958]: br1: dhcp_sendudp: Network is unreachable Feb 8 18:11:56 Tower webGUI: Successful login user **** from ****217 Feb 8 18:16:36 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 18:44:58 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 19:00:16 Tower crond[3055]: exit status 1 from user **** /usr/local/sbin/mover &> /dev/null Feb 8 19:10:08 Tower dhcpcd[2958]: br1: dhcp_sendudp: Network is unreachable Feb 8 19:13:19 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 19:41:42 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 20:00:16 Tower crond[3055]: exit status 1 from user **** /usr/local/sbin/mover &> /dev/null Feb 8 20:10:02 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 20:10:08 Tower dhcpcd[2958]: br1: dhcp_sendudp: Network is unreachable Feb 8 20:38:23 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 20:58:27 Tower sSMTP[34633]: Creating SSL connection to host Feb 8 20:58:27 Tower sSMTP[34633]: SSL connection using TLS_AES_256_GCM_SHA384 Feb 8 20:58:29 Tower sSMTP[34633]: Sent mail for ********** (221 2.0.0 closing connection y3sm6443052oix.41 - gsmtp) uid=0 username=**** outbytes=707 Feb 8 21:00:16 Tower crond[3055]: exit status 1 from user **** /usr/local/sbin/mover &> /dev/null Feb 8 21:06:44 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 21:10:08 Tower dhcpcd[2958]: br1: dhcp_sendudp: Network is unreachable Feb 8 21:35:06 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 22:00:07 Tower crond[3055]: exit status 1 from user **** /usr/local/sbin/mover &> /dev/null Feb 8 22:03:28 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 22:10:08 Tower dhcpcd[2958]: br1: dhcp_sendudp: Network is unreachable Feb 8 22:31:50 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 23:00:12 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 23:00:16 Tower crond[3055]: exit status 1 from user **** /usr/local/sbin/mover &> /dev/null Feb 8 23:10:08 Tower dhcpcd[2958]: br1: dhcp_sendudp: Network is unreachable Feb 8 23:28:34 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 8 23:56:55 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 9 00:00:16 Tower crond[3055]: exit status 1 from user **** /usr/local/sbin/mover &> /dev/null Feb 9 00:08:15 Tower emhttpd: read SMART /dev/sdh Feb 9 00:08:45 Tower emhttpd: read SMART /dev/sdi Feb 9 00:10:08 Tower dhcpcd[2958]: br1: dhcp_sendudp: Network is unreachable Feb 9 00:19:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:19:01 Tower sSMTP[36648]: Creating SSL connection to host Feb 9 00:19:01 Tower sSMTP[36648]: SSL connection using TLS_AES_256_GCM_SHA384 Feb 9 00:19:03 Tower sSMTP[36648]: Sent mail for ********** (221 2.0.0 closing connection h2sm6140912ots.51 - gsmtp) uid=0 username=**** outbytes=715 Feb 9 00:19:15 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only 4096 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:20:00 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only 4096 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:20:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:20:45 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only 4096 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:21:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:21:31 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only 4096 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:22:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:22:17 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only 4096 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:23:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:23:03 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only 4096 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:23:49 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only 4096 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:24:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:24:34 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only 4096 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:25:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:25:17 Tower ipmiseld[7365]: ****6: Error connecting: session timeout Feb 9 00:25:20 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only 4096 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:26:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:26:05 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only 4096 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:26:51 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only 4096 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:27:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:27:37 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only 4096 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:28:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:28:22 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only 4096 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:28:31 Tower winbindd[1400]: [2022/02/09 00:28:31.484103, 0] ../../source3/lib/util.c:500(reinit_after_fork) Feb 9 00:28:31 Tower winbindd[1400]: messaging_reinit() failed: NT_STATUS_DISK_FULL Feb 9 00:28:31 Tower nmbd[1401]: [2022/02/09 00:28:31.494604, 0] ../../source3/lib/util.c:500(reinit_after_fork) Feb 9 00:28:31 Tower nmbd[1401]: messaging_reinit() failed: NT_STATUS_DISK_FULL Feb 9 00:28:37 Tower smbd[1437]: [2022/02/09 00:28:37.208898, 0] ../../source3/lib/util.c:500(reinit_after_fork) Feb 9 00:28:37 Tower smbd[1437]: messaging_reinit() failed: NT_STATUS_DISK_FULL Feb 9 00:29:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:29:09 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:29:54 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:30:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:30:40 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:31:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:31:26 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:32:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:32:11 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:32:57 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:33:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:33:43 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:34:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:34:29 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:35:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:35:15 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:36:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:36:02 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:36:47 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:37:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:37:32 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:38:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:38:19 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:39:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:39:04 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:39:50 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:40:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:40:35 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:41:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:41:22 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:42:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:42:08 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:42:54 Tower rc.diskinfo[12839]: PHP Warning: file_put_contents(): Only -1 of 10928 bytes written, possibly out of free disk space in /etc/rc.d/rc.diskinfo on line 266 Feb 9 00:43:01 Tower crond[3055]: exit status 2 from user **** /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null Feb 9 00:43:31 Tower winbindd[36574]: [2022/02/09 00:43:31.492601, 0] ../../source3/lib/util.c:500(reinit_after_fork) Feb 9 00:43:31 Tower winbindd[36574]: messaging_reinit() failed: NT_STATUS_DISK_FULL Feb 9 00:43:31 Tower nmbd[36575]: [2022/02/09 00:43:31.499934, 0] ../../source3/lib/util.c:500(reinit_after_fork) Feb 9 00:43:31 Tower nmbd[36575]: messaging_reinit() failed: NT_STATUS_DISK_FULL Feb 9 00:43:37 Tower smbd[36975]: [2022/02/09 00:43:37.244701, 0] ../../source3/lib/util.c:500(reinit_after_fork) Feb 9 00:43:37 Tower smbd[36975]: messaging_reinit() failed: NT_STATUS_DISK_FULL tower-diagnostics-20220209-0800.zip
February 9, 20224 yr Author Crashed again, I was able to grab syslog this time. This may be a side-effect of also adjusting my rclone mount settings the same day. I had increased the cache size and I had not supplied a cache-dir setting. I am assuming it exceeded the RAM and possibly lead to this problem. I've updated my rclone mount scripts and I will continue to monitor. Snip of syslog: eb 9 08:54:16 Tower ipmiseld[7519]: ******6: Error connecting: session timeout Feb 9 08:57:43 Tower kernel: docker0: port 9(veth62b81ae) entered blocking state Feb 9 08:57:43 Tower kernel: docker0: port 9(veth62b81ae) entered disabled state Feb 9 08:57:43 Tower kernel: device veth62b81ae entered promiscuous mode Feb 9 08:57:43 Tower kernel: docker0: port 9(veth62b81ae) entered blocking state Feb 9 08:57:43 Tower kernel: docker0: port 9(veth62b81ae) entered forwarding state Feb 9 08:57:43 Tower kernel: docker0: port 9(veth62b81ae) entered disabled state Feb 9 08:57:44 Tower kernel: eth0: renamed from veth1d85cfc Feb 9 08:57:44 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth62b81ae: link becomes ready Feb 9 08:57:44 Tower kernel: docker0: port 9(veth62b81ae) entered blocking state Feb 9 08:57:44 Tower kernel: docker0: port 9(veth62b81ae) entered forwarding state Feb 9 09:00:07 Tower crond[3070]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null Feb 9 09:04:15 Tower emhttpd: cmd: /usr/local/emhttp/plugins/user.scripts/backgroundScript.sh /tmp/user.scripts/tmpScripts/plex_preloader/script Feb 9 09:04:17 Tower emhttpd: cmd: /usr/local/emhttp/plugins/user.scripts/showLog.php plex_preloader Feb 9 09:22:38 Tower ipmiseld[7519]:******6: Error connecting: session timeout Feb 9 09:26:09 Tower root: error: /webGui/include/Notify.php: uninitialized csrf_token Feb 9 09:26:09 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:12 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:12 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:12 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:12 Tower root: error: /plugins/preclear.disk/Preclear.php: uninitialized csrf_token Feb 9 09:26:15 Tower root: error: /plugins/dynamix.system.temp/include/SystemTemp.php: uninitialized csrf_token Feb 9 09:26:15 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:19 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:19 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:19 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:19 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:21 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:26 Tower root: error: /plugins/dynamix.system.temp/include/SystemTemp.php: uninitialized csrf_token Feb 9 09:26:27 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:28 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:28 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:28 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:33 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:37 Tower root: error: /plugins/dynamix.system.temp/include/SystemTemp.php: uninitialized csrf_token Feb 9 09:26:39 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:44 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:44 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:44 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:45 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token Feb 9 09:26:45 Tower root: error: /webGui/include/DashUpdate.php: uninitialized csrf_token
August 26, 20223 yr Community Expert Looks like they may have filled rootfs. 2 minutes ago, benloveslamp said: similar issue Attach Diagnostics to your NEXT post in this thread
August 26, 20223 yr Thanks for the reply. Diagnostics zip attached. tower-diagnostics-20220826-1425.zip
August 27, 20223 yr Community Expert Nothing in those diagnostics, literally no contents, just empty files. What do you get from the command line with this df -h
August 27, 20223 yr Oh sheesh that's annoying! I had just done a syslog before I requested the diagnostics files and it had all the errors. But I had to reboot the server so I could grab the zip file via SMB. I figured it had written successfully, but could be to do with the disfunction occurring. Everything's working again now because it's a time dependant problem. I've deleted the CA backup plugin I was using because I have a hunch that it was happening after the backup request was made at 02:00 hours and I've recently disconnected the hard drive which the back was directed to. I'll report back.
August 27, 20223 yr Community Expert 9 hours ago, benloveslamp said: had just done a syslog before I requested the diagnostics files and it had all the errors If you have that syslog, zip and attach to your NEXT post
August 27, 20223 yr Community Expert 11 hours ago, trurl said: What do you get from the command line with this df -h The reason I asked for this, which is a result that would have been in diagnostics if they worked, is because I suspect you have filled rootfs, where the OS keeps its own files. If you fill that the OS has no space to work in and will behave erratically.
August 27, 20223 yr Community Expert 9 hours ago, benloveslamp said: Everything's working again now because it's a time dependant problem. I've deleted the CA backup plugin I was using because I have a hunch that it was happening after the backup request was made at 02:00 hours and I've recently disconnected the hard drive which the back was directed to. Another reason to suspect you had filled rootfs. If you had the backup set to go to a path that wasn't mounted storage, that would be a path in rootfs. I thought Unassigned Devices handled this better though by limiting the amount of space an unmounted UD path could have. Do you know what path you specified for that backup?
August 28, 20223 yr Unfortunately I didn't save the syslog before reboot, but here is the result of df -h. Everything is running fine still at this point. Sorry, I am very much a noob. root@ruffellserver:~# df -h Filesystem Size Used Avail Use% Mounted on rootfs 3.8G 708M 3.1G 19% / tmpfs 32M 856K 32M 3% /run /dev/sda1 15G 516M 14G 4% /boot overlay 3.8G 708M 3.1G 19% /lib/firmware overlay 3.8G 708M 3.1G 19% /lib/modules devtmpfs 8.0M 0 8.0M 0% /dev tmpfs 3.9G 0 3.9G 0% /dev/shm cgroup_root 8.0M 0 8.0M 0% /sys/fs/cgroup tmpfs 128M 212K 128M 1% /var/log /dev/md1 1.9T 852G 1011G 46% /mnt/disk1 /dev/md2 3.7T 1007G 2.7T 28% /mnt/disk2 shfs 5.5T 1.9T 3.7T 34% /mnt/user0 shfs 5.5T 1.9T 3.7T 34% /mnt/user /dev/loop2 20G 5.0G 14G 28% /var/lib/docker /dev/loop3 1.0G 3.8M 905M 1% /etc/libvirt tmpfs 784M 0 784M 0% /run/user/0 root@ruffellserver:~#
August 28, 20223 yr Community Expert 19 hours ago, benloveslamp said: Everything is running fine still at this point. Can you get diagnostics now?
August 29, 20223 yr Latest diagnostics attached. Everything still running fine. I think what you're saying must have been what it was. Probably rootfs filling up after external drive was disconnected. ruffellserver-diagnostics-20220829-1952.zip
December 6, 20223 yr Hi, I started having this problem a month or so ago. I'm currently on 6.11.5. If I stop the array and start it again, things begin to function again however the GUI summary on the right of Dashboard no longer lists the drives. The only docker I'm running is Plex-Media-Server and occasionally binhex-krusader. To get full functionality in the GUI i have to reboot the server. I copied the errors from the log while the issue was taking place. I was unable to access save diagnostics until after I stopped and re-started the array. I'm attaching the syslog w/ the errors and the diagnostics from after a re-start of the array. Any help appreciated, thanks. Unraid_SysLogErrors_.txt tower-diagnostics-20221206-0828_GUIprobs.zip Edited December 6, 20223 yr by ez009
December 25, 20223 yr bump. This is happening every week. It's a major problem. I'm about to leave town and I really need my server to stable. Can anyone provide an explanation or advise to help solve this issue? thx,
December 25, 20223 yr Community Expert One possibility suggested by that syslog is that you are filling rootfs. Any path that isn't a mounted disk or user share is a path in rootfs, the space in RAM the OS runs in. If you fill it, all sorts of things can go wrong since the OS no longer has any space to work in. A common way to fill rootfs is a docker host path that isn't on a disk or user share. What to you get from the command line with this? df -h /
December 25, 20223 yr Community Expert 1 minute ago, trurl said: filling rootfs I see I already suggested that to other users on this thread. Did you read those posts?
December 26, 20223 yr Hi thanks for the reply. I did see the previous post and did not see anything out of the ordinary when running that command. Currently, After a recent re-boot, I'm seeing this: Filesystem Size Used Avail Use% Mounted on rootfs 12G 1.1G 11G 10% / I have Plex transcoding set to TMP (RAM) - However last time I was having the issue, I did check RAM usage in the GUI and I never saw it above 40%. But it must be Plex, considering it's the only docker running. So I will setup a disk share on the Cache Pool and hope that was the cause. Will report back either way. - Thx again
January 2, 20233 yr So it was Plex transcoding. After setting it to use a disk share on the cache drive, I could see the following day it was using over 200 GB for transcoding (4k movies). All is well in the GUI since. Thx for your help.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.