unRaide Posted February 20, 2021 Share Posted February 20, 2021 I'm having issues with my server locking up and becoming unresponsive. What I know so far: My server locks up consistently every 2- 3 days, just long enough to get through the Parity check When this happens I cannot ssh into the box and the UI/dockers/etc.. are unresponsive. I don't remember changing anything specific but it started happening a few weeks ago I've disabled all new dockers that were installed since then to try and isolate the issue with no luck I turned on the Method 1 recommend by @Squid in this forum post which I recently checked after the server rebooted and it didn’t show anything out of the ordinary other than a ton of "shfs: share cache full" msgs Does anyone have any guidance on how to troubleshoot this issue. Also attached my diagnostics. Thx unraider-diagnostics-20210219-1852.zip Quote Link to comment
JorgeB Posted February 20, 2021 Share Posted February 20, 2021 Try this and then post that log after a crash. Quote Link to comment
unRaide Posted February 20, 2021 Author Share Posted February 20, 2021 Thx, I just enabled the Local syslog sever and hopefully that will give me more info after the next lockup. Quote Link to comment
unRaide Posted February 21, 2021 Author Share Posted February 21, 2021 (edited) Just had another lockup requiring a hard reboot and there doesn’t seem to be anything helpful in the logs as they also seemed to have locked up/stopping updating for a few hours? Feb 20 16:47:01 UnRaider root: Fix Common Problems Version 2021.02.18 Feb 20 16:47:01 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored Feb 20 16:47:05 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored Feb 20 16:47:05 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored Feb 20 17:47:01 UnRaider root: Fix Common Problems Version 2021.02.18 Feb 20 17:47:02 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored Feb 20 17:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored Feb 20 17:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored Feb 20 18:47:01 UnRaider root: Fix Common Problems Version 2021.02.18 Feb 20 18:47:02 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored Feb 20 18:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored Feb 20 18:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored Feb 20 19:47:01 UnRaider root: Fix Common Problems Version 2021.02.18 Feb 20 19:47:02 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored Feb 20 19:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored Feb 20 19:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored Feb 20 20:47:01 UnRaider root: Fix Common Problems Version 2021.02.18 Feb 20 20:47:01 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored Feb 20 20:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored Feb 20 20:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored Feb 20 21:47:01 UnRaider root: Fix Common Problems Version 2021.02.18 Feb 20 21:47:02 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored Feb 20 21:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored Feb 20 21:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored Feb 20 23:39:43 UnRaider cache_dirs: Arguments=-i backup -i domains -i downloads -i files -i isos -i multimedia -i nextcloud -i sync -i sync-downloads -i windows Feb 20 23:39:43 UnRaider cache_dirs: Max Scan Secs=10, Min Scan Secs=1 Feb 20 23:39:43 UnRaider cache_dirs: Scan Type=adaptive Feb 20 23:39:43 UnRaider cache_dirs: Min Scan Depth=4 Feb 20 23:39:43 UnRaider cache_dirs: Max Scan Depth=none Feb 20 23:39:43 UnRaider cache_dirs: Use Command='find -noleaf' Feb 20 23:39:43 UnRaider cache_dirs: ---------- Caching Directories --------------- Feb 20 23:39:43 UnRaider cache_dirs: backup Feb 20 23:39:43 UnRaider cache_dirs: domains Feb 20 23:39:43 UnRaider cache_dirs: downloads Feb 20 23:39:43 UnRaider cache_dirs: files Feb 20 23:39:43 UnRaider cache_dirs: isos Feb 20 23:39:43 UnRaider cache_dirs: multimedia Feb 20 23:39:43 UnRaider cache_dirs: nextcloud Feb 20 23:39:43 UnRaider cache_dirs: sync Feb 20 23:39:43 UnRaider cache_dirs: sync-downloads Feb 20 23:39:43 UnRaider cache_dirs: windows Feb 20 23:39:43 UnRaider cache_dirs: ---------------------------------------------- Feb 20 23:39:43 UnRaider cache_dirs: Setting Included dirs: backup,domains,downloads,files,isos,multimedia,nextcloud,sync,sync-downloads,windows Feb 20 23:39:43 UnRaider cache_dirs: Setting Excluded dirs: Feb 20 23:39:43 UnRaider cache_dirs: min_disk_idle_before_restarting_scan_sec=60 Feb 20 23:39:43 UnRaider cache_dirs: scan_timeout_sec_idle=150 Feb 20 23:39:43 UnRaider cache_dirs: scan_timeout_sec_busy=30 Feb 20 23:39:43 UnRaider cache_dirs: scan_timeout_sec_stable=30 Feb 20 23:39:43 UnRaider cache_dirs: frequency_of_full_depth_scan_sec=604800 Feb 20 23:39:43 UnRaider cache_dirs: cache_dirs service rc.cachedirs: Started: '/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts/cache_dirs -i "backup" -i "domains" -i "downloads" -i "files" -i "isos" -i "multimedia" -i "nextcloud" -i "sync" -i "sync-downloads" -i "windows" 2>/dev/null' Feb 20 23:39:43 UnRaider root: Delaying execution of fix common problems scan for 10 minutes Feb 20 23:39:43 UnRaider unassigned.devices: Mounting 'Auto Mount' Devices... Feb 20 23:39:43 UnRaider unassigned.devices: Adding disk '/dev/sdb1'... Feb 20 23:39:43 UnRaider unassigned.devices: Mount drive command: /sbin/mount -t xfs -o rw,noatime,nodiratime,discard '/dev/sdb1' '/mnt/disks/ssd' Feb 20 23:39:43 UnRaider kernel: XFS (sdb1): Mounting V5 Filesystem Feb 20 23:39:43 UnRaider kernel: XFS (sdb1): Starting recovery (logdev: internal) Feb 20 23:39:43 UnRaider kernel: XFS (sdb1): Ending recovery (logdev: internal) Feb 20 23:39:43 UnRaider unassigned.devices: Successfully mounted '/dev/sdb1' on '/mnt/disks/ssd'. Feb 20 23:39:43 UnRaider unassigned.devices: Disk with serial 'WDC_WD140EDFZ-11A0VA0_9LG83JPA', mountpoint 'easystore' is not set to auto mount. Feb 20 23:39:43 UnRaider emhttpd: Starting services... Feb 20 23:39:43 UnRaider emhttpd: shcmd (65): /etc/rc.d/rc.samba restart Feb 20 23:39:43 UnRaider rsyslogd: [origin software="rsyslogd" swVersion="8.1908.0" x-pid="6884" x-info="https://www.rsyslog.com"] start Feb 20 23:39:45 UnRaider root: Starting Samba: /usr/sbin/smbd -D Feb 20 23:39:45 UnRaider root: /usr/sbin/nmbd -D Feb 20 23:39:45 UnRaider root: /usr/sbin/wsdd Feb 20 23:39:45 UnRaider root: /usr/sbin/winbindd -D Feb 20 23:39:45 UnRaider emhttpd: shcmd (79): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 50 Feb 20 23:39:45 UnRaider kernel: BTRFS: device fsid a8e1d964-f72d-42b6-a377-79a34367d3fd devid 1 transid 2223184 /dev/loop2 Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): disk space caching is enabled Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): has skinny extents Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): start tree-log replay Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): checking UUID tree Feb 20 23:39:45 UnRaider root: Resize '/var/lib/docker' of 'max' Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): new size for /dev/loop2 is 53687091200 Feb 20 23:39:45 UnRaider emhttpd: shcmd (81): /etc/rc.d/rc.docker start Feb 20 23:39:45 UnRaider root: starting dockerd ... Really at a loss on how to stabilize my server 😩 Also attached the diagnostic taken after the server booted back up. unraider-diagnostics-20210220-2347.zip Edited February 21, 2021 by unRaide Quote Link to comment
JorgeB Posted February 21, 2021 Share Posted February 21, 2021 2 hours ago, unRaide said: and there doesn’t seem to be anything helpful in the logs That suggests a hardware issue, one other thing you can try it to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. Quote Link to comment
Howboys Posted August 17, 2022 Share Posted August 17, 2022 Did you ever figure this out @unRaide? Quote Link to comment
unRaide Posted August 17, 2022 Author Share Posted August 17, 2022 I didn’t unfortunately. Just had the server lock up on me again earlier today. Sad to say but I just hard reboot it every few days 😩 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.