Server consistently locking up and unresponsive


Recommended Posts

I'm having issues with my server locking up and becoming unresponsive.

 

What I know so far:

  • My server locks up consistently every 2- 3 days, just long enough to get through the Parity check :( 
  • When this happens I cannot ssh into the box and the UI/dockers/etc.. are unresponsive.
  • I don't remember changing anything specific but it started happening a few weeks ago
  • I've disabled all new dockers that were installed since then to try and isolate the issue with no luck
  • I turned on the Method 1 recommend by @Squid in this forum post which I recently checked after the server rebooted and it didn’t show anything out of the ordinary other than a ton of "shfs: share cache full" msgs

 

Does anyone have any guidance on how to troubleshoot this issue.

 

Also attached my diagnostics.

 

Thx

 

unraider-diagnostics-20210219-1852.zip

Link to comment

Just had another lockup requiring a hard reboot and there doesn’t seem to be anything helpful in the logs as they also seemed to have locked up/stopping updating for a few hours?

 

Feb 20 16:47:01 UnRaider root: Fix Common Problems Version 2021.02.18
Feb 20 16:47:01 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored
Feb 20 16:47:05 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored
Feb 20 16:47:05 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored
Feb 20 17:47:01 UnRaider root: Fix Common Problems Version 2021.02.18
Feb 20 17:47:02 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored
Feb 20 17:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored
Feb 20 17:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored
Feb 20 18:47:01 UnRaider root: Fix Common Problems Version 2021.02.18
Feb 20 18:47:02 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored
Feb 20 18:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored
Feb 20 18:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored
Feb 20 19:47:01 UnRaider root: Fix Common Problems Version 2021.02.18
Feb 20 19:47:02 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored
Feb 20 19:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored
Feb 20 19:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored
Feb 20 20:47:01 UnRaider root: Fix Common Problems Version 2021.02.18
Feb 20 20:47:01 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored
Feb 20 20:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored
Feb 20 20:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored
Feb 20 21:47:01 UnRaider root: Fix Common Problems Version 2021.02.18
Feb 20 21:47:02 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored
Feb 20 21:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored
Feb 20 21:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored
Feb 20 23:39:43 UnRaider cache_dirs: Arguments=-i backup -i domains -i downloads -i files -i isos -i multimedia -i nextcloud -i sync -i sync-downloads -i windows
Feb 20 23:39:43 UnRaider cache_dirs: Max Scan Secs=10, Min Scan Secs=1
Feb 20 23:39:43 UnRaider cache_dirs: Scan Type=adaptive
Feb 20 23:39:43 UnRaider cache_dirs: Min Scan Depth=4
Feb 20 23:39:43 UnRaider cache_dirs: Max Scan Depth=none
Feb 20 23:39:43 UnRaider cache_dirs: Use Command='find -noleaf'
Feb 20 23:39:43 UnRaider cache_dirs: ---------- Caching Directories ---------------
Feb 20 23:39:43 UnRaider cache_dirs: backup
Feb 20 23:39:43 UnRaider cache_dirs: domains
Feb 20 23:39:43 UnRaider cache_dirs: downloads
Feb 20 23:39:43 UnRaider cache_dirs: files
Feb 20 23:39:43 UnRaider cache_dirs: isos
Feb 20 23:39:43 UnRaider cache_dirs: multimedia
Feb 20 23:39:43 UnRaider cache_dirs: nextcloud
Feb 20 23:39:43 UnRaider cache_dirs: sync
Feb 20 23:39:43 UnRaider cache_dirs: sync-downloads
Feb 20 23:39:43 UnRaider cache_dirs: windows
Feb 20 23:39:43 UnRaider cache_dirs: ----------------------------------------------
Feb 20 23:39:43 UnRaider cache_dirs: Setting Included dirs: backup,domains,downloads,files,isos,multimedia,nextcloud,sync,sync-downloads,windows
Feb 20 23:39:43 UnRaider cache_dirs: Setting Excluded dirs: 
Feb 20 23:39:43 UnRaider cache_dirs: min_disk_idle_before_restarting_scan_sec=60
Feb 20 23:39:43 UnRaider cache_dirs: scan_timeout_sec_idle=150
Feb 20 23:39:43 UnRaider cache_dirs: scan_timeout_sec_busy=30
Feb 20 23:39:43 UnRaider cache_dirs: scan_timeout_sec_stable=30
Feb 20 23:39:43 UnRaider cache_dirs: frequency_of_full_depth_scan_sec=604800
Feb 20 23:39:43 UnRaider cache_dirs: cache_dirs service rc.cachedirs: Started: '/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts/cache_dirs -i "backup" -i "domains" -i "downloads" -i "files" -i "isos" -i "multimedia" -i "nextcloud" -i "sync" -i "sync-downloads" -i "windows" 2>/dev/null'
Feb 20 23:39:43 UnRaider root: Delaying execution of fix common problems scan for 10 minutes
Feb 20 23:39:43 UnRaider unassigned.devices: Mounting 'Auto Mount' Devices...
Feb 20 23:39:43 UnRaider unassigned.devices: Adding disk '/dev/sdb1'...
Feb 20 23:39:43 UnRaider unassigned.devices: Mount drive command: /sbin/mount -t xfs -o rw,noatime,nodiratime,discard '/dev/sdb1' '/mnt/disks/ssd'
Feb 20 23:39:43 UnRaider kernel: XFS (sdb1): Mounting V5 Filesystem
Feb 20 23:39:43 UnRaider kernel: XFS (sdb1): Starting recovery (logdev: internal)
Feb 20 23:39:43 UnRaider kernel: XFS (sdb1): Ending recovery (logdev: internal)
Feb 20 23:39:43 UnRaider unassigned.devices: Successfully mounted '/dev/sdb1' on '/mnt/disks/ssd'.
Feb 20 23:39:43 UnRaider unassigned.devices: Disk with serial 'WDC_WD140EDFZ-11A0VA0_9LG83JPA', mountpoint 'easystore' is not set to auto mount.
Feb 20 23:39:43 UnRaider emhttpd: Starting services...
Feb 20 23:39:43 UnRaider emhttpd: shcmd (65): /etc/rc.d/rc.samba restart
Feb 20 23:39:43 UnRaider rsyslogd: [origin software="rsyslogd" swVersion="8.1908.0" x-pid="6884" x-info="https://www.rsyslog.com"] start
Feb 20 23:39:45 UnRaider root: Starting Samba:  /usr/sbin/smbd -D
Feb 20 23:39:45 UnRaider root:                  /usr/sbin/nmbd -D
Feb 20 23:39:45 UnRaider root:                  /usr/sbin/wsdd 
Feb 20 23:39:45 UnRaider root:                  /usr/sbin/winbindd -D
Feb 20 23:39:45 UnRaider emhttpd: shcmd (79): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 50
Feb 20 23:39:45 UnRaider kernel: BTRFS: device fsid a8e1d964-f72d-42b6-a377-79a34367d3fd devid 1 transid 2223184 /dev/loop2
Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): disk space caching is enabled
Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): has skinny extents
Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): start tree-log replay
Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): checking UUID tree
Feb 20 23:39:45 UnRaider root: Resize '/var/lib/docker' of 'max'
Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): new size for /dev/loop2 is 53687091200
Feb 20 23:39:45 UnRaider emhttpd: shcmd (81): /etc/rc.d/rc.docker start
Feb 20 23:39:45 UnRaider root: starting dockerd ...

 

Really at a loss on how to stabilize my server 😩

 

Also attached the diagnostic taken after the server booted back up.

 

unraider-diagnostics-20210220-2347.zip

 

Edited by unRaide
Link to comment
2 hours ago, unRaide said:

and there doesn’t seem to be anything helpful in the logs

That suggests a hardware issue, one other thing you can try it to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.