Server consistently locking up and unresponsive


unRaide

Recommended Posts

I'm having issues with my server locking up and becoming unresponsive.

 

What I know so far:

  • My server locks up consistently every 2- 3 days, just long enough to get through the Parity check :( 
  • When this happens I cannot ssh into the box and the UI/dockers/etc.. are unresponsive.
  • I don't remember changing anything specific but it started happening a few weeks ago
  • I've disabled all new dockers that were installed since then to try and isolate the issue with no luck
  • I turned on the Method 1 recommend by @Squid in this forum post which I recently checked after the server rebooted and it didn’t show anything out of the ordinary other than a ton of "shfs: share cache full" msgs

 

Does anyone have any guidance on how to troubleshoot this issue.

 

Also attached my diagnostics.

 

Thx

 

unraider-diagnostics-20210219-1852.zip

Link to comment

Just had another lockup requiring a hard reboot and there doesn’t seem to be anything helpful in the logs as they also seemed to have locked up/stopping updating for a few hours?

 

Feb 20 16:47:01 UnRaider root: Fix Common Problems Version 2021.02.18
Feb 20 16:47:01 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored
Feb 20 16:47:05 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored
Feb 20 16:47:05 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored
Feb 20 17:47:01 UnRaider root: Fix Common Problems Version 2021.02.18
Feb 20 17:47:02 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored
Feb 20 17:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored
Feb 20 17:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored
Feb 20 18:47:01 UnRaider root: Fix Common Problems Version 2021.02.18
Feb 20 18:47:02 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored
Feb 20 18:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored
Feb 20 18:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored
Feb 20 19:47:01 UnRaider root: Fix Common Problems Version 2021.02.18
Feb 20 19:47:02 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored
Feb 20 19:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored
Feb 20 19:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored
Feb 20 20:47:01 UnRaider root: Fix Common Problems Version 2021.02.18
Feb 20 20:47:01 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored
Feb 20 20:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored
Feb 20 20:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored
Feb 20 21:47:01 UnRaider root: Fix Common Problems Version 2021.02.18
Feb 20 21:47:02 UnRaider root: Fix Common Problems: Warning: Share backup set to not use the cache, but files / folders exist on the cache drive ** Ignored
Feb 20 21:47:06 UnRaider root: Fix Common Problems: Warning: Docker application duckdns2 has moderator comments listed ** Ignored
Feb 20 21:47:06 UnRaider root: Fix Common Problems: Warning: Docker application letsencrypt has moderator comments listed ** Ignored
Feb 20 23:39:43 UnRaider cache_dirs: Arguments=-i backup -i domains -i downloads -i files -i isos -i multimedia -i nextcloud -i sync -i sync-downloads -i windows
Feb 20 23:39:43 UnRaider cache_dirs: Max Scan Secs=10, Min Scan Secs=1
Feb 20 23:39:43 UnRaider cache_dirs: Scan Type=adaptive
Feb 20 23:39:43 UnRaider cache_dirs: Min Scan Depth=4
Feb 20 23:39:43 UnRaider cache_dirs: Max Scan Depth=none
Feb 20 23:39:43 UnRaider cache_dirs: Use Command='find -noleaf'
Feb 20 23:39:43 UnRaider cache_dirs: ---------- Caching Directories ---------------
Feb 20 23:39:43 UnRaider cache_dirs: backup
Feb 20 23:39:43 UnRaider cache_dirs: domains
Feb 20 23:39:43 UnRaider cache_dirs: downloads
Feb 20 23:39:43 UnRaider cache_dirs: files
Feb 20 23:39:43 UnRaider cache_dirs: isos
Feb 20 23:39:43 UnRaider cache_dirs: multimedia
Feb 20 23:39:43 UnRaider cache_dirs: nextcloud
Feb 20 23:39:43 UnRaider cache_dirs: sync
Feb 20 23:39:43 UnRaider cache_dirs: sync-downloads
Feb 20 23:39:43 UnRaider cache_dirs: windows
Feb 20 23:39:43 UnRaider cache_dirs: ----------------------------------------------
Feb 20 23:39:43 UnRaider cache_dirs: Setting Included dirs: backup,domains,downloads,files,isos,multimedia,nextcloud,sync,sync-downloads,windows
Feb 20 23:39:43 UnRaider cache_dirs: Setting Excluded dirs: 
Feb 20 23:39:43 UnRaider cache_dirs: min_disk_idle_before_restarting_scan_sec=60
Feb 20 23:39:43 UnRaider cache_dirs: scan_timeout_sec_idle=150
Feb 20 23:39:43 UnRaider cache_dirs: scan_timeout_sec_busy=30
Feb 20 23:39:43 UnRaider cache_dirs: scan_timeout_sec_stable=30
Feb 20 23:39:43 UnRaider cache_dirs: frequency_of_full_depth_scan_sec=604800
Feb 20 23:39:43 UnRaider cache_dirs: cache_dirs service rc.cachedirs: Started: '/usr/local/emhttp/plugins/dynamix.cache.dirs/scripts/cache_dirs -i "backup" -i "domains" -i "downloads" -i "files" -i "isos" -i "multimedia" -i "nextcloud" -i "sync" -i "sync-downloads" -i "windows" 2>/dev/null'
Feb 20 23:39:43 UnRaider root: Delaying execution of fix common problems scan for 10 minutes
Feb 20 23:39:43 UnRaider unassigned.devices: Mounting 'Auto Mount' Devices...
Feb 20 23:39:43 UnRaider unassigned.devices: Adding disk '/dev/sdb1'...
Feb 20 23:39:43 UnRaider unassigned.devices: Mount drive command: /sbin/mount -t xfs -o rw,noatime,nodiratime,discard '/dev/sdb1' '/mnt/disks/ssd'
Feb 20 23:39:43 UnRaider kernel: XFS (sdb1): Mounting V5 Filesystem
Feb 20 23:39:43 UnRaider kernel: XFS (sdb1): Starting recovery (logdev: internal)
Feb 20 23:39:43 UnRaider kernel: XFS (sdb1): Ending recovery (logdev: internal)
Feb 20 23:39:43 UnRaider unassigned.devices: Successfully mounted '/dev/sdb1' on '/mnt/disks/ssd'.
Feb 20 23:39:43 UnRaider unassigned.devices: Disk with serial 'WDC_WD140EDFZ-11A0VA0_9LG83JPA', mountpoint 'easystore' is not set to auto mount.
Feb 20 23:39:43 UnRaider emhttpd: Starting services...
Feb 20 23:39:43 UnRaider emhttpd: shcmd (65): /etc/rc.d/rc.samba restart
Feb 20 23:39:43 UnRaider rsyslogd: [origin software="rsyslogd" swVersion="8.1908.0" x-pid="6884" x-info="https://www.rsyslog.com"] start
Feb 20 23:39:45 UnRaider root: Starting Samba:  /usr/sbin/smbd -D
Feb 20 23:39:45 UnRaider root:                  /usr/sbin/nmbd -D
Feb 20 23:39:45 UnRaider root:                  /usr/sbin/wsdd 
Feb 20 23:39:45 UnRaider root:                  /usr/sbin/winbindd -D
Feb 20 23:39:45 UnRaider emhttpd: shcmd (79): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 50
Feb 20 23:39:45 UnRaider kernel: BTRFS: device fsid a8e1d964-f72d-42b6-a377-79a34367d3fd devid 1 transid 2223184 /dev/loop2
Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): disk space caching is enabled
Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): has skinny extents
Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): start tree-log replay
Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): checking UUID tree
Feb 20 23:39:45 UnRaider root: Resize '/var/lib/docker' of 'max'
Feb 20 23:39:45 UnRaider kernel: BTRFS info (device loop2): new size for /dev/loop2 is 53687091200
Feb 20 23:39:45 UnRaider emhttpd: shcmd (81): /etc/rc.d/rc.docker start
Feb 20 23:39:45 UnRaider root: starting dockerd ...

 

Really at a loss on how to stabilize my server 😩

 

Also attached the diagnostic taken after the server booted back up.

 

unraider-diagnostics-20210220-2347.zip

 

Edited by unRaide
Link to comment
2 hours ago, unRaide said:

and there doesn’t seem to be anything helpful in the logs

That suggests a hardware issue, one other thing you can try it to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

Link to comment
  • 1 year later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.