(SOLVED) unRAID hangs after a few days


ohj

Recommended Posts

I've god unRAID installed on an old stationary computer, with 2x4TB data storage, 4TB parity and 256GB SSD cache.

On it I have one Debian VM running Home Assistant, and a number of docker containers.

 

My problem is that my computer just freezes at some point, and I get no contact with any running services or the unRAID GUI.  When starting the computer, it boots just fine into unRAID, with the typical white on black text screen.  When the computer freezes, however, the text is gone and the screen is completely blank (black).

 

How can I troubleshoot this?  The only way to get the system up again is to hold the power button in for 5 seconds to force a shutdown, and then boot the computer again.  So I have no way to get log files before rebooting.

unraid-diagnostics-20210122-0855.zip

Edited by ohj
I've added the diagnostics file, downloaded after forcing a shutdown and rebooting unRAID.
Link to comment

Since I activate logging the server has hung twice.  Unfortunately I don't see anything happening at the point when the server is hanging.  The server is sending me status emails every once in a while, and in the log I'm only seeing an entry of sending a status email a few hours before the server hangs, and then new log entries once the server reboots.  So I'm uncertain how I ought to handle this...

 

Hang #1:

Jan 27 12:20:01 unraid sSMTP[23451]: Creating SSL connection to host
Jan 27 12:20:02 unraid sSMTP[23451]: SSL connection using TLS_AES_256_GCM_SHA384
Jan 27 12:20:03 unraid sSMTP[23451]: Sent mail for ***@***.no (221 2.0.0 Bye) uid=0 username=root outbytes=1134
Jan 27 12:23:46 unraid kernel: veth10fae82: renamed from eth0
Jan 27 12:23:46 unraid kernel: docker0: port 6(veth882542c) entered disabled state
Jan 27 12:23:46 unraid kernel: docker0: port 6(veth882542c) entered disabled state
Jan 27 12:23:46 unraid kernel: device veth882542c left promiscuous mode
Jan 27 12:23:46 unraid kernel: docker0: port 6(veth882542c) entered disabled state
Jan 27 12:24:44 unraid kernel: docker0: port 6(vethe18f83a) entered blocking state
Jan 27 12:24:44 unraid kernel: docker0: port 6(vethe18f83a) entered disabled state
Jan 27 12:24:44 unraid kernel: device vethe18f83a entered promiscuous mode
Jan 27 12:24:44 unraid kernel: IPv6: ADDRCONF(NETDEV_UP): vethe18f83a: link is not ready
Jan 27 12:24:45 unraid kernel: eth0: renamed from vethc36c89c
Jan 27 12:24:45 unraid kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethe18f83a: link becomes ready
Jan 27 12:24:45 unraid kernel: docker0: port 6(vethe18f83a) entered blocking state
Jan 27 12:24:45 unraid kernel: docker0: port 6(vethe18f83a) entered forwarding state
Jan 27 18:20:01 unraid sSMTP[26939]: Creating SSL connection to host
Jan 27 18:20:01 unraid sSMTP[26939]: SSL connection using TLS_AES_256_GCM_SHA384
Jan 27 18:20:03 unraid sSMTP[26939]: Sent mail for ***@***.no (221 2.0.0 Bye) uid=0 username=root outbytes=1134
<== SERVER HUNG.  FORCING SHUTDOWN AND REBOOTING SERVER.
Jan 28 00:00:16 unraid root: Delaying execution of fix common problems scan for 10 minutes
Jan 28 00:00:16 unraid emhttpd: Starting services...
Jan 28 00:00:16 unraid emhttpd: shcmd (44): /etc/rc.d/rc.samba restart
Jan 28 00:00:17 unraid rsyslogd: [origin software="rsyslogd" swVersion="8.1908.0" x-pid="2653" x-info="https://www.rsyslog.com"] start
Jan 28 00:00:18 unraid root: Starting Samba:  /usr/sbin/smbd -D
Jan 28 00:00:18 unraid root:                  /usr/sbin/nmbd -D
Jan 28 00:00:18 unraid root:                  /usr/sbin/wsdd 
Jan 28 00:00:18 unraid root:                  /usr/sbin/winbindd -D
Jan 28 00:00:19 unraid emhttpd: shcmd (58): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 20
Jan 28 00:00:19 unraid kernel: BTRFS: device fsid 900e8dad-69d0-4748-b5c1-c657b528f60e devid 1 transid 53866 /dev/loop2
Jan 28 00:00:19 unraid kernel: BTRFS info (device loop2): disk space caching is enabled
Jan 28 00:00:19 unraid kernel: BTRFS info (device loop2): has skinny extents
Jan 28 00:00:19 unraid kernel: BTRFS info (device loop2): start tree-log replay
Jan 28 00:00:21 unraid kernel: BTRFS info (device loop2): checking UUID tree
Jan 28 00:00:21 unraid root: Resize '/var/lib/docker' of 'max'
Jan 28 00:00:21 unraid kernel: BTRFS info (device loop2): new size for /dev/loop2 is 21474836480
Jan 28 00:00:21 unraid emhttpd: shcmd (60): /etc/rc.d/rc.docker start
Jan 28 00:00:21 unraid root: starting dockerd ...

 

Hang #2:

Jan 29 02:15:40 unraid CA Backup/Restore: #######################
Jan 29 02:15:40 unraid CA Backup/Restore: appData Backup complete
Jan 29 02:15:40 unraid CA Backup/Restore: #######################
Jan 29 02:15:40 unraid sSMTP[32198]: Creating SSL connection to host
Jan 29 02:15:41 unraid sSMTP[32198]: SSL connection using TLS_AES_256_GCM_SHA384
Jan 29 02:15:42 unraid sSMTP[32198]: Sent mail for ***@***.no (221 2.0.0 Bye) uid=0 username=root outbytes=628
Jan 29 02:15:43 unraid CA Backup/Restore: Deleting /mnt/user/backups/CommunityApplicationsAppdataBackup/[email protected]
Jan 29 02:15:43 unraid CA Backup/Restore: Backup / Restore Completed
Jan 29 03:02:38 unraid kernel: traps: OZW-driver[32583] general protection ip:15441fa9b5d0 sp:15441b875358 error:0 in libopenzwave.so.1.6[15441f9b2000+115000]
Jan 29 03:40:01 unraid crond[1679]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null
Jan 29 06:20:01 unraid sSMTP[12148]: Creating SSL connection to host
Jan 29 06:20:02 unraid sSMTP[12148]: SSL connection using TLS_AES_256_GCM_SHA384
Jan 29 06:20:03 unraid sSMTP[12148]: Sent mail for ***@***.no (221 2.0.0 Bye) uid=0 username=root outbytes=1137
<== SERVER HUNG.  FORCING SHUTDOWN AND REBOOTING SERVER.
Jan 29 09:01:14 unraid root: Delaying execution of fix common problems scan for 10 minutes
Jan 29 09:01:14 unraid emhttpd: Starting services...
Jan 29 09:01:14 unraid emhttpd: shcmd (44): /etc/rc.d/rc.samba restart
Jan 29 09:01:15 unraid rsyslogd: [origin software="rsyslogd" swVersion="8.1908.0" x-pid="2655" x-info="https://www.rsyslog.com"] start
Jan 29 09:01:16 unraid root: Starting Samba:  /usr/sbin/smbd -D
Jan 29 09:01:16 unraid root:                  /usr/sbin/nmbd -D
Jan 29 09:01:16 unraid root:                  /usr/sbin/wsdd 
Jan 29 09:01:16 unraid root:                  /usr/sbin/winbindd -D
Jan 29 09:01:16 unraid emhttpd: shcmd (58): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 20
Jan 29 09:01:16 unraid kernel: BTRFS: device fsid 900e8dad-69d0-4748-b5c1-c657b528f60e devid 1 transid 57069 /dev/loop2
Jan 29 09:01:16 unraid kernel: BTRFS info (device loop2): disk space caching is enabled
Jan 29 09:01:16 unraid kernel: BTRFS info (device loop2): has skinny extents
Jan 29 09:01:16 unraid kernel: BTRFS info (device loop2): start tree-log replay
Jan 29 09:01:16 unraid kernel: BTRFS info (device loop2): checking UUID tree
Jan 29 09:01:16 unraid root: Resize '/var/lib/docker' of 'max'
Jan 29 09:01:16 unraid kernel: BTRFS info (device loop2): new size for /dev/loop2 is 21474836480
Jan 29 09:01:16 unraid emhttpd: shcmd (60): /etc/rc.d/rc.docker start
Jan 29 09:01:16 unraid root: starting dockerd ...

 

Link to comment
4 minutes ago, JorgeB said:

One thing you can try it to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

Thank you for the input.  Yes, I will do that if it still crashes with C states disabled.

Edited by ohj
Link to comment
  • 3 weeks later...

Well, it's been a couple of weeks since the last crash/hang.  I've restarted unRAID for other reasons a couple of times, but as of now my uptime is a little more than 8 days - more than double of the longest uptime period I had before disabling the C states.  So I think it's safe to say that disabling the C states was the solution in my case.

 

Thanks for all input.

  • Like 1
Link to comment
  • ohj changed the title to (SOLVED) unRAID hangs after a few days

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.