Hi, I've been running Unraid 6.12.9 for around 7 months and all was great up until recently. Suddenly my server becomes unresponsive to GUI and SSH every few days or so (occurred twice on one day and latest I got 8 days before it happened). At the moment this occurs I notice a bunch of disk activity (audibly) then on some occasions it sounds like there's a bunch more at intervals. Some of the times this hasn't been the case though.

At the time it first occurred I had recently introduced a UPS and a bunch of Docker containers would have also updated including Frigate to 0.14 (I have auto-update on though).

I've been capturing the syslog for the past 6 or 7 times it's occurred and see no pattern. This latest time around 30 minutes ago there's nothing at all logged. It had previously looked like the UPS or Plugin and Docker update checks might have been the cause but I have disabled and/or removed both and the issue is still occurring.
 

I've noticed that UPS and other entries are still logging in the syslog after the lockup, as well as once being able to get the GUI login page to show and being able to just login via SSH (both of which were still unresponsive). It's quite annoying and concerning since it requires a hard power off each time.

 

Can I configure a higher level of logging? Any ideas or advice greatly appreciated.

Edited September 3, 20241 yr by Varmint

Enable the syslog server and post that after a crash

Hi, I am writing syslog to my boot USB. There was what looked like common logging around the time (plugin and docker update check plugin) but today when it occurred there was nothing logged.

Crashed around 15:15. The "nut_libusb_get_report: Input/Output Error" are innocuous and server still crashes if I remove the UPS / these log entries:
 

Sep  3 00:22:00 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 00:23:02 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 00:29:18 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 00:38:51 NAZ emhttpd: spinning down /dev/sdf
Sep  3 00:46:56 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 01:08:48 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 01:31:55 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 01:39:20 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 01:55:09 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 02:05:58 NAZ emhttpd: read SMART /dev/sdf
Sep  3 02:59:31 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 03:36:00 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 03:49:31 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 03:53:37 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 03:54:39 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 03:55:42 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 04:22:47 NAZ emhttpd: spinning down /dev/sdf
Sep  3 04:29:38 NAZ emhttpd: spinning down /dev/sdg
Sep  3 05:29:55 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 06:05:05 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 06:31:52 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 06:34:55 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 07:11:02 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 07:31:36 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 07:56:34 NAZ emhttpd: read SMART /dev/sdg
Sep  3 08:39:06 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 08:49:23 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 08:55:31 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 09:43:34 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 09:45:36 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 10:17:06 NAZ emhttpd: spinning down /dev/sdg
Sep  3 10:18:54 NAZ emhttpd: read SMART /dev/sdg
Sep  3 10:21:35 NAZ emhttpd: read SMART /dev/sdf
Sep  3 10:45:19 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 11:03:42 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 11:49:24 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 11:51:30 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 12:06:18 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 12:11:35 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 13:00:36 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 13:28:47 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 14:11:56 NAZ usbhid-ups[13442]: nut_libusb_get_string: Input/Output Error
Sep  3 14:17:09 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 14:28:36 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 14:37:00 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 14:38:02 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error
Sep  3 15:14:34 NAZ usbhid-ups[13442]: nut_libusb_get_report: Input/Output Error

Edited September 3, 20241 yr by Varmint

That's from the NUT plugin, and should be unrelated to this issue, but suggest you ask in the plugin support thread, to see if it can stop the spam.

 

 

Correct, it's not related as server was still hanging with UPS removed.

Just hung again 30 mins ago after only 16 hours online. Same as most times, heard disk/s spin up then a bunch of activity then unresponsive. Every few minutes a bunch of disk activity and sounds like my Windows 10 VM is going on and offline or at least the passed through USB (have a Aquacomputer Quattro passed through via USB and can hear the fans spool up indicating loss of connectivity to the Aqua service and running fans at failback value).

Went bad at ~07:32. Nothing logged besides SMART:

Sep  4 06:45:08 NAZ emhttpd: spinning down /dev/sdc
Sep  4 06:45:46 NAZ emhttpd: spinning down /dev/sdf
Sep  4 06:45:48 NAZ emhttpd: spinning down /dev/sdd
Sep  4 07:00:06 NAZ emhttpd: spinning down /dev/sdg
Sep  4 07:31:17 NAZ emhttpd: read SMART /dev/sdg
Sep  4 07:31:17 NAZ emhttpd: read SMART /dev/sdc
Sep  4 08:11:44 NAZ emhttpd: read SMART /dev/sdd
Sep  4 08:11:44 NAZ emhttpd: read SMART /dev/sdf

Edited September 3, 20241 yr by Varmint

Since there's nothing relevant logged, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

Thanks, will give it a go.