Hey everyone,

 

I need some help with an issue that I've been dealing with for about 10 days (I haven't changed anything (at least nothing I'm aware of) during this time) now. My Unraid server keeps crashing, on average about every 1.5 days with crashing meaning that neither the server nor any of the running services are accessible in any way. Unfortunately, the syslog doesn’t contain any relevant information (the last crash was on Dec 3rd at 23:50)

 

Dec  3 11:00:47 Tower Docker Auto Update: Community Applications Docker Autoupdate finished
Dec  3 12:00:13 Tower emhttpd: read SMART /dev/sde
Dec  3 12:00:13 Tower emhttpd: read SMART /dev/sdb
Dec  3 12:31:44 Tower emhttpd: spinning down /dev/sde
Dec  3 12:42:14 Tower emhttpd: spinning down /dev/sdb
Dec  3 13:17:54 Tower emhttpd: read SMART /dev/sde
Dec  3 13:51:13 Tower emhttpd: spinning down /dev/sde
Dec  3 17:57:19 Tower emhttpd: read SMART /dev/sde
Dec  3 18:27:52 Tower emhttpd: read SMART /dev/sdb
Dec  3 18:29:08 Tower emhttpd: spinning down /dev/sde
Dec  3 19:54:07 Tower emhttpd: spinning down /dev/sdc
Dec  3 19:58:49 Tower emhttpd: spinning down /dev/sdd
Dec  3 19:58:49 Tower emhttpd: spinning down /dev/sdb
Dec  3 20:11:28 Tower emhttpd: read SMART /dev/sdc
Dec  3 20:15:45 Tower emhttpd: read SMART /dev/sdd
Dec  3 20:15:45 Tower emhttpd: read SMART /dev/sdb
Dec  3 20:41:35 Tower emhttpd: spinning down /dev/sdc
Dec  3 20:46:08 Tower emhttpd: spinning down /dev/sdd
Dec  3 20:46:08 Tower emhttpd: spinning down /dev/sdb
Dec  3 22:32:10 Tower emhttpd: read SMART /dev/sde
Dec  3 23:07:38 Tower emhttpd: spinning down /dev/sde
Dec  4 07:56:18 Tower root: Delaying execution of fix common problems scan for 10 minutes
Dec  4 07:56:18 Tower unassigned.devices: Mounting 'Auto Mount' Devices...
Dec  4 07:56:19 Tower emhttpd: Starting services...
Dec  4 07:56:19 Tower emhttpd: shcmd (155): /etc/rc.d/rc.samba restart
Dec  4 07:56:19 Tower wsdd2[5279]: 'Terminated' signal received.
Dec  4 07:56:19 Tower nmbd[5269]: [2024/12/04 07:56:19.041749,  0] ../../source3/nmbd/nmbd.c:59(terminate)
Dec  4 07:56:19 Tower nmbd[5269]:   Got SIGTERM: going down...
Dec  4 07:56:19 Tower winbindd[5284]: [2024/12/04 07:56:19.041758,  0] ../../source3/winbindd/winbindd_dual.c:1957(winbindd_sig_term_handler)
Dec  4 07:56:19 Tower winbindd[5282]: [2024/12/04 07:56:19.041760,  0] ../../source3/win

 

 

This to me sounds like a hardware issue. My first guess was a faulty cache drive (since the docker image was always corrupted after the crashes), but based on the smart logs everything appears to be fine. Now I'm thinking power supply, but it's only 2 years old and the crashes also occur when there is virtually no load on the server.

 

Any pointers on how I can figure out what exactly is going wrong would be greatly appreciated.

Edited December 4, 20241 yr by _monad

Since there's nothing relevant logged, this can also be a hardware issue, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the individual docker containers. 

I think it most likely is a hardware issue. The question is just what hardware. I run btrfs scrub on the cache drive and the syslog shows two errors. However I'm missing the expertise to tell how critical this is. Can this be the root of my issues or is it more likely that it is a result of my issue (i.e. the crashes during filesystem operations)


 

Dec  4 10:33:05 Tower kernel: BTRFS info (device dm-3): scrub: started on devid 1
Dec  4 10:33:07 Tower kernel: BTRFS warning (device dm-3): checksum error at logical 5420457984 on dev /dev/mapper/nvme0n1p1, physical 5420457984, root 5, inode 634208, offset 2830336: path resolving failed with ret=1
Dec  4 10:33:07 Tower kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 107, gen 0
Dec  4 10:33:07 Tower kernel: BTRFS error (device dm-3): unable to fixup (regular) error at logical 5420457984 on dev /dev/mapper/nvme0n1p1
Dec  4 10:33:07 Tower kernel: BTRFS warning (device dm-3): checksum error at logical 5420462080 on dev /dev/mapper/nvme0n1p1, physical 5420462080, root 5, inode 634208, offset 2834432: path resolving failed with ret=1
Dec  4 10:33:07 Tower kernel: BTRFS error (device dm-3): bdev /dev/mapper/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 108, gen 0
Dec  4 10:33:07 Tower kernel: BTRFS error (device dm-3): unable to fixup (regular) error at logical 5420462080 on dev /dev/mapper/nvme0n1p1
Dec  4 10:34:18 Tower emhttpd: spinning down /dev/sde

 

Edited December 4, 20241 yr by _monad

Data corruption is typically from bad RAM, start by running memtest, but note that memtest is only definitive if it finds errors, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.

After a lot of testing it appears to not have been the RAM itself but one of the slots that is defect. Thanks for bringing me onto the right track!

Edited December 11, 20241 yr by _monad