Hello Unraid Forum!

 

I have been having intermittent crashing issues with my server for the last year or so where I come home to find no signs of life from Unraid. This occurs maybe every week or every other week. Sometimes I can go several weeks without a crash. When I check on the server, I find that the lights are still on and fans are still spinning, but there is no read/write activity shown by the LED and no screen output when I connect a monitor for troubleshooting. I always end up having to do an unsafe reset and everything starts back up like normal.

 

I enabled syslog and was patiently waiting for some good logs to figure out what this issue is. It took several instances of this occurring before I could find anything relevant in the syslog. It seems when it had happened the last two times, the crash was so critical that syslog stopped reporting entirely leaving me with no clues to go on.

 

The good news is that I finally have some good logs to show! Below is the syslog from around the time of the crash. The sudden jump in time after the last event is the time I turned the server back on. Would anyone mind taking a look at these to see if they can make sense of this? I will attach my diagnostics to this post as well.

 

Thank you!

 

Aug 26 04:44:33 Unraid emhttpd: spinning down /dev/sdd
Aug 26 04:44:52 Unraid emhttpd: spinning down /dev/sdb
Aug 26 11:11:21 Unraid kernel: BUG: Bad page map in process sleep  pte:ffff8883008ef990 pmd:685139067
Aug 26 11:11:21 Unraid kernel: addr:000014c164793000 vm_flags:00000071 anon_vma:0000000000000000 mapping:ffff8881eabeca78 index:1b8
Aug 26 11:11:21 Unraid kernel: file:libc.so.6 fault:filemap_fault mmap:btrfs_file_mmap read_folio:btrfs_read_folio
Aug 26 11:11:21 Unraid kernel: CPU: 2 PID: 21908 Comm: sleep Tainted: P        W  O       6.1.38-Unraid #2
Aug 26 11:11:21 Unraid kernel: Hardware name: System manufacturer System Product Name/Z170I PRO GAMING, BIOS 3405 05/12/2017
Aug 26 11:11:21 Unraid kernel: Call Trace:
Aug 26 11:11:21 Unraid kernel: <TASK>
Aug 26 11:11:21 Unraid kernel: dump_stack_lvl+0x44/0x5c
Aug 26 11:11:21 Unraid kernel: print_bad_pte+0x1bc/0x1d6
Aug 26 11:11:21 Unraid kernel: vm_normal_page+0x81/0x9b
Aug 26 11:11:21 Unraid kernel: unmap_page_range+0x384/0x67b
Aug 26 11:11:21 Unraid kernel: unmap_vmas+0xb6/0x100
Aug 26 11:11:21 Unraid kernel: exit_mmap+0xdb/0x22e
Aug 26 11:11:21 Unraid kernel: __mmput+0x43/0xe3
Aug 26 11:11:21 Unraid kernel: do_exit+0x31b/0x923
Aug 26 11:11:21 Unraid kernel: ? preempt_latency_start+0x1e/0x46
Aug 26 11:11:21 Unraid kernel: ? up_read+0x47/0x5d
Aug 26 11:11:21 Unraid kernel: do_group_exit+0x7a/0x7a
Aug 26 11:11:21 Unraid kernel: __x64_sys_exit_group+0x14/0x14
Aug 26 11:11:21 Unraid kernel: do_syscall_64+0x68/0x81
Aug 26 11:11:21 Unraid kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd
Aug 26 11:11:21 Unraid kernel: RIP: 0033:0x14c1646afbad
Aug 26 11:11:21 Unraid kernel: Code: Unable to access opcode bytes at 0x14c1646afb83.
Aug 26 11:11:21 Unraid kernel: RSP: 002b:00007fff043250b8 EFLAGS: 00000206 ORIG_RAX: 00000000000000e7
Aug 26 11:11:21 Unraid kernel: RAX: ffffffffffffffda RBX: 000014c1647b8208 RCX: 000014c1646afbad
Aug 26 11:11:21 Unraid kernel: RDX: 00000000000000e7 RSI: ffffffffffffff88 RDI: 0000000000000000
Aug 26 11:11:21 Unraid kernel: RBP: 0000000000000002 R08: 00007fff04325060 R09: 00007fff04324fef
Aug 26 11:11:21 Unraid kernel: R10: 00007fff04324f70 R11: 0000000000000206 R12: 0000000000000000
Aug 26 11:11:21 Unraid kernel: R13: 0000000000000000 R14: 000014c1647b6760 R15: 000014c1647b8220
Aug 26 11:11:21 Unraid kernel: </TASK>
Aug 26 12:00:02 Unraid emhttpd: read SMART /dev/sde
Aug 26 12:00:02 Unraid emhttpd: read SMART /dev/sdc
Aug 26 12:31:57 Unraid emhttpd: spinning down /dev/sde
Aug 26 12:31:57 Unraid emhttpd: spinning down /dev/sdc
Aug 26 22:26:16 Unraid kernel: BUG: Bad rss-counter state mm:00000000bed77b23 type:MM_FILEPAGES val:1
Aug 27 17:10:50 Unraid root: Delaying execution of fix common problems scan for 10 minutes
Aug 27 17:10:50 Unraid emhttpd: Starting services...

 

unraid-diagnostics-20230827-1712.zip

Edited August 27, 20232 yr by ethan_carter

That error is not very helpful, at least not to me, did this start out of the blue without any change? One thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

1 minute ago, JorgeB said:

That error is not very helpful, at least not to me, did this start out of the blue without any change? One thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

Yes, this server has been basically unchanged hardware-wise since built. The only change I made recently was to upgrade from 16GB of RAM to 32GB, but this same behavior occurred with both sets of RAM. I host 2 Minecraft servers, a calibre-web instance, homebridge, and more... so there are many things that could be causing this from a container / services perspective.

 

I'm trying to think of weak links here hardware-wise and I feel it's worth mentioning I had my cache pool (single Samsung 970 EVO 250GB) become unmountable last month which I posted about here. I ended up having to repair the filesystem which allowed me to copy over the entire contents of the cache drive to the array, format the SSD, and then copy the contents back. Everything has worked the same since that. The SMART data for this SSD doesn't seem bad to me but I could be wrong...

Unlikely that the SSD even if bad would lock up the server, especially without leaving anything logged.