for the past few weeks, when I reboot my unraid server, it just comes back up and is stuck on starting services in the GUI.

I can start the array in maintenance mode without issue.

it will rarely come back up fully stated without issue, but the next reboot sees the same "starting services" problem.

 

I've collected the diagnostics(attached)  and it looks like there is a crash of some kind in the syslog file.

brief snippet of the crash and right before:

Feb 11 09:38:56 UnRaid  emhttpd: shcmd (78): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 20
Feb 11 09:38:56 UnRaid kernel: loop2: detected capacity change from 0 to 41943040
Feb 11 09:38:56 UnRaid kernel: BTRFS: device fsid 97dd2387-71dd-44df-a651-770f67eddb06 devid 1 transid 1033478 /dev/loop2 scanned by mount (3173)
Feb 11 09:38:56 UnRaid kernel: BTRFS info (device loop2): using free space tree
Feb 11 09:38:56 UnRaid kernel: BTRFS info (device loop2): has skinny extents
Feb 11 09:38:56 UnRaid kernel: BUG: unable to handle page fault for address: 0000000000247930
Feb 11 09:38:56 UnRaid kernel: #PF: supervisor write access in kernel mode
Feb 11 09:38:56 UnRaid kernel: #PF: error_code(0x0002) - not-present page
Feb 11 09:38:56 UnRaid kernel: PGD 8000000c46688067 P4D 8000000c46688067 PUD ca5aad067 PMD 0 
Feb 11 09:38:56 UnRaid kernel: Oops: 0002 [#1] PREEMPT SMP PTI
Feb 11 09:38:56 UnRaid kernel: CPU: 10 PID: 3173 Comm: mount Tainted: G          I       5.19.17-Unraid #2
Feb 11 09:38:56 UnRaid kernel: Hardware name: Dell Inc. PowerEdge R710/0YDJK3, BIOS 2.0.13 04/06/2010
Feb 11 09:38:56 UnRaid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x153/0x1d0
Feb 11 09:38:56 UnRaid kernel: Code: b9 01 00 00 00 f0 0f b1 0b 74 76 eb cc c1 ee 12 83 e0 03 ff ce 48 c1 e0 05 48 63 f6 48 05 00 ce 02 00 48 03 04 f5 e0 6a 16 82 <48> 89 10 8b 42 08 85 c0 75 04 f3 90 eb f5 48 8b 32 48 85 f6 74 bc
Feb 11 09:38:56 UnRaid kernel: RSP: 0018:ffffc9000801f818 EFLAGS: 00010006
Feb 11 09:38:56 UnRaid kernel: RAX: 0000000000247930 RBX: ffff888100000200 RCX: 00000000002c0000
Feb 11 09:38:56 UnRaid kernel: RDX: ffff88980fb6ce00 RSI: 000000000000073f RDI: ffff888100000200
Feb 11 09:38:56 UnRaid kernel: RBP: 000000000000000a R08: ffff888c43767800 R09: 00000000002a0014
Feb 11 09:38:56 UnRaid kernel: R10: 0000000000000003 R11: 0000000000000001 R12: ffff88980fb6ce00
Feb 11 09:38:56 UnRaid kernel: R13: 0000000000000000 R14: 0000000000000246 R15: ffffffff810822b3
Feb 11 09:38:56 UnRaid kernel: FS:  0000151e3770ab80(0000) GS:ffff88980fb40000(0000) knlGS:0000000000000000
Feb 11 09:38:56 UnRaid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 11 09:38:56 UnRaid kernel: CR2: 0000000000247930 CR3: 0000000c45952005 CR4: 00000000000206e0
Feb 11 09:38:56 UnRaid kernel: Call Trace:
Feb 11 09:38:56 UnRaid kernel: <TASK>
Feb 11 09:38:56 UnRaid kernel: do_raw_spin_lock+0x14/0x1a
Feb 11 09:38:56 UnRaid kernel: _raw_spin_lock_irqsave+0x2c/0x37
Feb 11 09:38:56 UnRaid kernel: get_partial_node+0x41/0x172
Feb 11 09:38:56 UnRaid kernel: ? alloc_unbound_pwq+0x207/0x365
Feb 11 09:38:56 UnRaid kernel: ? __alloc_pages+0x107/0x1e8
Feb 11 09:38:56 UnRaid kernel: ? alloc_worker+0x1d/0x4d
Feb 11 09:38:56 UnRaid kernel: ___slab_alloc+0x205/0x590
Feb 11 09:38:56 UnRaid kernel: ? alloc_worker+0x1d/0x4d
Feb 11 09:38:56 UnRaid kernel: ? xas_store+0x19f/0x3f4
Feb 11 09:38:56 UnRaid kernel: ? alloc_worker+0x1d/0x4d
Feb 11 09:38:56 UnRaid kernel: __slab_alloc.constprop.0+0x4e/0x79
Feb 11 09:38:56 UnRaid kernel: ? alloc_worker+0x1d/0x4d
Feb 11 09:38:56 UnRaid kernel: kmem_cache_alloc_node_trace+0xb3/0x177
Feb 11 09:38:56 UnRaid kernel: alloc_worker+0x1d/0x4d
Feb 11 09:38:56 UnRaid kernel: create_worker+0x4e/0x173
 

Any ideas would be appreciated.

unraid-diagnostics-20230211-0939.zip

Since the crash appears to be happening after the docker image starts to mount disable the docker service, if it doesn't crash like that recreate the docker image.

I ran a parity check and it found 1 issue and corrected it.

the array did start fully after the parity check.

 

Next time I have this issue, I'll check and see if the stack is the same and try disabling the docker service if it is.

 

Thanks.