I tried to turn off the server to replace my RAM and it's stuck at "System is going down... XX"
I can't get to the log file itself so I copied what I could see. Does anyone know what the deal is? This is what the log says:
Jan 6 10:34:54 TrueSource root: root 27108 .r..m php-fpm
Jan 6 10:34:54 TrueSource root: root 27143 .rc.. kworker/u32:6-bond0
Jan 6 10:34:54 TrueSource root: root 27144 .rc.. kworker/u32:10-btrfs-endio-meta
Jan 6 10:34:54 TrueSource root: root 27237 .rc.. kworker/12:1-rcu_gp
Jan 6 10:34:54 TrueSource root: root 28907 .rc.. kworker/u32:20-poll_mpt2sas0_statu
Jan 6 10:34:54 TrueSource root: root 28908 .rc.. kworker/u32:23-btrfs-endio
Jan 6 10:34:54 TrueSource root: root 28909 .rc.. kworker/u32:25-btrfs-endio
Jan 6 10:34:54 TrueSource root: root 28911 .rc.. kworker/u32:30-btrfs-endio
Jan 6 10:34:54 TrueSource root: root 28912 .rc.. kworker/u32:32-btrfs-endio
Jan 6 10:34:54 TrueSource root: root 28913 .rc.. kworker/u32:33-btrfs-endio-meta
Jan 6 10:34:54 TrueSource root: root 28914 .rc.. kworker/u32:34-btrfs-endio
Jan 6 10:34:54 TrueSource root: root 28915 .rc.. kworker/u32:37-kcryptd/254:14
Jan 6 10:34:54 TrueSource root: root 28917 .rc.. kworker/u32:39-kcryptd/254:14
Jan 6 10:34:54 TrueSource root: root 28918 .rc.. kworker/u32:40-kcryptd/254:14
Jan 6 10:34:54 TrueSource root: root 29301 .rc.. kworker/4:2-events
Jan 6 10:34:54 TrueSource root: root 29484 .rc.. kworker/5:1-cgroup_destroy
Jan 6 10:34:54 TrueSource root: root 29522 .rc.. kworker/12:2-events
Jan 6 10:34:54 TrueSource root: root 30590 .rc.. kworker/5:0-events
Jan 6 10:34:54 TrueSource root: root 31038 .rc.. kworker/8:2-events
Jan 6 10:34:54 TrueSource root: root 31167 .r..m php-fpm
Jan 6 10:34:54 TrueSource root: root 31294 .rc.. kworker/0:0-events
Jan 6 10:34:54 TrueSource root: root 31557 .rc.. kworker/u32:12-events_power_efficient
Jan 6 10:34:54 TrueSource root: root 31947 Fr..m tmux: server
Jan 6 10:34:54 TrueSource root: root 31951 .rce. bash
Jan 6 10:34:54 TrueSource root: root 32223 .rc.. kworker/u32:15-bond0
Jan 6 10:34:54 TrueSource root: root 32226 .rc.. kworker/u32:27-btrfs-endio-meta
Jan 6 10:34:54 TrueSource root: root 32234 .rc.. kworker/0:1-events
Jan 6 10:34:54 TrueSource root: /mnt/disk14: root kernel mount /mnt/disk14
Jan 6 10:34:54 TrueSource root: /mnt/disk15: root kernel mount /mnt/disk15
Jan 6 10:34:54 TrueSource root: /mnt/disk16: root kernel mount /mnt/disk16
Jan 6 10:34:54 TrueSource root: /mnt/disks: root kernel mount /mnt/disks
Jan 6 10:34:54 TrueSource root: /mnt/remotes: root kernel mount /mnt/remotes
Jan 6 10:34:54 TrueSource root: /mnt/rootshare: root kernel mount /mnt/rootshare
Jan 6 10:34:54 TrueSource root: Active pids left on /dev/md*
Jan 6 10:34:54 TrueSource root: Generating diagnostics...
Jan 6 10:34:59 TrueSource kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
Jan 6 10:34:59 TrueSource kernel: rcu: 12-....: (59999 ticks this GP) idle=49ac/1/0x4000000000000000 softirq=110945897/110945897 fqs=13317
Jan 6 10:34:59 TrueSource kernel: (t=60001 jiffies g=198847037 q=41532 ncpus=16)
Jan 6 10:34:59 TrueSource kernel: CPU: 12 PID: 9340 Comm: umount Tainted: P O 6.1.64-Unraid #1
Jan 6 10:34:59 TrueSource kernel: Hardware name: System manufacturer System Product Name/PRIME B450M-A, BIOS 0409 08/24/2018
Jan 6 10:34:59 TrueSource kernel: RIP: 0010:preempt_count_sub+0x87/0x93
Jan 6 10:34:59 TrueSource kernel: Code: 00 85 c0 74 2c 83 3d e8 a5 31 01 00 75 23 48 c7 c6 c3 43 0c 82 48 c7 c7 24 24 0b 82 e8 5c 79 fd ff 0f 0b e9 64 c3 b6 00 f7 df <65> 01 3d 1b 6e f8 7e e9 56 c3 b6 00 0f 1f 44 00 00 85 ff 74 05 e9
Jan 6 10:34:59 TrueSource kernel: RSP: 0018:ffffc90003887e18 EFLAGS: 00000297
Jan 6 10:34:59 TrueSource kernel: RAX: 0000000000000002 RBX: ffff88802ae42538 RCX: ffff8881663cc238
Jan 6 10:34:59 TrueSource kernel: RDX: 0000000000000002 RSI: ffff8881663cce38 RDI: 00000000ffffffff
Jan 6 10:34:59 TrueSource kernel: RBP: ffff888105add000 R08: ffffffff8125be65 R09: ffff88816c4a48c0
Jan 6 10:34:59 TrueSource kernel: R10: 00000000003a38a5 R11: ffff8881353aeb90 R12: ffff88802ae42538
Jan 6 10:34:59 TrueSource kernel: R13: ffff888105add540 R14: ffff88802ae425b8 R15: ffff8880ddb9af40
Jan 6 10:34:59 TrueSource kernel: FS: 0000149f302aeb80(0000) GS:ffff888216f00000(0000) knlGS:0000000000000000
Jan 6 10:34:59 TrueSource kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 6 10:34:59 TrueSource kernel: CR2: 0000149f3044f213 CR3: 000000010309e000 CR4: 00000000003506e0
Jan 6 10:34:59 TrueSource kernel: Call Trace:
Jan 6 10:34:59 TrueSource kernel: <IRQ>
Jan 6 10:34:59 TrueSource kernel: ? rcu_dump_cpu_stacks+0x95/0xb9
Jan 6 10:34:59 TrueSource kernel: ? rcu_sched_clock_irq+0x345/0xa45
Jan 6 10:34:59 TrueSource kernel: ? perf_event_task_tick+0x5e/0x1e3
Jan 6 10:34:59 TrueSource kernel: ? __default_send_IPI_dest_field+0x32/0x50
Jan 6 10:34:59 TrueSource kernel: ? tick_init_jiffy_update+0x7c/0x7c
Jan 6 10:34:59 TrueSource kernel: ? update_process_times+0x62/0x81
Jan 6 10:34:59 TrueSource kernel: ? tick_sched_timer+0x43/0x71
Jan 6 10:34:59 TrueSource kernel: ? __hrtimer_run_queues+0xeb/0x190
Jan 6 10:34:59 TrueSource kernel: ? hrtimer_interrupt+0x9c/0x16e
Jan 6 10:34:59 TrueSource kernel: ? __sysvec_apic_timer_interrupt+0xc5/0x12f
Jan 6 10:34:59 TrueSource kernel: ? sysvec_apic_timer_interrupt+0x80/0xa6
Jan 6 10:34:59 TrueSource kernel: </IRQ>
Jan 6 10:34:59 TrueSource kernel: <TASK>
Jan 6 10:34:59 TrueSource kernel: ? asm_sysvec_apic_timer_interrupt+0x16/0x20
Jan 6 10:34:59 TrueSource kernel: ? destroy_inode+0x5d/0x6b
Jan 6 10:34:59 TrueSource kernel: ? preempt_count_sub+0x87/0x93
Jan 6 10:34:59 TrueSource kernel: ? _raw_spin_lock+0x13/0x1c
Jan 6 10:34:59 TrueSource kernel: _raw_spin_unlock+0x14/0x29
Jan 6 10:34:59 TrueSource kernel: evict_inodes+0xa6/0x159
Jan 6 10:34:59 TrueSource kernel: generic_shutdown_super+0x3d/0x17f
Jan 6 10:34:59 TrueSource kernel: kill_block_super+0x1f/0x3e
Jan 6 10:34:59 TrueSource kernel: deactivate_locked_super+0x2f/0x61
Jan 6 10:34:59 TrueSource kernel: cleanup_mnt+0xab/0xd3
Jan 6 10:34:59 TrueSource kernel: task_work_run+0x6b/0x80
Jan 6 10:34:59 TrueSource kernel: exit_to_user_mode_prepare+0x75/0x112
Jan 6 10:34:59 TrueSource kernel: syscall_exit_to_user_mode+0x18/0x2c
Jan 6 10:34:59 TrueSource kernel: do_syscall_64+0x77/0x81
Jan 6 10:34:59 TrueSource kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Jan 6 10:34:59 TrueSource kernel: RIP: 0033:0x149f303c59e7
Jan 6 10:34:59 TrueSource kernel: Code: 0d 00 f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 01 24 0d 00 f7 d8 64 89 02 b8
Jan 6 10:34:59 TrueSource kernel: RSP: 002b:00007ffe662a6fe8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
Jan 6 10:34:59 TrueSource kernel: RAX: 0000000000000000 RBX: 000000000040a498 RCX: 0000149f303c59e7
Jan 6 10:34:59 TrueSource kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000040a5b0
Jan 6 10:34:59 TrueSource kernel: RBP: 0000000000000000 R08: 0000000000000073 R09: 000000000040a5d0
Jan 6 10:34:59 TrueSource kernel: R10: 0000149f302c2410 R11: 0000000000000246 R12: 0000149f3055afa4
Jan 6 10:34:59 TrueSource kernel: R13: 000000000040a5b0 R14: 000000000040a380 R15: 0000000000000000
Jan 6 10:34:59 TrueSource kernel: </TASK>
Jan 6 10:37:59 TrueSource kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
Jan 6 10:37:59 TrueSource kernel: rcu: 12-....: (240003 ticks this GP) idle=49ac/1/0x4000000000000000 softirq=110945897/110945897 fqs=66179
Jan 6 10:37:59 TrueSource kernel: (t=240005 jiffies g=198847037 q=437916 ncpus=16)
Jan 6 10:37:59 TrueSource kernel: CPU: 12 PID: 9340 Comm: umount Tainted: P O 6.1.64-Unraid #1
Jan 6 10:37:59 TrueSource kernel: Hardware name: System manufacturer System Product Name/PRIME B450M-A, BIOS 0409 08/24/2018
Jan 6 10:37:59 TrueSource kernel: RIP: 0010:evict_inodes+0x5f/0x159
Jan 6 10:37:59 TrueSource kernel: Code: cb 01 00 4c 8d ad 40 05 00 00 4c 89 ef e8 8d 36 62 00 48 8b 85 48 05 00 00 4c 8b 20 48 8d 98 f0 fe ff ff 49 81 ec 10 01 00 00 <48> 8d 93 10 01 00 00 48 8d 85 48 05 00 00 48 39 c2 0f 84 ab 00 00
Jan 6 10:37:59 TrueSource kernel: RSP: 0018:ffffc90003887e28 EFLAGS: 00000202
Jan 6 10:37:59 TrueSource kernel: RAX: ffff88802ae42648 RBX: ffff88802ae42538 RCX: ffff8881663cc238
Jan 6 10:37:59 TrueSource kernel: RDX: 0000000000000002 RSI: ffff8881663cce38 RDI: 00000000ffffffff
Jan 6 10:37:59 TrueSource kernel: RBP: ffff888105add000 R08: ffffffff8125be65 R09: ffff88816c4a48c0
Jan 6 10:37:59 TrueSource kernel: R10: 00000000003a38a5 R11: ffff8881353aeb90 R12: ffff88802ae42538
Jan 6 10:37:59 TrueSource kernel: R13: ffff888105add540 R14: ffff88802ae425b8 R15: ffff8880ddb9af40
Jan 6 10:37:59 TrueSource kernel: FS: 0000149f302aeb80(0000) GS:ffff888216f00000(0000) knlGS:0000000000000000
Jan 6 10:37:59 TrueSource kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 6 10:37:59 TrueSource kernel: CR2: 0000149f3044f213 CR3: 000000010309e000 CR4: 00000000003506e0
Jan 6 10:37:59 TrueSource kernel: Call Trace:
Jan 6 10:37:59 TrueSource kernel: <IRQ>
Jan 6 10:37:59 TrueSource kernel: ? rcu_dump_cpu_stacks+0x95/0xb9
Jan 6 10:37:59 TrueSource kernel: ? rcu_sched_clock_irq+0x345/0xa45
Jan 6 10:37:59 TrueSource kernel: ? perf_event_task_tick+0x5e/0x1e3
Jan 6 10:37:59 TrueSource kernel: ? tick_init_jiffy_update+0x7c/0x7c
Jan 6 10:37:59 TrueSource kernel: ? update_process_times+0x62/0x81
Jan 6 10:37:59 TrueSource kernel: ? tick_sched_timer+0x43/0x71
Jan 6 10:37:59 TrueSource kernel: ? __hrtimer_run_queues+0xeb/0x190
Jan 6 10:37:59 TrueSource kernel: ? hrtimer_interrupt+0x9c/0x16e
Jan 6 10:37:59 TrueSource kernel: ? __sysvec_apic_timer_interrupt+0xc5/0x12f
Jan 6 10:37:59 TrueSource kernel: ? sysvec_apic_timer_interrupt+0x80/0xa6
Jan 6 10:37:59 TrueSource kernel: </IRQ>
Jan 6 10:37:59 TrueSource kernel: <TASK>
Jan 6 10:37:59 TrueSource kernel: ? asm_sysvec_apic_timer_interrupt+0x16/0x20
Jan 6 10:37:59 TrueSource kernel: ? destroy_inode+0x5d/0x6b
Jan 6 10:37:59 TrueSource kernel: ? evict_inodes+0x5f/0x159
Jan 6 10:37:59 TrueSource kernel: ? evict_inodes+0xa6/0x159
Jan 6 10:37:59 TrueSource kernel: generic_shutdown_super+0x3d/0x17f
Jan 6 10:37:59 TrueSource kernel: kill_block_super+0x1f/0x3e
Jan 6 10:37:59 TrueSource kernel: deactivate_locked_super+0x2f/0x61
Jan 6 10:37:59 TrueSource kernel: cleanup_mnt+0xab/0xd3
Jan 6 10:37:59 TrueSource kernel: task_work_run+0x6b/0x80
Jan 6 10:37:59 TrueSource kernel: exit_to_user_mode_prepare+0x75/0x112
Jan 6 10:37:59 TrueSource kernel: syscall_exit_to_user_mode+0x18/0x2c
Jan 6 10:37:59 TrueSource kernel: do_syscall_64+0x77/0x81
Jan 6 10:37:59 TrueSource kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Jan 6 10:37:59 TrueSource kernel: RIP: 0033:0x149f303c59e7
Jan 6 10:37:59 TrueSource kernel: Code: 0d 00 f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 01 24 0d 00 f7 d8 64 89 02 b8
Jan 6 10:37:59 TrueSource kernel: RSP: 002b:00007ffe662a6fe8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
Jan 6 10:37:59 TrueSource kernel: RAX: 0000000000000000 RBX: 000000000040a498 RCX: 0000149f303c59e7
Jan 6 10:37:59 TrueSource kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000040a5b0
Jan 6 10:37:59 TrueSource kernel: RBP: 0000000000000000 R08: 0000000000000073 R09: 000000000040a5d0
Jan 6 10:37:59 TrueSource kernel: R10: 0000149f302c2410 R11: 0000000000000246 R12: 0000149f3055afa4
Jan 6 10:37:59 TrueSource kernel: R13: 000000000040a5b0 R14: 000000000040a380 R15: 0000000000000000
Jan 6 10:37:59 TrueSource kernel: </TASK>
an 6 10:40:59 TrueSource kernel: task_work_run+0x6b/0x80
Jan 6 10:40:59 TrueSource kernel: exit_to_user_mode_prepare+0x75/0x112
Jan 6 10:40:59 TrueSource kernel: syscall_exit_to_user_mode+0x18/0x2c
Jan 6 10:40:59 TrueSource kernel: do_syscall_64+0x77/0x81
Jan 6 10:40:59 TrueSource kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Jan 6 10:40:59 TrueSource kernel: RIP: 0033:0x149f303c59e7
Jan 6 10:40:59 TrueSource kernel: Code: 0d 00 f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 01 24 0d 00 f7 d8 64 89 02 b8
Jan 6 10:40:59 TrueSource kernel: RSP: 002b:00007ffe662a6fe8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
Jan 6 10:40:59 TrueSource kernel: RAX: 0000000000000000 RBX: 000000000040a498 RCX: 0000149f303c59e7
Jan 6 10:40:59 TrueSource kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000040a5b0
Jan 6 10:40:59 TrueSource kernel: RBP: 0000000000000000 R08: 0000000000000073 R09: 000000000040a5d0
Jan 6 10:40:59 TrueSource kernel: R10: 0000149f302c2410 R11: 0000000000000246 R12: 0000149f3055afa4
Jan 6 10:40:59 TrueSource kernel: R13: 000000000040a5b0 R14: 000000000040a380 R15: 0000000000000000
Jan 6 10:40:59 TrueSource kernel: </TASK>
Jan 6 10:41:00 TrueSource nginx: 2024/01/06 10:41:00 [error] 11302#11302: *3667102 limiting requests, excess: 20.116 by zone "authlimit", client: 10.0.0.54, server: , request: "PROPFIND /login HTTP/1.1", host: "truesource"
Jan 6 10:41:03 TrueSource nginx: 2024/01/06 10:41:03 [error] 11302#11302: *3667146 limiting requests, excess: 20.291 by zone "authlimit", client: 10.0.0.54, server: , request: "PROPFIND /login HTTP/1.1", host: "truesource"
Jan 6 10:41:06 TrueSource nginx: 2024/01/06 10:41:06 [error] 11302#11302: *3667178 limiting requests, excess: 20.027 by zone "authlimit", client: 10.0.0.54, server: , request: "PROPFIND /login HTTP/1.1", host: "truesource"
Jan 6 10:41:10 TrueSource nginx: 2024/01/06 10:41:10 [error] 11302#11302: *3667218 limiting requests, excess: 20.182 by zone "authlimit", client: 10.0.0.54, server: , request: "PROPFIND /login HTTP/1.1", host: "truesource"
Jan 6 10:41:21 TrueSource nginx: 2024/01/06 10:41:21 [error] 11302#11302: *3667358 limiting requests, excess: 20.223 by zone "authlimit", client: 10.0.0.54, server: , request: "PROPFIND /login HTTP/1.1", host: "truesource"
Jan 6 10:41:25 TrueSource nginx: 2024/01/06 10:41:25 [error] 11302#11302: *3667404 limiting requests, excess: 20.345 by zone "authlimit", client: 10.0.0.54, server: , request: "PROPFIND /login HTTP/1.1", host: "truesource"
Jan 6 10:41:28 TrueSource nginx: 2024/01/06 10:41:28 [error] 11302#11302: *3667431 limiting requests, excess: 20.122 by zone "authlimit", client: 10.0.0.54, server: , request: "PROPFIND /login HTTP/1.1", host: "truesource"
Jan 6 10:41:31 TrueSource nginx: 2024/01/06 10:41:31 [error] 11302#11302: *3667477 limiting requests, excess: 20.238 by zone "authlimit", client: 10.0.0.54, server: , request: "PROPFIND /login HTTP/1.1", host: "truesource"
Jan 6 10:41:35 TrueSource nginx: 2024/01/06 10:41:35 [error] 11302#11302: *3667525 limiting requests, excess: 20.353 by zone "authlimit", client: 10.0.0.54, server: , request: "PROPFIND /login HTTP/1.1", host: "truesource"
Jan 6 10:41:38 TrueSource nginx: 2024/01/06 10:41:38 [error] 11302#11302: *3667553 limiting requests, excess: 20.051 by zone "authlimit", client: 10.0.0.54, server: , request: "PROPFIND /login HTTP/1.1", host: "truesource"
Probably relevant but also probably obvious (?), but one of the cores is maxed out. I assume I could just (and will probably have to) do an unclean shutdown but I'd like to know what caused this so it doesn't happen again.