BTRFS Issues, I think? Dockers and other parts of the OS not responsive

[TheWooginator]

Unraid was up and running and humming along but when I got home after a couple weeks, I couldn't access it and neither Plex or Deluge were responsive, so it was hung up somehow. I'm running headless with no VM's, so I figured I'd do a hard reset and that might sort it out. After doing that, I have access to the main page and the array is up and running a parity check, but the Docker is still not functioning, running a diagnostic just stalls and says its running but doesn't actually do anything. Tried starting and stopping the Docker and that doesn't respond either. Really strange. Here's the call trace errors I'm getting over and over:

 

Jan 30 20:46:06 Tower kernel: rcu: INFO: rcu_bh self-detected stall on CPU
Jan 30 20:46:06 Tower kernel: rcu: 4-....: (27433363 ticks this GP) idle=29e/1/0x4000000000000002 softirq=10384/11371 fqs=6394571
Jan 30 20:46:06 Tower kernel: rcu: (t=27240467 jiffies g=-1191 q=6)
Jan 30 20:46:06 Tower kernel: NMI backtrace for cpu 4
Jan 30 20:46:06 Tower kernel: CPU: 4 PID: 9726 Comm: btrfs-transacti Tainted: G D 4.19.56-Unraid #1
Jan 30 20:46:06 Tower kernel: Hardware name: System manufacturer System Product Name/SABERTOOTH X79, BIOS 4701 05/06/2014
Jan 30 20:46:06 Tower kernel: Call Trace:
Jan 30 20:46:06 Tower kernel: <IRQ>
Jan 30 20:46:06 Tower kernel: dump_stack+0x5d/0x79
Jan 30 20:46:06 Tower kernel: nmi_cpu_backtrace+0x71/0x83
Jan 30 20:46:06 Tower kernel: ? lapic_can_unplug_cpu+0x8e/0x8e
Jan 30 20:46:06 Tower kernel: nmi_trigger_cpumask_backtrace+0x57/0xd7
Jan 30 20:46:06 Tower kernel: rcu_dump_cpu_stacks+0x91/0xbb
Jan 30 20:46:06 Tower kernel: rcu_check_callbacks+0x28f/0x58e
Jan 30 20:46:06 Tower kernel: ? tick_sched_handle.isra.5+0x2f/0x2f
Jan 30 20:46:06 Tower kernel: update_process_times+0x23/0x45
Jan 30 20:46:06 Tower kernel: tick_sched_timer+0x36/0x64
Jan 30 20:46:06 Tower kernel: __hrtimer_run_queues+0xb1/0x105
Jan 30 20:46:06 Tower kernel: hrtimer_interrupt+0xf4/0x20d
Jan 30 20:46:06 Tower kernel: smp_apic_timer_interrupt+0x79/0x91
Jan 30 20:46:06 Tower kernel: apic_timer_interrupt+0xf/0x20
Jan 30 20:46:06 Tower kernel: </IRQ>
Jan 30 20:46:06 Tower kernel: RIP: 0010:queued_write_lock_slowpath+0x5e/0x6b
Jan 30 20:46:06 Tower kernel: Code: 00 00 00 eb 25 ba ff 00 00 00 f0 0f b1 13 85 c0 75 e5 48 89 ef c6 07 00 66 66 66 90 5b 5d c3 f0 0f b1 13 3d 00 01 00 00 74 e8 <8b> 03 3d 00 01 00 00 74 ec f3 90 eb f3 48 63 ff 48 8b 04 fd 40 92
Jan 30 20:46:06 Tower kernel: RSP: 0018:ffffc90003ce7af8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13
Jan 30 20:46:06 Tower kernel: RAX: 00000000000001ff RBX: ffff888780ed12f8 RCX: 0000000000000000
Jan 30 20:46:06 Tower kernel: RDX: 00000000000000ff RSI: 0000000000000001 RDI: ffff888780ed12f8
Jan 30 20:46:06 Tower kernel: RBP: ffff888780ed12fc R08: ffff8887c81d8000 R09: ffffc90003ce7c0e
Jan 30 20:46:06 Tower kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8887f84c9800
Jan 30 20:46:06 Tower kernel: R13: 0000000000000001 R14: 0000000000000001 R15: ffff8887a3f24af8
Jan 30 20:46:06 Tower kernel: btrfs_try_tree_write_lock+0x1d/0x55
Jan 30 20:46:06 Tower kernel: btrfs_search_slot+0x798/0x832
Jan 30 20:46:06 Tower kernel: lookup_inline_extent_backref+0x114/0x542
Jan 30 20:46:06 Tower kernel: __btrfs_free_extent+0xf7/0x8e4
Jan 30 20:46:06 Tower kernel: ? load_balance+0x14e/0x70c
Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x41/0x70
Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x35/0x70
Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x41/0x70
Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x35/0x70
Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x41/0x70
Jan 30 20:46:06 Tower kernel: __btrfs_run_delayed_refs+0xa3e/0xb96
Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x35/0x70
Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x41/0x70
Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x35/0x70
Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x41/0x70
Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x41/0x70
Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x41/0x70
Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x35/0x70
Jan 30 20:46:06 Tower kernel: btrfs_run_delayed_refs+0x5a/0x15c
Jan 30 20:46:06 Tower kernel: ? try_to_del_timer_sync+0x4f/0x6e
Jan 30 20:46:06 Tower kernel: btrfs_commit_transaction+0x50/0x76b
Jan 30 20:46:06 Tower kernel: ? start_transaction+0x29e/0x30e
Jan 30 20:46:06 Tower kernel: transaction_kthread+0xca/0x136
Jan 30 20:46:06 Tower kernel: ? btrfs_cleanup_transaction+0x4a1/0x4a1
Jan 30 20:46:06 Tower kernel: kthread+0x10b/0x113
Jan 30 20:46:06 Tower kernel: ? kthread_park+0x71/0x71
Jan 30 20:46:06 Tower kernel: ret_from_fork+0x35/0x40

 

 

I would attach a diagnostics file if it actually worked, but even that doesn't want to play nice. I ran a BTRFS scrub on the cache, which gave a single Checksum error, but don't know what to do beyond that.

 

I tried stopping the parity check and doing a clean reboot/shutdown and neither of those are responding..... wtf?

 

Everything was good, haven't changed anything, it's running on a UPC, and I come home to find it limping along and can't even pull a diag??

 

Any help would be appreciated. Thanks!

Edited January 31, 2020 by TheWooginator