TheWooginator

I've been in the process of rebuilding after my OS decided to stop working and couldn't figure out why the Deluge GUI wouldn't work. Thanks for the assist fellas! Sweden is working for me for the port forwarding.

Unraid was up and running and humming along but when I got home after a couple weeks, I couldn't access it and neither Plex or Deluge were responsive, so it was hung up somehow. I'm running headless with no VM's, so I figured I'd do a hard reset and that might sort it out. After doing that, I have access to the main page and the array is up and running a parity check, but the Docker is still not functioning, running a diagnostic just stalls and says its running but doesn't actually do anything. Tried starting and stopping the Docker and that doesn't respond either. Really strange. Here's the call trace errors I'm getting over and over: Jan 30 20:46:06 Tower kernel: rcu: INFO: rcu_bh self-detected stall on CPU Jan 30 20:46:06 Tower kernel: rcu: 4-....: (27433363 ticks this GP) idle=29e/1/0x4000000000000002 softirq=10384/11371 fqs=6394571 Jan 30 20:46:06 Tower kernel: rcu: (t=27240467 jiffies g=-1191 q=6) Jan 30 20:46:06 Tower kernel: NMI backtrace for cpu 4 Jan 30 20:46:06 Tower kernel: CPU: 4 PID: 9726 Comm: btrfs-transacti Tainted: G D 4.19.56-Unraid #1 Jan 30 20:46:06 Tower kernel: Hardware name: System manufacturer System Product Name/SABERTOOTH X79, BIOS 4701 05/06/2014 Jan 30 20:46:06 Tower kernel: Call Trace: Jan 30 20:46:06 Tower kernel: <IRQ> Jan 30 20:46:06 Tower kernel: dump_stack+0x5d/0x79 Jan 30 20:46:06 Tower kernel: nmi_cpu_backtrace+0x71/0x83 Jan 30 20:46:06 Tower kernel: ? lapic_can_unplug_cpu+0x8e/0x8e Jan 30 20:46:06 Tower kernel: nmi_trigger_cpumask_backtrace+0x57/0xd7 Jan 30 20:46:06 Tower kernel: rcu_dump_cpu_stacks+0x91/0xbb Jan 30 20:46:06 Tower kernel: rcu_check_callbacks+0x28f/0x58e Jan 30 20:46:06 Tower kernel: ? tick_sched_handle.isra.5+0x2f/0x2f Jan 30 20:46:06 Tower kernel: update_process_times+0x23/0x45 Jan 30 20:46:06 Tower kernel: tick_sched_timer+0x36/0x64 Jan 30 20:46:06 Tower kernel: __hrtimer_run_queues+0xb1/0x105 Jan 30 20:46:06 Tower kernel: hrtimer_interrupt+0xf4/0x20d Jan 30 20:46:06 Tower kernel: smp_apic_timer_interrupt+0x79/0x91 Jan 30 20:46:06 Tower kernel: apic_timer_interrupt+0xf/0x20 Jan 30 20:46:06 Tower kernel: </IRQ> Jan 30 20:46:06 Tower kernel: RIP: 0010:queued_write_lock_slowpath+0x5e/0x6b Jan 30 20:46:06 Tower kernel: Code: 00 00 00 eb 25 ba ff 00 00 00 f0 0f b1 13 85 c0 75 e5 48 89 ef c6 07 00 66 66 66 90 5b 5d c3 f0 0f b1 13 3d 00 01 00 00 74 e8 <8b> 03 3d 00 01 00 00 74 ec f3 90 eb f3 48 63 ff 48 8b 04 fd 40 92 Jan 30 20:46:06 Tower kernel: RSP: 0018:ffffc90003ce7af8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff13 Jan 30 20:46:06 Tower kernel: RAX: 00000000000001ff RBX: ffff888780ed12f8 RCX: 0000000000000000 Jan 30 20:46:06 Tower kernel: RDX: 00000000000000ff RSI: 0000000000000001 RDI: ffff888780ed12f8 Jan 30 20:46:06 Tower kernel: RBP: ffff888780ed12fc R08: ffff8887c81d8000 R09: ffffc90003ce7c0e Jan 30 20:46:06 Tower kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8887f84c9800 Jan 30 20:46:06 Tower kernel: R13: 0000000000000001 R14: 0000000000000001 R15: ffff8887a3f24af8 Jan 30 20:46:06 Tower kernel: btrfs_try_tree_write_lock+0x1d/0x55 Jan 30 20:46:06 Tower kernel: btrfs_search_slot+0x798/0x832 Jan 30 20:46:06 Tower kernel: lookup_inline_extent_backref+0x114/0x542 Jan 30 20:46:06 Tower kernel: __btrfs_free_extent+0xf7/0x8e4 Jan 30 20:46:06 Tower kernel: ? load_balance+0x14e/0x70c Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x41/0x70 Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x35/0x70 Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x41/0x70 Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x35/0x70 Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x41/0x70 Jan 30 20:46:06 Tower kernel: __btrfs_run_delayed_refs+0xa3e/0xb96 Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x35/0x70 Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x41/0x70 Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x35/0x70 Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x41/0x70 Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x41/0x70 Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x41/0x70 Jan 30 20:46:06 Tower kernel: ? __switch_to_asm+0x35/0x70 Jan 30 20:46:06 Tower kernel: btrfs_run_delayed_refs+0x5a/0x15c Jan 30 20:46:06 Tower kernel: ? try_to_del_timer_sync+0x4f/0x6e Jan 30 20:46:06 Tower kernel: btrfs_commit_transaction+0x50/0x76b Jan 30 20:46:06 Tower kernel: ? start_transaction+0x29e/0x30e Jan 30 20:46:06 Tower kernel: transaction_kthread+0xca/0x136 Jan 30 20:46:06 Tower kernel: ? btrfs_cleanup_transaction+0x4a1/0x4a1 Jan 30 20:46:06 Tower kernel: kthread+0x10b/0x113 Jan 30 20:46:06 Tower kernel: ? kthread_park+0x71/0x71 Jan 30 20:46:06 Tower kernel: ret_from_fork+0x35/0x40 I would attach a diagnostics file if it actually worked, but even that doesn't want to play nice. I ran a BTRFS scrub on the cache, which gave a single Checksum error, but don't know what to do beyond that. I tried stopping the parity check and doing a clean reboot/shutdown and neither of those are responding..... wtf? Everything was good, haven't changed anything, it's running on a UPC, and I come home to find it limping along and can't even pull a diag?? Any help would be appreciated. Thanks!

So no crashes overnight other than when my power actually went out at house briefly which was odd, but these were some errors that popped up multiple times in the middle of the night that did't cause a meltdown: Jan 25 04:49:15 Tower kernel: CPU: 1 PID: 5069 Comm: shfs Tainted: G D W 4.14.13-unRAID #1 Jan 25 04:47:34 Tower kernel: CPU: 4 PID: 350 Comm: khugepaged Tainted: G D W 4.14.13-unRAID #1 It seems that as long as I'm not moving chunks of data onto or around inside of the array, it stays happy. Baby steps.

Did it again. Moving files onto the array from my iMac through the network. Nothing special. I'm going to continue posting these until someone can give me a definitive answer on what the hell is going on. I'm not a linux expert, so this reads like Swahili to me. Jan 24 12:03:49 Tower root: move: file /mnt/cache/MyMedia/Movies/Power Rangers (2017) [1080p] [YTS.AG]/WWW.YTS.AG.jpg Jan 24 12:03:49 Tower root: move_object: /mnt/cache/MyMedia/Movies: Directory not empty Jan 24 12:03:49 Tower root: move_object: /mnt/cache/MyMedia: Directory not empty Jan 24 12:03:49 Tower root: mover: finished Jan 24 12:11:16 Tower emhttpd: req (3): shareMoverSchedule=0+0+*+*+*&shareMoverLogging=yes&changeMover=Apply&csrf_token=**************** Jan 24 12:11:16 Tower emhttpd: shcmd (229): /usr/local/sbin/update_cron Jan 24 16:08:42 Tower shfs: error: shfs_rmdir, 1517: Directory not empty (39): rmdir: /mnt/cache/appdata/binhex-plexpass/Plex Media Server/Cache/Transcode/Sessions/plex-transcode-fab9f93d4063e672-com-plexapp-android-24983809-a3ee-4af3-bcf6-231800306c36 Jan 24 16:28:21 Tower kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000080 Jan 24 16:28:21 Tower kernel: IP: workingset_eviction+0x40/0x85 Jan 24 16:28:21 Tower kernel: PGD 0 P4D 0 Jan 24 16:28:21 Tower kernel: Oops: 0000 [#1] PREEMPT SMP PTI Jan 24 16:28:21 Tower kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables vhost_net tun vhost tap veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs md_mod nct6775 hwmon_vid x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd e1000e intel_cstate intel_uncore intel_rapl_perf i2c_i801 i2c_core ahci libahci ptp mxm_wmi wmi_bmof wmi pps_core button Jan 24 16:28:21 Tower kernel: CPU: 9 PID: 864 Comm: kswapd0 Not tainted 4.14.13-unRAID #1 Jan 24 16:28:21 Tower kernel: Hardware name: System manufacturer System Product Name/SABERTOOTH X79, BIOS 4701 05/06/2014 Jan 24 16:28:21 Tower kernel: task: ffff8807fa3b3800 task.stack: ffffc9000364c000 Jan 24 16:28:21 Tower kernel: RIP: 0010:workingset_eviction+0x40/0x85 Jan 24 16:28:21 Tower kernel: RSP: 0018:ffffc9000364fb88 EFLAGS: 00010047 Jan 24 16:28:21 Tower kernel: RAX: 0000000000000000 RBX: ffffea0009172900 RCX: 0000000000000000 Jan 24 16:28:21 Tower kernel: RDX: 0000000000000000 RSI: ffff88081fff9000 RDI: ffff8807c03c7470 Jan 24 16:28:21 Tower kernel: RBP: ffff8807c03c7488 R08: 0000000000024b80 R09: ffffea0009172801 Jan 24 16:28:21 Tower kernel: R10: 0000000000000001 R11: ffffea0009172900 R12: 0000000000000286 Jan 24 16:28:21 Tower kernel: R13: 0000000000000001 R14: 0000000000000000 R15: ffff8807c03c7470 Jan 24 16:28:21 Tower kernel: FS: 0000000000000000(0000) GS:ffff8807ff440000(0000) knlGS:0000000000000000 Jan 24 16:28:21 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 24 16:28:21 Tower kernel: CR2: 0000000000000080 CR3: 0000000004c0a002 CR4: 00000000000606e0 Jan 24 16:28:21 Tower kernel: Call Trace: Jan 24 16:28:21 Tower kernel: __remove_mapping+0x177/0x1bc Jan 24 16:28:21 Tower kernel: shrink_page_list+0x8a5/0xa8f Jan 24 16:28:21 Tower kernel: shrink_inactive_list+0x25f/0x3d5 Jan 24 16:28:21 Tower kernel: shrink_node_memcg+0x4c9/0x680 Jan 24 16:28:21 Tower kernel: ? shrink_node+0xce/0x29b Jan 24 16:28:21 Tower kernel: shrink_node+0xce/0x29b Jan 24 16:28:21 Tower kernel: kswapd+0x437/0x55a Jan 24 16:28:21 Tower kernel: ? __switch_to+0xd4/0x2e8 Jan 24 16:28:21 Tower kernel: ? mem_cgroup_shrink_node+0x89/0x89 Jan 24 16:28:21 Tower kernel: kthread+0x10f/0x117 Jan 24 16:28:21 Tower kernel: ? kthread_create_on_node+0x3a/0x3a Jan 24 16:28:21 Tower kernel: ret_from_fork+0x1f/0x30 Jan 24 16:28:21 Tower kernel: Code: 66 66 90 0f b7 91 b8 00 00 00 eb 02 31 d2 66 66 66 66 90 48 63 86 40 3a 00 00 48 8b 8c c1 b0 03 00 00 eb 07 48 8d 8e 20 3b 00 00 <48> 39 b1 80 00 00 00 74 07 48 89 b1 80 00 00 00 b8 01 00 00 00 Jan 24 16:28:21 Tower kernel: RIP: workingset_eviction+0x40/0x85 RSP: ffffc9000364fb88 Jan 24 16:28:21 Tower kernel: CR2: 0000000000000080 Jan 24 16:28:21 Tower kernel: ---[ end trace 68f0b4a0a40ba233 ]--- Jan 24 16:28:21 Tower kernel: note: kswapd0[864] exited with preempt_count 1

This hardware was running fine as a normal PC before it became an UnRaid array. I'm not ruling it out, but I don't think it's hardware. Memtest+ on the boot flash doesn't seem to want to run for me, but the bios seems to think the memory is ok.

Same as before. root@Tower:~# btrfs dev stats /mnt/cache [/dev/sdf1].write_io_errs 0 [/dev/sdf1].read_io_errs 0 [/dev/sdf1].flush_io_errs 0 [/dev/sdf1].corruption_errs 0 [/dev/sdf1].generation_errs 0 [/dev/sde1].write_io_errs 0 [/dev/sde1].read_io_errs 0 [/dev/sde1].flush_io_errs 0 [/dev/sde1].corruption_errs 0 [/dev/sde1].generation_errs 0

Snagged it! I ran a logger on my iMac to keep an eye on it in case it crashed on me and I was able to grab this before it froze up and forced me to reset it: Jan 24 10:39:31 Tower root: Fix Common Problems Version 2018.01.21 Jan 24 10:39:32 Tower root: Fix Common Problems: /var/log currently 2 % full Jan 24 10:39:32 Tower root: Fix Common Problems: rootfs (/) currently 5 % full Jan 24 10:49:32 Tower root: Fix Common Problems Version 2018.01.21 Jan 24 10:49:33 Tower root: Fix Common Problems: /var/log currently 2 % full Jan 24 10:49:33 Tower root: Fix Common Problems: rootfs (/) currently 5 % full Jan 24 10:59:33 Tower root: Fix Common Problems Version 2018.01.21 Jan 24 10:59:34 Tower root: Fix Common Problems: /var/log currently 2 % full Jan 24 10:59:34 Tower root: Fix Common Problems: rootfs (/) currently 5 % full Jan 24 11:00:01 Tower root: mover: started Jan 24 11:02:28 Tower kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000088 Jan 24 11:02:28 Tower kernel: IP: account_page_dirtied+0xaf/0x13b Jan 24 11:02:28 Tower kernel: PGD 0 P4D 0 Jan 24 11:02:28 Tower kernel: Oops: 0000 [#1] PREEMPT SMP PTI Jan 24 11:02:28 Tower kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables vhost_net tun vhost tap veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs md_mod nct6775 hwmon_vid x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_uncore intel_rapl_perf e1000e i2c_i801 i2c_core ahci mxm_wmi wmi_bmof libahci ptp wmi pps_core button Jan 24 11:02:28 Tower kernel: CPU: 4 PID: 12156 Comm: kworker/u24:6 Not tainted 4.14.13-unRAID #1 Jan 24 11:02:28 Tower kernel: Hardware name: System manufacturer System Product Name/SABERTOOTH X79, BIOS 4701 05/06/2014 Jan 24 11:02:28 Tower kernel: Workqueue: btrfs-extent-refs btrfs_extent_refs_helper Jan 24 11:02:28 Tower kernel: task: ffff8807005d7000 task.stack: ffffc9000fafc000 Jan 24 11:02:28 Tower kernel: RIP: 0010:account_page_dirtied+0xaf/0x13b Jan 24 11:02:28 Tower kernel: RSP: 0018:ffffc9000faff9e8 EFLAGS: 00010047 Jan 24 11:02:28 Tower kernel: RAX: 0000000000000000 RBX: ffffea0009172900 RCX: 0000000000024b90 Jan 24 11:02:28 Tower kernel: RDX: ffff8807fcc19400 RSI: 000000000000000f RDI: ffff88081fff9000 Jan 24 11:02:28 Tower kernel: RBP: ffff8807f79c8458 R08: 0000000000024b80 R09: 0000000000000000 Jan 24 11:02:28 Tower kernel: R10: ffff8807f69f8368 R11: 000000000007c439 R12: ffff8807f69f8378 Jan 24 11:02:28 Tower kernel: R13: 0000000000000286 R14: 0000000000000000 R15: 0000000000000000 Jan 24 11:02:28 Tower kernel: FS: 0000000000000000(0000) GS:ffff8807ff300000(0000) knlGS:0000000000000000 Jan 24 11:02:28 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 24 11:02:28 Tower kernel: CR2: 0000000000000088 CR3: 0000000004c0a006 CR4: 00000000000606e0 Jan 24 11:02:28 Tower kernel: Call Trace: Jan 24 11:02:28 Tower kernel: __set_page_dirty_nobuffers+0x98/0x12c Jan 24 11:02:28 Tower kernel: set_extent_buffer_dirty+0x6a/0x76 Jan 24 11:02:28 Tower kernel: btrfs_mark_buffer_dirty+0x75/0x98 Jan 24 11:02:28 Tower kernel: __btrfs_cow_block+0x49e/0x4b8 Jan 24 11:02:28 Tower kernel: btrfs_cow_block+0x106/0x114 Jan 24 11:02:28 Tower kernel: btrfs_search_slot+0x330/0x83c Jan 24 11:02:28 Tower kernel: btrfs_del_csums+0xaa/0x340 Jan 24 11:02:28 Tower kernel: ? release_extent_buffer+0x7e/0x85 Jan 24 11:02:28 Tower kernel: __btrfs_free_extent+0x8dc/0x9e8 Jan 24 11:02:28 Tower kernel: __btrfs_run_delayed_refs+0xa7f/0xc84 Jan 24 11:02:28 Tower kernel: ? kmem_cache_free+0x12e/0x131 Jan 24 11:02:28 Tower kernel: btrfs_run_delayed_refs+0x68/0x1e9 Jan 24 11:02:28 Tower kernel: delayed_ref_async_start+0x54/0x90 Jan 24 11:02:28 Tower kernel: btrfs_worker_helper+0xbc/0x16f Jan 24 11:02:28 Tower kernel: process_one_work+0x146/0x239 Jan 24 11:02:28 Tower kernel: ? rescuer_thread+0x258/0x258 Jan 24 11:02:28 Tower kernel: worker_thread+0x1c3/0x292 Jan 24 11:02:28 Tower kernel: kthread+0x10f/0x117 Jan 24 11:02:28 Tower kernel: ? kthread_create_on_node+0x3a/0x3a Jan 24 11:02:28 Tower kernel: ? SyS_exit_group+0xb/0xb Jan 24 11:02:28 Tower kernel: ret_from_fork+0x1f/0x30 Jan 24 11:02:28 Tower kernel: Code: 43 38 48 85 c0 74 30 66 66 66 66 90 48 8b 80 c0 02 00 00 65 48 ff 40 78 48 8b 03 48 8b 53 38 48 c1 e8 3a 48 8b 84 c2 b0 03 00 00 <48> 8b 80 88 00 00 00 65 48 ff 40 78 48 89 df be 06 00 00 00 e8 Jan 24 11:02:28 Tower kernel: RIP: account_page_dirtied+0xaf/0x13b RSP: ffffc9000faff9e8 Jan 24 11:02:28 Tower kernel: CR2: 0000000000000088 Jan 24 11:02:28 Tower kernel: ---[ end trace a3e8bf5e33c2ee49 ]--- Jan 24 11:02:28 Tower kernel: note: kworker/u24:6[12156] exited with preempt_count 1 Jan 24 11:02:28 Tower kernel: ------------[ cut here ]------------ Jan 24 11:02:28 Tower kernel: WARNING: CPU: 4 PID: 12156 at kernel/rcu/tree_plugin.h:329 rcu_note_context_switch+0x27/0x281 Jan 24 11:02:28 Tower kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables vhost_net tun vhost tap veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs md_mod nct6775 hwmon_vid x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_uncore intel_rapl_perf e1000e i2c_i801 i2c_core ahci mxm_wmi wmi_bmof libahci ptp wmi pps_core button Jan 24 11:02:28 Tower kernel: CPU: 4 PID: 12156 Comm: kworker/u24:6 Tainted: G D 4.14.13-unRAID #1 Jan 24 11:02:28 Tower kernel: Hardware name: System manufacturer System Product Name/SABERTOOTH X79, BIOS 4701 05/06/2014 Jan 24 11:02:28 Tower kernel: Workqueue: btrfs-extent-refs btrfs_extent_refs_helper Jan 24 11:02:28 Tower kernel: task: ffff8807005d7000 task.stack: ffffc9000fafc000 Jan 24 11:02:28 Tower kernel: RIP: 0010:rcu_note_context_switch+0x27/0x281 Jan 24 11:02:28 Tower kernel: RSP: 0018:ffffc9000faffe68 EFLAGS: 00010002 Jan 24 11:02:28 Tower kernel: RAX: 0000000000020000 RBX: ffff8807005d7000 RCX: ffff8807005d7330 Jan 24 11:02:28 Tower kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Jan 24 11:02:28 Tower kernel: RBP: ffff8807005d7000 R08: 0000000000000001 R09: ffffffff8104a700 Jan 24 11:02:28 Tower kernel: R10: ffffea0004724d80 R11: ffffffff8200fe01 R12: 0000000000000000 Jan 24 11:02:28 Tower kernel: R13: 0000000000000000 R14: ffff8807005d75a0 R15: 0000000000020900 Jan 24 11:02:28 Tower kernel: FS: 0000000000000000(0000) GS:ffff8807ff300000(0000) knlGS:0000000000000000 Jan 24 11:02:28 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 24 11:02:28 Tower kernel: CR2: 0000000000000088 CR3: 0000000004c0a006 CR4: 00000000000606e0 Jan 24 11:02:28 Tower kernel: Call Trace: Jan 24 11:02:28 Tower kernel: __schedule+0x88/0x4e9 Jan 24 11:02:28 Tower kernel: do_task_dead+0x38/0x3a Jan 24 11:02:28 Tower kernel: do_exit+0x896/0x896 Jan 24 11:02:28 Tower kernel: rewind_stack_do_exit+0x17/0x20 Jan 24 11:02:28 Tower kernel: Code: 5c 41 5d c3 41 56 41 55 41 54 41 89 fc 55 53 65 48 8b 2c 25 00 5c 01 00 e8 c1 e6 ff ff 45 84 e4 75 0b 83 bd 28 03 00 00 00 7e 02 <0f> ff 83 bd 28 03 00 00 00 0f 8e ce 01 00 00 80 bd 2c 03 00 00 Jan 24 11:02:28 Tower kernel: ---[ end trace a3e8bf5e33c2ee4a ]--- Jan 24 11:02:35 Tower kernel: traps: emhttpd[6918] trap divide error ip:419f15 sp:14596af71e00 error:0 in emhttpd[400000+26000] ^^This was the last transmission that went through before it froze. If it was continuing to spit out info after that, I don't think there's a way to grab it as it locks up the GUI interface I have set up on the array as well. It ran all night just fine and got fussy as soon as I tried to move some data onto the array through the network. That's what I was in the process of doing when it locked up this time.

Well, it lasted longer this time than any other previous time, so... progress? It bugged out shortly after I decided to experiment with a VM, but it never got past the point of enabling it in the settings. I was prepping to do a Win 10 VM when it crapped out. The really weird part that I still can't figure out is that when it crashes, it locks up my modem which is connected directly to it. Once I power down the array, the modem magically comes back and re-enables the wifi. I'm set up as static at both ends soooooo.... tower-diagnostics-20180124-0023.zip

Roger. Array is up and running, so we'll see if it lasts the night. Is there anything I should be doing/logging in the meantime so I can capture that actual event if it happens again? Every time it quits on me I lose access and have to do a hard reset, so I can't actually grab the live errors.

Linux 4.14.13-unRAID. root@Tower:~# btrfs dev stats /mnt/cache [/dev/sdf1].write_io_errs 0 [/dev/sdf1].read_io_errs 0 [/dev/sdf1].flush_io_errs 0 [/dev/sdf1].corruption_errs 0 [/dev/sdf1].generation_errs 0 [/dev/sde1].write_io_errs 0 [/dev/sde1].read_io_errs 0 [/dev/sde1].flush_io_errs 0 [/dev/sde1].corruption_errs 0 [/dev/sde1].generation_errs 0 root@Tower:~#

I'm really new to this. Where am I looking to see these errors?

Thanks for that. I went ahead and updated the firmware on both SSD's and rebuilt the cache pool, so now we'll see if that helps.

So I went ahead and made sure that the system is on a static IP from the router and I'm still getting errors where the system becomes unresponsive or just shuts me out entirely. It just did it again with no warning while I was moving some data from my iMac to the array via the network. This is getting really frustrating. It seems like it's working fine, running through a parity check, moving data off the cache drives, moving data onto the array via the network, and then it just takes a $h*t. Please help guys. If I can't make this work before the trial period is over I'm likely going to mothball this project and get a QNAP or something. tower-diagnostics-20180122-2205.zip

Hey guys. I'm working through moving data from an external drive onto my array via Krusader and I've been getting intermittent freezes on the system. Some of it still seems to respond (I just successfully downloaded and installed the SSD Trim Plugin) but the system simply won't respond when trying to do a clean reboot. I was able to collect diagnostic data and I'm waiting to see if there's something I can do short of doing a push-button reset which would restart my 10 hr parity check. Initially it seemed like some of the freezing was due to the cache drive running out of space while moving data into the array, so I ran the mover function overnight to give me a solid 200gb of space to "move into". My last dump was about 100gb and it froze up unexpectedly somewhere at the 95% ish complete mark. Cycling docker on/off didn't work and restarting/shutting down Krusader doesn't appear to do anything. I've attached my diagnostic zip file. Please let me know f there's anything else that would be beneficial to have. tower-diagnostics-20180121-1110.zip EDIT: Adding this to top post for visibility Snagged it! I ran a logger on my iMac to keep an eye on it in case it crashed on me and I was able to grab this before it froze up and forced me to reset it: Jan 24 10:39:31 Tower root: Fix Common Problems Version 2018.01.21 Jan 24 10:39:32 Tower root: Fix Common Problems: /var/log currently 2 % full Jan 24 10:39:32 Tower root: Fix Common Problems: rootfs (/) currently 5 % full Jan 24 10:49:32 Tower root: Fix Common Problems Version 2018.01.21 Jan 24 10:49:33 Tower root: Fix Common Problems: /var/log currently 2 % full Jan 24 10:49:33 Tower root: Fix Common Problems: rootfs (/) currently 5 % full Jan 24 10:59:33 Tower root: Fix Common Problems Version 2018.01.21 Jan 24 10:59:34 Tower root: Fix Common Problems: /var/log currently 2 % full Jan 24 10:59:34 Tower root: Fix Common Problems: rootfs (/) currently 5 % full Jan 24 11:00:01 Tower root: mover: started Jan 24 11:02:28 Tower kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000088 Jan 24 11:02:28 Tower kernel: IP: account_page_dirtied+0xaf/0x13b Jan 24 11:02:28 Tower kernel: PGD 0 P4D 0 Jan 24 11:02:28 Tower kernel: Oops: 0000 [#1] PREEMPT SMP PTI Jan 24 11:02:28 Tower kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables vhost_net tun vhost tap veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs md_mod nct6775 hwmon_vid x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_uncore intel_rapl_perf e1000e i2c_i801 i2c_core ahci mxm_wmi wmi_bmof libahci ptp wmi pps_core button Jan 24 11:02:28 Tower kernel: CPU: 4 PID: 12156 Comm: kworker/u24:6 Not tainted 4.14.13-unRAID #1 Jan 24 11:02:28 Tower kernel: Hardware name: System manufacturer System Product Name/SABERTOOTH X79, BIOS 4701 05/06/2014 Jan 24 11:02:28 Tower kernel: Workqueue: btrfs-extent-refs btrfs_extent_refs_helper Jan 24 11:02:28 Tower kernel: task: ffff8807005d7000 task.stack: ffffc9000fafc000 Jan 24 11:02:28 Tower kernel: RIP: 0010:account_page_dirtied+0xaf/0x13b Jan 24 11:02:28 Tower kernel: RSP: 0018:ffffc9000faff9e8 EFLAGS: 00010047 Jan 24 11:02:28 Tower kernel: RAX: 0000000000000000 RBX: ffffea0009172900 RCX: 0000000000024b90 Jan 24 11:02:28 Tower kernel: RDX: ffff8807fcc19400 RSI: 000000000000000f RDI: ffff88081fff9000 Jan 24 11:02:28 Tower kernel: RBP: ffff8807f79c8458 R08: 0000000000024b80 R09: 0000000000000000 Jan 24 11:02:28 Tower kernel: R10: ffff8807f69f8368 R11: 000000000007c439 R12: ffff8807f69f8378 Jan 24 11:02:28 Tower kernel: R13: 0000000000000286 R14: 0000000000000000 R15: 0000000000000000 Jan 24 11:02:28 Tower kernel: FS: 0000000000000000(0000) GS:ffff8807ff300000(0000) knlGS:0000000000000000 Jan 24 11:02:28 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 24 11:02:28 Tower kernel: CR2: 0000000000000088 CR3: 0000000004c0a006 CR4: 00000000000606e0 Jan 24 11:02:28 Tower kernel: Call Trace: Jan 24 11:02:28 Tower kernel: __set_page_dirty_nobuffers+0x98/0x12c Jan 24 11:02:28 Tower kernel: set_extent_buffer_dirty+0x6a/0x76 Jan 24 11:02:28 Tower kernel: btrfs_mark_buffer_dirty+0x75/0x98 Jan 24 11:02:28 Tower kernel: __btrfs_cow_block+0x49e/0x4b8 Jan 24 11:02:28 Tower kernel: btrfs_cow_block+0x106/0x114 Jan 24 11:02:28 Tower kernel: btrfs_search_slot+0x330/0x83c Jan 24 11:02:28 Tower kernel: btrfs_del_csums+0xaa/0x340 Jan 24 11:02:28 Tower kernel: ? release_extent_buffer+0x7e/0x85 Jan 24 11:02:28 Tower kernel: __btrfs_free_extent+0x8dc/0x9e8 Jan 24 11:02:28 Tower kernel: __btrfs_run_delayed_refs+0xa7f/0xc84 Jan 24 11:02:28 Tower kernel: ? kmem_cache_free+0x12e/0x131 Jan 24 11:02:28 Tower kernel: btrfs_run_delayed_refs+0x68/0x1e9 Jan 24 11:02:28 Tower kernel: delayed_ref_async_start+0x54/0x90 Jan 24 11:02:28 Tower kernel: btrfs_worker_helper+0xbc/0x16f Jan 24 11:02:28 Tower kernel: process_one_work+0x146/0x239 Jan 24 11:02:28 Tower kernel: ? rescuer_thread+0x258/0x258 Jan 24 11:02:28 Tower kernel: worker_thread+0x1c3/0x292 Jan 24 11:02:28 Tower kernel: kthread+0x10f/0x117 Jan 24 11:02:28 Tower kernel: ? kthread_create_on_node+0x3a/0x3a Jan 24 11:02:28 Tower kernel: ? SyS_exit_group+0xb/0xb Jan 24 11:02:28 Tower kernel: ret_from_fork+0x1f/0x30 Jan 24 11:02:28 Tower kernel: Code: 43 38 48 85 c0 74 30 66 66 66 66 90 48 8b 80 c0 02 00 00 65 48 ff 40 78 48 8b 03 48 8b 53 38 48 c1 e8 3a 48 8b 84 c2 b0 03 00 00 <48> 8b 80 88 00 00 00 65 48 ff 40 78 48 89 df be 06 00 00 00 e8 Jan 24 11:02:28 Tower kernel: RIP: account_page_dirtied+0xaf/0x13b RSP: ffffc9000faff9e8 Jan 24 11:02:28 Tower kernel: CR2: 0000000000000088 Jan 24 11:02:28 Tower kernel: ---[ end trace a3e8bf5e33c2ee49 ]--- Jan 24 11:02:28 Tower kernel: note: kworker/u24:6[12156] exited with preempt_count 1 Jan 24 11:02:28 Tower kernel: ------------[ cut here ]------------ Jan 24 11:02:28 Tower kernel: WARNING: CPU: 4 PID: 12156 at kernel/rcu/tree_plugin.h:329 rcu_note_context_switch+0x27/0x281 Jan 24 11:02:28 Tower kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables vhost_net tun vhost tap veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs md_mod nct6775 hwmon_vid x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_uncore intel_rapl_perf e1000e i2c_i801 i2c_core ahci mxm_wmi wmi_bmof libahci ptp wmi pps_core button Jan 24 11:02:28 Tower kernel: CPU: 4 PID: 12156 Comm: kworker/u24:6 Tainted: G D 4.14.13-unRAID #1 Jan 24 11:02:28 Tower kernel: Hardware name: System manufacturer System Product Name/SABERTOOTH X79, BIOS 4701 05/06/2014 Jan 24 11:02:28 Tower kernel: Workqueue: btrfs-extent-refs btrfs_extent_refs_helper Jan 24 11:02:28 Tower kernel: task: ffff8807005d7000 task.stack: ffffc9000fafc000 Jan 24 11:02:28 Tower kernel: RIP: 0010:rcu_note_context_switch+0x27/0x281 Jan 24 11:02:28 Tower kernel: RSP: 0018:ffffc9000faffe68 EFLAGS: 00010002 Jan 24 11:02:28 Tower kernel: RAX: 0000000000020000 RBX: ffff8807005d7000 RCX: ffff8807005d7330 Jan 24 11:02:28 Tower kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Jan 24 11:02:28 Tower kernel: RBP: ffff8807005d7000 R08: 0000000000000001 R09: ffffffff8104a700 Jan 24 11:02:28 Tower kernel: R10: ffffea0004724d80 R11: ffffffff8200fe01 R12: 0000000000000000 Jan 24 11:02:28 Tower kernel: R13: 0000000000000000 R14: ffff8807005d75a0 R15: 0000000000020900 Jan 24 11:02:28 Tower kernel: FS: 0000000000000000(0000) GS:ffff8807ff300000(0000) knlGS:0000000000000000 Jan 24 11:02:28 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 24 11:02:28 Tower kernel: CR2: 0000000000000088 CR3: 0000000004c0a006 CR4: 00000000000606e0 Jan 24 11:02:28 Tower kernel: Call Trace: Jan 24 11:02:28 Tower kernel: __schedule+0x88/0x4e9 Jan 24 11:02:28 Tower kernel: do_task_dead+0x38/0x3a Jan 24 11:02:28 Tower kernel: do_exit+0x896/0x896 Jan 24 11:02:28 Tower kernel: rewind_stack_do_exit+0x17/0x20 Jan 24 11:02:28 Tower kernel: Code: 5c 41 5d c3 41 56 41 55 41 54 41 89 fc 55 53 65 48 8b 2c 25 00 5c 01 00 e8 c1 e6 ff ff 45 84 e4 75 0b 83 bd 28 03 00 00 00 7e 02 <0f> ff 83 bd 28 03 00 00 00 0f 8e ce 01 00 00 80 bd 2c 03 00 00 Jan 24 11:02:28 Tower kernel: ---[ end trace a3e8bf5e33c2ee4a ]--- Jan 24 11:02:35 Tower kernel: traps: emhttpd[6918] trap divide error ip:419f15 sp:14596af71e00 error:0 in emhttpd[400000+26000] ^^This was the last transmission that went through before it froze. If it was continuing to spit out info after that, I don't think there's a way to grab it as it locks up the GUI interface I have set up on the array as well. It ran all night just fine and got fussy as soon as I tried to move some data onto the array through the network. That's what I was in the process of doing when it locked up this time.

Edited original post. Thanks for the help guys! Onward to more struggles

So, I managed to get it working last night after some trial and error. I had changed the boot order and disabled other drives from being bootable media in the bios initially, but that didn't seem to work so I went back to all default settings in the bios, enabled the XMP profile, re-enabled secure boot, but switched it to "other OS". It won't directly boot the flash drive as UEFI, but once I get into bios, I can select "UEFI Kingston etc etc" and it will boot up successfully. I've been messing with the array setup, building the parity drive and installing apps/docker containers at this point. I'm still going to have to go back in and mess with the bios a bit more as I would very much like the system to boot directly in with UEFI without forcing me into setup mode to select it. Seems strange as I have that selected as first in the boot priority order, but it won't do it unless I manually select it in the bios. I'll work on it and report back! Progress!!

Hey guys. I'm totally new to this, but I've been doing as much reading as I possibly can prior to posting in case I missed something simple. I can't get past the black screen with the cursor when trying to boot form the USB. Hardware: Asus sabertooth x79 mobo w/ 3930k v1 chip running 4701 bios version (it's the latest non-beta version according to the asus website) kingston traveler se9 16gb flash drive, formatted fat32 with MBR I've tried all sorts of options on the bios including turning secure boot on/off, UEFI and legacy booting, disabled all other bootables, basically everything the troubleshooting guides recommend and it still just won't load. Am I missing something here? I'm using a mac running OS X to create the USB and I've tried it using the usb creator program as well as just doing it the legacy way where you copy the files and run the "make bootable" file. Please help! I didn't think it would be this tough to get it up and running!! EDIT: So I was able to get things working by: 1. Going back to the base bios settings and running my XMP profile that came with my DDR3 mem sticks so I can actually run them at 1833mhz, 2. left secure boot ON 3. made sure the UEFI Kingston had boot priority 4. Disabled all other bootables except bootable CD override 5. I found out that my CPU fans from noctua run at a lower RPM than the "alarm" state for the bios which threw it into an error that isn't really an error. I set the bios to NOT require me to press F1 to enter setup to "fix" this "error" and it allowed the OS to boot straight through from a cold start with no help from me. SUCCESS!