Just one of those weeks....

[TechnoBabble28]

Hey guys,

 

Long story short, one of my drives had a corrupted filesystem. I did not have a backup of the data and it wasn't that big of a deal because it was all replaceable. I formatted the drive and reinstalled. Everything was fine for a day or so, then I started having all sorts of random errors. To further aggravate things, in my carelessness I deleted my appdata folder (i did have a backup of this) and decided to use the opportunity to just reformat my cache drives since I was getting some csum errors from btrfs. Now I have a brand new cache pool and a brand new drive. The problem is the errors I am still getting and the server is locking up.

 

Here is a snippet of all the crap going on....

Apr 12 11:11:03 MediaServer root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token
Apr 12 11:11:15 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 217452544 csum 0x9883fc4e expected csum 0x9a9ee9e3 mirror 1
Apr 12 11:11:15 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 263610368 csum 0x54ed82f1 expected csum 0x58eef065 mirror 1
Apr 12 11:11:19 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 1244962816 csum 0xa8f2f821 expected csum 0x9bee6e99 mirror 1
Apr 12 11:11:19 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 1365680128 csum 0x9e50867b expected csum 0xc91194b2 mirror 2
Apr 12 11:11:20 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 1482903552 csum 0xdba4a14a expected csum 0x866ac217 mirror 1
Apr 12 11:11:20 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 1482903552 csum 0xdba4a14a expected csum 0x866ac217 mirror 1
Apr 12 11:11:21 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 1482903552 csum 0xdba4a14a expected csum 0x866ac217 mirror 2
Apr 12 11:11:22 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 1482903552 csum 0xdba4a14a expected csum 0x866ac217 mirror 1
Apr 12 11:11:22 MediaServer root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
Apr 12 11:11:22 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 1482903552 csum 0xdba4a14a expected csum 0x866ac217 mirror 2
Apr 12 11:11:22 MediaServer root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token
Apr 12 11:11:29 MediaServer root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token
Apr 12 11:11:40 MediaServer kernel: general protection fault: 0000 [#2] PREEMPT SMP NOPTI
Apr 12 11:11:40 MediaServer kernel: Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs nfsd lockd grace sunrpc md_mod it87 hwmon_vid mlx4_en mlx4_core igb ptp pps_core i2c_algo_bit hid_logitech_hidpp edac_mce_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc mpt3sas aesni_intel aes_x86_64 crypto_simd glue_helper ahci libahci i2c_piix4 i2c_core raid_class cryptd scsi_transport_sas hid_logitech_dj mxm_wmi wmi_bmof ccp wmi button acpi_cpufreq [last unloaded: mlx4_core]
Apr 12 11:11:40 MediaServer kernel: CPU: 6 PID: 6426 Comm: kworker/u32:11 Tainted: G D 4.14.26-unRAID #1
Apr 12 11:11:40 MediaServer kernel: Hardware name: Gigabyte Technology Co., Ltd. AX370-Gaming K5/AX370-Gaming K5-CF, BIOS F22 03/15/2018
Apr 12 11:11:40 MediaServer kernel: Workqueue: btrfs-worker btrfs_worker_helper
Apr 12 11:11:40 MediaServer kernel: task: ffff8803c5686200 task.stack: ffffc90009c58000
Apr 12 11:11:40 MediaServer kernel: RIP: 0010:prefetch_freepointer.isra.11+0x8/0x10
Apr 12 11:11:40 MediaServer kernel: RSP: 0018:ffffc90009c5bd98 EFLAGS: 00010286
Apr 12 11:11:40 MediaServer kernel: RAX: 0000000000000000 RBX: fff78803baef8c00 RCX: 00000000023e5b06
Apr 12 11:11:40 MediaServer kernel: RDX: 00000000023e5a06 RSI: fff78803baef8c00 RDI: ffff88041e807620
Apr 12 11:11:40 MediaServer kernel: RBP: ffff88041e807600 R08: 00000000457cf000 R09: 0000000000000000
Apr 12 11:11:40 MediaServer kernel: R10: 0000000000000003 R11: ffff88041eda09e8 R12: ffff8803baef8ba0
Apr 12 11:11:40 MediaServer kernel: R13: 0000000001408040 R14: ffffffff811d33da R15: 0000000000000000
Apr 12 11:11:40 MediaServer kernel: FS: 0000000000000000(0000) GS:ffff88041ed80000(0000) knlGS:0000000000000000
Apr 12 11:11:40 MediaServer kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 12 11:11:40 MediaServer kernel: CR2: 00001456e50f7ff8 CR3: 00000003c213e000 CR4: 00000000003406e0
Apr 12 11:11:40 MediaServer kernel: Call Trace:
Apr 12 11:11:40 MediaServer kernel: __kmalloc+0xd6/0x121
Apr 12 11:11:40 MediaServer kernel: ? btrfs_async_submit_limit+0x1c/0x1c
Apr 12 11:11:40 MediaServer kernel: btrfs_csum_one_bio+0x51/0x3cd
Apr 12 11:11:40 MediaServer kernel: ? __accumulate_pelt_segments+0x1d/0x2a
Apr 12 11:11:40 MediaServer kernel: ? native_apic_wait_icr_idle+0x18/0x23
Apr 12 11:11:40 MediaServer kernel: ? irq_work_queue+0x47/0x7c
Apr 12 11:11:40 MediaServer kernel: ? dequeue_entity+0x49d/0x4c2
Apr 12 11:11:40 MediaServer kernel: ? btrfs_async_submit_limit+0x1c/0x1c
Apr 12 11:11:40 MediaServer kernel: __btrfs_submit_bio_start+0x9/0x12
Apr 12 11:11:40 MediaServer kernel: run_one_async_start+0x20/0x29
Apr 12 11:11:40 MediaServer kernel: btrfs_worker_helper+0xc2/0x185
Apr 12 11:11:40 MediaServer kernel: process_one_work+0x14c/0x23f
Apr 12 11:11:40 MediaServer kernel: ? rescuer_thread+0x258/0x258
Apr 12 11:11:40 MediaServer kernel: worker_thread+0x1c3/0x292
Apr 12 11:11:40 MediaServer kernel: kthread+0x111/0x119
Apr 12 11:11:40 MediaServer kernel: ? kthread_create_on_node+0x3a/0x3a
Apr 12 11:11:40 MediaServer kernel: ? SyS_exit_group+0xb/0xb
Apr 12 11:11:40 MediaServer kernel: ret_from_fork+0x22/0x40
Apr 12 11:11:40 MediaServer kernel: Code: 44 06 ff a5 41 f6 40 09 04 74 17 49 63 40 1c 41 8b 48 50 48 01 c6 29 c1 48 89 f7 88 d0 48 63 c9 f3 aa c3 48 85 f6 74 0a 48 63 07 <48> 8b 04 06 0f 18 08 c3 31 c0 c3 55 48 89 fd 53 48 63 47 20 48
Apr 12 11:11:40 MediaServer kernel: RIP: prefetch_freepointer.isra.11+0x8/0x10 RSP: ffffc90009c5bd98
Apr 12 11:11:40 MediaServer kernel: ---[ end trace 35dac60ee19e4866 ]---
Apr 12 11:11:40 MediaServer kernel: general protection fault: 0000 [#3] PREEMPT SMP NOPTI
Apr 12 11:11:40 MediaServer kernel: Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs nfsd lockd grace sunrpc md_mod it87 hwmon_vid mlx4_en mlx4_core igb ptp pps_core i2c_algo_bit hid_logitech_hidpp edac_mce_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc mpt3sas aesni_intel aes_x86_64 crypto_simd glue_helper ahci libahci i2c_piix4 i2c_core raid_class cryptd scsi_transport_sas hid_logitech_dj mxm_wmi wmi_bmof ccp wmi button acpi_cpufreq [last unloaded: mlx4_core]
Apr 12 11:11:40 MediaServer kernel: CPU: 6 PID: 6560 Comm: kworker/u32:12 Tainted: G D 4.14.26-unRAID #1
Apr 12 11:11:40 MediaServer kernel: Hardware name: Gigabyte Technology Co., Ltd. AX370-Gaming K5/AX370-Gaming K5-CF, BIOS F22 03/15/2018
Apr 12 11:11:40 MediaServer kernel: Workqueue: btrfs-worker btrfs_worker_helper
Apr 12 11:11:40 MediaServer kernel: task: ffff88033b442a00 task.stack: ffffc90009e30000
Apr 12 11:11:40 MediaServer kernel: RIP: 0010:__kmalloc+0xb7/0x121
Apr 12 11:11:40 MediaServer kernel: RSP: 0018:ffffc90009e33da0 EFLAGS: 00010286
Apr 12 11:11:40 MediaServer kernel: RAX: 0000000000000000 RBX: ffff88033b712040 RCX: 00000000023e5c06
Apr 12 11:11:40 MediaServer kernel: RDX: 00000000023e5b06 RSI: 00000000023e5b06 RDI: 0000000000024060
Apr 12 11:11:40 MediaServer kernel: RBP: ffff88041e807600 R08: 0000000047def000 R09: 0000000000000000
Apr 12 11:11:40 MediaServer kernel: R10: 0000000000000001 R11: ffff88041eda09e8 R12: fff78803baef8c00
Apr 12 11:11:40 MediaServer kernel: R13: 0000000001408040 R14: ffffffff811d33da R15: 0000000000000000
Apr 12 11:11:40 MediaServer kernel: FS: 0000000000000000(0000) GS:ffff88041ed80000(0000) knlGS:0000000000000000
Apr 12 11:11:40 MediaServer kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 12 11:11:40 MediaServer kernel: CR2: 00001456e50f7ff8 CR3: 00000003c213e000 CR4: 00000000003406e0
Apr 12 11:11:40 MediaServer kernel: Call Trace:
Apr 12 11:11:40 MediaServer kernel: ? btrfs_async_submit_limit+0x1c/0x1c
Apr 12 11:11:40 MediaServer kernel: btrfs_csum_one_bio+0x51/0x3cd
Apr 12 11:11:40 MediaServer kernel: ? __accumulate_pelt_segments+0x1d/0x2a
Apr 12 11:11:40 MediaServer kernel: ? native_apic_wait_icr_idle+0x18/0x23
Apr 12 11:11:40 MediaServer kernel: ? irq_work_queue+0x47/0x7c
Apr 12 11:11:40 MediaServer kernel: ? dequeue_entity+0x49d/0x4c2
Apr 12 11:11:40 MediaServer kernel: ? btrfs_async_submit_limit+0x1c/0x1c
Apr 12 11:11:40 MediaServer kernel: __btrfs_submit_bio_start+0x9/0x12
Apr 12 11:11:40 MediaServer kernel: run_one_async_start+0x20/0x29
Apr 12 11:11:40 MediaServer kernel: btrfs_worker_helper+0xc2/0x185
Apr 12 11:11:40 MediaServer kernel: process_one_work+0x14c/0x23f
Apr 12 11:11:40 MediaServer kernel: ? rescuer_thread+0x258/0x258
Apr 12 11:11:40 MediaServer kernel: worker_thread+0x1c3/0x292
Apr 12 11:11:40 MediaServer kernel: kthread+0x111/0x119
Apr 12 11:11:40 MediaServer kernel: ? kthread_create_on_node+0x3a/0x3a
Apr 12 11:11:40 MediaServer kernel: ret_from_fork+0x22/0x40
Apr 12 11:11:40 MediaServer kernel: Code: 34 01 f0 7e 48 8b 70 08 48 39 f2 75 e7 48 83 78 10 00 4c 8b 20 74 35 4d 85 e4 74 30 48 63 45 20 48 8d 8a 00 01 00 00 48 8b 7d 00 <49> 8b 1c 04 4c 89 e0 65 48 0f c7 0f 0f 94 c0 84 c0 74 b2 48 8d
Apr 12 11:11:40 MediaServer kernel: RIP: __kmalloc+0xb7/0x121 RSP: ffffc90009e33da0
Apr 12 11:11:40 MediaServer kernel: ---[ end trace 35dac60ee19e4867 ]---
Apr 12 11:11:40 MediaServer kernel: general protection fault: 0000 [#4] PREEMPT SMP NOPTI
Apr 12 11:11:40 MediaServer kernel: Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs nfsd lockd grace sunrpc md_mod it87 hwmon_vid mlx4_en mlx4_core igb ptp pps_core i2c_algo_bit hid_logitech_hidpp edac_mce_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc mpt3sas aesni_intel aes_x86_64 crypto_simd glue_helper ahci libahci i2c_piix4 i2c_core raid_class cryptd scsi_transport_sas hid_logitech_dj mxm_wmi wmi_bmof ccp wmi button acpi_cpufreq [last unloaded: mlx4_core]

Apr 12 11:18:38 MediaServer nginx: 2018/04/12 11:18:38 [error] 9461#9461: *17726 connect() to unix:/var/run/php5-fpm.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.28.1, server: , request: "POST /plugins/unassigned.devices/UnassignedDevices.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.28.2", referrer: "http://192.168.28.2/Main"

Apr 12 11:06:22 MediaServer root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token
Apr 12 11:06:37 MediaServer root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token

 

Any ideas???

[JorgeB]

Please post your diagnostics: Tools -> Diagnostics

[TechnoBabble28]

Sorry, forgot about that. Here you go!

mediaserver-diagnostics-20180412-1129.zip

[JorgeB]

Since there's corruption on both cache devices, and according to you they were recently formatted, I'd start by running memtest for a few hours, then run a correcting scrub you the pool and make sure all errors are corrected.

 

For the locking up make sure you are disabling C-states, see the Ryzen notes:

 

[TechnoBabble28]

I will give the memtest a try. I tried a scrub once already and it came back with 20 uncorrectable errors from what i remember. But I will try it again after a memtest. As for the cstates/zenstates thing, i definately have cstates disabled in the MB BIOS and when i tried the zenstates code it made the system very unstable so i havent been using it.

[TechnoBabble28]

Wellllll, I think we found the problem.