TechnoBabble28 Posted April 12, 2018 Share Posted April 12, 2018 Hey guys, Long story short, one of my drives had a corrupted filesystem. I did not have a backup of the data and it wasn't that big of a deal because it was all replaceable. I formatted the drive and reinstalled. Everything was fine for a day or so, then I started having all sorts of random errors. To further aggravate things, in my carelessness I deleted my appdata folder (i did have a backup of this) and decided to use the opportunity to just reformat my cache drives since I was getting some csum errors from btrfs. Now I have a brand new cache pool and a brand new drive. The problem is the errors I am still getting and the server is locking up. Here is a snippet of all the crap going on.... Apr 12 11:11:03 MediaServer root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Apr 12 11:11:15 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 217452544 csum 0x9883fc4e expected csum 0x9a9ee9e3 mirror 1 Apr 12 11:11:15 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 263610368 csum 0x54ed82f1 expected csum 0x58eef065 mirror 1 Apr 12 11:11:19 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 1244962816 csum 0xa8f2f821 expected csum 0x9bee6e99 mirror 1 Apr 12 11:11:19 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 1365680128 csum 0x9e50867b expected csum 0xc91194b2 mirror 2 Apr 12 11:11:20 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 1482903552 csum 0xdba4a14a expected csum 0x866ac217 mirror 1 Apr 12 11:11:20 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 1482903552 csum 0xdba4a14a expected csum 0x866ac217 mirror 1 Apr 12 11:11:21 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 1482903552 csum 0xdba4a14a expected csum 0x866ac217 mirror 2 Apr 12 11:11:22 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 1482903552 csum 0xdba4a14a expected csum 0x866ac217 mirror 1 Apr 12 11:11:22 MediaServer root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token Apr 12 11:11:22 MediaServer kernel: BTRFS warning (device sde1): csum failed root 5 ino 1020051 off 1482903552 csum 0xdba4a14a expected csum 0x866ac217 mirror 2 Apr 12 11:11:22 MediaServer root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token Apr 12 11:11:29 MediaServer root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Apr 12 11:11:40 MediaServer kernel: general protection fault: 0000 [#2] PREEMPT SMP NOPTI Apr 12 11:11:40 MediaServer kernel: Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs nfsd lockd grace sunrpc md_mod it87 hwmon_vid mlx4_en mlx4_core igb ptp pps_core i2c_algo_bit hid_logitech_hidpp edac_mce_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc mpt3sas aesni_intel aes_x86_64 crypto_simd glue_helper ahci libahci i2c_piix4 i2c_core raid_class cryptd scsi_transport_sas hid_logitech_dj mxm_wmi wmi_bmof ccp wmi button acpi_cpufreq [last unloaded: mlx4_core] Apr 12 11:11:40 MediaServer kernel: CPU: 6 PID: 6426 Comm: kworker/u32:11 Tainted: G D 4.14.26-unRAID #1 Apr 12 11:11:40 MediaServer kernel: Hardware name: Gigabyte Technology Co., Ltd. AX370-Gaming K5/AX370-Gaming K5-CF, BIOS F22 03/15/2018 Apr 12 11:11:40 MediaServer kernel: Workqueue: btrfs-worker btrfs_worker_helper Apr 12 11:11:40 MediaServer kernel: task: ffff8803c5686200 task.stack: ffffc90009c58000 Apr 12 11:11:40 MediaServer kernel: RIP: 0010:prefetch_freepointer.isra.11+0x8/0x10 Apr 12 11:11:40 MediaServer kernel: RSP: 0018:ffffc90009c5bd98 EFLAGS: 00010286 Apr 12 11:11:40 MediaServer kernel: RAX: 0000000000000000 RBX: fff78803baef8c00 RCX: 00000000023e5b06 Apr 12 11:11:40 MediaServer kernel: RDX: 00000000023e5a06 RSI: fff78803baef8c00 RDI: ffff88041e807620 Apr 12 11:11:40 MediaServer kernel: RBP: ffff88041e807600 R08: 00000000457cf000 R09: 0000000000000000 Apr 12 11:11:40 MediaServer kernel: R10: 0000000000000003 R11: ffff88041eda09e8 R12: ffff8803baef8ba0 Apr 12 11:11:40 MediaServer kernel: R13: 0000000001408040 R14: ffffffff811d33da R15: 0000000000000000 Apr 12 11:11:40 MediaServer kernel: FS: 0000000000000000(0000) GS:ffff88041ed80000(0000) knlGS:0000000000000000 Apr 12 11:11:40 MediaServer kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 12 11:11:40 MediaServer kernel: CR2: 00001456e50f7ff8 CR3: 00000003c213e000 CR4: 00000000003406e0 Apr 12 11:11:40 MediaServer kernel: Call Trace: Apr 12 11:11:40 MediaServer kernel: __kmalloc+0xd6/0x121 Apr 12 11:11:40 MediaServer kernel: ? btrfs_async_submit_limit+0x1c/0x1c Apr 12 11:11:40 MediaServer kernel: btrfs_csum_one_bio+0x51/0x3cd Apr 12 11:11:40 MediaServer kernel: ? __accumulate_pelt_segments+0x1d/0x2a Apr 12 11:11:40 MediaServer kernel: ? native_apic_wait_icr_idle+0x18/0x23 Apr 12 11:11:40 MediaServer kernel: ? irq_work_queue+0x47/0x7c Apr 12 11:11:40 MediaServer kernel: ? dequeue_entity+0x49d/0x4c2 Apr 12 11:11:40 MediaServer kernel: ? btrfs_async_submit_limit+0x1c/0x1c Apr 12 11:11:40 MediaServer kernel: __btrfs_submit_bio_start+0x9/0x12 Apr 12 11:11:40 MediaServer kernel: run_one_async_start+0x20/0x29 Apr 12 11:11:40 MediaServer kernel: btrfs_worker_helper+0xc2/0x185 Apr 12 11:11:40 MediaServer kernel: process_one_work+0x14c/0x23f Apr 12 11:11:40 MediaServer kernel: ? rescuer_thread+0x258/0x258 Apr 12 11:11:40 MediaServer kernel: worker_thread+0x1c3/0x292 Apr 12 11:11:40 MediaServer kernel: kthread+0x111/0x119 Apr 12 11:11:40 MediaServer kernel: ? kthread_create_on_node+0x3a/0x3a Apr 12 11:11:40 MediaServer kernel: ? SyS_exit_group+0xb/0xb Apr 12 11:11:40 MediaServer kernel: ret_from_fork+0x22/0x40 Apr 12 11:11:40 MediaServer kernel: Code: 44 06 ff a5 41 f6 40 09 04 74 17 49 63 40 1c 41 8b 48 50 48 01 c6 29 c1 48 89 f7 88 d0 48 63 c9 f3 aa c3 48 85 f6 74 0a 48 63 07 <48> 8b 04 06 0f 18 08 c3 31 c0 c3 55 48 89 fd 53 48 63 47 20 48 Apr 12 11:11:40 MediaServer kernel: RIP: prefetch_freepointer.isra.11+0x8/0x10 RSP: ffffc90009c5bd98 Apr 12 11:11:40 MediaServer kernel: ---[ end trace 35dac60ee19e4866 ]--- Apr 12 11:11:40 MediaServer kernel: general protection fault: 0000 [#3] PREEMPT SMP NOPTI Apr 12 11:11:40 MediaServer kernel: Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs nfsd lockd grace sunrpc md_mod it87 hwmon_vid mlx4_en mlx4_core igb ptp pps_core i2c_algo_bit hid_logitech_hidpp edac_mce_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc mpt3sas aesni_intel aes_x86_64 crypto_simd glue_helper ahci libahci i2c_piix4 i2c_core raid_class cryptd scsi_transport_sas hid_logitech_dj mxm_wmi wmi_bmof ccp wmi button acpi_cpufreq [last unloaded: mlx4_core] Apr 12 11:11:40 MediaServer kernel: CPU: 6 PID: 6560 Comm: kworker/u32:12 Tainted: G D 4.14.26-unRAID #1 Apr 12 11:11:40 MediaServer kernel: Hardware name: Gigabyte Technology Co., Ltd. AX370-Gaming K5/AX370-Gaming K5-CF, BIOS F22 03/15/2018 Apr 12 11:11:40 MediaServer kernel: Workqueue: btrfs-worker btrfs_worker_helper Apr 12 11:11:40 MediaServer kernel: task: ffff88033b442a00 task.stack: ffffc90009e30000 Apr 12 11:11:40 MediaServer kernel: RIP: 0010:__kmalloc+0xb7/0x121 Apr 12 11:11:40 MediaServer kernel: RSP: 0018:ffffc90009e33da0 EFLAGS: 00010286 Apr 12 11:11:40 MediaServer kernel: RAX: 0000000000000000 RBX: ffff88033b712040 RCX: 00000000023e5c06 Apr 12 11:11:40 MediaServer kernel: RDX: 00000000023e5b06 RSI: 00000000023e5b06 RDI: 0000000000024060 Apr 12 11:11:40 MediaServer kernel: RBP: ffff88041e807600 R08: 0000000047def000 R09: 0000000000000000 Apr 12 11:11:40 MediaServer kernel: R10: 0000000000000001 R11: ffff88041eda09e8 R12: fff78803baef8c00 Apr 12 11:11:40 MediaServer kernel: R13: 0000000001408040 R14: ffffffff811d33da R15: 0000000000000000 Apr 12 11:11:40 MediaServer kernel: FS: 0000000000000000(0000) GS:ffff88041ed80000(0000) knlGS:0000000000000000 Apr 12 11:11:40 MediaServer kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 12 11:11:40 MediaServer kernel: CR2: 00001456e50f7ff8 CR3: 00000003c213e000 CR4: 00000000003406e0 Apr 12 11:11:40 MediaServer kernel: Call Trace: Apr 12 11:11:40 MediaServer kernel: ? btrfs_async_submit_limit+0x1c/0x1c Apr 12 11:11:40 MediaServer kernel: btrfs_csum_one_bio+0x51/0x3cd Apr 12 11:11:40 MediaServer kernel: ? __accumulate_pelt_segments+0x1d/0x2a Apr 12 11:11:40 MediaServer kernel: ? native_apic_wait_icr_idle+0x18/0x23 Apr 12 11:11:40 MediaServer kernel: ? irq_work_queue+0x47/0x7c Apr 12 11:11:40 MediaServer kernel: ? dequeue_entity+0x49d/0x4c2 Apr 12 11:11:40 MediaServer kernel: ? btrfs_async_submit_limit+0x1c/0x1c Apr 12 11:11:40 MediaServer kernel: __btrfs_submit_bio_start+0x9/0x12 Apr 12 11:11:40 MediaServer kernel: run_one_async_start+0x20/0x29 Apr 12 11:11:40 MediaServer kernel: btrfs_worker_helper+0xc2/0x185 Apr 12 11:11:40 MediaServer kernel: process_one_work+0x14c/0x23f Apr 12 11:11:40 MediaServer kernel: ? rescuer_thread+0x258/0x258 Apr 12 11:11:40 MediaServer kernel: worker_thread+0x1c3/0x292 Apr 12 11:11:40 MediaServer kernel: kthread+0x111/0x119 Apr 12 11:11:40 MediaServer kernel: ? kthread_create_on_node+0x3a/0x3a Apr 12 11:11:40 MediaServer kernel: ret_from_fork+0x22/0x40 Apr 12 11:11:40 MediaServer kernel: Code: 34 01 f0 7e 48 8b 70 08 48 39 f2 75 e7 48 83 78 10 00 4c 8b 20 74 35 4d 85 e4 74 30 48 63 45 20 48 8d 8a 00 01 00 00 48 8b 7d 00 <49> 8b 1c 04 4c 89 e0 65 48 0f c7 0f 0f 94 c0 84 c0 74 b2 48 8d Apr 12 11:11:40 MediaServer kernel: RIP: __kmalloc+0xb7/0x121 RSP: ffffc90009e33da0 Apr 12 11:11:40 MediaServer kernel: ---[ end trace 35dac60ee19e4867 ]--- Apr 12 11:11:40 MediaServer kernel: general protection fault: 0000 [#4] PREEMPT SMP NOPTI Apr 12 11:11:40 MediaServer kernel: Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs nfsd lockd grace sunrpc md_mod it87 hwmon_vid mlx4_en mlx4_core igb ptp pps_core i2c_algo_bit hid_logitech_hidpp edac_mce_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc mpt3sas aesni_intel aes_x86_64 crypto_simd glue_helper ahci libahci i2c_piix4 i2c_core raid_class cryptd scsi_transport_sas hid_logitech_dj mxm_wmi wmi_bmof ccp wmi button acpi_cpufreq [last unloaded: mlx4_core] Apr 12 11:18:38 MediaServer nginx: 2018/04/12 11:18:38 [error] 9461#9461: *17726 connect() to unix:/var/run/php5-fpm.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.28.1, server: , request: "POST /plugins/unassigned.devices/UnassignedDevices.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.28.2", referrer: "http://192.168.28.2/Main" Apr 12 11:06:22 MediaServer root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Apr 12 11:06:37 MediaServer root: error: /plugins/preclear.disk/Preclear.php: wrong csrf_token Any ideas??? Quote Link to comment
JorgeB Posted April 12, 2018 Share Posted April 12, 2018 Please post your diagnostics: Tools -> Diagnostics Quote Link to comment
TechnoBabble28 Posted April 12, 2018 Author Share Posted April 12, 2018 Sorry, forgot about that. Here you go! mediaserver-diagnostics-20180412-1129.zip Quote Link to comment
JorgeB Posted April 12, 2018 Share Posted April 12, 2018 Since there's corruption on both cache devices, and according to you they were recently formatted, I'd start by running memtest for a few hours, then run a correcting scrub you the pool and make sure all errors are corrected. For the locking up make sure you are disabling C-states, see the Ryzen notes: Quote Link to comment
TechnoBabble28 Posted April 12, 2018 Author Share Posted April 12, 2018 I will give the memtest a try. I tried a scrub once already and it came back with 20 uncorrectable errors from what i remember. But I will try it again after a memtest. As for the cstates/zenstates thing, i definately have cstates disabled in the MB BIOS and when i tried the zenstates code it made the system very unstable so i havent been using it. Quote Link to comment
TechnoBabble28 Posted April 12, 2018 Author Share Posted April 12, 2018 Wellllll, I think we found the problem. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.