March 22, 20233 yr I noticed all my docker containers weren't working, but my server was still running and everything appeared to be working fine on the dashboard. I clicked on the logs for my cache drive and saw a bunch of red BTRFS errors and saying it was now mounted read only. I have attached my diagnostics. I rebooted and everything is running fine now, but it did detect an unclean shutdown and started a parity check. I did some troubleshooting for unclean shutdowns previously and have my docker stop timeout at 240 seconds, so I think the issue is all in the cache drive. unraid-diagnostics-20230321-2030.zip Edited March 22, 20233 yr by colev14
March 22, 20233 yr Community Expert Mar 21 18:59:48 Unraid kernel: BTRFS error (device nvme1n1p1): block=196110008320 write time tree block corruption detected This is usually caused by bad RAM or other kernel memory corruption, and Ryzen with overclocked RAM like you have is known to corrupt data, so start here.
March 22, 20233 yr Author 2 minutes ago, JorgeB said: Mar 21 18:59:48 Unraid kernel: BTRFS error (device nvme1n1p1): block=196110008320 write time tree block corruption detected This is usually caused by bad RAM or other kernel memory corruption, and Ryzen with overclocked RAM like you have is known to corrupt data, so start here. I have disabled C states before. I used to have this issue every other day and that solved it. As part of solving the original issue I changed to this new cache drive and got a new ram kit. I went from 4 8Gb sticks to 2 new 8gb sticks. Would just turning off XMP solve this? I'm not sure if I did that or not. I will have to check when I get back from work tonight.
March 22, 20233 yr Community Expert The RAM is running at 3200MT/s which is an overclock for your CPU, see link.
March 26, 20233 yr Author On 3/22/2023 at 6:24 AM, JorgeB said: The RAM is running at 3200MT/s which is an overclock for your CPU, see link. I switched off XMP in the bios. I am still getting this error now after a couple of days. Am I doing something else wrong? unraid-diagnostics-20230326-1409.zip
March 27, 20233 yr Community Expert Was the pool clear after the change, i.e., did you run a scrub without errors?
March 27, 20233 yr Author 1 hour ago, JorgeB said: Was the pool clear after the change, i.e., did you run a scrub without errors? There are 6 uncorrectable errors
March 27, 20233 yr Community Expert Solution First you must delete/restore those files from a backup, scrub must return 0 errors, only after that can you retest to see if the problem is solved or not.
April 9, 20233 yr Author Restored cache drive. I'm now 15+ days of uptime with no crashes or any other issues. Thank you!
March 18, 20242 yr @JorgeB Here the same problem Cache Pool no longer accessible after a restart. I was able to solve it with btrfs rescue zero-log /dev/nvme1n1p1. Unfortunately the same error came back after a few days. I then ran btrfs resuce zero-log again and wanted to move the data to the array via the mover. Unfortunately, I have massive access errors. Can you possibly help? Mar 18 01:03:07 Sugo kernel: btrfs_print_data_csum_error: 1494 callbacks suppressed Mar 18 01:03:07 Sugo kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 36649273 off 0 csum 0x8941f998 expected csum 0x36d489f9 mirror 1 Mar 18 01:03:07 Sugo kernel: btrfs_dev_stat_inc_and_print: 1494 callbacks suppressed Mar 18 01:03:07 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 25951, gen 0 Mar 18 01:03:07 Sugo kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 36649273 off 4096 csum 0x8941f998 expected csum 0xa4cb39d0 mirror 1 Translated with DeepL.com (free version) sugo-diagnostics-20240318-0032.zip
March 18, 20242 yr Community Expert Mar 10 17:11:54 Sugo kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 122, gen 0 Mar 10 17:11:54 Sugo kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 122, gen 0 Btrfs is detecting data corruption on both pool devices, start by running memtest.
March 19, 20242 yr 23 hours ago, JorgeB said: Mar 10 17:11:54 Sugo kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 122, gen 0 Mar 10 17:11:54 Sugo kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 122, gen 0 Btrfs is detecting data corruption on both pool devices, start by running memtest. I have now run it twice without any results. The first run took over 14 hours. On the second run I reset all BIOS values beforehand without any errors.
March 19, 20242 yr Community Expert Memtest is only definitive if it finds errors, reset the btrfs stats and run a correcting scrub, then post he output.
March 19, 20242 yr Community Expert To get rid of the errors you'd need to delete the corrupt files, they will be listed in the syslog after the scrub, but note that if there are still RAM issues, it may be finding false positives, and you would possibly be deleting good files.
March 19, 20242 yr @Jorge I have now deleted some data but unfortunately the next scrub will show new files as corrupt? The docker file is also affected, how can I restore it or have it recreated? To be honest, I don't really understand why btrfs and a raid 1 system and two ECC RAMs play such an important role? :-(
March 19, 20242 yr Community Expert 1 minute ago, Smolo said: the next scrub will show new files as corrupt? That suggests there's still a hardware issue causing the data corruption.
March 19, 20242 yr I'm completely at a loss, I've never had anything like this before. Theoretically it can only be the power supply(?), memory, NVME or mainboard. I've already ordered a new power supply, but I can't really imagine what's going on anymore. The memory shouldn't play a role at this point, should it?
March 19, 20242 yr Community Expert You can try with just one stick of RAM, if the same try the other one, that will basically rule out a RAM issue.
March 20, 20242 yr @Jorge After I successively deleted the files, it stopped at some point, i.e. no more display of corrupt files. I am currently using a RAM bar. What do I do now with the remaining errors? Mar 20 09:12:21 Sugo ool www[14282]: /usr/local/emhttp/plugins/dynamix/scripts/btrfs_scrub 'start' '/mnt/cache' '' Mar 20 09:12:21 Sugo kernel: BTRFS info (device nvme0n1p1): scrub: started on devid 1 Mar 20 09:12:21 Sugo kernel: BTRFS info (device nvme0n1p1): scrub: started on devid 2 Mar 20 09:12:21 Sugo kernel: scrub_handle_errored_block: 6910 callbacks suppressed Mar 20 09:12:21 Sugo kernel: btrfs_dev_stat_inc_and_print: 6892 callbacks suppressed Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15786, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15787, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15788, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15789, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15790, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15791, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15793, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15792, gen 0 Mar 20 09:12:21 Sugo kernel: scrub_handle_errored_block: 6868 callbacks suppressed Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208361881600 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208362405888 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208361488384 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208362274816 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208361750528 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208361619456 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208362012672 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208362143744 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15795, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15794, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208361885696 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208361754624 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS info (device nvme0n1p1): scrub: finished on devid 1 with status: 0 Mar 20 09:12:21 Sugo kernel: BTRFS info (device nvme0n1p1): scrub: finished on devid 2 with status: 0
March 20, 20242 yr Community Expert If it's still finding errors recommend backing up and recreating the pool.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.