colev14 Posted March 22, 2023 Share Posted March 22, 2023 (edited) I noticed all my docker containers weren't working, but my server was still running and everything appeared to be working fine on the dashboard. I clicked on the logs for my cache drive and saw a bunch of red BTRFS errors and saying it was now mounted read only. I have attached my diagnostics. I rebooted and everything is running fine now, but it did detect an unclean shutdown and started a parity check. I did some troubleshooting for unclean shutdowns previously and have my docker stop timeout at 240 seconds, so I think the issue is all in the cache drive. unraid-diagnostics-20230321-2030.zip Edited March 22, 2023 by colev14 Quote Link to comment
pointdewu Posted March 22, 2023 Share Posted March 22, 2023 I've had the same problem and it's been days, even rebooting doesn't fix it. Quote Link to comment
JorgeB Posted March 22, 2023 Share Posted March 22, 2023 Mar 21 18:59:48 Unraid kernel: BTRFS error (device nvme1n1p1): block=196110008320 write time tree block corruption detected This is usually caused by bad RAM or other kernel memory corruption, and Ryzen with overclocked RAM like you have is known to corrupt data, so start here. Quote Link to comment
colev14 Posted March 22, 2023 Author Share Posted March 22, 2023 2 minutes ago, JorgeB said: Mar 21 18:59:48 Unraid kernel: BTRFS error (device nvme1n1p1): block=196110008320 write time tree block corruption detected This is usually caused by bad RAM or other kernel memory corruption, and Ryzen with overclocked RAM like you have is known to corrupt data, so start here. I have disabled C states before. I used to have this issue every other day and that solved it. As part of solving the original issue I changed to this new cache drive and got a new ram kit. I went from 4 8Gb sticks to 2 new 8gb sticks. Would just turning off XMP solve this? I'm not sure if I did that or not. I will have to check when I get back from work tonight. Quote Link to comment
JorgeB Posted March 22, 2023 Share Posted March 22, 2023 The RAM is running at 3200MT/s which is an overclock for your CPU, see link. Quote Link to comment
colev14 Posted March 26, 2023 Author Share Posted March 26, 2023 On 3/22/2023 at 6:24 AM, JorgeB said: The RAM is running at 3200MT/s which is an overclock for your CPU, see link. I switched off XMP in the bios. I am still getting this error now after a couple of days. Am I doing something else wrong? unraid-diagnostics-20230326-1409.zip Quote Link to comment
JorgeB Posted March 27, 2023 Share Posted March 27, 2023 Was the pool clear after the change, i.e., did you run a scrub without errors? Quote Link to comment
colev14 Posted March 27, 2023 Author Share Posted March 27, 2023 1 hour ago, JorgeB said: Was the pool clear after the change, i.e., did you run a scrub without errors? There are 6 uncorrectable errors Quote Link to comment
Solution JorgeB Posted March 27, 2023 Solution Share Posted March 27, 2023 First you must delete/restore those files from a backup, scrub must return 0 errors, only after that can you retest to see if the problem is solved or not. Quote Link to comment
colev14 Posted April 9, 2023 Author Share Posted April 9, 2023 Restored cache drive. I'm now 15+ days of uptime with no crashes or any other issues. Thank you! 1 Quote Link to comment
Smolo Posted March 18 Share Posted March 18 @JorgeB Here the same problem Cache Pool no longer accessible after a restart. I was able to solve it with btrfs rescue zero-log /dev/nvme1n1p1. Unfortunately the same error came back after a few days. I then ran btrfs resuce zero-log again and wanted to move the data to the array via the mover. Unfortunately, I have massive access errors. Can you possibly help? Mar 18 01:03:07 Sugo kernel: btrfs_print_data_csum_error: 1494 callbacks suppressed Mar 18 01:03:07 Sugo kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 36649273 off 0 csum 0x8941f998 expected csum 0x36d489f9 mirror 1 Mar 18 01:03:07 Sugo kernel: btrfs_dev_stat_inc_and_print: 1494 callbacks suppressed Mar 18 01:03:07 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 25951, gen 0 Mar 18 01:03:07 Sugo kernel: BTRFS warning (device nvme0n1p1): csum failed root 5 ino 36649273 off 4096 csum 0x8941f998 expected csum 0xa4cb39d0 mirror 1 Translated with DeepL.com (free version) sugo-diagnostics-20240318-0032.zip Quote Link to comment
JorgeB Posted March 18 Share Posted March 18 Mar 10 17:11:54 Sugo kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 122, gen 0 Mar 10 17:11:54 Sugo kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 122, gen 0 Btrfs is detecting data corruption on both pool devices, start by running memtest. 1 Quote Link to comment
Smolo Posted March 19 Share Posted March 19 23 hours ago, JorgeB said: Mar 10 17:11:54 Sugo kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 122, gen 0 Mar 10 17:11:54 Sugo kernel: BTRFS info (device nvme0n1p1): bdev /dev/nvme1n1p1 errs: wr 0, rd 0, flush 0, corrupt 122, gen 0 Btrfs is detecting data corruption on both pool devices, start by running memtest. I have now run it twice without any results. The first run took over 14 hours. On the second run I reset all BIOS values beforehand without any errors. Quote Link to comment
JorgeB Posted March 19 Share Posted March 19 Memtest is only definitive if it finds errors, reset the btrfs stats and run a correcting scrub, then post he output. 1 Quote Link to comment
JorgeB Posted March 19 Share Posted March 19 To get rid of the errors you'd need to delete the corrupt files, they will be listed in the syslog after the scrub, but note that if there are still RAM issues, it may be finding false positives, and you would possibly be deleting good files. Quote Link to comment
Smolo Posted March 19 Share Posted March 19 @Jorge I have now deleted some data but unfortunately the next scrub will show new files as corrupt? The docker file is also affected, how can I restore it or have it recreated? To be honest, I don't really understand why btrfs and a raid 1 system and two ECC RAMs play such an important role? :-( Quote Link to comment
JorgeB Posted March 19 Share Posted March 19 1 minute ago, Smolo said: the next scrub will show new files as corrupt? That suggests there's still a hardware issue causing the data corruption. Quote Link to comment
Smolo Posted March 19 Share Posted March 19 I'm completely at a loss, I've never had anything like this before. Theoretically it can only be the power supply(?), memory, NVME or mainboard. I've already ordered a new power supply, but I can't really imagine what's going on anymore. The memory shouldn't play a role at this point, should it? Quote Link to comment
JorgeB Posted March 19 Share Posted March 19 You can try with just one stick of RAM, if the same try the other one, that will basically rule out a RAM issue. Quote Link to comment
Smolo Posted March 20 Share Posted March 20 @Jorge After I successively deleted the files, it stopped at some point, i.e. no more display of corrupt files. I am currently using a RAM bar. What do I do now with the remaining errors? Mar 20 09:12:21 Sugo ool www[14282]: /usr/local/emhttp/plugins/dynamix/scripts/btrfs_scrub 'start' '/mnt/cache' '' Mar 20 09:12:21 Sugo kernel: BTRFS info (device nvme0n1p1): scrub: started on devid 1 Mar 20 09:12:21 Sugo kernel: BTRFS info (device nvme0n1p1): scrub: started on devid 2 Mar 20 09:12:21 Sugo kernel: scrub_handle_errored_block: 6910 callbacks suppressed Mar 20 09:12:21 Sugo kernel: btrfs_dev_stat_inc_and_print: 6892 callbacks suppressed Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15786, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15787, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15788, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15789, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15790, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15791, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15793, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15792, gen 0 Mar 20 09:12:21 Sugo kernel: scrub_handle_errored_block: 6868 callbacks suppressed Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208361881600 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208362405888 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208361488384 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208362274816 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208361750528 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208361619456 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208362012672 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208362143744 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15795, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 0, rd 0, flush 0, corrupt 15794, gen 0 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208361885696 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS error (device nvme0n1p1): unable to fixup (regular) error at logical 208361754624 on dev /dev/nvme0n1p1 Mar 20 09:12:21 Sugo kernel: BTRFS info (device nvme0n1p1): scrub: finished on devid 1 with status: 0 Mar 20 09:12:21 Sugo kernel: BTRFS info (device nvme0n1p1): scrub: finished on devid 2 with status: 0 Quote Link to comment
JorgeB Posted March 20 Share Posted March 20 If it's still finding errors recommend backing up and recreating the pool. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.