March 2, 20242 yr Hi all, I have read hundreds of threads trying to troubleshoot this issue, swapped out my motherboard, swapped out my power supply but still having issues. some background details: Server was stable for 6+ months, no major changes or updates were made. One day, there was a power outage, and in the following days all docker containers started throwing errors, mostly database errors in the form of I/O errors. After tons of troubleshooting, I ended up wiping everything and rebuilding from scratch. The problem was still there. I stumbled on this thread: where @JorgeB suggested change docker from macvlan to ipvlan when getting "write time tree block corruption" I have made the change to ipvlan yet the problems persist, now I'm not getting any more tree block corruptions. But critical "kernel: BTRFS error (device nvme0n1p1)" is still there. After running dozens of memtests for a few days, I realized one of the RAM modules was throwing errors. Once that module was out of the system, it passed extended 24h memtests. Now, I have formatted the cache drives in the bios, and rebuilt them as new cache disks in unraid. I am still getting the same "kernel: BTRFS error (device nvme0n1p1)" What can I do next?
March 2, 20242 yr Author maven-diagnostics-20240302-1147.zip Attached is my latest diagnostics file. I have a number of backdated ones as well if that is useful.
March 2, 20242 yr Community Expert 2 hours ago, connect0r said: where @JorgeB suggested change docker from macvlan to ipvlan when getting "write time tree block corruption" I suggested to run memtest, since this is usually a RAM problem, also changing to ipvlan was because of a different issue that user was also having. 2 hours ago, connect0r said: I am still getting the same "kernel: BTRFS error (device nvme0n1p1)" Btrfs is detecting data corruption, likely from the previous RAM issues, run a correcting scrub and post the results.
March 2, 20242 yr Author 3 hours ago, JorgeB said: I suggested to run memtest, since this is usually a RAM problem, also changing to ipvlan was because of a different issue that user was also having. Btrfs is detecting data corruption, likely from the previous RAM issues, run a correcting scrub and post the results. Thanks for your help ❤️ Here are the results from the correcting scrub run: UUID: 82c4600e-84bf-4150-bec7-51f5cfca0556 Scrub started: Sat Mar 2 18:13:42 2024 Status: finished Duration: 0:01:47 Total to scrub: 650.91GiB Rate: 6.08GiB/s Error summary: csum=6 Corrected: 0 Uncorrectable: 6 Unverified: 0 Edited March 2, 20242 yr by connect0r
March 3, 20242 yr Community Expert Solution Look in the syslog for the list of corrupt files and delete/restore from a backup, then run another scrub to confirm 0 errors.
March 4, 20242 yr Author Worked!!! Results: UUID: 82c4600e-84bf-4150-bec7-51f5cfca0556 Scrub started: Mon Mar 4 12:54:57 2024 Status: finished Duration: 0:01:14 Total to scrub: 429.28GiB Rate: 5.84GiB/s Error summary: no errors found Rebuilding my docker stacks now. no errors so far. 😭❤️ Thank you so much for your help @JorgeB Server was down for 2 weeks, had late nights with the server in pieces trying to isolate the hardware issues. None of the threads I found guided me to a solution and usually just lead to different errors. After countless parity checks, scrubs and reinstalls I was at a loss of what to do. I ripped the server apart, replaced most hardware and still nothing worked. In panic I nuked the cache, lost data, made some stupid mistakes while sleep deprived and not thinking straight. Thank you for the peace you have brought me.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.