BTRFS corruption on 1 drive and stuck on a fix..

[rclifton]

Hi,

On Sunday I had a freakout moment and noticed some data was missing (thread here).

 I followed the link that was posted in one of the responses and ran the restore command and copied the data to a spare drive.  I then went one extra and copied all the data off the array using krusader to some usb drives.  After doing all that I ran the  btrfs check --repair /dev/md4 command and it found and corrected errors.  The problem is after rebooting the system is still reporting the same errors on drive 4.  I'm not really sure what to do at this point.  I know I could just blow it up and restore from my backups, but not sure I really want to go that route before exhausting any other avenues..  At this point could I run the BTRFS check --repair command again and then power down the system, remove the drive, format a spare and copy the contents from the old drive that I already backed up onto the new drive and then restart? Or something else?

I'm kind of stuck at this point as I'm not real sure where to go from here at this point...  I pasted the output of the BTRFS check --repair command below :

 

The operation will start in 10 seconds.
        Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/md4
UUID: f10d28a7-144b-4b86-8489-c8be6efcc9f8
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
parent transid verify failed on 87638016 wanted 25935 found 25928
parent transid verify failed on 87638016 wanted 25935 found 25928
Ignoring transid failure
bad block 87638016
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
parent transid verify failed on 87638016 wanted 25935 found 25928
Ignoring transid failure
(the above line repeated several 100x and was removed)
Ignoring transid failure
Wrong key of child node/leaf, wanted: (65781, 1, 0), have: (256, 1, 0)
Wrong generation of child node/leaf, wanted: 25928, have: 25935
Deleting bad dir index [6934,96,5] root 5
Deleting bad dir index [63235,96,3] root 5
Deleting bad dir index [63235,96,4] root 5
Deleting bad dir index [63235,96,5] root 5
Deleting bad dir index [1264,96,18] root 5
Deleting bad dir index [1254,96,5] root 5
Deleting bad dir index [6934,96,6] root 5
ERROR: errors found in fs roots
found 4022054645760 bytes used, error(s) found
total csum bytes: 0
total tree bytes: 30851072
total fs tree bytes: 2523136
total extent tree bytes: 26148864
btree space waste bytes: 7056862
file data blocks allocated: 545990963200
 referenced 544234590208

 

Thanks for any help!

Edited November 11, 2020 by rclifton

[JorgeB]

After doing all that I ran the  btrfs check --repair /dev/md4 command and it found and corrected errors.  The problem is after rebooting the system is still reporting the same errors on drive 4.  I'm not really sure what to do at this point.

Like I already posted on your other topic that error is fatal, the disk needs to e re-formatted.

 

 

[rclifton]

8 hours ago, JorgeB said:

Like I already posted on your other topic that error is fatal, the disk needs to e re-formatted.

 

 

Yes, you did.  And I thank you for the help, but at the risk of sounding like an idiot basically I'm asking what do I do after I format it? Do I recopy the data back to the same drive and put it back and Unraid thinks everythings a-ok now? Do I put the drive I copied everything to into the old drives spot and Unraid will figure it out? Or something entirely different that I'm not seeing?  I guess I'm kind of confused because telling me to format the drive doesn't help when you also said parity can't help me, but if I format the drive what fixes the problem of getting the data back safely?  I hope all of this actually makes sense, thanks... 

[JorgeB]

You need to re-format the disk (and parity will be updated accordingly), to do that stop the array, click on that disk, change the filesystem to a different one, start array, that disk will appear unmountable and can be formatted next to array start stop button, then repeat the process, change the filesystem back to the original, start array, format once more, now you can restore the data.

[rclifton]

Thanks!  And now that I've done this and can see what has happened.  I'll mention in case someone in the future finds this post.  Reformatting the drive I assume essentially zeros out the parity for that drive as Unraid will not attempt to rebuild it after you format it so make sure you have a backup of what's on that drive specifically and not just a full backup.  I have now disabled my parity drive and am copying over the contents of what was on that specific drive now.  Once that is finished I will put the parity drive back in place and run a parity sync.  

 

Thank you for all the help!

[trurl]

2 minutes ago, rclifton said:

Reformatting the drive I assume essentially zeros out the parity for that drive

Format writes an empty filesystem to the disk, and Unraid treats that write operation just as it does any other, by updating parity.

[JonathanM]

6 hours ago, rclifton said:

I have now disabled my parity drive

Why?

[rclifton]

11 minutes ago, jonathanm said:

Why?

Because I will do a parity sync once everything is back in place, But mainly because copying 4TB+ of data without parity copies at about 160MBps and with parity it's more like 70-90MBps..  I've got backups of everything so I really don't see the need to write parity for all that data when I'm going to sync it all after the copy anyway..

[JonathanM]

Ok. The copy speed is a good reason.

 

Parity syncs all the bits on all the drives, regardless of whether or not the specific bits contain any data. That's why parity can't help with corrupted data, it doesn't know the difference between a file and random ones and zeroes from long deleted files. Parity recreates a complete duplicate of the missing drive, corrupt, formatted, or fully intact, parity doesn't care, it's all just meaningless ones and zeros as far as parity is concerned.

https://wiki.unraid.net/Parity