convert reiserfs to xfs: parity check

[pfp]

I'm currently converting 8 disks from reiserfs to xfs and this process is going along just fine.

 

I move all the data off a drive reiserfs drive to the other drives in the array.  I've done this with midnight commander and with unbalance.

Once all the data is moved I make sure any folders that didn't get deleted are deleted.

I stop the array and change the disk to xfs

I start the array and format the drive.

Then I run a parity check writing corrections to parity

 

To my surprise there are no parity corrections - I don't understand this.   How could there not be any parity errors?

[JonathanM]

3 minutes ago, pfp said:

To my surprise there are no parity corrections - I don't understand this.   How could there not be any parity errors?

Parity doesn't have any files or filesystems in it. It ONLY emulates the total drive, no matter what the contents. It cannot recover individual files.

http://lime-technology.com/wiki/index.php/FAQ#How_does_parity_work.3F

[Frank1940]

Because parity is being updated as you performed each every step along the way.  Example, when a a file is copied from the drive, parity was updated.  When the file was the deleted from the drive, parity was updated.  When you formatted that drive, parity was updated.  So there should have been no parity corrections since parity was always being updated so that it would always be correct.  If you had had a correction during your check THAT would have indicated a problem!!!

[pfp]

18 minutes ago, Frank1940 said:

Because parity is being updated as you performed each every step along the way.  Example, when a a file is copied from the drive, parity was updated.  When the file was the deleted from the drive, parity was updated.  When you formatted that drive, parity was updated.  So there should have been no parity corrections since parity was always being updated so that it would always be correct.  If you had had a correction during your check THAT would have indicated a problem!!!

 

When data is deleted is unraid setting all the bits that held that data to 0 then?

[Frank1940]

It does not zero the actual  data bits when a file is deleted, the OS will simply set a bit (or byte) in the file allocation area that the file has been deleted.  This also informs the OS that the disk space is now available to overwrite with a new file when required.  All of the data is still there until a new file is written.  (That is why undelete programs can often recover a file that has been accidentally deleted.)  If a disk is in the array and has bytes that are not all zeros, parity 'knows' about them and if the disk is rebuilt, it will be rebuilt using those bytes.  If another disk has too be rebuilt, the contents of those 'garbage' bytes are used when Rebuilt operation is calculating what the data has to be written on the replacement disk in order to maintain parity. 

 

I am not completely sure why unRAID zeros out a new disk when it is added to the array unless it is to make sure that a write operation can successfully write to every byte on the disk which provides some assurance that the disk is most likely good.  unRAID could just as easily been designed not to zero a new disk but rather to rebuilt parity using whatever data was on that new disk.  However, if it is found (an unlikely event) that some part(s) of the new disk can't be written to then parity is now hosed up since parity has been updated to reflect that stuff that could be read from that disk up to that point and things are now a nightmare from data recovery standpoint.

 

When a new disk that has been zeroed (whether by unRAID or a preclear operation)  is added to the array, unRAID knows that the disk contains all zeros and that parity will not be changed if it were to actually be rebuilt so it does not do it.  Instead, it now adds the disk to array

(now all zeros) and formats the disk with a file system.  This formatting operation is actually a special kind of data file and parity will treat the writing of that information in the same way as it does the write of any file--- it updates parity to reflect the data being written.  

 

So basically, once any disk becomes  a part of the array, any write operation to it will require that parity be updated to reflect that operation.

 

Hope this helps.

Edited May 18, 2017 by Frank1940

[ken-ji]

56 minutes ago, Frank1940 said:

I am not completely sure why unRAID zeros out a new disk when it is added to the array unless it is to make sure that a write operation can successfully write to every byte on the disk which provides some assurance that the disk is most likely good.  unRAID could just as easily been designed not to zero a new disk but rather to rebuilt parity using whatever data was on that new disk.  However, if it is found (an unlikely event) that some part(s) of the new disk can't be written to then parity is now hosed up since parity has been updated to reflect that stuff that could be read from that disk up to that point and things are now a nightmare from data recovery standpoint.

 

Actually, unRAID zeroes a new disk so that it doesn't have to rebuild parity.

The mathematics for computing parity is a simple as count the bits for a specific address of all data drives and if odd write 1 in the parity drive.

So if you are adding a zero, then parity is still good. but if you are adding some random data, parity needs to be rechecked, thus waking up all the drives and going through all of them.

I suppose they could have just read the data on the new drive and update parity, as if new drive data was written and parity updated, but I'm sure zeroing a new drive is a simpler decision and step to take.

[Frank1940]

1 minute ago, ken-ji said:

I suppose they could have just read the data on the new drive and update parity, as if new drive data was written and parity updated, but I'm sure zeroing a new drive is a simpler decision and step to take.

 

Either method requires about the same amount of time (At least today with modern processors being so much faster).  But, as I think more about it, the zeroing(clearing)  of the drive most probably takes place before the drive is actually becomes part of the array.  So if it is found defective, you would right back where you started from and the original array is still protected by a valid parity.  Once cleared (or the preclear signature found), the disk is added logically to the array and from that point on is protected by parity (still valid) and will be formatted with the desired file system.  Thus completing the process.

 

It was a design decision made back when unRAID was first developed and, today, arguments could be made for doing it either way and neither method might stand out as a better choice.  Back then, I am not so sure how and which factors weighted the decision to the direction that was chosen. 

[itimpi]

The big advantage of zeroing the disk before adding it as an active array drive is that you remain protected the whole time.   If parity was rebuilt instead then you would be unprotected until the parity rebuild finished.   There is also the side-effect that at you do a level of confidence check on the new drive before using it to store data.

[gubbgnutten]

Zeroing a new drive before it is actually added keeps the parity correct at all times and is extremely straightforward to implement.

 

Adding a drive by reading from it and updating the parity on the other hand, yikes. Not only would the parity be invalid (or at best require special handling) during the update, it would also in virtually all cases be way slower than filling a drive outside of the array with zeros.

[SSD]

12 hours ago, pfp said:

I'm currently converting 8 disks from reiserfs to xfs and this process is going along just fine.

 

I move all the data off a drive reiserfs drive to the other drives in the array.  I've done this with midnight commander and with unbalance.

Once all the data is moved I make sure any folders that didn't get deleted are deleted.

I stop the array and change the disk to xfs

I start the array and format the drive.

Then I run a parity check writing corrections to parity

 

To my surprise there are no parity corrections - I don't understand this.   How could there not be any parity errors?

 

Good idea to get off of RFS!

 

There is no need to run a parity check (step 5). As has been explained thoroughly above, nothing you are doing should throw off parity. It is a good idea to verify that the copying / moving of data (step 1) was successful. I computed and compared MD5s, but there are multiple methods.

 

In step 1, you mention you "move all of the data".

And in step 2 "Once all the data is moved I make sure any folders that didn't get deleted are deleted"

 

Moving involves two steps - copying and then deleting. There is no need to delete the files from the RFS disk. RFS is slow to delete especially large files. So instead of "moving" the data, you could simply copy it. And as for step 2, that is completely unnecessary. Anything on the disk will be gone after the format to xfs (step 4).

 

If "moving" is used to keep track of what has been moved vs what has not, that is a perfectly valid use case. That might happen if you are not monolithically copying all of the data to the same XFS disk, but reorganizing and parceling it out to several different disk. It might just add some time to the process, but by moving the files you know which ones remain.

 

Good luck and hope all goes smoothly.

[pfp]

This now makes sense.  Thank you

[jbrodriguez]

If a disk fails, you swap in a new disk and unRAID will rebuild it from parity right ?

 

I can't remember if you can choose which filesystem the new disk will have, but I think you can.

 

An alternative could be to somehow fake a disk failure, preclear it, insert it into the array as xfs and let unRAID rebuild it.

 

Not sure if it will let you "re"-insert the disk with the same serial though.

 

Another downside is the data in that disk only exists "virtually", but it's the same concept as if your hard disk had died.

 

Any thoughts ?

[JorgeB]

32 minutes ago, jbrodriguez said:

If a disk fails, you swap in a new disk and unRAID will rebuild it from parity right ?

 

I can't remember if you can choose which filesystem the new disk will have, but I think you can.

 

An alternative could be to somehow fake a disk failure, preclear it, insert it into the array as xfs and let unRAID rebuild it.

 

Not sure if it will let you "re"-insert the disk with the same serial though.

 

Another downside is the data in that disk only exists "virtually", but it's the same concept as if your hard disk had died.

 

Any thoughts ?

 

You can't rebuild a disk with a different filesystem, it's always rebuilt with the same filesystem as the old disk.

[SSD]

3 hours ago, jbrodriguez said:

If a disk fails, you swap in a new disk and unRAID will rebuild it from parity right ?

 

I can't remember if you can choose which filesystem the new disk will have, but I think you can.

 

An alternative could be to somehow fake a disk failure, preclear it, insert it into the array as xfs and let unRAID rebuild it.

 

Not sure if it will let you "re"-insert the disk with the same serial though.

 

Another downside is the data in that disk only exists "virtually", but it's the same concept as if your hard disk had died.

 

Any thoughts ?

 

A filesystem, like rfs and xfs, define ways of organizing files on a disk. Different file systems, different organizations, different bytes on the disk. UnRaid adds redundancy so it can regenerate a disk as an exact clone of the original bytes on the disk. How could the clone be in anything but the filesystem format of the original?

 

It's like using a copier to make a copy (clone) of a sheet of paper containing something in English, announcing the copy will be in Russian, and handing it to a Russian person to read! Just like a photocopier has no idea what it is copying, and no ability to translate it, unRaid's parity has no idea what format your data is in and no ability to convert it to something else.

 

[garycase]

On ‎5‎/‎18‎/‎2017 at 6:48 AM, Frank1940 said:

I am not completely sure why unRAID zeros out a new disk when it is added to the array unless it is to make sure that a write operation can successfully write to every byte on the disk which provides some assurance that the disk is most likely good.  unRAID could just as easily been designed not to zero a new disk but rather to rebuilt parity using whatever data was on that new disk.  However, if it is found (an unlikely event) that some part(s) of the new disk can't be written to then parity is now hosed up since parity has been updated to reflect that stuff that could be read from that disk up to that point and things are now a nightmare from data recovery standpoint.

 

You've essentially answered the question in your first sentence with your 2nd sentence .     Note that by zeroing a new disk before adding it, there is NO change to parity ... so the system still has valid parity after the new disk is added.    If UnRAID did as you suggest in the 2nd sentence ("... but rather to rebuild parity using whatever data was on that new disk ...") that would also work fine ... EXCEPT you wouldn't have valid parity during that operation.     You can, by the way, EASILY add a disk in this manner if that's what you'd prefer -- just do a New Config that includes the new disk; then when you Start the array it will do a new parity sync. 

 

[garycase]

On ‎5‎/‎17‎/‎2017 at 9:40 PM, pfp said:

To my surprise there are no parity corrections - I don't understand this.   How could there not be any parity errors?

 

Everything you've done was done within the parity protected array; so parity has always been valid.    It would actually be a surprise (and an indication that you've had errors), if you DID have any parity errors.

 

[trurl]

Just in case anybody reading this doesn't understand the more elaborate responses to this idea:

11 hours ago, jbrodriguez said:

If a disk fails, you swap in a new disk and unRAID will rebuild it from parity right ?

 

I can't remember if you can choose which filesystem the new disk will have, but I think you can.

 

An alternative could be to somehow fake a disk failure, preclear it, insert it into the array as xfs and let unRAID rebuild it.

 

...

 

Any thoughts ?

No. You absolutely cannot do this. And any attempt to change the filesystem of a disk to be rebuilt is likely to lead to data loss since changing the filesystem formats the disk. In fact, people have lost data by formatting a disk that needed to be rebuilt. Don't try it.

[garycase]

12 hours ago, jbrodriguez said:

I can't remember if you can choose which filesystem the new disk will have, but I think you can.

 

An alternative could be to somehow fake a disk failure, preclear it, insert it into the array as xfs and let unRAID rebuild it.

 

 

Whoa !!   Just to emphasize what trurl noted above, you CANNOT do this -- and will absolutely LOSE ALL OF YOUR DATA if you attempt it.   When you change the file system of the disk, UnRAID will reformat it to the new system and this will erase all of the data ... even if it's a red-balled (failed) disk (it will be emulated using all the other disks in the array).    Then when you do a rebuild with a new disk, the newly rebuilt disk will match the newly formatted BLANK disk you just created by changing the file system.

 

[jbrodriguez]

oh I see, will not do that then