Migrate from XFS to BTRFS

[shaunmccloud]

I would like to migrate from XFS to BTRFS as my wife and I store a lot of pictures of our kids on our server and I do not want them to get corrupted.  Is the best way to do this to stop the array, copy the data from each disk to the others in the array, format it as BTRFS in the CLI, copy the data back then restart the array for each disk.  Or does the format have to be done in the GUI?

[BRiT]

Switching filesystems is not a replacement for proper backups. You should have a backup plan first.

[JorgeB]

Basically you have to empty your largest disk, format it, then move data from another disk to that one, format the now empty disk, rinse and repeat.

 

If you need, more details about fs conversion here:

https://forums.unraid.net/topic/54769-format-xfs-on-replacement-drive-convert-from-rfs-to-xfs/

 

[trurl]

9 minutes ago, shaunmccloud said:

does the format have to be done in the GUI?

There are definitely ways you could do it wrong in the CLI so the GUI would be the right way to get it right.

 

12 minutes ago, shaunmccloud said:

I store a lot of pictures of our kids on our server and I do not want them to get corrupted.

Lots of ways to lose files that have nothing to do with corruption or disk failure, including simple user error.

 

2 minutes ago, BRiT said:

Switching filesystems is not a replacement for proper backups. You should have a backup plan first.

 

You don't have to have a backup of everything, but you absolutely must have another copy of anything important and irreplaceable, such as

13 minutes ago, shaunmccloud said:

pictures of our kids

 

[shaunmccloud]

9 minutes ago, BRiT said:

Switching filesystems is not a replacement for proper backups. You should have a backup plan first.

Already doing backups, but the bitrot will be backed up too.

7 minutes ago, johnnie.black said:

Basically you have to empty your largest disk, format it, then move data from another disk to that one, format the now empty disk, rinse and repeat.

 

If you need, more details about fs conversion here:

https://forums.unraid.net/topic/54769-format-xfs-on-replacement-drive-convert-from-rfs-to-xfs/

 

Copying via the CLI is ok right?  Probably using rsync since that way I can easily get changes.

3 minutes ago, trurl said:

There are definitely ways you could do it wrong in the CLI so the GUI would be the right way to get it right.

 

Lots of ways to lose files that have nothing to do with corruption or disk failure, including simple user error.

 

 

You don't have to have a backup of everything, but you absolutely must have another copy of anything important and irreplaceable, such as

 

And how do I force a reformat in the GUI on a drive that already has data on it?  That is my issue.

[JorgeB]

1 minute ago, shaunmccloud said:

Copying via the CLI is ok right?  Probably using rsync since that way I can easily get changes.

Yes, rsync is also what I always used, first when converting from reiser to xfs and then from xfs to btrfs.

 

BTW, currently and by default Unraid formats all disks with single metadata, unlike btrfs default which is dup metadata for hard disks, this should change on the next release but for now recommend you do this for each disk after formatting:

 

btrfs balance start -mconvert=dup /mnt/diskX

 

 

[trurl]

Just now, shaunmccloud said:

And how do I force a reformat in the GUI on a drive that already has data on it?  That is my issue.

The link johnnie gave has everything you need, but short answer: With the array stopped, click on the disk you want to reformat and change its filesystem. Obviously reformatting will lose any data on the disk so be sure you have copied it elsewhere.

[shaunmccloud]

Ok, going to take a while to copy data for me.  I may have to shut down a lot of stuff to get it to work which will make the wife unhappy.

[Frank1940]

In my opinion, 'Bitrot' does not exist on modern hard drives.  The error checking and correction codes catch "it".  Most of the time, they fix it.  When they can't, the drive returns an read error code.  By the way, read errors are a rather common on modern hard disks and most of them are simply corrected on the fly.  You never even know that they happened.  

 

You should really be more concerned about the physical safety of your server.  Have you considered the possibility of fire, floods, hurricanes, theft, lightning strikes, etc.  You are far more likely to lose your data because of one of those...

[shaunmccloud]

Just now, Frank1940 said:

In my opinion, 'Bitrot' does not exist on modern hard drives.  The error checking and correction codes catch "it".  Most of the time, they fix it.  When they can't, the drive returns an read error code.  By the way, read errors are a rather common on modern hard disks and most of them are simply corrected on the fly.  You never even know that they happened.  

 

You should really be more concerned about the physical safety of your server.  Have you considered the possibility of fire, floods, hurricanes, theft, lightning strikes, etc.  You are far more likely to lose your data because of one of those...

Actually, I have.  I am just paranoid because I did manage to lose all our data once.  Including the pictures of my wife holding our oldest niece in the hospital right after she was born.

 

Is switching from XFS to BTRFS required?  Nope.  But again I am paranoid so I want to in order to do everything I can to protect my data.

[JorgeB]

IMO "bitrot" is not a big concern, but having checksums of all your data, or using a checksum enable fs like btrfs, can be invaluable when something happens that makes you doubt the data integrity, like for example errors on multiple disks during a rebuild.

 

 

[Frank1940]

@shaunmccloud, You should also inquire about the status of the File System Correcting/Fixing  Tools for both btrfs and XFS.   You are far more likely to have these types of problems. 

[Frank1940]

One more thing to consider is the high probability that the "bitrot" actually occurs in RAM prior or during the transfer/from to the hard disk or during a copy operation.  ECC RAM should help here.  

[JonathanM]

I would think twice about doing this conversion. In my experience, BTRFS, while it has a very desirable feature set, is more "brittle" than XFS.

 

If you have a rock solid server with ECC that never glitches, with a high quality UPS properly configured, then ok.

 

If you are running on consumer grade stuff with no UPS, then I'd stick with XFS.

 

This is based on personal and anecdotal evidence from here on the unraid forums.

[JorgeB]

3 minutes ago, Frank1940 said:

ECC RAM should help here.  

Yes, though ECC isn't required, it should be used by anyone that cares about data integrity.

[shaunmccloud]

1 hour ago, jonathanm said:

I would think twice about doing this conversion. In my experience, BTRFS, while it has a very desirable feature set, is more "brittle" than XFS.

 

If you have a rock solid server with ECC that never glitches, with a high quality UPS properly configured, then ok.

 

If you are running on consumer grade stuff with no UPS, then I'd stick with XFS.

 

This is based on personal and anecdotal evidence from here on the unraid forums.

No ECC RAM but I do have a good UPS.

[JorgeB]

Just be aware that you're much more likely to get corruption from non ECC RAM that from any kind of "bitrot", and btrfs can't help if the corruption happens in RAM while writing the data the first time, though it can help if different data is returned on subsequent reads.