Re: Format XFS on replacement drive / Convert from RFS to XFS (discussion only)

[dchamb]

I am preparing to convert my reiserfs drives to xfs and I've been over the recommended procedure for doing this. If the group could indulge me, I would like to show my procedure, adapted from the one on this forum, to see if I have this down correctly. I have the steps outlined in the attached pdf along with my current array configuration.

 

 

 

Converting REISERFS to XFS.pdf

[dchamb]

On 5/18/2023 at 11:24 AM, dchamb said:

I am preparing to convert my reiserfs drives to xfs and I've been over the recommended procedure for doing this. If the group could indulge me, I would like to show my procedure, adapted from the one on this forum, to see if I have this down correctly. I have the steps outlined in the attached pdf along with my current array configuration.

 

 

 

Converting REISERFS to XFS.pdf 34.5 kB · 5 downloads

I was hoping someone could validate my understand of this process. Does caching need to be turned off before rsyncing files? Does parity get affected by the format of the disk? i.e. when swapping an reiserfs for an xfs, why doesn't this affect parity?

[Frank1940]

31 minutes ago, dchamb said:

I was hoping someone could validate my understand of this process. Does caching need to be turned off before rsyncing files? Does parity get affected by the format of the disk? i.e. when swapping an reiserfs for an xfs, why doesn't this affect parity?

Looking at what you propose to do:

 

You need to get the array to un-remember  Parity2.   (Otherwise, it will assume that you are going to do a 'parity swap'.)  See here:

 

https://wiki.unraid.net/Manual/Storage_Management#Removing_parity_disk.28s.29

 

Looking at the remainder of your PDF, I have a comment.  While you are doing this conversion, don't use the server to anything else!   Shutdown Docker.  Don't be writing stuff to that array.   This will rid you of worrying about files that Mover might be moving onto the array.  This would eliminate the need for Step 4 which will move than double the time required for complete conversion. 

 

I don't think you understand what is involved in steps 11 thru 16 of the procedure.  Make a printout of those instructions and go through it very carefully.  Cross out the example disk numbers and put your own disk numbers in place of them. (I made up a big table/(spreadsheet) for each disk that I converted showing each step for that disk.  As I did each step, I crossed it off.) 

 

One more important thing.  Look at this instruction:

 

 

If you use SSH, you must be using 'screen' session.  (Otherwise, if the SSH session is interrupted, what ever the shell is running will terminate instantly!)  I strongly urge to consider using the console. 

[Frank1940]

3 hours ago, dchamb said:

Does parity get affected by the format of the disk?

Yes, because formatting is being done to an drive that has been included as a member of the array.  Every write operation to that drive will cause a corresponding write operation to parity to reflect those bytes which were changed on the data drive.  (Rebuilding a disk rewrites EVERY byte on that disk which includes the File Structure which formatting creates.)

 

See here for how Parity1 works:

 

    https://wiki.unraid.net/Manual/Overview#Parity-Protected_Array

 

Parity2 works in a similiar manner but the calculation of the actual parity byte is done with a matrix operation.

 

If you eventually decide to remove that 12TB disk from the array when you are done, parity will have to be rebuilt because the removal of all of those bytes on that 12TB disk have to be accounted for in the parity information. 

 

Edited May 20, 2023 by Frank1940

[dchamb]

4 hours ago, Frank1940 said:

You need to get the array to un-remember  Parity2.   (Otherwise, it will assume that you are going to do a 'parity swap'.)  See here:

 

https://wiki.unraid.net/Manual/Storage_Management#Removing_parity_disk.28s.29

Thanks for this catch! I will add this to my procedure.

 

5 hours ago, Frank1940 said:

Looking at the remainder of your PDF, I have a comment.  While you are doing this conversion, don't use the server to anything else!   Shutdown Docker.  Don't be writing stuff to that array.   This will rid you of worrying about files that Mover might be moving onto the array.  This would eliminate the need for Step 4 which will move than double the time required for complete conversion. 

 

I don't think you understand what is involved in steps 11 thru 16 of the procedure.  Make a printout of those instructions and go through it very carefully.  Cross out the example disk numbers and put your own disk numbers in place of them. (I made up a big table/(spreadsheet) for each disk that I converted showing each step for that disk.  As I did each step, I crossed it off.) 

I was really hoping to only keep my Plex docker running during the conversion. Can I disable mover during this time?

 

Yes I am aware of the drive numbers needing to change for each iteration but I like your idea of listing them out and crossing them off as I go through it.

5 hours ago, Frank1940 said:

If you use SSH, you must be using 'screen' session.  (Otherwise, if the SSH session is interrupted, what ever the shell is running will terminate instantly!)  I strongly urge to consider using the console. 

I will likely use the console. I am not an unRAID/Linux expert so I'm not sure what screen mode is.

2 hours ago, Frank1940 said:

If you eventually decide to remove that 12TB disk from the array when you are done, parity will have to be rebuilt because the removal of all of those bytes on that 12TB disk have to be accounted for in the parity information. 

The 12TB will need to be removed as I don't have enough slots for another drive and I have a new 22TB to add in to replace the Parity 2 drive.

 

You've been very helpful. I appreciate it!

 

Dale

[dn7172]

I found out I need to perform the conversion for one of my disks in RFS format.  This disk has documents/files I don't want to loose, so it's important that I do this accurately.  So I like to check here and confirm if my approach is sound before starting the process.  

 

Current array:

Parity 4TB
Disk1 1TB RFS (99% utilized)
Disk2 4TB XFS  (25% utilized)
Disk3 1TB XFS
Disk4 230GB XFS

Cache 250GB 

 

Future array:
Parity 4TB
Disk1 1TB XFS
Disk2 4TB XFS
Disk3 1TB XFS
Disk4 230GB XFS

 

My plan is to use MC to move/copy files, which seems the the most straightforward and understandable to me of some of the methods I've read.  Should I be OK?

 

I have two ideas to keep all my files and convert the disk1.  Which one is the best/safest approach?

 

Approach 1

Move all my files from disk1 to disk2

Stop the array

Convert the FS

Optional Move files back to disk1

Done

 

Approach 2

Create a folder "temp" on disk2

Copy all my files from disk1 to disk2/temp

Stop the array

Convert the FS

Copy all files from disk2/temp to disk1

Done

 

Question:

I noticed there is folders called /isos and /system on disk1.  Should I move or copy them along with my files?

/system has /docker/docker.img and /libvirt/libvirt.img

 

Question 2:

As I'm a little nervous about doing this, I want to copy just the contents from disk1 to a Windows enabled external USB harddrive to create a secondary backup in case something goes terribly wrong.  I don't think I can do this on a Windows PC via network shares without having to identify each file that resides on disk1.  Is what I want to do even possible and how would I go about doing this?

 

Edited June 10, 2023 by dn7172

[trurl]

8 hours ago, dn7172 said:

/system has /docker/docker.img and /libvirt/libvirt.img

Ideally, those files would be on a pool outside the array, such as cache. You don't mention, do you have cache or other pool?

 

If you use approach 2, you will have the "backup" already on disk2 until you are satisfied with the results. Copy is always safer than move since nothing gets deleted from source.

[dn7172]

14 hours ago, trurl said:

Ideally, those files would be on a pool outside the array, such as cache. You don't mention, do you have cache or other pool?

 

If you use approach 2, you will have the "backup" already on disk2 until you are satisfied with the results. Copy is always safer than move since nothing gets deleted from source.

 

Sorry I missed mentioned the one cache drive (updated my previous post). What should I do?

 

I have a follow up question to approach 2, please forgive my limited knowledge and terminology.  I believe my files under folders are distributed across the array (e.g. Documents folder resides on disk1, disk2 and disk3 with various files across those disks).  After I copy the files to "backup" to disk2 and convert disk1 to XFS, do I simply just use the MC copy command to copy back those files?  Will unraid know and need to recreate the reference points on disk1 to the files?  
 

[trurl]

On 6/10/2023 at 12:02 AM, dn7172 said:

do I simply just use the MC copy command to copy back those files?

yes

[dn7172]

I have copied over my files to disk2.  For next step, do I need to do something with the /isos and /system on disk1  OR will it be safe to leave it and go ahead to convert the disk to NFS? 

 

Best,

D

[Frank1940]

8 hours ago, dn7172 said:

I have copied over my files to disk2.  For next step, do I need to do something with the /isos and /system on disk1  OR will it be safe to leave it and go ahead to convert the disk to NFS? 

 

Best,

D

Disk 1 must be empty of all files that you wish to preserve.    Formatting a disk with Unraid does the same thing as formatting does with any OS.  It creates an empty file system!  If there were files still on the disk when the formatting was started they are now gone!

 

EDIT:  Quote from here:

 

       https://legacy.wiki.unraid.net/index.php/File_System_Conversion

 

Quote

It's likely that some of those who started this without planning or care have lost data. Why? Because it's easy to think that when you issue a copy command (doesn't matter which tool you use), that when it finishes, then everything on the drive has been copied, and now you can format the drive and move on to the next drive. But what if something is writing to the drive while you are copying everything off? Every time the copy tool finishes copying a folder, it moves on to the next, and it never comes back! So if new files were added to folders already copied, they won't be seen, and they won't be copied, and they will be lost when you format the drive! And you won't even know it, until sometime later when you wonder where that file went.

 

Edited June 15, 2023 by Frank1940

[trurl]

11 hours ago, dn7172 said:

/system on disk1

Did you get /system copied to cache where it belongs? Disable Docker and VM Manager in Settings until you are done with all this.

[dn7172]

9 hours ago, trurl said:

Did you get /system copied to cache where it belongs? Disable Docker and VM Manager in Settings until you are done with all this.

 

Thanks for letting me know.  I don't have any docker apps or VMs, i.e. my system is pretty bare and strictly used as a file repository but I have disabled them in settings as instructed.

 

I just checked and saw there is a /system folder on my cache disk too. For some background, my unraid system originally started without a cache disk.  I think it was this way for many years, but last year I added a cache disk.  Should I really copy disk1/system/ to cache disk or is it better to save it elsewhere and copy back?

 

 

Edited June 15, 2023 by dn7172

[trurl]

If you don't intend to use Docker or VM Manager, you don't need a system share

[dn7172]

29 minutes ago, trurl said:

If you don't intend to use Docker or VM Manager, you don't need a system share

 

There might be a reason to use either in the future.  As of now, I haven't done more with it because my system specs are quite outdated and I plan to upgrade it. 

Edited June 16, 2023 by dn7172

[trurl]

When you enable Docker and VM Manager, the specified paths in their configurations would be created anyway. Default shares:

https://docs.unraid.net/unraid-os/manual/shares#default-shares

 

[dn7172]

7 hours ago, trurl said:

When you enable Docker and VM Manager, the specified paths in their configurations would be created anyway. Default shares:

https://docs.unraid.net/unraid-os/manual/shares#default-shares

 

 

Appreciates the info on default shares, I have further understanding of what files would be in here.

I checked the /system folder on disk1, and discovered it's no longer there so I guess I'm ok 😕 

The /isos folder is empty and that would make sense since I never used VM.

 

I think I've covered all my bases, so pretty much ready to convert the filesystem.  But first I am running parity check now as I noticed error counts.

 

 

While the parity check is running, I was thought I should verify the MC copy was ok.  I have double checked total size of directories and count of files - I'm 99% satisfied.  For those also learning like me, I'm sharing the resources I used https://linuxhandbook.com/find-directory-size-du-command/ and https://linuxize.com/post/count-files-in-directory-on-linux/

re: 99% satisfied is because there's a discrepancy for one of my folders which the original displayed 92GB vs copied 91GB.  However the file count is the same. I'm not sure why the size difference.

Edited June 17, 2023 by dn7172