I am preparing to convert my reiserfs drives to xfs and I've been over the recommended procedure for doing this. If the group could indulge me, I would like to show my procedure, adapted from the one on this forum, to see if I have this down correctly. I have the steps outlined in the attached pdf along with my current array configuration.

 

 

 

Converting REISERFS to XFS.pdf

On 5/18/2023 at 11:24 AM, dchamb said:

I am preparing to convert my reiserfs drives to xfs and I've been over the recommended procedure for doing this. If the group could indulge me, I would like to show my procedure, adapted from the one on this forum, to see if I have this down correctly. I have the steps outlined in the attached pdf along with my current array configuration.

 

 

 

Converting REISERFS to XFS.pdf 34.5 kB · 5 downloads

I was hoping someone could validate my understand of this process. Does caching need to be turned off before rsyncing files? Does parity get affected by the format of the disk? i.e. when swapping an reiserfs for an xfs, why doesn't this affect parity?

31 minutes ago, dchamb said:

I was hoping someone could validate my understand of this process. Does caching need to be turned off before rsyncing files? Does parity get affected by the format of the disk? i.e. when swapping an reiserfs for an xfs, why doesn't this affect parity?

Looking at what you propose to do:

 

You need to get the array to un-remember  Parity2.   (Otherwise, it will assume that you are going to do a 'parity swap'.)  See here:

 

https://wiki.unraid.net/Manual/Storage_Management#Removing_parity_disk.28s.29

 

Looking at the remainder of your PDF, I have a comment.  While you are doing this conversion, don't use the server to anything else!   Shutdown Docker.  Don't be writing stuff to that array.   This will rid you of worrying about files that Mover might be moving onto the array.  This would eliminate the need for Step 4 which will move than double the time required for complete conversion. 

 

I don't think you understand what is involved in steps 11 thru 16 of the procedure.  Make a printout of those instructions and go through it very carefully.  Cross out the example disk numbers and put your own disk numbers in place of them. (I made up a big table/(spreadsheet) for each disk that I converted showing each step for that disk.  As I did each step, I crossed it off.) 

 

One more important thing.  Look at this instruction:

 

 

If you use SSH, you must be using 'screen' session.  (Otherwise, if the SSH session is interrupted, what ever the shell is running will terminate instantly!)  I strongly urge to consider using the console. 

3 hours ago, dchamb said:

Does parity get affected by the format of the disk?

Yes, because formatting is being done to an drive that has been included as a member of the array.  Every write operation to that drive will cause a corresponding write operation to parity to reflect those bytes which were changed on the data drive.  (Rebuilding a disk rewrites EVERY byte on that disk which includes the File Structure which formatting creates.)

 

See here for how Parity1 works:

 

    https://wiki.unraid.net/Manual/Overview#Parity-Protected_Array

 

Parity2 works in a similiar manner but the calculation of the actual parity byte is done with a matrix operation.

 

If you eventually decide to remove that 12TB disk from the array when you are done, parity will have to be rebuilt because the removal of all of those bytes on that 12TB disk have to be accounted for in the parity information. 

 

Edited May 20, 20233 yr by Frank1940

4 hours ago, Frank1940 said:

You need to get the array to un-remember  Parity2.   (Otherwise, it will assume that you are going to do a 'parity swap'.)  See here:

 

https://wiki.unraid.net/Manual/Storage_Management#Removing_parity_disk.28s.29

Thanks for this catch! I will add this to my procedure.

 

5 hours ago, Frank1940 said:

Looking at the remainder of your PDF, I have a comment.  While you are doing this conversion, don't use the server to anything else!   Shutdown Docker.  Don't be writing stuff to that array.   This will rid you of worrying about files that Mover might be moving onto the array.  This would eliminate the need for Step 4 which will move than double the time required for complete conversion. 

 

I don't think you understand what is involved in steps 11 thru 16 of the procedure.  Make a printout of those instructions and go through it very carefully.  Cross out the example disk numbers and put your own disk numbers in place of them. (I made up a big table/(spreadsheet) for each disk that I converted showing each step for that disk.  As I did each step, I crossed it off.) 

I was really hoping to only keep my Plex docker running during the conversion. Can I disable mover during this time?

 

Yes I am aware of the drive numbers needing to change for each iteration but I like your idea of listing them out and crossing them off as I go through it.

5 hours ago, Frank1940 said:

If you use SSH, you must be using 'screen' session.  (Otherwise, if the SSH session is interrupted, what ever the shell is running will terminate instantly!)  I strongly urge to consider using the console. 

I will likely use the console. I am not an unRAID/Linux expert so I'm not sure what screen mode is.

2 hours ago, Frank1940 said:

If you eventually decide to remove that 12TB disk from the array when you are done, parity will have to be rebuilt because the removal of all of those bytes on that 12TB disk have to be accounted for in the parity information. 

The 12TB will need to be removed as I don't have enough slots for another drive and I have a new 22TB to add in to replace the Parity 2 drive.

 

You've been very helpful. I appreciate it!

 

Dale

I found out I need to perform the conversion for one of my disks in RFS format.  This disk has documents/files I don't want to loose, so it's important that I do this accurately.  So I like to check here and confirm if my approach is sound before starting the process.  

 

Current array:

Parity 4TB
Disk1 1TB RFS (99% utilized)
Disk2 4TB XFS  (25% utilized)
Disk3 1TB XFS
Disk4 230GB XFS

Cache 250GB 

 

Future array:
Parity 4TB
Disk1 1TB XFS
Disk2 4TB XFS
Disk3 1TB XFS
Disk4 230GB XFS

 

My plan is to use MC to move/copy files, which seems the the most straightforward and understandable to me of some of the methods I've read.  Should I be OK?

 

I have two ideas to keep all my files and convert the disk1.  Which one is the best/safest approach?

 

Approach 1

Move all my files from disk1 to disk2

Stop the array

Convert the FS

Optional Move files back to disk1

Done

 

Approach 2

Create a folder "temp" on disk2

Copy all my files from disk1 to disk2/temp

Stop the array

Convert the FS

Copy all files from disk2/temp to disk1

Done

 

Question:

I noticed there is folders called /isos and /system on disk1.  Should I move or copy them along with my files?

/system has /docker/docker.img and /libvirt/libvirt.img

 

Question 2:

As I'm a little nervous about doing this, I want to copy just the contents from disk1 to a Windows enabled external USB harddrive to create a secondary backup in case something goes terribly wrong.  I don't think I can do this on a Windows PC via network shares without having to identify each file that resides on disk1.  Is what I want to do even possible and how would I go about doing this?

 

Edited June 10, 20233 yr by dn7172

8 hours ago, dn7172 said:

/system has /docker/docker.img and /libvirt/libvirt.img

Ideally, those files would be on a pool outside the array, such as cache. You don't mention, do you have cache or other pool?

 

If you use approach 2, you will have the "backup" already on disk2 until you are satisfied with the results. Copy is always safer than move since nothing gets deleted from source.

14 hours ago, trurl said:

Ideally, those files would be on a pool outside the array, such as cache. You don't mention, do you have cache or other pool?

 

If you use approach 2, you will have the "backup" already on disk2 until you are satisfied with the results. Copy is always safer than move since nothing gets deleted from source.

 

Sorry I missed mentioned the one cache drive (updated my previous post). What should I do?

 

I have a follow up question to approach 2, please forgive my limited knowledge and terminology.  I believe my files under folders are distributed across the array (e.g. Documents folder resides on disk1, disk2 and disk3 with various files across those disks).  After I copy the files to "backup" to disk2 and convert disk1 to XFS, do I simply just use the MC copy command to copy back those files?  Will unraid know and need to recreate the reference points on disk1 to the files?  
 

On 6/10/2023 at 12:02 AM, dn7172 said:

do I simply just use the MC copy command to copy back those files?

yes

I have copied over my files to disk2.  For next step, do I need to do something with the /isos and /system on disk1  OR will it be safe to leave it and go ahead to convert the disk to NFS? 

 

Best,

D

8 hours ago, dn7172 said:

I have copied over my files to disk2.  For next step, do I need to do something with the /isos and /system on disk1  OR will it be safe to leave it and go ahead to convert the disk to NFS? 

 

Best,

D

Disk 1 must be empty of all files that you wish to preserve.    Formatting a disk with Unraid does the same thing as formatting does with any OS.  It creates an empty file system!  If there were files still on the disk when the formatting was started they are now gone!

 

EDIT:  Quote from here:

 

       https://legacy.wiki.unraid.net/index.php/File_System_Conversion

 

Quote

It's likely that some of those who started this without planning or care have lost data. Why? Because it's easy to think that when you issue a copy command (doesn't matter which tool you use), that when it finishes, then everything on the drive has been copied, and now you can format the drive and move on to the next drive. But what if something is writing to the drive while you are copying everything off? Every time the copy tool finishes copying a folder, it moves on to the next, and it never comes back! So if new files were added to folders already copied, they won't be seen, and they won't be copied, and they will be lost when you format the drive! And you won't even know it, until sometime later when you wonder where that file went.

 

Edited June 15, 20233 yr by Frank1940

11 hours ago, dn7172 said:

/system on disk1

Did you get /system copied to cache where it belongs? Disable Docker and VM Manager in Settings until you are done with all this.

9 hours ago, trurl said:

Did you get /system copied to cache where it belongs? Disable Docker and VM Manager in Settings until you are done with all this.

 

Thanks for letting me know.  I don't have any docker apps or VMs, i.e. my system is pretty bare and strictly used as a file repository but I have disabled them in settings as instructed.

 

I just checked and saw there is a /system folder on my cache disk too. For some background, my unraid system originally started without a cache disk.  I think it was this way for many years, but last year I added a cache disk.  Should I really copy disk1/system/ to cache disk or is it better to save it elsewhere and copy back?

 

 

Edited June 15, 20233 yr by dn7172

If you don't intend to use Docker or VM Manager, you don't need a system share

29 minutes ago, trurl said:

If you don't intend to use Docker or VM Manager, you don't need a system share

 

There might be a reason to use either in the future.  As of now, I haven't done more with it because my system specs are quite outdated and I plan to upgrade it. 

Edited June 16, 20233 yr by dn7172

When you enable Docker and VM Manager, the specified paths in their configurations would be created anyway. Default shares:

https://docs.unraid.net/unraid-os/manual/shares#default-shares

 

7 hours ago, trurl said:

When you enable Docker and VM Manager, the specified paths in their configurations would be created anyway. Default shares:

https://docs.unraid.net/unraid-os/manual/shares#default-shares

 

 

Appreciates the info on default shares, I have further understanding of what files would be in here.

I checked the /system folder on disk1, and discovered it's no longer there so I guess I'm ok 😕 

The /isos folder is empty and that would make sense since I never used VM.

 

I think I've covered all my bases, so pretty much ready to convert the filesystem.  But first I am running parity check now as I noticed error counts.

 

 

While the parity check is running, I was thought I should verify the MC copy was ok.  I have double checked total size of directories and count of files - I'm 99% satisfied.  For those also learning like me, I'm sharing the resources I used https://linuxhandbook.com/find-directory-size-du-command/ and https://linuxize.com/post/count-files-in-directory-on-linux/

re: 99% satisfied is because there's a discrepancy for one of my folders which the original displayed 92GB vs copied 91GB.  However the file count is the same. I'm not sure why the size difference.

Edited June 17, 20233 yr by dn7172

My server has been running for 15+ years and all data drives are on ReiserFS.  It's time to convert them to XFS, but I wanted to check one thing first.  I have a new 14TB drive which I need to add to the array and utilize to convert my 10TB drives to XFS.   I'd like to pre-clear the 14TB, but I don't have any free SATA ports. I have moved all, or almost all, data off of a 2TB drive. So does this make sense:

 

1) Remove 100% of files from 2TB drive

2) Clear drive and shrink array using Unraid/Spaceinvader instructions

3) Add 14TB as unassigned drive and pre-clear (XFS format)

4) Add 14TB into array

5) Start process of moving data to 14TB, converting drive, moving data back to original drive

 

I think the other option would be to upgrade the 2TB to the 14TB, but then I wouldn't be able to pre-clear the 14TB (or I'd have to stop the array for an extended period while pre-clear runs).

 

Any advice would be greatly appreciated.

 

Parity - 14TBx2

Data - 17 drives from 2TB to 10TB (all ReiserFS) 98TB total storage

Cache - 1TB SSD (btrfs)

System cache - 480GB SSD (xfs)

 

 

 

tower-diagnostics-20240805-0946.zip

30 minutes ago, dpc2 said:

My server has been running for 15+ years and all data drives are on ReiserFS.  It's time to convert them to XFS, but I wanted to check one thing first.  I have a new 14TB drive which I need to add to the array and utilize to convert my 10TB drives to XFS.   I'd like to pre-clear the 14TB, but I don't have any free SATA ports. I have moved all, or almost all, data off of a 2TB drive. So does this make sense:

 

1) Remove 100% of files from 2TB drive

2) Clear drive and shrink array using Unraid/Spaceinvader instructions

3) Add 14TB as unassigned drive and pre-clear (XFS format)

4) Add 14TB into array

5) Start process of moving data to 14TB, converting drive, moving data back to original drive

 

I think the other option would be to upgrade the 2TB to the 14TB, but then I wouldn't be able to pre-clear the 14TB (or I'd have to stop the array for an extended period while pre-clear runs).

 

Any advice would be greatly appreciated.

 

Parity - 14TBx2

Data - 17 drives from 2TB to 10TB (all ReiserFS) 98TB total storage

Cache - 1TB SSD (btrfs)

System cache - 480GB SSD (xfs)

 

 

 

tower-diagnostics-20240805-0946.zip 344.87 kB · 1 download

Pre-clear using a USB3.0 port before adding it to your array. Then slowly convert over the other drives with unbalanced. 

31 minutes ago, dpc2 said:

5) Start process of moving data to 14TB, converting drive, moving data back to original drive

 

 

First don't format the drive in UD's preclear.  That will erase the preclear flag.  Instead add the drive to the array.  Unraid should recognize it as precleared and format it.  Formatting will take about two-three minutes. 

 

What I would do is to copy the files from the next disk to be converted (One of the present 10TB disks) to that newly formatted XFS 14TB disk.  Then format the newly emptied 10TB as XFS  And copy the data from the next disk.  (You have to make sure that you convert the largest disks first...)

 

Many years ago, there was a full set of instructions written to do this conversion in a minimum amount of time.  When the new documentation was prepared, it was deleted and lost forever as near as I can determine.   I do have a copy of the this original procedure and the notes that I made about what I thought should be included in the new documentation--- It never made it.  IF you want a copy, send me a personal message.

5 hours ago, foo_fighter said:

Pre-clear using a USB3.0 port before adding it to your array. Then slowly convert over the other drives with unbalanced. 

That makes total sense.  For some reason I never think about doing operations via USB ports.

It turns out that I have a Marked-up PDF file of the original "Mirror Method while Preserving Parity".  I have attached a copy of it.

 

Now let me state that parity is only preserved for the Parity disk.  It is not preserved for the Parity 2 disk!  So if you have a dual Parity setup, you will have only single parity protection during the procedure.  You will have to rebuild parity after all drive are converted to restore dual parity protection. I would suggest that you unassign the Parity 2 disk while doing the conversion and reassign it until you have finished the conversion of all of the reiserfs formatted disks.

 

The procedure  is written for conversion to XFS but you can using any file system that is supported for use on the array.

 

I have not used these instructions in several years.  And they have not been available on the Unraid website for more than eight months.  Read them carefully as they are a bit confusing at first reading. (I woould suggest printing them out on paper so that you have them in hand as you perform each step.) What I did was set up a step-by-step table where I entered the disk # involved for each step of the instruction.  I check off each step of the procedure as I started it.  (It can take a couple of days to the rsync portion for each disk depending on the disk size and how far full it was.   The largest disk I had  at that time was 3TB so it could take up to eight hours for it to finish!)

 

You might want to install the three Unassigned Devices plugins as one them contains tmux which will allow you to use the GUI terminal and be able to keep the Shell session running in the event you lose the connection between the server and the terminal.  (Google  "tmux" for a description and quick sheets to using it.  I have attached a 'cheatsheet' for TMUX that I use. ) 

 

If you have questions, search this thread for your answer...  

FileConversionWIKI(1).pdf

osdc_cheatsheet-tmux-2021.6.25(1).pdf

Edited August 22, 20241 yr by Frank1940

Doing the highlighted step:

File_Conversion_WIKI---pg_7.pdf

Seeking advice on my situation described below ..

 

Goal: Replace Array Disk3 4T (rfs) with a new 14T disk and change FS to xfs.

 

Steps followed so far:
1. Added new 14T disk using "Unassigned Devices" plugin and precleard using "Unassigned Devices Preclear".
2. Stopped the array and added the new 14T disk into the array as "Disk11".
3. Started the array and formated Disk11 as "xfs" during mount.
4. Used "Unbalanced" plugin to "copy" all data from Disk3(rfs) to Disk11(xfs).
5. Stopped the array, changed Disk3 FS to "xfs" and started with reformat.
6. Used "Unbalanced" plugin again to "copy" all data back into Disk3 (now xfs) from Disk11(xfs).

 

At this point both Disk3 (4T) and Disk11 (14T) are in the array having the same FS and both containing identical data.

Now I want to remove Disk3(4T) from the array completely and replace that slot with Disk11 (14T).
What's the best and safest way to do this?

2 hours ago, kolla said:

What's the best and safest way to do this?

You can do a new config and resync parity without that disk.

 

P.S. isn't Unbalanced used to move the data from on disk to another, not copy?