Changing File System

[senrab]

Hey guys, 

 

I am in the middle of changing my parity drive and replacing one of data drives. I thought now would be a good time to change my files system from reiser to XFS for each drive. 

 

So far, I have swapped out the 3TB partiy drive for a 4TB. All is well. 

 

I have the 3TB sitting as unmounted (It will eventually replace one of the disks as its starting to give some SMART errors). Do I want to click 'Mount' then format it to XFS and then pick a disk in the array and copy the data and put it on the unmounted drive? Then format the drive in the array to XFS and put the data back? (and so on for the rest of the drives). 

 

I am just wondering the best procedure for doing this? 

 

Thanks!

[JonathanM]

Stick thread just above your current post.

[senrab]

I've read through that a few times and before I make an attempt I'd like to clearify a couple of questions I have. 

 

1) Currently I have my former 3TB parity drive sitting as an 'Unassigned' disk. (I also have a brand new 4TB drive that will be added to the array at some point) 

    -Do I want to click format and format that as XFS first then add it to my array? Or do I just add it to the array and it will allow me to select which file system       to format the drive to?  

 

2) Should I replace the 2TB disk that is giving CRC errors before I do any of this? Everything is working fine now so I assume I would be ok. 

 

Should I even worry about making this change? I figured I would do it now before I start adding more drives and accumulating more data which would likely be more work in the future. 

 

Thanks for the help!

[JonathanM]

How much total free space do you have right now on the array without any other drive changes?

[Frank1940]

32 minutes ago, senrab said:

 

2) Should I replace the 2TB disk that is giving CRC errors before I do any of this? Everything is working fine now so I assume I would be ok. 

CRC errors are almost never caused by the HD itself.  It is a communication issue between the HD and the buss (forget which one) on the MB.  It is usually caused by a SATA data cable or its connection.  Another point is that is not really a critical failure.  The data will be retransmitted until it is received correctly.  Of course, the side effort is that the data transfer rate will plummet during these corrections if there are very many errors occurring close together.   

Edited August 9, 2019 by Frank1940

[senrab]

4 hours ago, jonathanm said:

How much total free space do you have right now on the array without any other drive changes?

I have about 100gigs. 

 

I have a 3TB and a 4TB that I want to add to the array. I figured now would be a good time to change the file system if it is something that is recommended. 

[senrab]

3 hours ago, Frank1940 said:

CRC errors are almost never caused by the HD itself.  It is a communication issue between the HD and the buss (forget which one) on the MB.  It is usually caused by a SATA data cable or its connection.  Another point is that is not really a critical failure.  The data will be retransmitted until it is received correctly.  Of course, the side effort is that the data transfer rate will plummet during these corrections if there are very many errors occurring close together.   

Great thanks for the info! I will try changing the sata cable and see if that corrects the CRC errors. 

[JonathanM]

Ok. With that little free space, you definitely need to add a drive.

Is the new 4TB tested?

[senrab]

The 4TB is brand new. I haven't put it in the server yet. The 3TB that is sitting as unassigned is the former parity drive. I figured I would start with that one because the drives in my array are 3TB(2 of them) and a 2TB. 

[JonathanM]

Brand new != tested. I've seen several cases of brand new drives failing shortly after being put into service, which is the last thing you want to happen with unraid. The drive could have been used as a football by the shipping company at some point, and you wouldn't know it.

 

Do you have the physical connections to attach both the 4TB and 3TB along with the rest of the drives? If so, you can get rolling with the filesystem conversion using the 3TB while testing the 4TB at the same time.

[senrab]

I know that new doesn't mean tested. Sorry, I was just stating that it wasn't in the server yet. I wasn't planning on putting it in the server(then testing it) until I was at that point.

 

I am just a little confused on the order this all should happen. 

 

For the 3TB drive that I have sitting as unassigned, does that get added to the array as a new drive to the array? I am assuming that the drives in the array can be different file systems. 

[JonathanM]

3 minutes ago, senrab said:

For the 3TB drive that I have sitting as unassigned, does that get added to the array as a new drive to the array? I am assuming that the drives in the array can be different file systems. 

I was trying to get a feel for what the state of things were so I could recommend the most efficient order of operations.

 

Yes, add the 3TB as an additional array member and let unraid clear it. Be sure it is set to the format type you wish to use. Verify that only that drive is listed as unmountable, and select the option to format it. Copy (not move)* the entire content of one of the ReiserFS drives to the newly formatted drive. Verify the copy was complete and accurate, change the requested format type of the source drive, format it, lather rinse repeat.

 

*Moving the files is much slower than a copy, and eliminates the option for a do over if things go south.

[itimpi]

11 minutes ago, senrab said:

I am assuming that the drives in the array can be different file systems. 

There is no requirement that disks in the array have the same file system.    Parity works as the physical sector level and does not care what file system (if any) is on the array disks.

[senrab]

6 minutes ago, jonathanm said:

I was trying to get a feel for what the state of things were so I could recommend the most efficient order of operations.

 

Yes, add the 3TB as an additional array member and let unraid clear it. Be sure it is set to the format type you wish to use. Verify that only that drive is listed as unmountable, and select the option to format it. Copy (not move)* the entire content of one of the ReiserFS drives to the newly formatted drive. Verify the copy was complete and accurate, change the requested format type of the source drive, format it, lather rinse repeat.

 

*Moving the files is much slower than a copy, and eliminates the option for a do over if things go south.

Great! Thank you for the help!

 

Reading the directions on the other thread, it says I should make a folder on the destination drive(their example was the folder 't') and copy everything in there using:

cp -rpv /mnt/[source]/* /mnt/[dest]/t

 

Is this still the best way? 

 

Sorry for all of the questions, I am learning that things are a lot different than when I first started with my unraid server(v5). I am just making sure everything is correct.( I am always nervous when I am moving my data around). lol

 

Edited August 9, 2019 by senrab

[JonathanM]

5 minutes ago, senrab said:

it says I should make a folder on the destination drive(their example was the folder 't') and copy everything in there using:

cp -rpv /mnt/[source]/* /mnt/[dest]/t

 

Is this still the best way? 

It complicates some things, and simplifies others.

 

Is it acceptable to you to temporarily stop all array activity while the migration is happening, or will you be run out of town on a rail if certain people can't access what they want, when they want it?

[senrab]

1 minute ago, jonathanm said:

It complicates some things, and simplifies others.

 

Is it acceptable to you to temporarily stop all array activity while the migration is happening, or will you be run out of town on a rail if certain people can't access what they want, when they want it?

LOL. I can manage either. If the array has to be stopped, it has to be stopped. I could also do this at certain times of the day when no one needs to access the server. 

[JonathanM]

The array isn't going to be stopped, but if someone has files open, or adds files to a share that is being accessed, things get complicated. If you can tell people not to add or change files, and preferably temporarily not use the server, so much the better.

 

Best is to disable the docker and vm service so nothing is interfering there either.

 

If nobody is going to be accessing the files, then no need for a temporary folder, simply make an identical copy of all the folders. The side effect of this is that the temporarily duplicated files will be hidden from the user share system, so someone could make a change that you would subsequently format away. The file verification step should catch that, but easier not to have to mess with it at all.

[senrab]

5 minutes ago, jonathanm said:

The array isn't going to be stopped, but if someone has files open, or adds files to a share that is being accessed, things get complicated. If you can tell people not to add or change files, and preferably temporarily not use the server, so much the better.

 

Best is to disable the docker and vm service so nothing is interfering there either.

 

If nobody is going to be accessing the files, then no need for a temporary folder, simply make an identical copy of all the folders. The side effect of this is that the temporarily duplicated files will be hidden from the user share system, so someone could make a change that you would subsequently format away. The file verification step should catch that, but easier not to have to mess with it at all.

Ok sounds good. This makes total sense. 

 

I am the only one who writes to the server. The rest of the household uses it viewing media using Plex. 

 

I just need to:

Add 3TB drive to the array as XFS

Run: cp -rpv /mnt/[source]/* /mnt/[dest]

format 'source' drive from previous step to XFS

Pick another drive and do the same.

 

Also, will I have to make any changes to my shares? 

 

Thanks again for all of the help! It is greatly appreciated!

[JonathanM]

2 minutes ago, senrab said:

 

Also, will I have to make any changes to my shares?

Do you have any includes or excludes set currently?

[senrab]

2 minutes ago, jonathanm said:

Do you have any includes or excludes set currently?

No. All shares have access to all disks. 

[Frank1940]

1 hour ago, senrab said:

Great thanks for the info! I will try changing the sata cable and see if that corrects the CRC errors. 

A new cable will not correct the CRC errors.  This count is the total number of CRC errors that the disk has ever experienced.   If you change the cable and that was the source of the errors, they will simply stop increasing.  That is the best, you can hope for with CRC errors!  (I have always wondered why they were logged as a part of the disk statistics in the first place but that was a decision made by the HD manufacturers.  I further wondered why it was given the high level of significance in Unraid since unless you have hundreds of them per day, they are really not that important to data security.  It is actually more important to know if they happened last night or back three years ago.) 

[Frank1940]

This is the method that I used when I did the conversion from reiserfs to XFS.  It is a well tested method that has been successfully used by  a lot of users.

 

   https://wiki.unraid.net/index.php/File_System_Conversion#Mirror_each_disk_with_rsync.2C_preserving_parity\

[JonathanM]

9 minutes ago, Frank1940 said:

This is the method that I used when I did the conversion from reiserfs to XFS.  It is a well tested method that has been successfully used by  a lot of users.

 

   https://wiki.unraid.net/index.php/File_System_Conversion#Mirror_each_disk_with_rsync.2C_preserving_parity\

That procedure contains the meat of what I've recommend, but it adds some extra steps designed to keep the content on the same drive slot after you are done, and some precautions so that the temporarily duplicated files won't cause issues with ongoing usage during conversion. The OP has indicated he doesn't really care what drive slot contains specific file or shares, so no need to go through the extra steps of swapping slots. Simply keeping accurate track of which drive is source and which drive is destination for each iteration is enough.

 

It does, however, spell out the use of rsync instead of cp, and I personally use the rsync method to do the copy and then the compare before nuking the old source drive and moving on to the next transfer.

 

In the OP's specific case the steps outlined in 10-18 can be skipped and instead simply change the format type of the last source disk number to become the new blank destination disk.

 

As long as you keep track of which disk number is being currently read from and written to, you should be fine.

[senrab]

Is it a problem to still have Plex running so my kids can still have access to their shows and movies?

[JonathanM]

1 hour ago, senrab said:

Is it a problem to still have Plex running so my kids can still have access to their shows and movies?

I don't know. I use Emby, not Plex, and I know Emby sometimes will add files to the media folders. If plex is the same way, then it's possible plex will operate on the source drive after the copy has happened, so the change will be lost. With emby it wouldn't be a big deal, but the verification step would show differences that you would need to track down.

 

It shouldn't be a show stopper to leave plex running, but it may slow things down or complicate them a little, or may have no effect. Honestly I'd be tempted to try leaving it running, the possible downsides aren't that big.