Jump to content
savestheday

Re: Format XFS on replacement drive / Convert from RFS to XFS (discussion only)

900 posts in this topic Last Reply

Recommended Posts

I've also added a red note to it, to indicate that the swap trick doesn't work any more, as of 6.2, as johnnie.black has reminded me several times.  This 6.2 change really disappointed me, as the swap trick made it so much easier.  For 6.2, I need to replace steps 10 through 12, but I don't have a test system, to see what's on the screen and write new steps up.  I haven't quite known what to do about it, so just left it alone for awhile.

 

For v6.2 users using P parity only you could make small changes to 4 steps to make it work, but it involves doing a new config and that can make it much more prone to a user making a mistake:

 

10.  Stop the array; we are now going to swap their assignments; when stopped, click on Tools -> New Config -> on "Retain current configuration:" select "All", check "Yes I want to do this", click Apply, click Done; go back to the main page and click on the dropdown for the new drive (e.g. Disk 11) and unassign it;

11. Click on the dropdown for the other drive (e.g. Disk 10), the one being converted, and reassign it as the physical drive that was just added, the new drive that was empty;

12. Click on the dropdown for the slot of the new drive (e.g. Disk 11) and reassign it to the physical drive that was being converted (e.g. Disk 10); you have just swapped the 2 drives, which is fine as they are identical (except for file system format)

13. (as it was)

14. You should see all array disks with a blue icon, a warning that the parity disk will be erased, and a check box "Parity is already valid"; important click it before the Start button to start the array; it should start up without issue and look almost identical to what it looked like before the swap, with no parity check needed; however the XFS disk is now online and its files are now being shared as they normally would; check it all if you are in doubt

 

I checked and I believe the procedure is correct but in case you decide to use it it should be double checked by someone else.

Share this post


Link to post

OK, I'm going to complain again.  For those of us who use unraid as a NAS, set it up and forget it until we have to do something, there's no central repository of information that's easy and quick to understand and follow.

 

Central repository/Wiki: http://lime-technology.com/wiki/index.php?title=UnRAID_Wiki

 

I lost about 10 hours worth of work that was on the cache drive (and I thought was also at least partially on the array, although I could not find it even with repair of the file system), which for me as an attorney who gets paid for the work I actually do, that's a significant loss of income I had to completely make up.

 

That is very unfortunate, and a hard lesson learned.  I never work more than an hour without making a backup of the data I’m currently working with.  I also save every few minutes to my local copy.

 

Now, I've reconfigured the server and my system to make it more secure (copy the cache to the array every hour, moved all my work files to a share that does not use the cache, set up a Windows backup to another computer to back up every hour, etc.).

 

Sounds like you’ve got a much better system in place now to prevent future loss.

 

So, am I wasting all this time converting from RFS to XFS?  I have no idea.  However, since I've started it and am on drive 2 of 4, I might as well continue.  But at least there could be some easily accessible resource that would tell me why I would want to do this and how.  You have the how in post 2 but not the why.

 

It would have been prudent to ask questions prior to starting such a labor intensive process.  That is what these forums are for.  I always make inquiries before diving in.

 

Anyway, if there's one thing that would improve unraid is a way to easily find information.  I literally stumbled on this post, for instance.

 

Unfortunately that is the nature of a free public forum.  There aren’t any paid employees reorganizing it.  Every individual here is nothing more than another community member.  Be appreciative that we can come together as a community to help each other.  This forum may not be organized like a tech manual, but what can you expect for free assistance?  You can always reference the Wiki instead if the personal help is not satisfactory, but I find it much better to talk with others.

 

I don't want to end on bad note.

 

To be honest I think you are expecting a lot for the minimal cost of unRAID.  How much of your time would the price of unRAID buy?  I would imagine perhaps enough time to write one letter.  Can I bill you for my reply here?

Share this post


Link to post

I've also added a red note to it, to indicate that the swap trick doesn't work any more, as of 6.2, as johnnie.black has reminded me several times.  This 6.2 change really disappointed me, as the swap trick made it so much easier.  For 6.2, I need to replace steps 10 through 12, but I don't have a test system, to see what's on the screen and write new steps up.  I haven't quite known what to do about it, so just left it alone for awhile.

 

For v6.2 users using P parity only you could make small changes to 4 steps to make it work, but it involves doing a new config and that can make it much more prone to a user making a mistake:

 

10.  Stop the array; we are now going to swap their assignments; when stopped, click on Tools -> New Config -> on "Retain current configuration:" select "All", check "Yes I want to do this", click Apply, click Done; go back to the main page and click on the dropdown for the new drive (e.g. Disk 11) and unassign it;

11. Click on the dropdown for the other drive (e.g. Disk 10), the one being converted, and reassign it as the physical drive that was just added, the new drive that was empty;

12. Click on the dropdown for the slot of the new drive (e.g. Disk 11) and reassign it to the physical drive that was being converted (e.g. Disk 10); you have just swapped the 2 drives, which is fine as they are identical (except for file system format)

13. (as it was)

14. You should see all array disks with a blue icon, a warning that the parity disk will be erased, and a check box "Parity is already valid"; important click it before the Start button to start the array; it should start up without issue and look almost identical to what it looked like before the swap, with no parity check needed; however the XFS disk is now online and its files are now being shared as they normally would; check it all if you are in doubt

 

I checked and I believe the procedure is correct but in case you decide to use it it should be double checked by someone else.

 

Thank you, Johnnie!  What I decided to do was copy the post to a wiki page ->  File System Conversion

 

Then I incorporated your steps, and added comments that it's only for 6.2 or later, if earlier go to my original post.  I've modified my post to indicate it's only for 6.0 and 6.1, and if 6.2 go to the wiki page.  I think that covers it.  I also added a comment about dual parity.  I've added links to my section on File Systems and whether to convert or not, but I hope to pull that stuff into the page, later.

 

And I added a comment about it being untested yet, so I would really like to hear from anyone willing to try it!  I'm sure it can be improved.

Share this post


Link to post

Thank you, Johnnie!  What I decided to do was copy the post to a wiki page ->  File System Conversion

 

Then I incorporated your steps, and added comments that it's only for 6.2 or later, if earlier go to my original post.  I've modified my post to indicate it's only for 6.0 and 6.1, and if 6.2 go to the wiki page.  I think that covers it.  I also added a comment about dual parity.  I've added links to my section on File Systems and whether to convert or not, but I hope to pull that stuff into the page, later.

 

And I added a comment about it being untested yet, so I would really like to hear from anyone willing to try it!  I'm sure it can be improved.

 

I tried to edit the wiki myself but can't, I always get I'm not logged in, this is what I wanted to edit:

 

on step 12: "Go back to the Main page and click on the dropdown for the new drive (e.g. Disk 11) and unassign it (click on Unassigned???)" - that should be changed to (click on "no device").

 

And more importantly, after step 11 there should be a warning about the new config, something like this:

 

"Doing a new config will reset the file system type for all disks to auto, while usually this is not a problem, especially with the latest unRAID, in some circumstances can lead to unmountable disk(s), if that happens you just need to select the correct file system for that disk(s), if in doubt ask for help."

Share this post


Link to post

Thank you, Johnnie!  What I decided to do was copy the post to a wiki page ->  File System Conversion

 

Then I incorporated your steps, and added comments that it's only for 6.2 or later, if earlier go to my original post.  I've modified my post to indicate it's only for 6.0 and 6.1, and if 6.2 go to the wiki page.  I think that covers it.  I also added a comment about dual parity.  I've added links to my section on File Systems and whether to convert or not, but I hope to pull that stuff into the page, later.

 

And I added a comment about it being untested yet, so I would really like to hear from anyone willing to try it!  I'm sure it can be improved.

 

I tried to edit the wiki myself but can't, I always get I'm not logged in, this is what I wanted to edit:

 

on step 12: "Go back to the Main page and click on the dropdown for the new drive (e.g. Disk 11) and unassign it (click on Unassigned???)" - that should be changed to (click on "no device").

 

And more importantly, after step 11 there should be a warning about the new config, something like this:

 

"Doing a new config will reset the file system type for all disks to auto, while usually this is not a problem, especially with the latest unRAID, in some circumstances can lead to unmountable disk(s), if that happens you just need to select the correct file system for that disk(s), if in doubt ask for help."

 

Done, and thanks!  I've added your changes.

 

Now we need to find out why you can't edit the wiki.  I'll post a question about it in the moderator board, but you may want to ask LimeTech support yourself.

Share this post


Link to post

Wow - hard to believe 29 pages of discussion about converting to XFS. It was almost exactly 2 years ago I wrote the initial comprehensive guide to convert. Thanks to RobJ, gary, trurl and the rest of the crew for continuing to enhance the process and supporting users enhancing their servers.

 

There was quite a debate about the value of converting when XFS first became an option, with reasons on both sides of the aisle. But at this point, XFS is proven. There are compelling reasons to convert, and I would encourage virtually all users to upgrade to XFS, at least on their primary arrays. Here are some of the reasons ...

 

1 - Reiserfs (RFS) is not being actively supported or enhanced. Just enough is being done to keep the lights on. We've had one situation where a Kernel update required a small update to RFS. The fix was not made properly, and allowed data corruption that made it into the golden version of Slackware Linux and unRAID. The corruption was insidious and did not just affect new files written to the disk - it could just as easily corrupt old files. Several unRAID users were affected (due to the nature of the issue, knowing if you were impacted and if so, exactly what files were involved, was virtually impossible for most people. I cannot say with certainty that I was not affected). The problem was fixed pretty quickly, and no new issues (that we know of) have occurred, but given the population using RFS is getting smaller and smaller, the testing of new updates will be less and less of a priority. RFS is not a good place for your family jewels IMO.

 

2. As one who has been around unRAID for nearly 10 years, since unRAID 2, I can tell you that unRAID always had its quirks with RFS. I had inconsistent performance, timeouts, and a number of oddities, especially as disks get full. As disk sizes grew to 1T, 2T and beyond, the problems got worse, and eventually problems were happening with half full disks. I can say that since moving to XFS, I have had NONE of these annoying issues. My XFS powered array is rock solid, except for the very occasional real disk problem which unRAID handles quite well.

 

3. Everyone uses their servers a little different, but if you're like most, your unRAID server is getting more and more data to hold and stuff to do over time. Initially mine was a toy in my basement, and wife could have cared less if it was up or down. But now it is an indispensable tool for my family. Converting to XFS takes some time and attention, but next month, or next year it will be worse. And imagine hearing that RFS has started corrupting people's servers. That is not the time you want to begin thinking about your update journey. How'd you feel about shutting down your server for a few weeks while it gets sorted? Two years ago I might have agreed that there is no rush, wait for an opportune moment. But at this point, I suggest you get started now.

 

4. Dual parity is a nice feature, but it would behoove you to upgrade before installing that second parity drive, which just makes the upgrade process more complicated.

 

5. Maybe low on the totem pole, but the author and namesake of RFS, Hans Reiser, brutally murdered his wife and is rotting in prison. He's a real winner - having just lost a lawsuit from his children. Read about him HERE. I'd just as soon not use his file system, and let it fall into obscurity, with its user based shriveling to zero.

 

Good luck everyone! I am most easily reached via PM, as I am not able to monitor these forums like I once did.

 

Enjoy your arrays!!

Share this post


Link to post

I think I'm ready to start the conversion process on 8 RFS disks. Couple questions, if someone would be so kind:

 

I know it will vary based on hardware, but generally, how long should a full 4TB drive take to copy ? Most of my disks are 5400rpm (parity is 7200 though), so I'm already prepared for a long process. I would imagine it would be less than a preclear if I don't do the second checksum validation...

 

If I have a cache pool and disable the mover, assuming I don't delete or change any files, anything added to a user share should stay on the cache and not hit my actual disks, right? I ask because I'd rather speed up the process and not evoke the rsync -rcvPX command. At least not for every disk conversion.

 

Has anyone tried performing 2 simultaneous copies to separate disks before? I wonder if that would speed up the process.

 

Thanks! And thanks to all who contributed to documenting this conversion process.

Share this post


Link to post

...If I have a cache pool and disable the mover, assuming I don't delete or change any files, anything added to a user share should stay on the cache and not hit my actual disks, right?...

Perhaps you know this, but any user share set to cache-no (the default) are never written to cache but instead directly to the array.

Share this post


Link to post

Perhaps you know this, but any user share set to cache-no (the default) are never written to cache but instead directly to the array.

Thanks for pointing that out. I don't have any user shares set to cache-no, so I think I'm good in that respect.

Share this post


Link to post

Thank you, Johnnie!  What I decided to do was copy the post to a wiki page ->  File System Conversion

 

Then I incorporated your steps, and added comments that it's only for 6.2 or later, if earlier go to my original post.  I've modified my post to indicate it's only for 6.0 and 6.1, and if 6.2 go to the wiki page.  I think that covers it.  I also added a comment about dual parity.  I've added links to my section on File Systems and whether to convert or not, but I hope to pull that stuff into the page, later.

 

And I added a comment about it being untested yet, so I would really like to hear from anyone willing to try it!  I'm sure it can be improved.

Minor change suggestion to the wiki:

Go back to the Main page and click on the dropdown for the new drive (e.g. Disk 11) and unassign it (click on "no device")

It actually is "unassigned" not "no device" on 6.2.4.

Share this post


Link to post

Minor change suggestion to the wiki:

Go back to the Main page and click on the dropdown for the new drive (e.g. Disk 11) and unassign it (click on "no device")

It actually is "unassigned" not "no device" on 6.2.4.

 

Do you mind posting a screenshot, is yours different from this?

Screenshot_2017-01-11_08_51_47.png.1688cab4d523ed4e51800b63f16999cc.png

Share this post


Link to post

Do you mind posting a screenshot, is yours different from this?

 

Of course. I'm about halfway through converting another drive, so I should have a screenshot this evening after work.

Share this post


Link to post

Here it is:

 

You're right, the option is different when it's done with assigned disks (no device) or after a new config (unassigned), so for this procedure since it's after a new config the correct option is "unassigned".

 

Thanks for pointing it out.

 

@Rob, I still can't edit the wiki, please correct that when you can.

Share this post


Link to post

Here it is:

 

You're right, the option is different when it's done with assigned disks (no device) or after a new config (unassigned), so for this procedure since it's after a new config the correct option is "unassigned".

 

Thanks for pointing it out.

 

@Rob, I still can't edit the wiki, please correct that when you can.

 

Thank you guys, changed it.  I decided to keep both, with 'unassigned' first, in case an advanced user is 'adapting' the instructions to their particular use.

 

Johnnie, I've posted about it here, no response yet.

Share this post


Link to post

Really thanks for that great tutorial.

 

I am just about to convert all my disks from ReiserFS to XFS. Unfortunately it takes about 24hrs to copy over 4TB of data from one drive to another. Write speeds never really rocked on unRaid  ;)

Share this post


Link to post

I wrapped up my conversion from RFS to XFS last week. Since I have only one parity disk, I was able to do the drive reassignments with the "new config" tool and assignment swaps as detailed without mucking with my user shares.

 

I followed the tutorial on the wiki and I'm happy to report that a parity check after all 10 drives were converted came back with zero sync errors! Looks like it even slightly sped up my parity check speed as well. But most importantly, I'm just glad my array doesn't spin up now when I update or stop a Docker as it was doing with my RFS-formatted drives.

 

Not sure if you'd like to reflect in the wiki that it's now confirmed that the process works... Thanks to all who contributed!

Share this post


Link to post

Looks like it even slightly sped up my parity check speed as well.

Not possible that changing filesystems would affect parity check speed since parity has no concept of filesystems, just bits.

Share this post


Link to post

Looks like it even slightly sped up my parity check speed as well.

Not possible that changing filesystems would affect parity check speed since parity has no concept of filesystems, just bits.

Taken as an isolated system, very true. However... if there are writes to the array during the parity check, then parity check speed is a product of the ideal maximum for the hardware plus overhead for any reads or writes that occurred during the check.

 

trurl's point stands, file system doesn't effect parity check speed.

 

File system type very definitely can effect read and write speed to the array. Given that very few people shut down all file activity during a parity check, I can see where there could be an observed difference, but it's only due to file access, not the actual parity check.

Share this post


Link to post

guys i was hoping you could take a look at my current config and tell me the best way to go about changing my filesystems from reiserfs to xfs, currently as i understand it, i need to ADD a 4TB BLANK disk to copy the contents of the other disk's too in from reiserfs -> xfs, here is my connundrum though, currently this server is packed with all the HDD it can hold, and the 4TB unassigned in the photo above is replacing a 2tb (array is running in emulated mode from the 4tb parity drive),.  So how would i go about doing this when i cant add a blank disk ?  could i move the contents of the 2tb drives to the first two drives ? (i have 3.8tb of free space total right now) then un-assign one ? 

 

thank you for the input

 

 

EDIT: just found UNBALANCE https://github.com/jbrodriguez/unbalance  but i am worried about doing this on an emulated array, should i first rebuild my array, then run unbalance ? or would it be just as safe to perform these actions from the parity drive ?

 

thanks.  also, is there a concise tutorial to follow to do the xfs conversion via unbalance ?  thank you for all your help

Share this post


Link to post

I simply moved all data off one disk, formatted to XFS and then kept leap frogging all my data around until every disk was empty so I could format to XFS. It certainly took a while, but got it done eventually.

 

I personally would certainly get your drive backup before I did anything. When you rebuild your drive its going to build bit by bit back to your reiser disk so your going to have to reformat it after the fact when your data is backup and moved.

 

If your new disk is going to be a 4TB you have plenty of room to shuffle your files around based on the amount of free space you have.

Share this post


Link to post

I simply moved all data off one disk, formatted to XFS and then kept leap frogging all my data around until every disk was empty so I could format to XFS. It certainly took a while, but got it done eventually.

 

I personally would certainly get your drive backup before I did anything. When you rebuild your drive its going to build bit by bit back to your reiser disk so your going to have to reformat it after the fact when your data is backup and moved.

 

If your new disk is going to be a 4TB you have plenty of room to shuffle your files around based on the amount of free space you have.

 

Agree'd, i currently have the drive rebuilding, i will remote in after i get to the office and start shuffling files around, thanks for the info kizer! your post was the post that made me realize i could do this ;)!

 

-D8A7ED

Share this post


Link to post

I've recently upgraded my unRAID server and while I'm at it, I'd like to switch from reiserfs to XFS.  I added a new pre-clear XFS disk3 to the array and executed rsync -avPX /mnt/disk2/ /mnt/disk3/ from the console to kick off the copy.  Two questions.

 

1. The folder structure on the copy to disk3 looks like this: /mnt/disk3/disk2/myDataFolders.  Did I foul up the command?  How would I get it to copy as /mnt/disk3/myDataFolders?

2. The copy is parity protected (by design), but is there a best practice to bypass this for the sake of increasing transfer speed?  I understand there is a risk of data loss.

Share this post


Link to post

I've recently upgraded my unRAID server and while I'm at it, I'd like to switch from reiserfs to XFS.  I added a new pre-clear XFS disk3 to the array and executed rsync -avPX /mnt/disk2/ /mnt/disk3/ from the console to kick off the copy.  Two questions.

 

1. The folder structure on the copy to disk3 looks like this: \mnt\disk3\disk2\myDataFolders.  Did I foul up the command?  How would I get it to copy as \mnt\disk3\myDataFolders?

 

That means you used:

 

rsync -avPX /mnt/disk2 /mnt/disk3/

 

It's missing one / at end of disk2

 

2. The copy is parity protected (by design), but is there a best practice to bypass this for the sake of increasing transfer speed?  I understand there is a risk of data loss.

 

If you want to disable parity stop array, unassign parity, start array, then re-assign it in the end, a parity sync will begin (obviously you'll be unprotected during the conversion)

Share this post


Link to post

Also some enable Turbo mode while copying to speed up the write speeds. I've never done it "personally" simply because I'm a run it and forget it for several hours kind of guy, but that could be an option too opposed to not using parity.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.