Re: Format XFS on replacement drive / Convert from RFS to XFS (discussion only)

[JorgeB]

Also some enable Turbo mode while copying to speed up the write speeds. I've never done it "personally" simply because I'm a run it and forget it for several hours kind of guy, but that could be an option too opposed to not using parity.

 

Turbo write should be a little faster than normal writes but because the source disk has to alternate between reading the data to copy and data do update parity it's considerably slower then normal turbo writes, IIRC it maxes out at ~60MB/s.

[dboonthego]

 

That means you used:

 

rsync -avPX /mnt/disk2 /mnt/disk3/

 

It's missing one / at end of disk2

 

 

I'm sure that's what I did.  Thanks!  I want to copy the data back to disk2 after the format.  Considering my mistake earlier, this should do it correct?

rsync -avPX /mnt/disk3/disk2/ /mnt/disk2/

[RobJ]

Also some enable Turbo mode while copying to speed up the write speeds. I've never done it "personally" simply because I'm a run it and forget it for several hours kind of guy, but that could be an option too opposed to not using parity.

 

Turbo write should be a little faster than normal writes but because the source disk has to alternate between reading the data to copy and data do update parity it's considerably slower then normal turbo writes, IIRC it maxes out at ~60MB/s.

 

Within the past week, I just tested and can confirm this to be true, the constant re-seeks slow down Turbo Write.

 

And another factor that can slow it down further is having a much slower drive in the array, because like parity checks, Turbo Write cannot go faster than the speed of your slowest drive.  If your array of drives has a mix of slow and fast drives, and your transfer will be between 2 fast drives, then you probably want read/modify/write, not turbo write.

 

My Turbo Write speeds and parity checks were getting slower and slower, dropping below 30MB/s, and I assumed I had a failing drive, but I couldn't identify it, all SMART reports looked fine.  (When I have time, I'll write it up - very interesting/distressing drive failure, some surprising SMART behavior, and some improvements needed in our SMART issue detection.)

[SSD]

(When I have time, I'll write it up - very interesting/distressing drive failure, some surprising SMART behavior, and some improvements needed in our SMART issue detection.)

 

Very much looking forward to that!

[garycase]

If your array of drives has a mix of slow and fast drives, and your transfer will be between 2 fast drives, then you probably want read/modify/write, not turbo write.

 

I'd think this would be true regardless of whether the two involved drives were "fast" or not.    Consider ...

 

If you're copying from Drive A to Drive B:

 

(a)  With turbo write:    Data is read from drive A (1 seek);  then to write it to drive B there's another seek on A (moving it away from where you're reading data);  seeks on every other drive in the system (to read the data from all other drives and position drive B to the correct sector); then writes to drive B and parity;  then (for the next sector/block) a seek back to the correct position on drive A ... and this cycle repeats until the end of the copy.

 

(b)  With read/write/modify:  Data is read from drive A (1 seek);  then drive B and parity seek to the correct destination and their current data is read;  then drive B and parity are re-written (NO additional seek).    Now the next sector/block is read from drive A, which is already either at the correct position or a single track away from it; and the cycle repeats.

 

Excluding the initial seek to the data on drive A, that results in the following:

 

For (a)  two seek delays (one for ALL drives, and one for drive A to get back to where you're reading the data from);  a read from all drives; and a write to drive B and parity (resulting in one rotational latency delay).  So 2 seeks, 2 reads, 1 write, and 1 rotational latency delay  (counting all simultaneous actions as a single action).

 

For (b)  One seek delay (while B and parity get to the proper location);  two reads (from drive B and parity); one rotational latency delay (while B and parity rotate); and a write to drive B and parity.  So 1 seek, 2 reads, 1 write, and 1 rotational latency delay.

 

Note that this does not factor in the drive's buffering -- which would likely help the turbo-write case by eliminating some of the re-seeks back to the data location on drive A ... but nevertheless any benefit of turbo-write is largely offset by the additional thrashing of the heads between the location where data is being read for the copy and where it needs to read the current data as part of the turbo-write parity calculation.

 

There would be a BIG difference if a copy utility was being used that buffered a significant amount of data before doing the actual writes -- but I don't know what Linux utilities do this (or can be configured to do it).

 

 

[chip]

Guys

 

I would like to convert my disks to XFS.

I have a Parity drive and 4- 3tb drives in the array. How can I convert the 3tb drives to XFS if I don't have a spare drive?

 

Drives are 45%-75% full.

[trurl]

Guys

 

I would like to convert my disks to XFS.

I have a Parity drive and 4- 3tb drives in the array. How can I convert the 3tb drives to XFS if I don't have a spare drive?

 

Drives are 45%-75% full.

Any disk you change the filesystem on will be formatted, so you must copy the files somewhere before changing. Simple as that.

[chip]

Ok so I could start with disk 1 move all data between rest of disks and so on.

 

Is there a best solution\command for moving data and verifying data moved ok?

[trurl]

Ok so I could start with disk 1 move all data between rest of disks and so on.

 

Is there a best solution\command for moving data and verifying data moved ok?

That's what this thread is about. Start reading it from the end and you will find the latest recommendation a few pages back.

[dboonthego]

Ok so I could start with disk 1 move all data between rest of disks and so on.

 

Is there a best solution\command for moving data and verifying data moved ok?

 

This will copy the contents of disk1 to disk2; change the disk # as needed for your situation.  Include the trailing slashes.  I forgot to include them which altered my destination folder structure.

rsync -avPX /mnt/disk1/ /mnt/disk2/

 

More info here:

http://lime-technology.com/wiki/index.php/Upgrading_to_UnRAID_v6#Converting_Drives_from_ReiserFS_to_XFS

 

 

[dboonthego]

Any disk you change the filesystem on will be formatted, so you must copy the files somewhere before changing. Simple as that.

 

 

Makes complete sense.  I formatted disk2 to XFS, but my data is still showing up in windows explorer for \\tower\disk2.  Why is that?  If I look from the console /mnt/disk2 is empty as expected.  Is it being emulated from parity?  At what point can I trust what windows explorer says?

[trurl]

Any disk you change the filesystem on will be formatted, so you must copy the files somewhere before changing. Simple as that.

 

 

Makes complete sense.  I formatted disk2 to XFS, but my data is still showing up in windows explorer for \\tower\disk2.  Why is that?  If I look from the console /mnt/disk2 is empty as expected.  Is it being emulated from parity?  At what point can I trust what windows explorer says?

What you are seeing is the user share named "disk2" that you accidentally created when you botched the rsync command. Any top level folder on cache or array disks is automatically a user share.

[SSD]

Guys

 

I would like to convert my disks to XFS.

I have a Parity drive and 4- 3tb drives in the array. How can I convert the 3tb drives to XFS if I don't have a spare drive?

 

Drives are 45%-75% full.

 

I would suggest freeing up one of the 3T drives by copying to other disks. Now you have an extra disk in the array. You can format it to XFS and do the leap frog (I like that analogy ) to do the other disks. When you're all done you can re-balance - that's up to you.

[dboonthego]

What you are seeing is the user share named "disk2" that you accidentally created when you botched the rsync command. Any top level folder on cache or array disks is automatically a user share.

 

Interesting quagmire I got myself into.  Thanks.  I didn't think unRAID allowed user shares with reserved names like flash, disk1, disk2, etc.

[bonienl]

What you are seeing is the user share named "disk2" that you accidentally created when you botched the rsync command. Any top level folder on cache or array disks is automatically a user share.

 

Interesting quagmire I got myself into.  Thanks.  I didn't think unRAID allowed user shares with reserved names like flash, disk1, disk2, etc.

 

The GUI won't allow you to create shares with reserved names, but any tools (rsync) don't have that protection.

 

[JonathanM]

I think it's time for a new read only sticky post with the agreed upon best practice for migrating off of ReiserFS. This thread should be referenced from that post and kept alive as ongoing support for people migrating, but I think a succinct summary is needed as a sticky instead of this thread.

[chip]

So if my action is like this -

 

round [source] [dest] 

  ------- ----------- --------

  1      disk3  disk4

 

Disk3 is new XFS disk

 

  2      disk1  disk3 

Disk1 is new XFS disk

 

  3      disk2  disk1 

Disk2 is new XFS disk

 

  4      disk4    disk2 

Disk4 is new XFS disk

  5     

 

My other questions are about Parity, shares, etc still confused on that part even after reading the links. Since everything will now be moved to different drives.

 

It was raised from these statements from http://lime-technology.com/wiki/index.php/Upgrading_to_UnRAID_v6#Converting_Drives_from_ReiserFS_to_XFS

 

    Correct all of the drive includes and excludes for all shares, plus all other drive references

    Or use New Config to re-assign all drives back to their original drive number (marking parity as valid)

 

 

 

 

 

 

 

 

 

[SSD]

Parity will take care of itself. To unRAID, a disk is a string of bits. So when you format the disk, to unRAID you are just changing some of those bits. Parity will be kept straight throughout the process.

 

Share include/exclude disks settings need to be revisited after the process. Depends on how much you've customized them. I think even share settings survive a new config. So you'd need to adjust them regardless.

 

No need to do a new config. In fact, the whole idea is to do all this within the protected array.

[trurl]

1)  Correct all of the drive includes and excludes for all shares, plus all other drive references

2)  Or use New Config to re-assign all drives back to their original drive number (marking parity as valid)

2) would work, but the concern I would have would be deciding what was the original drive number. Especially if you had disks of different sizes and needed to copy the contents of one disk to multiple disks, etc.

 

1) makes me ask another question though. Where else do you have drive references except in the user share settings? If you only had user share references elsewhere you wouldn't have that problem.

[chip]

so after I move my data

ound [source] [dest]

  ------- ----------- --------

  1      disk3  disk4

 

to format disk 3 to xfs I will actually need to add it as a new disk with will become disk5. Won't this throw off parity?

 

Disk 1 would become disk6 etc.

 

[optiman]

I think it's time for a new read only sticky post with the agreed upon best practice for migrating off of ReiserFS. This thread should be referenced from that post and kept alive as ongoing support for people migrating, but I think a succinct summary is needed as a sticky instead of this thread.

 

+1  I'm ready to convert my fs too and after reading this thread, I have a headache.  I'm looking for the safest process with easy step by step instructions.  I did read the instructions on wiki, and they are good, but conflict with other advice in this thread.  That is th e only reason I have not started.

 

I would be happy to see the experts agree on best process to follow (how they would convert their fs) and put on post #1

[chip]

Yeah I am a little confused as well and may hold off though I would like to see if this would resolve some issues of mine.

[trurl]

so after I move my data

ound [source] [dest]

  ------- ----------- --------

  1      disk3  disk4

 

to format disk 3 to xfs I will actually need to add it as a new disk with will become disk5. Won't this throw off parity?

 

Disk 1 would become disk6 etc.

No. It is not even possible to do what you suggest without setting a new config.

 

Assuming you have already moved all the files off the disk. Just stop the array, click on the disk to get to its settings page, choose the filesystem, and start the array to reformat it.

 

 

[dboonthego]

Just stop the array, click on the disk to get to its settings page, choose the filesystem, and start the array to reformat it.

 

How do you reformat an existing xfs disk as xfs?  Do these steps twice by formatting with a different filesystem first, then putting it back to xfs?

[JorgeB]

Just stop the array, click on the disk to get to its settings page, choose the filesystem, and start the array to reformat it.

 

How do you reformat an existing xfs disk as xfs?  Do these steps twice by formatting with a different filesystem first, then putting it back to xfs?

Yes.