New build -- a few questions

[adrayic]

I'm just finishing up building my first unRAID box and have a few questions.  The system will consist of the following drives:

 

3TB -- parity

3TB -- disk1

2TB -- disk2

1.5TB -- disk 3

320GB -- cache

 

I have just finished pre-clearing my new 3TB drives and the 320GB cache drive.  The 2TB and 1.5TB are both from an old media server (formatted with XFS) and I need to transfer the contents of these drives to the unRAID array.  Since unRAID cannot read XFS, I will probably mount these drives in a spare linux box and transfer the files to the unRAID array via gigabit ethernet.  Once the data has been successfully transferred (and verified), I can install these drives in the unRAID box, pre-clear them, and add them to the array.  Hopefully I got that right.

 

I followed the configuration tutorial and assigned one of my 3TB drives as disk1 and the other 3TB drive as parity (nothing else has been assigned).  I then started the array and it proceeded to run the parity-sync.  While this was happening I formatted disk1 (following the configuration tutorial).  Hopefully its ok to format the disk while the parity-sync is in process.  The format finished and the sync is still going.  Is this ok or did i screw something up by formatting disk1 while the parity-sync was in process?

 

Now i'm waiting for this parity sync to complete (currently moving at 120 MB/s with ~6 hours left).  Im not sure how to proceed from here.  Do I have to wait for this sync to finish before I can start adding data to the array?  I also forgot to enable user shares before I started the array   Should I stop the parity sync, stop the array, enable user shares, start the array, start the sync and then copy my data over while the sync is in progress?  Or should I just wait for the array to finish the parity sync and then enable user shares and start copying?

 

Sorry for all the questions -- lots to learn

 

 

[Rudy81]

Yes, there is lots to learn. I'm a noob myself and recently built my first unRaid box with 5.0beta14. 

 

Since you are just starting, I don't think you can go wrong as long as things keep working.  I might suggest NOT having the parity disk on line when you first transfer your data.  The configuration wiki suggests first moving all your data and then setting the parity disk. Reason is speed.  It will take a long time to transfer all that data with the parity disk running.  It will go much faster if you first transfer the data and then set the parity disk.  Your gigabit network will be much more useful if you transfer the files before you set the parity disk in the array. 

 

I might also suggest that you don't get in a hurry to install your current data disks until you are happy with the server and its performance. Keep your original data intact until you are REALLY sure things are set up and working the way you want.  I only have two disks and the parity disk thus far.  I have yet to add any other disks since they still hold all my data.  I am playing with the server for the time being to become more comfortable with the system and its uses.

 

Enjoy the process, I'm still learning, but overall the build and configuration has gone better than I could have hoped.....and I'm new to linux as well.

 

[JonathanM]

I might suggest NOT having the parity disk on line when you first transfer your data.  The configuration wiki suggests first moving all your data and then setting the parity disk. Reason is speed.  It will take a long time to transfer all that data with the parity disk running.  It will go much faster if you first transfer the data and then set the parity disk.  Your gigabit network will be much more useful if you transfer the files before you set the parity disk in the array.

I respectfully disagree. Since his parity sync is going at 120MB/s, I really don't see where transferring without parity is going to help his transfer speed much. What you give away by not having parity enabled isn't worth the small speed gains in my opinion.

 

If you want to see what the difference could be, try transferring to the cache drive directly and benchmark that transfer, then try the same thing again to the protected array.

 

I do agree that keeping your old drives intact for as long as practical is very good advice, I also would run binary file comparison on a not insignificant portion of your transferred data before calling it all good.

 

Parity calculations are file system agnostic. It matters not what you do the array disks, as long as parity is being updated, and you operate on the md devices, you can do any kind of file system operations and parity will keep up with the changes. Formatting or adding content is just fine. Keep in mind that you really do need to do a parity check after the parity is built to ensure that things are staying in sync. A clean parity check means everything went fine with the initial build, any errors need to be investigated.

[adrayic]

Thanks for the help.

 

I did read that having the parity disk in place before the initial data migration would slow things down but I also read that doing it this way was a good way to test the system.  I'm not in a big rush to get everything setup so I don't mind the extra time it takes to write data with the parity disk.

 

As for running a parity check after the sync is finished.... I was not aware of that.  Do you suggest running the parity check right after the sync finishes?  Can I start copying data to the array while the parity check is being done or should I wait for the check to come back (hopefully error free) before moving data?

 

[Rajahal]

You haven't done anything wrong (yet), so well done!   Congrats on your first server!

 

Since you say you aren't in a hurry, I would recommend running the parity check immediately after the parity sync completes, then start transferring your data once the check completes with no errors.  You technically can start transferring your data right now, since as johnathanm eloquently said "Parity calculations are file system agnostic," or to put it colloquially, 'Parity don't care'.  However, any disk activity (read or write) during a parity sync or parity check will just slow it down.  Also, you shouldn't trust your array until it can pass a parity sync and subsequent parity check without errors.  If you start transferring right away and then some errors crop up, then you may have to transfer all the data over again anyway.  Rudy81 also gave some good advice: run CRC or MD5 checks on your data after the transfer to make sure nothing went awry.

[Rudy81]

I certainly defer to the more experienced users.  Good point on comparing transfer speeds. 

[adrayic]

Thanks guys, I will wait for the sync to finish (almost done) and then check the parity before copying data to the array.  I appreciate the help and will report back with an update after I get the data copied

[adrayic]

Alright, the parity check has just finished with 0 errors!  Time to start copying my data.

 

I have installed the two XFS drives in another system and am accessing them through an Ubuntu 12.04 live CD.  The XFS drives are mounted and I can see the 3TB disk1 on the unRAID box from the Ubuntu box.  Given that I have about 2.5TB of data to transfer, what is the best way to do this?  Can you run CRC or MD5 on entire directories?  Should I be using rsync to move the data?  I came across the following howto for rsync:

 

http://lime-technology.com/forum/index.php?topic=13432.msg127670#msg127670

 

Is this something I should look into (setting up rsync daemon on the unraid box)?  I do not intend on using rsync for periodic backups so I'm not sure.  I simply want to get my data transferred to the new array. 

 

Thanks

 

EDIT:  I have decided to use MD5DEEP to verify the contents of the copy.  Data is still copying but so far so good