Jump to content

Moving a large amount of data into unRAID


jj_uk

Recommended Posts

The time has come for me to move a large amount of data from my old freenas box to my new UNRAID box.

 

I have to move 13TB.

 

I have precleared the drives for 3 full cycles over the past week so my array is currently 18T 2x8t, 2x1t) with no parity or cache disks- yet.

 

The plan is to move over all the data and then add a parity drive. It's on order and will also be getting a 3-cycle full preclear to stress-test the drive.

 

All my dockers are set up and ready. I just now need to move (copy) the data.

 

I was thinking of using rsync or sftp,  both are fault tolerant due to compression. It's over a local 1gbps network so encryption is not required.

 

My freenas box can saturate the network in read mode (disks are striped) and has rsync server/client included.

 

What's the fastest way to do this? I don't want to have to keep watching it, I just want it to do the copy then maybe email me when done, or something... it would be useful if it auto-restarts in case there is network interruption etc. 

 

 

Link to comment

A quick one line script example:

rsync command && mail command if good || mail command if error

Basically, if your rsync (or sftp) command ends with a good return code then the good mail command runs. If it returns something other than good, it sends a error mail.

 

Not sure if freenas has a mail command and on my unraid box it doesn't. You may have to mount each server onto a VM or other box and sync that way.

 

Hope this helps.

Link to comment

Thanks. I didn't try this but I did manage to transfer 13T of data over the network successfully with rsync. 

 

I used 'screen' to allow multiple rsync sessions at the same time to xfer different folders etc.

 

Without a parity drive, the transfer rate was ~ 120MB/sec (full speed of the network), so i'm happy with that.

 

 

Link to comment

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...