Any recommendations for transferring large amounts of data?

[christobaass]

I apologize if this question has been done to death.

 

I've done a little bit of research into this, and everyone seems to have a different answer, and some software is not free. Would using a simple terminal copy/ paste be sufficient? I'm copying about 25TB from a RAID10 to my new unRAID build and want to make sure I'm safely transferring data.

 

Thanks in advance.

[Frank1940]

Patience is the first word I would give you.

 

Since you have a RAID 10 server,  I would set up an array of data disks and NOT assign any parity drive(s) on your Unraid server.  This will about double the transfer rate to the Unraid array.  When all of the data has been transferred, assign the parity disk(s) at that time and build parity at that point.   Use the fact that your RAID 10 server, is a backup of your data until you decommission that server.   

 

Important note:   Large files transfer faster than small files.  The file allocation overheard, disk head movement time between file space and file allocation tables, and head latency all become big factors with small files. 

Edited October 4, 2022 by Frank1940

[Frank1940]

55 minutes ago, christobaass said:

I've done a little bit of research into this, and everyone seems to have a different answer, and some software is not free. Would using a simple terminal copy/ paste be sufficient?

 

Consider the Unassigned Devices plugin.  That should allow you to mount the RAID 10 server to your Unraid server.  Then you can use any number of free file managers to move the data-- including the one built into the the GUI in ver 6.11.0.   Or if you are proficient in the Linux command line, you could use it to do the task. 

[christobaass]

10 minutes ago, Frank1940 said:

Since you have a RAID 10 server,  I would set up an array of data disks and NOT assign any parity drive(s) on your Unraid server.

Thanks so much for your response. Would you mind describing this process more in depth? Sorry for my total noobness to this! To be clear, I have a hardware RAID10 box that connects to my current server via USB cable. I don't think this part matters, but I currently run ubuntu server with portainer.

[Frank1940]

Here is the link to the Unraid Manual:

 

         https://wiki.unraid.net/Manual

 

I would suggest reading the first couple of sections to give you a bit of background as Unraid is NOT a conventional RAID setup.   Pay particular attention to this section. 

 

        https://wiki.unraid.net/Articles/Getting_Started#Assigning_Devices_to_the_Array_and_Pool.28s.29

 

Please observe that unlike RAID setups, Unraid does not actually require parity disk(s) to operate.  Without a parity disk being assigned, it is just a pool of hard drives.  Adding the parity drives makes it a parity protected array.  They can be added or removed without losing any data. 

[christobaass]

1 hour ago, Frank1940 said:

Here is the link to the Unraid Manual:

 

         https://wiki.unraid.net/Manual

 

I would suggest reading the first couple of sections to give you a bit of background as Unraid is NOT a conventional RAID setup.   Pay particular attention to this section. 

 

        https://wiki.unraid.net/Articles/Getting_Started#Assigning_Devices_to_the_Array_and_Pool.28s.29

 

Please observe that unlike RAID setups, Unraid does not actually require parity disk(s) to operate.  Without a parity disk being assigned, it is just a pool of hard drives.  Adding the parity drives makes it a parity protected array.  They can be added or removed without losing any data. 

 

This stuff I basically understand. I don't quite grasp exactly how the parity works, but oh well, I trust it.

 

So my steps would essentially be:

1. I plug my 4 HDD's from my RAID10 into my Unraid machine

2. keep the RAID10 drives unassigned

3. copy my data from the RAID10 drives over to the pool using the built in tool in Unraid [which I am attempting to locate]

4. preclear the HDD's from my RAID10 and assign them to the pool

5. assign the parity drive and sync

 

Does that sound right? Thanks again.

Edited October 4, 2022 by christobaass

[Frank1940]

3 hours ago, christobaass said:

1. I plug my 4 HDD's from my RAID10 into my Unraid machine

2. keep the RAID10 drives unassigned

3. copy my data from the RAID10 drives over to the pool using the built in tool in Unraid [which I am attempting to locate

 

You can not mount those RAID10  drives in your Unraid server.   There is no provision in Unraid to read the data back off the disks.  You can read about RAID10 here:

 

      https://www.techtarget.com/searchstorage/definition/RAID-10-redundant-array-of-independent-disks

 

Understand that all of those disks will have to remain their present configuration until you have copied all of the data to your new Unraid server.  (When you read the link you will find that your individuals files are split into pieces and each piece written on different disks in the RAID0 portion of your RAID10 configuration!) 

 

5 hours ago, christobaass said:

To be clear, I have a hardware RAID10 box that connects to my current server via USB cable.

 

If this is true, you would install the Unassigned Devices plugin and attach the RAID10 box using the USB connection.  I am assuming that this box appears as a drive to your present server.  If this is true, it would be mounted using Unassigned Devices and it would probably appear as a disk drive.

 

There is the  mc  (Midnight Commander) file manager built into Unraid.  (Type mc   at the command prompt in the Unraid terminal.) 

 

There is also the Dynamix File Manager which is a plugin.  Support thread here:

       https://forums.unraid.net/topic/120982-dynamix-file-manager/

 

The Kursader Docker is another file manager  Support thread here:       

      https://forums.unraid.net/topic/71764-support-binhex-krusader/

     

=============

A second option.  Use the Unassigned Devices plugin to mount your present server to your Unraid server.  (Unassigned Device is very flexible in its abilities to make 'outside' resources available for use with Unraid!)

=============

Third option.   Using a PC (or MAC) computer as a bridge between the two servers and copy the files over your LAN from one server to the other server.   This is the simplest setup for the neophyte to use as it uses a File Manager that you already know how to use.  It can also be the slowest method.  It will be constrained by both the network speed and the other processes being run on that PC.

[christobaass]

Thanks for your guidance on this. I plugged my RAID10 into the unraid server, and I'm using Krusader to copy files to shares. Once I'm done, I will build my parity drive, then move my RAID10 drives into the unraid machine, format, and add to the array. I'd love to keep my current RAID10 as a backup, but, the biggest reason I'm doing this is to increase storage space. I will build a backup machine soon enough.

[trurl]

1 hour ago, christobaass said:

then move my RAID10 drives into the unraid machine, format, and add to the array

You must let Unraid format the drives AFTER you add them to the array.

 

1 hour ago, christobaass said:

I'd love to keep my current RAID10 as a backup, but, the biggest reason I'm doing this is to increase storage space. I will build a backup machine soon enough.

You must always have another copy of anything important and irreplaceable. Parity is not a substitute for backup. You get to decide what qualifies as important and irreplaceable.

[Frank1940]

6 hours ago, trurl said:

You must always have another copy of anything important and irreplaceable. Parity is not a substitute for backup. You get to decide what qualifies as important and irreplaceable.

 

This is very important advice!  Just think about the recent events of Hurricane Ian.  Imagine you had a server in the area where the storm made landfall.  Think about Ramsom Malware.  Theft of the server.  Any data that is truly irreplaceable must be stored offsite and offline in a secure location.  I am not talking about old videos of Saturday Night Live but Family Photos, personal financial records, and business archives that must be kept for legal reasons. 

Edited October 5, 2022 by Frank1940

[christobaass]

ehhhh, you're right. i have a large collection of personal photos and documents i'll backup before getting rid of my RAID10