[SOLVED] File transfer process Old NAS to new UnRaid Setup


Recommended Posts

I can see quite a lot of threads on this and loathe to start a new one but they all seem to give conflicting advice based on particular hardware setups.

 

Going to hopefully build first UnRaid server over next few days. I then want to copy files from my Qnap TVS-471 onto it. UnRaid server will be main server with Qnap relegated to onsite backup. 

UnRaid setup will be to start [1 500gb SSD cache drive, 3 X iron wolf 14tb HDDs, one as parity]. 

 

With media library this is going to be about 20tb in total to move.  I can't work out a proper workflow of how to aporoach this?

 

1. Do I disable cache drive and setup a parity drive after migrating all the data? 

 

2. What's best way, that provides fastest transfer speeds, to connect the Qnap to the server?  I haven't got a switch on my network but can I ethernet cable it directly and simply mount it in UnRaid or is there an UnRaid specific utility that will handle this in a neater more reliable way?

 

3. Should I zip up / archive it all first to help with transfer? 

 

4. As it's mostly media files does UnRaid do anything odd where it may split files across discs that may cause playback problems unless copied in specific fashion? 

 

Thanks. 

Edited by 1300GT
Solved
Link to comment
4 hours ago, 1300GT said:

4. As it's mostly media files does UnRaid do anything odd where it may split files across discs that may cause playback problems unless copied in specific fashion? 

 

What type and how are your media files configured?  What I mean is do you have container folders which holds all the directories and files in the DVD and BluRay standard file layout scheme?  Or are using .ISO files which contain the contents of the standard layout?  Or have you transcoded your media files into a single file format?   Or is it a combination of formats?

 

EDIT: Unraid does not split a single file across more than one disk.  But it can split individual .VOB files of a DVD onto a second disk when the disk gets full.  There is a way to prevent that, but you don't want to implement it unless you truly need it. 

Edited by Frank1940
Link to comment
21 minutes ago, Frank1940 said:

What type and how are your media files configured?  What I mean is do you have container folders which holds all the directories and files in the DVD and BluRay standard file layout scheme?  Or are using .ISO files which contain the contents of the standard layout?  Or have you transcoded your media files into a single file format?   Or is it a combination of formats?

 

EDIT: Unraid does not split a single file across more than one disk.  But it can split individual .VOB files of a DVD onto a second disk when the disk gets full.  There is a way to prevent that, but you don't want to implement it unless you truly need it. 

Thanks for reply.

 

Should be ok on this point by the sounds of it. I've ripped and transcoded my Blu-ray's into mkv format single files and the other files are either FLACs and raw photos. 

 

Link to comment
42 minutes ago, 1300GT said:

Thanks for reply.

 

Should be ok on this point by the sounds of it. I've ripped and transcoded my Blu-ray's into mkv format single files and the other files are either FLACs and raw photos. 

 

However, you should set the   'Minimum Free Space' parameter   (  Shares   >>>   share-name    >>>   Share Settings   ) to the size of the largest file that you might write to that share.  Unraid will choke if you try to write a file which is larger than the space available on the disk selected by the 'Allocation method:' setting.   (If you are using very large data disks, this is not normally a problem for a very long time but since you are moving 20TB of data, you might run into it depending on the amount of free space you plan on having when you are finished with phase 1.)

Edited by Frank1940
  • Thanks 1
Link to comment
On 6/26/2020 at 4:27 PM, Frank1940 said:

However, you should set the   'Minimum Free Space' parameter   (  Shares   >>>   share-name    >>>   Share Settings   ) to the size of the largest file that you might write to that share.  Unraid will choke if you try to write a file which is larger than the space available on the disk selected by the 'Allocation method:' setting.   (If you are using very large data disks, this is not normally a problem for a very long time but since you are moving 20TB of data, you might run into it depending on the amount of free space you plan on having when you are finished with phase 1.)

Thanks for the tip, I'll make a note to do this. 

 

Do I just mount the Qnap as a share within UnRaid and let it (probably slowly) copy it all across or is there a program in UnRaid that manages this and would be better to use? 

Link to comment

I am no expert in this area but here are two threads that are active now:

 

     https://forums.unraid.net/topic/94090-unraid-and-rsync/

     https://forums.unraid.net/topic/94101-backup-options-unraid1-to-unraid2/

 

rsync is a standard Linux command.  If you are not familiar with its use, do a bit of Googling with   unraid.net rsync  as the search terms.

 

There is also     Krusader    (Docker app) and    mc     which is built into Unraid.  Both are full feature file managers.

 

I believe that an external server can be mounted onto Unraid using the  Unassigned Devices    plugin. 

 

PS:  You could consider setting up a SMB mapping on a PC and do the copy through it.  There is a bit of extra overhead involved but you would have to test to see how much of a hit you would get. 

 

Because the file overhead (on both ends), large files transfer at a much faster rate than small files.  I would also do the transfer without a Parity disk assigned on the Unraid server.  Having parity assigned will cut the transfer rate in half.  (You will have a backup on the other server in case something goes wrong.)  Don't use a cache drive for any share until you are finished with the initial transfer!

  • Thanks 1
Link to comment
11 hours ago, Frank1940 said:

I am no expert in this area but here are two threads that are active now:

 

     https://forums.unraid.net/topic/94090-unraid-and-rsync/

     https://forums.unraid.net/topic/94101-backup-options-unraid1-to-unraid2/

 

rsync is a standard Linux command.  If you are not familiar with its use, do a bit of Googling with   unraid.net rsync  as the search terms.

 

There is also     Krusader    (Docker app) and    mc     which is built into Unraid.  Both are full feature file managers.

 

I believe that an external server can be mounted onto Unraid using the  Unassigned Devices    plugin. 

 

PS:  You could consider setting up a SMB mapping on a PC and do the copy through it.  There is a bit of extra overhead involved but you would have to test to see how much of a hit you would get. 

 

Because the file overhead (on both ends), large files transfer at a much faster rate than small files.  I would also do the transfer without a Parity disk assigned on the Unraid server.  Having parity assigned will cut the transfer rate in half.  (You will have a backup on the other server in case something goes wrong.)  Don't use a cache drive for any share until you are finished with the initial transfer!

Thanks for the clarification re the parity & cache drive, that fits in with other things I'd heard researching this. 

 

I'm a Linux user so ok with rsync.   I'll have a look at Krusader and do some reading up. I'd mounted my Qnap as an SMB before and transfer rates were mind numbingly slow. 

 

Guess at this point will just have to get stuck in and work out best way.

 

Thanks for the guidance. 

Link to comment
  • JorgeB changed the title to [SOLVED] File transfer process Old NAS to new UnRaid Setup

for 20tb of space... I would suggest the program TeraCopy... it's a windows based program that is meant to move large pieces of data... the nice thing about it is that if a file cant be copied it passes over it and keeps going... so if an error were to happen when copying it wouldnt stop the job...

 

but as you said... SMB transfers are mind numbing...

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.