Quickest way to transfer files

[matt3]

Hi all,

 

I currently have two 8TB hdds installed in my unraid machine and I have two in my computer that I need to transfer all the files to and then format those to add to the unraid machine.

 

What I would like to know is what is the quickest way to move everything across. There's about 11TB of data. I have setup the share with Windows but this looks like it will take forever to transfer everything across. Is there a better way to do it?

 

I'm new to unraid so don't know everything to do with it so go easy on me 

Edited April 26, 2022 by matt3

[Frank1940]

You can unassign the Parity drive(s).  This will speed up the write speed on the Unraid side by a factor of (approximately) two.  (If you do this, copy the files over first. Assign the Parity drive.  Build parity to make sure everything is alright on the Unraid side before doing anything that will delete the files from those drives currently in the PC.) 

 

I also believe you can use the 'Unassigned Drives' plugin.  It may need an additional plugin to read NTFS file system.   That would eliminate the network speed bottleneck.  But it requires removing the disks from the PC and mounting them in the server.

[matt3]

5 hours ago, Frank1940 said:

You can unassign the Parity drive(s).  This will speed up the write speed on the Unraid side by a factor of (approximately) two.  (If you do this, copy the files over first. Assign the Parity drive.  Build parity to make sure everything is alright on the Unraid side before doing anything that will delete the files from those drives currently in the PC.) 

 

If I unassign the parity drive, would I then have to run the parity again? It took about 10 hours to complete so wouldn't want to do it again.

 

5 hours ago, Frank1940 said:

I also believe you can use the 'Unassigned Drives' plugin.  It may need an additional plugin to read NTFS file system.   That would eliminate the network speed bottleneck.  But it requires removing the disks from the PC and mounting them in the server.

 

I've no problem removing the drives and putting them into the server but how would I then transfer everything across to the new drives without having to wipe or format the drives before.

 

Thanks

[JonathanM]

What speeds are you averaging for large (multi GB) files?

[Frank1940]

4 hours ago, matt3 said:

If I unassign the parity drive, would I then have to run the parity again? It took about 10 hours to complete so wouldn't want to do it again.

Yes, but if you are transferring close to 16TB of data, the increased speed even with the 10 hour parity rebuild could result in a smaller total time required to complete the transfer.  You would have to look at the numbers as @JonathanM suggests.

 

4 hours ago, matt3 said:

I've no problem removing the drives and putting them into the server but how would I then transfer everything across to the new drives without having to wipe or format the drives before.

 

You mount them using the Unassigned Devices plugin.  Then copy the the files to the array using the command line, or the included mc (Midnight Commander) file manager, or the Krusader Docker (another file manager).  (I have not had Unassigned Devices installed on either of my servers for several years.  So I can't really provide you with complete step-by-step instructions.  However, it is available through the Unraid Apps manager and it has a support thread...)

 

Moving TB's of data is not a task that can be done quickly so put that thought behind you.  Max sustained speed will be in the 150MB/s range and goes south from there.   There are many factors involved including such things as SMR vs CMR, platter rotation speed, file size, etc., and this ignores the network speed bottleneck...

Edited April 27, 2022 by Frank1940
TYPO

[matt3]

10 hours ago, JonathanM said:

What speeds are you averaging for large (multi GB) files?

 

It starts of between 15 MB/s and 22 MB/s and then drops down between 5 MB/s and 10 MB/s. This is on WiFi as well.

 

6 hours ago, Frank1940 said:

Yes, but if you are transferring close to 16TB of data, the increased speed even with the 10 hour parity rebuild could result in a smaller total time required to complete the transfer.  You would have to look at the numbers as @JonathanM suggests.

 

Ah yeah! I didn't think of it that way.

 

6 hours ago, Frank1940 said:

You mount them using the Unassigned Devices plugin.  Then copy the the files to the array using the command line, or the included mc (Midnight Commander) file manager, or the Krusader Docker (another file manager).  (I have not had Unassigned Devices installed on either of my servers for several years.  So I can't really provide you with complete step-by-step instructions.  However, it is available through the Unraid Apps manager and it has a support thread...)

 

Moving TB's of data is not a task that can be done quickly so put that thought behind you.  Max sustained speed will be in the 150MB/s range and goes south from there.   There are many factors involved including such things as SMR vs CMR, platter rotation speed, file size, etc., and this ignores the network speed bottleneck...

 

I would imagine this will be the quicker way of the two as it wouldn't have to use the internet to transfer the files. I would probably have to use the file managers as I'm not sure how to transfer with command line.

[neural]

Here is what we did and its slow:

 

Setup

1. Array setup with one parity drive set to "Turbo" mode
2. A cache (NVMe 512 GB) for a share called "LargeFiles"
3. Attached a HDD 16TB drive, mounted it and use Krusader to migrate from direct connected HDD (mounted) to the share called LargeFiles

 

We tested with a folder of 200 Gb smaller documents 

We tested with large 1 GB files 

We tested with mixed large and small

.... all are slow

 

Is it a best practice to use the Network as the main conduit to migrate data into the Array ?

 

Txs

[itimpi]

1 hour ago, neural said:

all are slow

You did not define what you meant by ‘slow’ on this context     You also did not specify what you had for the USD Cache setting for this share.

 

Plugging a drive into the server and mounting it via the Unassigned Devices plugin is the commonest (and normally fastest) way to transfer large amounts of data.   Doing it over the network is almost invariably slower but can be more convenient.

[matt3]

11 hours ago, neural said:

1. Array setup with one parity drive set to "Turbo" mode

 

What's turbo mode for parity? Does it help with the transfer speed?

 

11 hours ago, neural said:

1. Attached a HDD 16TB drive, mounted it and use Krusader to migrate from direct connected HDD (mounted) to the share called LargeFiles

 

All my drives are 8TB. I actually need to purchase 2 more so I can get all info transferred across.

 

9 hours ago, itimpi said:

Plugging a drive into the server and mounting it via the Unassigned Devices plugin is the commonest (and normally fastest) way to transfer large amounts of data.   Doing it over the network is almost invariably slower but can be more convenient.

 

I'm currently transferring over the network using a terrible WiFi USB as my computer is too old for WiFi built in . I've been using Terracopy and it seems to be quicker but still not super fast.

[itimpi]

1 hour ago, matt3 said:

What's turbo mode for parity? Does it help with the transfer speed?

Turbo Mode is what is described here in the online documentation that can be accessed via the Manual link at the bottom of the GUI.

 

1 hour ago, matt3 said:

I'm currently transferring over the network using a terrible WiFi USB as my computer is too old for WiFi built in . I've been using Terracopy and it seems to be quicker but still not super fast.

 

Getting to get decent speeds over WiFi is very unlikely to happen.

[Frank1940]

3 hours ago, matt3 said:

I'm currently transferring over the network using a terrible WiFi USB as my computer is too old for WiFi built in . I've been using Terracopy and it seems to be quicker but still not super fast.

 

2 hours ago, itimpi said:

Getting to get decent speeds over WiFi is very unlikely to happen.

 

Move the client PC temporarily closer to the router/switch.  If that is not possible, look at a temporary Ethernet cable solution.  Flat cable can be a solution where round cable is a problem due to bulk.

 

     https://www.amazon.com/AmazonBasics-Cat-6-Gigabit-Ethernet-Internet/dp/B089MF22S6/ref=sr_1_6?crid=1KJ9F183YHBNF&keywords=flat%2Bcat6%2Bethernet%2Bcable&qid=1651146783&sprefix=flat%2Bcat%2Caps%2C95&sr=8-6&th=1

 

This type of cable will lie flat under carpet to cross foot traffic areas.  IF hard surface floors are issue, consider using a throw rug to eliminate trip hazard.

[matt3]

2 hours ago, itimpi said:

Turbo Mode is what is described here in the online documentation that can be accessed via the Manual link at the bottom of the GUI.

Thanks, I'll take a look

 

2 hours ago, itimpi said:

Getting to get decent speeds over WiFi is very unlikely to happen.

Yeah, I've been connected to Ethernet for years and now moved and the router is downstairs so having to use WiFi.

 

12 minutes ago, Frank1940 said:

Move the client PC temporarily closer to the router/switch.  If that is not possible, look at a temporary Ethernet cable solution.  Flat cable can be a solution where round cable is a problem due to bulk.

 

     https://www.amazon.com/AmazonBasics-Cat-6-Gigabit-Ethernet-Internet/dp/B089MF22S6/ref=sr_1_6?crid=1KJ9F183YHBNF&keywords=flat%2Bcat6%2Bethernet%2Bcable&qid=1651146783&sprefix=flat%2Bcat%2Caps%2C95&sr=8-6&th=1

 

This type of cable will lie flat under carpet to cross foot traffic areas.  IF hard surface floors are issue, consider using a throw rug to eliminate trip hazard.

 

I could try and move the PC next to the router for the transfer but wouldn't then be able to use the computer for anything else.

I'm guessing I can't take the HDD that has been in the unraid server and put it into my Windows computer to transfer everything over and then put it back?

[itimpi]

26 minutes ago, matt3 said:

I'm guessing I can't take the HDD that has been in the unraid server and put it into my Windows computer to transfer everything over and then put it back?

Assuming that you mean a disk already in the Unraid array, not unless you have software on your PC that can read the Linux file system (probably XFS which is the default for array disks).  
 

Having said that you could consider booting the PC off a Linux ‘Live’ dvd and use that to copy the files onto the dtive as that would understand the XFS file system format.   If going this route it is important that the disk is partitioned and formatted on the Unraid system to ensure it conforms to what Unraid expects.   After moving the drive back to Unraid you would have to rebuild parity (if you have a parity drive).

 

As was mentioned if you have a removable drive then it can be plugged into the Unraid server and mounted using the Unassigned Devices plugin (which supports all common PC/Mac file formats in addition to Linux ones).  

[matt3]

26 minutes ago, itimpi said:

Assuming that you mean a disk already in the Unraid array, not unless you have software on your PC that can read the Linux file system (probably XFS which is the default for array disks).  
 

Having said that you could consider booting the PC off a Linux ‘Live’ dvd and use that to copy the files onto the dtive as that would understand the XFS file system format.   If going this route it is important that the disk is partitioned and formatted on the Unraid system to ensure it conforms to what Unraid expects.   After moving the drive back to Unraid you would have to rebuild parity (if you have a parity drive).

 

That's what I meant, yes. I've not used Linux before so that might be a bit too much for me.

 

27 minutes ago, itimpi said:

As was mentioned if you have a removable drive then it can be plugged into the Unraid server and mounted using the Unassigned Devices plugin (which supports all common PC/Mac file formats in addition to Linux ones).

 

Are the file explorers on unraid similar to Windows or Mac where I can just drag and drop the files?

[Frank1940]

1 hour ago, matt3 said:

 

Are the file explorers on unraid similar to Windows or Mac where I can just drag and drop the files?

 

Yes and No.  They dual panel file managers and you do most file type operations between the two panels.  Krusader is the more powerful of the two as mc has a text based interface rather than a true GUI.  You can find YouTube videos on both of them by Googling  'midnight commander'  and     'Krusader'.   Those videos will show in a few minutes what would take a half-hour for me to write.

[matt3]

15 hours ago, Frank1940 said:

 

Yes and No.  They dual panel file managers and you do most file type operations between the two panels.  Krusader is the more powerful of the two as mc has a text based interface rather than a true GUI.  You can find YouTube videos on both of them by Googling  'midnight commander'  and     'Krusader'.   Those videos will show in a few minutes what would take a half-hour for me to write.

 

Thanks I'll have a look at them and see which one I think looks best for me.

[matt3]

Krusader looks good! Just watched this video on YouTube and it's very similar to an FTP client so that's great!

 

Thanks for letting me know about it.