Best way to copy/sync large data in high speed.


frodr

Recommended Posts

What is the recommended way to copy/sync large amount data (4TB) between Unraid server and a Qnap TVS471 NAS with best possible speed. Syncing from Unraid to Nas (WD Red 6TB in Raid1) using Resilio-Sync. Effective speed is about 10MB/s, which is......

 

Is there a way to do this to get close to 100MB/s? 

 

// Frode

 

Supermicro X11DAI-N, version 1.02 | AMI 3.1 | dual Xeon Silver 4116 | 64 MB RAM | 4 x Corsair Force MP510 (960GB) NvME ssd on Asus Hyper M.2x16 V2 l Unraid 6.72 l Quadro RTX4000 for Plex docker l RTX 2080 for Win kvm l Samsung 970 Pro as second ssd for Win Pro kvm l Asmedia PCIe USB card.

Link to comment
6 hours ago, frodr said:

What is the recommended way to copy/sync large amount data (4TB) between Unraid server and a Qnap TVS471 NAS with best possible speed. Syncing from Unraid to Nas (WD Red 6TB in Raid1) using Resilio-Sync. Effective speed is about 10MB/s, which is......

 

Is there a way to do this to get close to 100MB/s? 

 

// Frode

 

Supermicro X11DAI-N, version 1.02 | AMI 3.1 | dual Xeon Silver 4116 | 64 MB RAM | 4 x Corsair Force MP510 (960GB) NvME ssd on Asus Hyper M.2x16 V2 l Unraid 6.72 l Quadro RTX4000 for Plex docker l RTX 2080 for Win kvm l Samsung 970 Pro as second ssd for Win Pro kvm l Asmedia PCIe USB card.

The only fast way would be a 10GBit-Link between both machines...

Another question would be: How fast are your HDDs? They normaly max out at ~180MB/s

But in your case it seems Resilio-Sync is the limiting factor...

 

Edited by Zonediver
Link to comment

It seems that Resilio-Sync is slow. I guess a lot of initial indexing reduced the actual speed. I am considering 10G cards, but will like to understand if that's the limiting factor. Disks are, but the Array more so. What's the highest read speed of the array?

 

I have added the NAS as a smb share, and planned to copy with Krusader The smb share overview actual free size, but Krusader says only 32GB free. 

Link to comment
2 hours ago, frodr said:

Changing from binhex-Krusader to Krusader docker, fixed the problem not seeing the shares free space. 

 

Copying with Krusader show 15-20GB speed, so do ftp with FileZilla. Is the limitation in speed related to the smb share?

Can't answer this question because i use non of this things...

But i would say no. When i copy from my PC to unraid over the 10GBit-Link using SMB, i reach up to 560MB/s - the maximum what the SSDs can provide. The link itself is faster of corse - up to 1200MB/s.

Edited by Zonediver
Link to comment
9 hours ago, frodr said:

Changing from binhex-Krusader to Krusader docker, fixed the problem not seeing the shares free space. 

 

Copying with Krusader show 15-20GB speed, so do ftp with FileZilla. Is the limitation in speed related to the smb share?

I can't speak to Krusader as I haven't used it. Dolphin seems to greatly out perform Filezilla on my rig. I've done some testing and something is limiting my Filezilla to about 13MB/s (see image) this test is moving data within unraid from one share to another, I'm not sure why it isn't faster. Honestly I only use it out of laziness, if I'm already moving data from my seedbox to unraid with Filezilla then I will also use it to move internal folders as needed rather than opening dolphin. Note that internal moves are pretty rare for me... Next I tried Dolphin which is what I used to load my array for the first time, running the same exact test I used with Filezilla I got about 48MB/s. I'd say that's an expected speed considering my rig is really old. I may be wrong but I'd think you should get somewhere in the same neighborhood through USB 2.0, while not the 100 MB/s you're looking for it's better than the 15-20 you seem to be getting now.

 

filezilla.PNG

dolphin.PNG

Edited by Dissones4U
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.