June 5, 20224 yr hi all, i'm currently copying from one Unraid to another over 24TB of datas since 7days now (still currently copying 24h/7)... i was wondering what is the best was to copy large amount of datas from one Unraid to another... - i'm copying folders from my PC with source/destination with the network path mount (ie: \\server1\share1\*). - tryied with FreeFileCopy... workes but way too slow (copying on an average of 20-40kb/s). - tryied with TeraCopy... great since it's shows what files are now copied, but still slow. - tryied with robotcopy... workes well best one for far, but still way slow (can't know the datatransfer rate since it's a dos command). - tryied with default window copy... wrost one since it keeps asking if i want to copy/overwrite - and it's slow. - all my NIC are at 1000. Strangely, when i copy a file from my PC to one of the Unraid server, the transfer is around 115mb/s... wich is way faster that copying from one unraid to another... any ideas? Thx in advance!
June 5, 20224 yr Solution I don't venture much into different copying techniques but I always use this. Logged into destination server rsync -avzP --progress root@sourceIP:/mnt/user/WHATEVER_SHARE/ /mnt/user/WHATEVER_SHARE/ And any failures can easily be picked back up by running the command again. Once its done I would recommend running again to make sure no data changed on the source server in the meantime.
June 5, 20224 yr Community Expert Have you read this section of the Manual? https://wiki.unraid.net/Manual/Storage_Management#Array_Write_Modes
June 5, 20224 yr Community Expert One more thing, if you are just copying data to the new array, you have a backup copy of your data. You could remove the parity disk until you have finished the copy. The connect it back up and built parity at that point. That would allow transfer speeds over 100MB/s for large files (say, >10GB). Small files have a lot of file/disk overheard and depending on their size, it could go as low as 10MB/s. (Be a bit careful on the readings you are getting from the copy program. They have always seemed to be a bit suspect...)
June 5, 20224 yr Author 3 hours ago, jmztaylor said: I don't venture much into different copying techniques but I always use this. Logged into destination server rsync -avzP --progress root@sourceIP:/mnt/user/WHATEVER_SHARE/ /mnt/user/WHATEVER_SHARE/ And any failures can easily be picked back up by running the command again. Once its done I would recommend running again to make sure no data changed on the source server in the meantime. Works like a charm thank you !
June 5, 20224 yr Author 3 hours ago, Frank1940 said: One more thing, if you are just copying data to the new array, you have a backup copy of your data. You could remove the parity disk until you have finished the copy. The connect it back up and built parity at that point. That would allow transfer speeds over 100MB/s for large files (say, >10GB). Small files have a lot of file/disk overheard and depending on their size, it could go as low as 10MB/s. (Be a bit careful on the readings you are getting from the copy program. They have always seemed to be a bit suspect...) That's good to know, thank for the input! In my case, yes i want to copy large amount of data but the folders are re-organised... So it's not a mirror.
December 3, 20232 yr Hi, I'm trying to move data between two unraid servers. I've tried this command (rsync -avzP --progress root@sourceIP:/mnt/user/WHATEVER_SHARE/ /mnt/user/WHATEVER_SHARE/) but it is not working for me. This is the response: root@XXXXXXXXXX:~# rsync -avzP --progress [email protected]:/mnt/user/MEDIA/ /mnt/user/Media/ ssh: connect to host 192.168.1.150 port 22: Connection refused rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] rsync error: unexplained error (code 255) at io.c(231) [Receiver=3.2.7] Can you point me in the right direction? What I'm doing wrong? Double checked share's names and they're correct. Thx,
December 5, 20232 yr There it is. Does it matter that it is not asking me for the root password at the source server: 192.168.1.150 ? root@XXXXXXXX:~# ssh -vvv 192.168.1.150 OpenSSH_9.3p2, OpenSSL 1.1.1v 1 Aug 2023 debug1: Reading configuration data /etc/ssh/ssh_config debug2: resolve_canonicalize: hostname 192.168.1.150 is address debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/root/.ssh/known_hosts' debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/root/.ssh/known_hosts2' debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling debug3: ssh_connect_direct: entering debug1: Connecting to 192.168.1.150 [192.168.1.150] port 22. debug3: set_sock_tos: set socket 3 IP_TOS 0x48 debug1: connect to address 192.168.1.150 port 22: Connection refused ssh: connect to host 192.168.1.150 port 22: Connection refused root@XXXXXXXXX:~#
December 5, 20232 yr I just take out my HDD, put into another server, use Unassigned Devices mount the HDD, copy away. I am not joking, this was how I transfer files between my server and my friend's server.
December 16, 20232 yr On 6/5/2022 at 12:26 PM, Frank1940 said: Have you read this section of the Manual? https://wiki.unraid.net/Manual/Storage_Management#Array_Write_Modes https://docs.unraid.net/unraid-os/manual/storage-management/#array-write-modes
December 17, 20232 yr Community Expert On 12/4/2023 at 10:05 PM, xyzeratul said: I just take out my HDD, put into another server, use Unassigned Devices mount the HDD, copy away. I am not joking, this was how I transfer files between my server and my friend's server. This technique could invalidate parity on your server unless you mount the disk read-only.
December 17, 20232 yr I'm doing a large copy, 18TB. I was thinking it would be fastest to take the parity drive out of the array, do all the writes, then rebuild the parity. Hmm looks like even at 110MB/s it will take 1 day 21 hours, that's not too bad. Any other advice there? thanks!! Derp that's the saturation for gigabit ethernet. Looks like it's a hardware swap that needs to happen, 10G ethernet to make it any faster. 1.8 days it is! Can do. Oops, I need to turn my cache drive off. This will change things again. "Turbo write mode" Settings > Disk Settings > Tunable (md_write_method) options are Auto, read/modify/write, or reconstruct write. Reconstruct write is the "turbo mode" I'll do a short test without the cache drive, see where we land. https://wintelguy.com/transfertimecalc.pl https://www.gbmb.org/mbps-to-mbs Edited December 17, 20232 yr by rutherford
December 19, 20232 yr On 12/17/2023 at 8:20 AM, trurl said: This technique could invalidate parity on your server unless you mount the disk read-only. I forgot to mention that the array has no parity drive anyway, because nothing important on the drives.
December 19, 20232 yr Community Expert 7 hours ago, xyzeratul said: I forgot to mention that the array has no parity drive anyway, because nothing important on the drives. Probably bad advice for other users without that caveat, huh?
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.