cinereus Posted April 23, 2020 Share Posted April 23, 2020 I have to seed a new server with 30 TB+. I have a gigabit network connection and 12 TB USB 3.0 attached drives. I realise this will probably take the best part of a week. I looked at cp, rsync, rtrr, advanced copy, smb and rclone. Currently my MO is to copy via rclone (to try and saturate the 1 GbE and hopefully not miss any data that was forgotten or changed in the meantime) via smb with the command run on the target. Simultaneously I'm rcloning to the USB disk at about 100 MB/s and manually copying that over again for a net boost of about 50 MB/s. Hopefully I'm getting close to maxing out the disk read speed but increasing the speed even 10% would make a big difference. What are best practices for doing this with a particular emphasis on not missing anything and speed. Quote Link to comment
testdasi Posted April 23, 2020 Share Posted April 23, 2020 For massive data migration, I remove the parity drive and copy files by disk (so no single disk receives multiple IO streams simultaneously). Obviously no parity protection but I accepted the risks. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.