April 23, 20206 yr I have to seed a new server with 30 TB+. I have a gigabit network connection and 12 TB USB 3.0 attached drives. I realise this will probably take the best part of a week. I looked at cp, rsync, rtrr, advanced copy, smb and rclone. Currently my MO is to copy via rclone (to try and saturate the 1 GbE and hopefully not miss any data that was forgotten or changed in the meantime) via smb with the command run on the target. Simultaneously I'm rcloning to the USB disk at about 100 MB/s and manually copying that over again for a net boost of about 50 MB/s. Hopefully I'm getting close to maxing out the disk read speed but increasing the speed even 10% would make a big difference. What are best practices for doing this with a particular emphasis on not missing anything and speed.
April 23, 20206 yr For massive data migration, I remove the parity drive and copy files by disk (so no single disk receives multiple IO streams simultaneously). Obviously no parity protection but I accepted the risks.
Archived
This topic is now archived and is closed to further replies.