Finally broke down and bought a new unRaid server. It's a Ryzen box, with five 8TB drives. The cheapest 8TB drives I could find were actually external USB drives off newegg. When they arrived, I just popped the cases open and pulled the drives out. I made sure the motherboard had enough sata ports so I wouldn't even need to use my PCI sas card. So I've got two servers hooked up via gigabit ethernet, with a lot of data to transfer. I would rather wait than fuss with hardware plugging in/out etc. So this is going over ethernet.

 

I thought I'd report some speeds here, so you all know what to expect when doing a similar transfer. TLDR; 40Mbps, 3.7TB per 24 hours.

 

For the lowest overhead solution, I made sure that all dockers were shutdown. I installed Nerdtools (here support) that has Screen inside. With Screen, you can log into a terminal, start a screen console, run a really long command, detach from screen, and completely logout. That's necessary in case your session gets interrupted you aren't back and square one.

login: root
screen
<enter> to confirm
rsync -av [email protected]:/mnt/user/Movies /mnt/user/movies
ctrl-a, d (detaches from current screen)
exit

before running this command, you need to ensure you can log into the remote server with ssh, so, one time you'll do

login: root
ssh 192.168.11.52
"Store this computer in the blah blah thingy? yes/no"
<yes>

then you're connected to the other server. You can use that same IP in your commands in the future no problemo.

 

You can check on the command (not really the progress) by logging back into your server, screen -r, will reattach to the previous screen, and you'll see your files flipping by. Then ctrl-a, d, will detach you from the screen, and you can logout and go about your business.

 

There are a few other options in rsync, like --human and --progress (or something to that effect) that will round numbers like 34,556,467,456 bits into MB or GB, and progress will show you percentage on the current file - but even with those it won't show you overall progress; which would be nice.

 

There are some web-gui versions of file management things, this docker krusader that looks terrific, and I installed it. But I don't think it will persist when I logout, or close the browser window - specifically that I can reload it and see where it's at in it's copy progress.

 

I also found a thread somewhere about disabling the parity drive. I've done that, and the speed difference is negligible. I was getting 40Mbps, and now I'm getting something to the tune of 41 Mbps; plus I'll have to let the parity drive rebuild which takes more than a day for these 8TB drives.

 

Enable turbowrite, instructions but present in settings.

 

My recommendation: Screen, rsyc (see above options), all apps and dockers turned off, parity on

 

Hope this helps someone!

Edited October 4, 20178 yr by dkerlee

Hi, do you have Turbo Write enabled?

Whoops, didn't see that one until you mentioned it. Edited above post. Thanks! 

Click the link in my signature to see how to enable Turbo Write.  I would have expected that if you didn't have a Parity drive, or if you use Turbo Write, that with large modern hard drives you should be able to achieve speeds in excess of 50MB/s.  In fact I'd expect something around 70MB/s on average.  With Turbo Write enabled I sometimes (under optimal conditions, like copying a big .mkv) am able to hit full line speeds of 112MB/s.  It all depends, of course - copying lots of small files is a lot slower than larger ones.  Still, either dropping Parity or enabling Turbo Write usually results in noticeable performance increases, and 40MB/s is more typical of parity enabled, non Turbo Write performance.

2 minutes ago, tdallen said:

Click the link in my signature to see how to enable Turbo Write

For some reason, your signature doesn't show up. I'm on Chrome and Android. 

Go to your forum Account Settings, there's global switch to turn on Signatures and for some reason it's off by default.