Better way to copy files?

[levster]

I have two unRaid servers and wish to backup files on one of the drives to the other server, approximately 4tb in size total. What is the best way to do it?

 

I have a precleared 4tb drive. Should I place it into the original unit,  copy directly on to it and then move over the disk? Can I then simply make that drive a part off the array or will I have to copy the files from it to a drive that is already a part of the array?

 

Thanks!

[John_M]

You possibly could but it's asking for all sorts of problems. If I was going to use a separate hard disk to do it I wouldn't introduce it into the array but use the Unassigned Devices plugin to format and mount it outside the array. However, I personally wouldn't do that either. I'd use rsync to copy the files across the network. Logged into the destination server I'd use something like this:

 

rsync -avzAXP -e ssh root@source-ip-address:/mnt/user/Source-share/* /mnt/user/Dest-share

 

to copy an entire user share from one server to the other.

[levster]

You possibly could but it's asking for all sorts of problems. If I was going to use a separate hard disk to do it I wouldn't introduce it into the array but use the Unassigned Devices plugin to format and mount it outside the array. However, I personally wouldn't do that either. I'd use rsync to copy the files across the network. Logged into the destination server I'd use something like this:

 

rsync -avzAXP -e ssh root@source-ip-address:/mnt/user/Source-share/* /mnt/user/Dest-share

 

to copy an entire user share from one server to the other.

 

Thank you. I am copying the files, but thought that it maybe faster if I mounted internally, copied and then incorporated the disk into the array. Guess, that could not be done.

[gundamguy]

You possibly could but it's asking for all sorts of problems. If I was going to use a separate hard disk to do it I wouldn't introduce it into the array but use the Unassigned Devices plugin to format and mount it outside the array. However, I personally wouldn't do that either. I'd use rsync to copy the files across the network. Logged into the destination server I'd use something like this:

 

rsync -avzAXP -e ssh root@source-ip-address:/mnt/user/Source-share/* /mnt/user/Dest-share

 

to copy an entire user share from one server to the other.

 

Thank you. I am copying the files, but thought that it maybe faster if I mounted internally, copied and then incorporated the disk into the array. Guess, that could not be done.

 

I think it can be done, but I don't know if it would be faster since I think you would need a third disk to avoid formatting issues... (unRAID will format disks before adding them to the array for the first time)

 

Process would be like this

1: Mount Disk 1 in Array 1 using Unassigned Devices Plugin (already formatted in a way that works with Unassigned Devices...)

2: Copy Share X or Disk X from Array 1 to Disk 1

3: Un-mount Disk 1 from Array 1

4: Mount Disk 1 in Array 2 using Unassigned Devices Plugin

5: Copy from Disk 1 to Share X or Disk X in Array 2

 

... there might be a way to streamline this process.

 

 

 

[levster]

You possibly could but it's asking for all sorts of problems. If I was going to use a separate hard disk to do it I wouldn't introduce it into the array but use the Unassigned Devices plugin to format and mount it outside the array. However, I personally wouldn't do that either. I'd use rsync to copy the files across the network. Logged into the destination server I'd use something like this:

 

rsync -avzAXP -e ssh root@source-ip-address:/mnt/user/Source-share/* /mnt/user/Dest-share

 

to copy an entire user share from one server to the other.

 

Thank you. I am copying the files, but thought that it maybe faster if I mounted internally, copied and then incorporated the disk into the array. Guess, that could not be done.

 

I think it can be done, but I don't know if it would be faster since I think you would need a third disk to avoid formatting issues... (unRAID will format disks before adding them to the array for the first time)

 

Process would be like this

1: Mount Disk 1 in Array 1 using Unassigned Devices Plugin (already formatted in a way that works with Unassigned Devices...)

2: Copy Share X or Disk X from Array 1 to Disk 1

3: Un-mount Disk 1 from Array 1

4: Mount Disk 1 in Array 2 using Unassigned Devices Plugin

5: Copy from Disk 1 to Share X or Disk X in Array 2

 

... there might be a way to streamline this process.

 

Would be interesting to see what the time difference would be, considering I'm getting about 15MB/s copying  tower1 to tower2 via a switch.

[tdallen]

Would be interesting to see what the time difference would be, considering I'm getting about 15MB/s copying  tower1 to tower2 via a switch.

15 MB/s is pretty slow.  Are you working with all gigabit ethernet connections?

[levster]

Would be interesting to see what the time difference would be, considering I'm getting about 15MB/s copying  tower1 to tower2 via a switch.

15 MB/s is pretty slow.  Are you working with all gigabit ethernet connections?

 

I am. The switch lights are also confirming 1GB connection on both ports.

[tdallen]

I get consistent speeds in excess of 35 MB/s writing to my parity protected array over the network or from my cache drive (I'm not caching writes, though).  Do you have any older/slower hardware/drives on your servers that might be causing a bottleneck?

[levster]

I get consistent speeds in excess of 35 MB/s writing to my parity protected array over the network or from my cache drive (I'm not caching writes, though).  Do you have any older/slower hardware/drives on your servers that might be causing a bottleneck?

 

Writing to the array, also with or without the cache disk, I get much faster transfer rates. It's this unraid to unraid transfer for some reason is slow. I'll check the cables. Maybe they have something to do with this.

[cyrnel]

Just for reference, my brand new 6.1 system with a Xeon 1271v3, 16GB, a black parity drive and 5900rpm data drives receives long files at about 45-55MB/s with rsync over a smb connection. The same operation jumps to 55-60MB/s if the temp files are built on an ssd.

 

The same system rebuilds parity at about 150MB/s until the the last few percent where it slows to ~80MB/s.

 

I  use rsync unless the other system is being scavenged, in which case I pull and attach drives directly.

[levster]

I get consistent speeds in excess of 35 MB/s writing to my parity protected array over the network or from my cache drive (I'm not caching writes, though).  Do you have any older/slower hardware/drives on your servers that might be causing a bottleneck?

 

Writing to the array, also with or without the cache disk, I get much faster transfer rates. It's this unraid to unraid transfer for some reason is slow. I'll check the cables. Maybe they have something to do with this.

 

Changed cables and no difference. Still getting consistent 15MB/s.

[cyrnel]

Thinking back now... I do recall it's faster to add your parity disk after you copy files. Makes ~10MB/s difference on this system which could add up over multiple TB copies.

[JorgeB]

Changed cables and no difference. Still getting consistent 15MB/s.

 

I just tried this as I’m in the middle of backing up a 20TB server, and this method is much slower than using a windows pc, my backup server is parityless, I was getting between 80/90MB/s using windows as an intermediary and only get 16MB/s using ssh rsync, maybe some configuration needs to be changed for better speed.

[levster]

Changed cables and no difference. Still getting consistent 15MB/s.

 

I just tried this as I’m in the middle of backing up a 20TB server, and this method is much slower than using a windows pc, my backup server is parityless, I was getting between 80/90MB/s using windows as an intermediary and only get 16MB/s using ssh rsync, maybe some configuration needs to be changed for better speed.

 

Just tried copying from Win to unraid, with parity, and am getting 45 MB/s.

[tdallen]

Just tried copying from Win to unraid, with parity, and am getting 45 MB/s.

That's within the range for expected throughput. 

 

Maybe rsync is doing some additional read after write style validation?  Not an rsync expert...

[Helmonder]

Thinking back now... I do recall it's faster to add your parity disk after you copy files. Makes ~10MB/s difference on this system which could add up over multiple TB copies.

 

That would make an enormous difference... Without parity you should be able to get transfer speeds in exces of 100MBs

[SpaceInvaderOne]

That would make an enormous difference... Without parity you should be able to get transfer speeds in exces of 100MBs

 

Yes I recently had to copy alot of data from an old windows server that we were decommissioning. I had no parity drive assigned in the unraid server. I connected the windows server drive as an unassigned drive  then used the krusader docker to copy the data to the array at speeds of 120 to 130mbs. After the copy I left the drive in the unraid and then assigned it as the parity for the system

[JorgeB]

After looking up the rsync switchs used I’ve found the culprit:

 

-z, --compress

With this option, rsync compresses any data from the source file(s) which it sends to the destination machine. This option is useful on slow links. The compression method used is the same method that gzip uses. 

 

This is good for low bandwaith connections, for gigabit remove this switch, command should look kile this:

 

rsync -avAXP -e ssh root@source-ip-address:/mnt/user/Source-share/* /mnt/user/Dest-share

 

[levster]

After looking up the rsync switchs used I’ve found the culprit:

 

-z, --compress

With this option, rsync compresses any data from the source file(s) which it sends to the destination machine. This option is useful on slow links. The compression method used is the same method that gzip uses. 

 

This is good for low bandwaith connections, for gigabit remove this switch, command should look kile this:

 

rsync -avAXP -e ssh root@source-ip-address:/mnt/user/Source-share/* /mnt/user/Dest-share

 

Thanks. I'll give it a shot.