Migration of all data to a new Server?

[Zonediver]

Hi folks,

i need some advice to transfer the complete data of an existing unraid-server to a new one.

Data-content is about 35,78TB.

Can someone give me a recommendation, how to do this task? Is there a tool or something else to do this easily?

Thanks in advance

Edited May 28, 2018 by Zonediver

[pwm]

Just a couple of questions - someone else might add more.

 

Are you switching size of data disks, and is it important that the individual files/directories ends up identically distributed on the new disks?

 

Do you have concurrent access to both new and old machine?

 

Do you need full access to the RAID while copying? And if so - can you manage with read-protected access while the copy runs?

 

What file systems do you use on the old drives? Any changes for the new drives?

 

Do you care enough to do any copy complete with file integrity check to verify that the destination files has same checksum as the source files?

[Zonediver]

54 minutes ago, pwm said:

Just a couple of questions - someone else might add more.

Are you switching size of data disks, and is it important that the individual files/directories ends up identically distributed on the new disks?

Do you have concurrent access to both new and old machine?

Do you need full access to the RAID while copying? And if so - can you manage with read-protected access while the copy runs?

What file systems do you use on the old drives? Any changes for the new drives?

Do you care enough to do any copy complete with file integrity check to verify that the destination files has same checksum as the source files?

 

Hi pwm and thanks for your reply.

 

1) the new server will have less disk than the old one - only 6x 10TB WD-Red (5x Data, 1x parity).

2) identical distribution is not necessary.

3) The new server is only a backup-solution to store the data on it - no access needed for plex or something else (only for me).

4) All drives on the old server are XFS formated.

5) File integrity check is not needed as long as the parity is ok

Edited May 28, 2018 by Zonediver

[pwm]

If copying over the network, then I would most probably go for rsync. If you get 100 MB/s transfer rate that's 360 GB/hour so about 100 hours or 4 days. Way longer if you have lots of tiny files.

If you are in a hurry and can locate a second network interface for each machine, you could perform two copy operations at the same time for twice the speed - assuming you wait with building parity until the copying is done.

 

One issue with rsync is that it creates all destination directories at the start. But since you can fit three 3TB disks or two 4TB disks on each 10TB disk that shouldn't be an issue. The main problem is when people uses rsync and copy to a user share - then the split levels will not work correctly because the directories gets created on the first drive before it receives any files.

 

The other option would be if you have one or more spare connectors on the old machine. Then copy disk-to-disk locally - for example with Krusader. But you would still need to copy concurrently to two target disks to get much better copy speed than 4 days. And after you have filled the data disks you would have to move them to the new machine and then build parity. Doing it the other way - connecting source disks to the new machine would invalidate your parity unless the drives are explicitly mounted read-only.

 

What I meant about file integrity was if you wanted to compute hash files on the original machine before you start to copy. Then copy these hash files together with the data files to the new machine and validate all hashes. This is basically the only way to verify that each individual file has been copied without corruption - any corruption during the copying will be duplicated in the parity so you can't rely on correct parity to verify the transfer.

[Zonediver]

Again: "how" can i do this? Over the console? by hand? How?

Is there a tool to copy "all" Data from one server to an other?

Edited May 28, 2018 by Zonediver

[pwm]

I don't think there is any good tool to copy all data from one server to another while spanning all disks.

 

The closest you get is probably to copy each user share in sequence and have the destination machine spread the files according to configured split levels.

 

 

 

This should be a reasonably good thread if you go the rsync route.

Rsync is a command line tool - more or less a command-line recursive directory tree copy on steroids. You perform the copy over the network (hence the need to be able to do a ssh-connection from one machine to the other) and rsync also has the ability to restart the copy if it gets aborted by one machine being restarted or similar.

 

 

 

If connecting source/destination drives to same machine to copy, then alas this video is slightly off because the Krusader docker in the video has been deprecated. But it isn't too hard to install the Binhex docker instead.

Krusader is GUI where you browse source and target directories in left/right pane of window and then specify what content you want copied between the two panes. Krusader is very much based on the concept of the 30+ years old Norton Commander MS-DOS application. But the concept still works quite well.

 

[Zonediver]

I thought so - thanks for your help. Then I will deal with rsync and copy the shares one by one, keeping an eye on the splitlevel

[pwm]

1 hour ago, Zonediver said:

I thought so - thanks for your help. Then I will deal with rsync and copy the shares one by one, keeping an eye on the splitlevel

 

If you copy disk to disk, then you don't need to worry about the issue with rsync trying to overflow the first disk because it creates all the directories first.

 

You just have to start one rsync copy for every source disk and select any destination disk that has enough free space for the full source disk.