Need guidance for merging two large arrays into one

[oliver]

Asking for feedback for merging two large Unraid Arrays into one.  

Array # 1 - 
176tb, made of 8tb drives. 2 parity.  
1 tb SSD cache
No docker/no VM's 
This is primary an archival array.  Data is written to it once and rarely ever used.

 

 

Array # 2 - 
36TB, made of 18tb drives.  2 parity.

Cache pool of 1 SSD and 1 NVME, mirrored.  
Running Docker containers and VM's.

This is my main array and the one I want to be set as the primary new array.  I am going to add a bunch of 18tb drives and move over contents from #1 to a new share on this array.  The docker containers on this array are used quite a bit so I would prefer to minimize downtime to the array.

The massive amount of volume in array 1 is the main factor here.  

 

Approach 1 - Migrate direct from array 1 to array 2

1. Add the new drives to array two
2. Clear them 
3. Move the data from array #1

• What do I do with parity?  Leave my array with no redundancy? Or should I leave parity on since I'd have to do a rebuild at the end anyways.  
• Is it faster to turn off parity entirely and then rebuild it rather than leave it on the whole time?
• Also a con of this approach is degraded performance.  I don't really read/write intensive ops to Array #2, as long as Plex will still function, I think I should be ok?

Approach 2 - Setup a new (temporary) array

1. Setup a brand new array
2. Move over data from Array #1 to these disks
3. Import these disks to array #2
4. Rebuild parity

• Downside of this is I need new hardware to setup this temporary array but I think I can manage.  
• Also, how would "import" a disk from one array to another?  Do I just stop the array and it will let me add the disk?  Also note that both array 1 and 2 are encrypted XFS shares, but have the same password.
• Also, presumably the share names need to be identical from the temporary array and the main array for folder structures to work?  
• Can I use a trial version of  unraid for this?

 

Other questions - 

• What's the best way to actually move the massive amount of data from array #1?  I'm not in a huge hurry, I know it will take time, weeks possibly.
  • Doing it disk by disk is not practical since there's 22 drives or so and I also would prefer to have parity in place since im doing full reads on a lot of older drives.
  • This leaves network transfer as the only viable option.  Software wise, how to handle this?  Just load up Krusader via docker and send 170tb over SMB?  How gracefully would it handle errors?  Is there a better option?  
    • I have multiple NIC's, to avoid bottlenecking traffic, should I dedicated a private network between the machines?  I have a 2.5GBE network.

 

 

Other considerations:
Both arrays are xfs-encrypted however have the same password.
I would like to use the license from the #1 array, so I need to swap it with the license on array #2 when it's all done.  Is this doable?

Edited February 25 by oliver

[JorgeB]

Why not not just do a new config with all devices you want and resync parity?

[oliver]

1 hour ago, JorgeB said:

Why not not just do a new config with all devices you want and resync parity?

 

The drives in the 170TB array are not to be included in the final array, sorry I should have emphasized that better.  The new array will contain a smaller amount of drives with larger capacity.  So I have to copy all of that data over and then entirely get rid of array 1.  

Edited February 25 by oliver

[JorgeB]

You can copy the data first over LAN or do a new config with everything if case allows for it, copy locally, then do another new config.

[oliver]

6 hours ago, JorgeB said:

You can copy the data first over LAN or do a new config with everything if case allows for it, copy locally, then do another new config.

 

Questions on how exactly unraid works with existing drives and how the the new config option works.

 

Let's say I pull a drive from array 1 and insert it into array 2.  Now there's two things that can be done - 

 

• Before starting array 2, I add the disk via the dropdown menu and click start array  My understanding is that unraid will not simply allow you to add it array 1, it will format it, resulting in data loss of disk.  Is this correct?
• If the above is correct, the proper procedure here is to create a new config, preserve existing assignments, rebuild parity.  This will not result in data loss from the new disk?

 

How does unraid reconcile different share names.?  If the original disk from array 1 has a different root level share?  When creating the new config, will it just add a new share with the matching name?  

Edited February 25 by oliver

[oliver]

Ok so I setup two trial arrays and did a small test of the transfer process with some spare drives.  Reconfiguration seems to work quite well, creates the new shares.  

 

My last question is how does unraid handle a disk from another unraid system, which has the same shares and conflicting files?

 

In my case, the only conflict is the 'system' share, which im not familiar with (looks like docker/vms).  Will it cause problems if I merge the two arrays, which include the two different system shares? 

 

 

Edited February 26 by oliver

[itimpi]

If the same file exists in the same place on more than one drive then Unraid will use the one it finds on the first pool (alphabetically) or lowest number array drive.  
 

Ideally you want to avoid such duplicates as they waste space and can cause unexpected behaviour (e.g. you delete a file and it does not appear to work as Unraid now finds another copy on another drive).