Best solution data Migration to Unraid?


Recommended Posts

Hi All

 

After testing my new Unraid server I now feel secure enough to move all my data from my Synology server

But I have some problems how to do this, and I wanted to get a second opinion before I begin (Can't undo when started 😉)

 

First I am limited in both $ and my existing storage to do a nice 1:1 move, so this is not the "optimal" solution to do this!

 

My approach would be:

 

1) Remove the two dual parity drives and use them temporary to store some of the data from the Synology server

- All drives have been tested and cleared in Unraid

 

2) Move all data by USB3-Synology and maybee UAD in Unraid to single drives formatted in exFAT (That would make it possible to read and write between the NAS)

 

3) Use balancer to move data away from the smaller 3TB drives and replace them one by one with the new drives from the Synology volume

 

4) When all the data is moved back to the Unraid volume add 2 new larger Parity drives and wait a +1 week to let it rebuild 🙂

 

It's not a perfect plan but would it work?

Yes I would be running the risk moving large data without any parity drives...

And moving data that could have some data corruption

 

Servers:

Unraid: 21 drives + 2 Parity drives (1 free drive slot)

Synology: Primary volume: 11 drives (1 free drive slot)

 

As always your feedback and insight is highly appreciated

Link to comment

Hard to understand what you are trying to do. What is the total used capacity of your Synology server? What is the total available storage of your Unraid server?

 

I think Synology runs on RAID so you can't really do a one-by-one disk transfer between the Synology and Unraid servers since data is striped on the Synology volume. The entire volume has to be moved out together.

Link to comment
1 hour ago, testdasi said:

Hard to understand what you are trying to do. What is the total used capacity of your Synology server? What is the total available storage of your Unraid server?

 

I think Synology runs on RAID so you can't really do a one-by-one disk transfer between the Synology and Unraid servers since data is striped on the Synology volume. The entire volume has to be moved out together.

Sorry I wasn't clear when I wrote "can't do a nice 1:1 move" I mean I dont have the capacity to instigate a file transfer from one server to the other..

(I dont have 1:1 free storage available)

 

So I need to free some drives - In Unraid I can remove the two parity drives and move data to them by USB from the Synology

On the Synology I have Raid 6 so I "could" take the two 10TB parity drives  (Maybe just one) and move data to that drive

The rest is old 3TB drives I have 16 of them (All validated with pre-clear)

(This is going to take a very long time) but since I don't have a third server and no slots available I think this is my best option

USB3 on both servers

 

BTW: I am not even sure that exFat is supported by both servers or if they work with 8TB/10 drive but I am happy to see UAD plus is supporting this format

 

Link to comment
  • 2 weeks later...

I need to add many new drives and I want to know the best way to transfer data between drives?

 

Faster Method:


This method disables the parity drive while transferring data from the smaller drives to the larger drive, and then rebuilds parity at the end. Not having parity allows faster writing to the larger drive.

Stop array and power down.
Add new larger drive and power up.
Stop array and use Tools - New Config.
Assign new larger drive to new slot, unassign parity drive, and start array.
Copy all files from the smaller drives to the larger drive. For example, if the smaller drives are Disk 1 and Disk 2, and the larger drive is Disk 3, then you could use these commands in console or telnet:
rsync -avX /mnt/disk1/ /mnt/disk3  <-- Can I just do this in the command prompt browser? or would it be better to use Krusader?
rsync -avX /mnt/disk2/ /mnt/disk3
Stop array and power down.
Remove smaller drives and power up.
Use New Config to unassign removed drives and assign parity. You can also change the slot assignment of the new drive or any other drives at this point.
Start array and let parity rebuild.

 

Or would i need https://wiki.unraid.net/UnRAID_Add_Ons#UnMENU to run the command?

Edited by casperse
Link to comment
On 1/8/2020 at 2:11 AM, casperse said:

"can't do a nice 1:1 move" I mean I dont have the capacity to instigate a file transfer from one server to the other..

Even you haven't assign 2 party disks in Unraid and take out 2 disks from Synology. i.e. 20TB, you still need "1:1 - 20TB" storage in Unraid for data moving. Except Synology have several disk pool and you can free up more disk during moving.

 

And I don't think using USB3 with different filesystem disks for moving is a good idea or work. You should simple use rsync through network.

 

 

rsync -avX /mnt/disk1/ /mnt/disk3

rsync -avX /mnt/disk2/ /mnt/disk3


Above operation will mixup things and I usually do in below way. After ALL moving complete then move the data to a proper directory on each disk just on the fly.

 

rsync -avX /mnt/disk1/ /mnt/disk3/1/

rsync -avX /mnt/disk2/ /mnt/disk3/2/

 

 

Edited by Benson
Link to comment
2 minutes ago, Benson said:

Even you haven't assign 2 party disks in Unraid and take out 2 disks from Synology. i.e. 20TB, you still need "1:1 - 20TB" storage in Unraid for data moving. Except Synology have several disk pool and you can free up more disk during moving.

 

And I don't think using USB3 with different filesystem disks for moving is a good idea. You should just simple use rsync through network.

 

 

rsync -avX /mnt/disk1/ /mnt/disk3

rsync -avX /mnt/disk2/ /mnt/disk3

 

This will mixup things and I usually do in below way. After ALL moving complete then move the data to a proper directory on each disk just on the fly.

 

rsync -avX /mnt/disk1/ /mnt/disk3/1/

rsync -avX /mnt/disk2/ /mnt/disk3/2/

 

 

I couldn't because all drive slots where full on both Synology and Unraid server! - I have moved as much as I could over the network 90% full on Unraid now

I also had to use my USB3 dock and buy 3 x 12TB new drives so right now I am almost done emptying the volume and I can start to move the 10TB drives to Unraid server

 

My plan was to disable parity and then add one 10TB drive (one empty slot) and start moving 3x3TB to the new 1x10TB getting 3 empty drive slots for more 10TB drives - and then do a move again?

 

it should be safe to do a copy to the new drives using mc in the command prompt? or Krusader? no need to move the data since I will remove the drives when the transfer is successful right?

Link to comment
29 minutes ago, casperse said:

My plan was to disable parity and then add one 10TB drive (one empty slot) and start moving 3x3TB to the new 1x10TB getting 3 empty drive slots for more 10TB drives - and then do a move again?

So you should have enough space just no disk slot. Moving 3x3TB to 1x10TB won't have problem.

 

29 minutes ago, casperse said:

it should be safe to do a copy to the new drives using mc in the command prompt? or Krusader?

As mention, I like moving data in individual directory first then finally move to a proper directory within disk on the fly. It is because once mixup things then non reverseable.

 

I never use Kusader, I use MC and Rsync always.

 

All above tools haven't problem but human fault can't avoid, we just use some method to minimise it.

Link to comment
27 minutes ago, Benson said:

It is because once mixup things then non reverseable.

I let Unraid do its thing and use the "Automatic split" - different level for different media so example a single tv season is all on the same drive for a tvshow but other seasons of the same show  could be on other drives.... is this what you mean by mixup? in that case it's not a problem for me 😉 I think this helps not spinning up the same drive accessing multiple files and sofar it have proven to work great for me

Link to comment
17 minutes ago, casperse said:

I let Unraid do its thing and use the "Automatic split"

Not exactly.

What I means is I usually mix things in finally step. For example, if you moving data to array to a new directory, then you found not enough space or not expected disk allocated or cross unexpected disk .... etc. Then you can just simple remove that new directory. But if mixup first, then you can't simply remove it.

 

Besides, if moving between user share, use a new directory also a effective way to avoid the problem happen.

 

Like this case, you don't want last 10% affect the 90% job which done.

Edited by Benson
Link to comment
3 minutes ago, Benson said:

Not exactly.

What I means is I usually mix things in finally step. For example, if you moving data to array to a new directory, then you found not enough space or not expected disk allocated or cross unexpected disk .... etc. Then you can just simple remove that new directory. But if mixup first, then you can't simply remove it.

 

Like this case, you don't want last 10% affect the 90% job which done.

I think I understand now the mix of files into the same directory from multiple drives to one single drive! - But my 3x3TB will always fit on a 10TB so should be safe - But it will mix the folders if I have example Movies on all 3 drives I would get one single folder with all the combined folder on one new drive, but that okay i think...

Link to comment
On 1/22/2020 at 8:40 PM, Benson said:

Not exactly.

What I means is I usually mix things in finally step. For example, if you moving data to array to a new directory, then you found not enough space or not expected disk allocated or cross unexpected disk .... etc. Then you can just simple remove that new directory. But if mixup first, then you can't simply remove it.

 

Besides, if moving between user share, use a new directory also a effective way to avoid the problem happen.

 

Like this case, you don't want last 10% affect the 90% job which done.

This is strange I used MC and copied each disk to a folder on the new drive!

I stopped all dockers and VM's and today (its been running for some time) I see this new folder? "software"

Another of my shares.... is unraid coping files around while I do these transfers?

 

image.png.c9ea13828b1945ddba579f803e181a2f.png

 

UPDATE: A laptop on my network had a open excel file that it saved back to Unraid as a temp file... So I deleted the directory :-)

Edited by casperse
Link to comment

Should I have changed my share settings so they all exclude the drives I want to remove?

What happens to the data already on the drive after they are excluded?

Checking the file sizes after copying with Krusader, they show some differences?

 

image.png.2ec9f739d81c451879576ffb92ba8399.pngimage.png.24ae0ef990f566527faf76a6a0595edd.png

Running a compare check now....with Krusader

or maybee I could do a: rsync -narcv /mnt/disk1/ /mnt/disk21/1 ?

(I have disabled all docker an VM's so I should be relative safe)

Edited by casperse
Link to comment
21 hours ago, casperse said:

Should I have changed my share settings so they all exclude the drives I want to remove?

Exclude or not, no major difference

 

21 hours ago, casperse said:

What happens to the data already on the drive after they are excluded?

Exclude just affect file allocation to which disk, due to file were pooling,  file still showing in exclude disk.

 

21 hours ago, casperse said:

Checking the file sizes after copying with Krusader, they show some differences?

If you like, you may sync again and remove the different, that's why I always suggest moving data to new directory in first.

Edited by Benson
  • Thanks 1
Link to comment
On 1/25/2020 at 5:19 AM, Benson said:

If you like, you may sync again and remove the different, that's why I always suggest moving data to new directory in first.

I am now moving files from the last 4 x USB attached drives as Unassigned drive and using mc in a prompt to copy files to a specific new Disc 21 in a new root folder /USB1/ (which then pops up like before as a share until I move the files) - same as before except I am not moving "entire HD discs" into existing shares

 

Is it okay to just move the /USB1/ files to a existing shared drive on the same drive 21 when its done....? should be the same right?

BTW - I am actually getting better speed using this USB-C dock than before over the network :-)

 

Getting dual Parity setup again is going to take a very long time LOL

Link to comment
18 minutes ago, casperse said:

Is it okay to just move the /USB1/ files to a existing shared drive on the same drive 21 when its done....? should be the same right?

Yes, pls ensure on the same drive 21.

 

21 minutes ago, casperse said:

BTW - I am actually getting better speed using this USB-C dock than before over the network 🙂

Yes too, but usually true if you don't need perform copy_in_copy_out for the moving. Cppy_in also need time.

Link to comment
15 hours ago, Benson said:

Yes, pls ensure on the same drive 21.

 

Yes too, but usually true if you don't need perform copy_in_copy_out for the moving. Cppy_in also need time.

I just noticed some strange thing... after transfer from USB - HD (UAD mount) to Unraid (Using mc in a prompt) I have this view with wildcard *

image.png.db97f9adb3cb8ed49f9d2819b467bb61.png

UPDATE: I think all files are marked as executable - running a "New permission" to see if that fixes this... 🙂 

Edited by casperse
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.