Copying huge amount of data

[xtrips]

Hello,

 

I am merging a Readynas into Unraid. I need to do that over my 1Gb network.
I can do that using SMB or NFS.
I tried to use Krusader on Unraid but somehow I didn't manage to see my Readynas shares.
How do you recommend I do that?
How to optimize the writing into Unraid? Stop all dockers? If so what with Krusader? Maybe command line using Putty? If so what with long filenames or deep directory structures? Will that be a problem?
Maybe stop the cache drive and allow to write directly into the array?

Please I need your help.

Thank you

[JorgeB]

You can use rsync over SSH, enable turbo write in Unraid and make sure to bypass cache if there's one.

 

 

[ChatNoir]

Hello,

To connect a NFS or SMB share of another element of your network you will need the Unassigned Devices plugin.

 

As stated by JorgeB, it is best not to use cache and go directly to the Array for the initial data transfer as it will most probably be smaller than the data you have to transfer.

 

After that, you can chose the method you prefer for the transfer, it can be Krusader, CLI, etc. Personnaly, I prefer to use Midnight Commander (mc in the console).

[xtrips]

20 minutes ago, ChatNoir said:

Hello,

To connect a NFS or SMB share of another element of your network you will need the Unassigned Devices plugin.

 

As stated by JorgeB, it is best not to use cache and go directly to the Array for the initial data transfer as it will most probably be smaller than the data you have to transfer.

 

After that, you can chose the method you prefer for the transfer, it can be Krusader, CLI, etc. Personnaly, I prefer to use Midnight Commander (mc in the console).

I am really not comfortable with Rsync and CLI in general especially in Linux.
I already have my readynas directory mapped with Unassigned Devices plugin and I can see the data. But I don't know how to browse this same data in Krusader. Is Midnight Commander easier?
How do I neutralize the write to cache option temporarily while I copy the data? And where do I enable Turbo Write ? That is after I manage to see it in Krusader or Midnight Commander

 

P.S: In the meantime I found out about Turbo Write a.k.a Reconstruct Write. So I enabled it. I suppose I should switch it back to Auto after I finish with copying at risk of having my whole array spinning all the time, right? 

 

Thank you for taking the time

Edited March 3, 2021 by xtrips

[ChatNoir]

Difficult for me to get in too much details from work.  

 

MC vs Krusader:

Krusader might be easier if you prefer drag and drop instead of the all keyboard from MC.

Since Krusader is a Docker container, you will have to link give it access to your shares on Unraid in order to see it. It is not really complicated if you understand the way dockers work. MC just works for this, just go to /mnt/user/name_of_your_Unraid_share and /mnt/remotes/name_of_your_Readynas_share and start copying (ALT+5).

 

Not to use cache:

Be sure that the Share(s) you are copying your data is set to Cache : No in the Share options.

You can change the setting to what you'll need for the future later.

While you change your shares settings, be sure that each share has its Minimum free space set to the maximum file size you expect on this share.

[xtrips]

Thanks. I worked it out with Krusader. The problem is the copy is running at an average of 55 MiB/s ..... 
My LAN is 1Gbps. Wha could be the problem? What can I improve? Or maybe this is what is to be expected?

[trurl]

That speed is typical with writes to the parity array. Since you enabled turbo mode probably not much else can be done unless you want to get rid of parity until after the transfer is complete then rebuild parity after.

[xtrips]

I

12 minutes ago, trurl said:

That speed is typical with writes to the parity array. Since you enabled turbo mode probably not much else can be done unless you want to get rid of parity until after the transfer is complete then rebuild parity after.

It depends. What speed gain should I expect if I do that? BTW Rsync vs. Krusader, is there much to gain in speed? I'd rather not use Rsync but if I must.....

Edited March 3, 2021 by xtrips

[JorgeB]

Did you enable turbo write?

[xtrips]

5 minutes ago, JorgeB said:

Did you enable turbo write?

If you meant this then yes. If not I would appreciate that you show me how.

Edited March 3, 2021 by xtrips

[trurl]

Realtime parity is the reason Unraid can't write array disks at the full speed of the disk being written. If you study that link about Turbo Write you will get a better idea of what is involved.

 

I don't use Krusader since MC is builtin and has been around since long before Unraid V6. I don't think any of the "software" part of the transfer will make much difference to the speed.

 

One other thing to watch out for when transferring large amounts of data. If you are transferring to user shares, the software will often create the destination folders in advance, and since those empty folders don't take much space, they might all get created on one disk. Then it will try to copy the files to those precreated folders and that disk will run out of space. One workaround is to do smaller batches so another disk can be chosen for subsequent batches.

 

I usually use rsync for transfer between servers. My workaround is just to manually move some of the empty folders to other disks before they start getting filled. Since it works in alphabetical order I just move the empty folders starting at the end of the alphabet.

[xtrips]

6 minutes ago, trurl said:

Realtime parity is the reason Unraid can't write array disks at the full speed of the disk being written. If you study that link about Turbo Write you will get a better idea of what is involved.

 

I don't use Krusader since MC is builtin and has been around since long before Unraid V6. I don't think any of the "software" part of the transfer will make much difference to the speed.

 

One other thing to watch out for when transferring large amounts of data. If you are transferring to user shares, the software will often create the destination folders in advance, and since those empty folders don't take much space, they might all get created on one disk. Then it will try to copy the files to those precreated folders and that disk will run out of space. One workaround is to do smaller batches so another disk can be chosen for subsequent batches.

 

I usually use rsync for transfer between servers. My workaround is just to manually move some of the empty folders to other disks before they start getting filled. Since it works in alphabetical order I just move the empty folders starting at the end of the alphabet.

Noted. I will then copy specific amount of files/directories into specific disks to avoid all the problems you mentioned. What about disabling Parity? How much speed will I gain? 

Edited March 3, 2021 by xtrips

[trurl]

6 minutes ago, xtrips said:

What about disabling Parity? How much speed will I gain? 

Should be more like full speed of the disk and network. Of course you will have to follow that with a parity rebuild. But that technique is often used for  initial data load. With the original data still on the source there is little risk.

[xtrips]

12 minutes ago, trurl said:

Should be more like full speed of the disk and network. Of course you will have to follow that with a parity rebuild. But that technique is often used for  initial data load. With the original data still on the source there is little risk.

Aborted. Disabled parity. Restarted the array. This time I am copying controlled amounts into specific disks. And still the speed is barely improved. 59MiB/s. What's the bottleneck now?

 

P.S: Just out of curiosity. I tried to copy using my Win10 desktop, using 2 mapped drives. And the data flies..... 
And I thought putting the load on Unraid would help.....
There must be a problem somewhere....

Edited March 3, 2021 by xtrips

[JorgeB]

Krusader is known to be slower in some cases, hence why I suggested rsync.