I set up a new NAS for a relative. I have some media that I would like to transfer to him. Files are huge. I tried to copy using Midnight Commander from the disk where the files are located, to my SSD cache. After which, I copied files from my cache drive to his cache drive (spin drive), via Gigabit wired network. But as expected, it was very slow. Getting about 2-5 Mbps. I am about to transfer 70 GB. What's the most efficient, effective way to do this?

 

 

Sent from my iPhone using Tapatalk

I set up a new NAS for a relative. I have some media that I would like to transfer to him. Files are huge. I tried to copy using Midnight Commander from the disk where the files are located, to my SSD cache. After which, I copied files from my cache drive to his cache drive (spin drive), via Gigabit wired network. But as expected, it was very slow. Getting about 2-5 Mbps. I am about to transfer 70 GB. What's the most efficient, effective way to do this?
 
 
Sent from my iPhone using Tapatalk

By physically taking a drive either there or to your place

Sent via Tapatalk because I'm either at work or enjoying the summer

It is in my place, on the same Gigabit network. I am on Intel gigabit nic, the new unit is on Realtek.

In any case, is this the best transfer speed I can expect?


Sent from my iPhone using Tapatalk

Something is wrong if you get so slow speeds.

 

You should be able to write at full link speed to the cache. And if you turn on turbo write on the receiving machine, then you should be able to write at full link speed even when writing directly to the array. So 100+ MB/s transfer rates.

 

Have you verified that both machines actually connects to the network using gbit links?

 

No errors in the log on either machine?

 

Only your copy program that accesses all relevant disks? Note that the transfer rate will free-falling if you have multiple processes that makes disk accesses from the same HDD - then the disk has to constantly seek between the different locations. And every seek consumes a very significant amount of time - time when the disk can't perform any transfers.

Currently, NAS 1 (Intel NIC) is connected to Gigabit Switch #1, and NAS 2  (Realtek NIC) is connected to Gigabit Switch #2, and Gigabit Switch #1 and #2 are connected together.  I'll connect it directly and get back to you.

2 Mb/sec (megabits / sec) is really really slow. Are you sure it is not MB/sec (megabytes per second).

 

At 2Mb/sec 70 GB would take 3 1/4 days transfer. At 2MB/sec, it would take about 10 hours. 

 

At those speeds, probably using a USB stick would be faster.

 

But I would expect you'd see a minimum of 30MB/sec even if transferring to a protected array. At that speed it would finish in about 40 minutes.

 

There have been some issues with Realtec NICs over the years. Is it in the unRAID server? If so, suggest posting exact chipset so people can see if it is supported. 

I saw about 9 hours before I left home this morning.  

Moved the NAS boxes to side by side, both connected to a Gigabit Switch.  I'm not sure if this matters, the Windows Desktop wherein I have both File Explorer open, and dragged from 1 NAS to the other, is running on Wifi.  Does this have significance?

Oops - you draw unraid1 -> wifi-windows -> unraid2?

 

That's not even close to a gigabit network.

 

You first get the wifi link from router to the Windows machine.

And then next wifi link back to the router and to the second unraid machine.

 

It would have been so much faster if you unRAID machine had mounted a share from the second unRAID and then copied the data directly unRAID -> unRAID without involving the Windows machine and more importantly without involving two WiFi links.

Definitely a big factor.

The file now has to be read from the source to the windows Desktop over WiFi then written to the destination over the same WiFi - This automatically makes the maximum speed equal to a little less than half of however good your WiFi is - which will be very small comparatively. (usually 2MB/s)

You should wire the Desktop, which would give you about 30-50 MB/s

 

But for some serious speed, suggest you get Unassigned Devices installed on one machine and have it mount the other machine (via SMB).

Then use builtin mc to copy the files across - server to server with out additional intermediate devices and get closer to wirespeed 80MB/s.

 

And for the real way to do it is to shutdown both servers, pull the cache drive from one and add it to the other.

Use Unassigned Devices to mount the extra drive and proceed to copy away. <200MB/s depending on your cache drives 

 

Sorry that I can't point you to a tutorial on how to do this.

Edited June 25, 20188 yr by ken-ji

9 minutes ago, pwm said:

Oops - you draw unraid1 -> wifi-windows -> unraid2?

 

That's not even close to a gigabit network.

 

You first get the wifi link from router to the Windows machine.

And then next wifi link back to the router and to the second unraid machine.

 

It would have been so much faster if you unRAID machine had mounted a share from the second unRAID and then copied the data directly unRAID -> unRAID without involving the Windows machine and more importantly without involving two WiFi links.

Thanks, so that's why.  Will try to figure out how to mount a share from one nas to the other.  But mounting a share will mean parity/ array is involved, and speed will be slow.  How do I do it disk to disk?

 

@ken-ji, I know about Unassigned Devices.  Will try to figure out if I can do it.

I'm in business!   Thanks @pwm, @ken-ji.  Got it working.

 

Follow-up question.  I started with 80 MBps, now it's down to 53 MBps, why so low?  The nic has anything to do with it?  Using MC, SMB protocol.

Edited June 25, 20188 yr by jang430

2 hours ago, jang430 said:

I'm in business!   Thanks @pwm, @ken-ji.  Got it working.

 

Follow-up question.  I started with 80 MBps, now it's down to 53 MBps, why so low?  The nic has anything to do with it?  Using MC, SMB protocol.

 

 Not sure if you now have configuration to write to cache on destination machine but the speed seems to indicate that you might copy directly to the array. If turning on turbo write for parity updates - i.e. force the use of all disks - then you may manage over 100 MB/s directly to the array a bit depending on what disks and CPU you have.

I'm writing to cache of the other drive.  Although I have to admit, the cache of NAS 2 is only SATA 2 at best.  

I will still check out the Turbo write function.  I don't know where that is.  I'll look for it.

 

Thanks for your support guys.

 

 

Turbo write will have no effect if you are just accessing the cache drive.

Its intended for allowing maximum write speed to the array by running all the other drives in the array to read and just write to parity and disk instead the usual read from parity then write to disk

 

The drop to 53MB/s might be because of the true speed of the cache drive on NAS 2, which you mentioned is only a SATA2 HDD device.

Without the details such as drive model and state of fullness, you might be hitting the sustained data rate of your cache drive and the initial 80MB/s was just the initial write cached by system RAM.