Transfer speeds: Observations & Questions

[LennyNero]

Greetings!

 

I've been experimenting with Unraid for almost 2 weeks now. By now my new server is more or less ready to be deployed.

However I've got some questions regarding my findings: probably there is still something wrong with the settings... or perhaps I've got just wrong expectations?

 

To test the performance I did copy a 70,62 GB file back and forth a few times. I checked the transfer time with a stop watch and calculated the MB/s rate accordingly.

Docker and VM are disabled. This is intended to be a pure file server.

My desktop computer is equipped with a fast SSD, so that should not be a limiting factor.

 

 

1. Write test directly into the array: user share with no cache enabled:

7:57 min -> 152 MB/s

This is a slightly disappointing result; even my old external USB drive was faster than this. But I think the reason is: Parity needs to be updated at the same time(right?).

Disk settings (md_write_method) is set to reconstruct write. I understand this should be the fastest option?

As you can see in the picture above: there is no constant stream of data over the network. It looks more like pulsing. Same for the drive activity.

Is this considered to be normal?

 

 

2. Read test directly from the array: user share with no cache enabled:

4:37min -> 261 MB/s

As the read speed relates directly to the performance of a single drive... I think this result is acceptable.

As seen above: network and drive activity seem to be just the same... and a constant stream of data. Looks ok, right?

 

 

3. Write test to a cache only user share:

2:05min -> 579 MB/s

Now with this result I'm very disappointed. As far as I understand parity is not updated when writing to the SSD cache?

What setting could have an influence here? (Cache pool of 4 Samsung 970EVO)

Again there is no constant stream of data.

That's normal?

 

 

4. Read test from a cache only user share:

1:24 min -> 861 MB/s

Again a slightly disappointing result.

The stats above seem to be ok.... it's just too slow. I was expecting to get at least >1 GB/s from a 10Gbase-T network.

 

I set the MTU to 9000, and

in Global Share Settings I enabled Direct IO.

 

Is there any other setting that I overlooked so far?

 

 

Thank you for your time reading all this

and

have a nice day!

[JorgeB]

Try writing directly to a disk share, user shares always add some overhead, more noticeable in some hardware configurations than others.

[itimpi]

To be honest those speeds look quite reasonable.    Do not forget the disk speed is being measured in bytes while things like network speed are measured in bits.  I often assume as a rule of thumb about a 10-1 ratio allowing for overheads.

[LennyNero]

5 hours ago, johnnie.black said:

Try writing directly to a disk share, user shares always add some overhead, more noticeable in some hardware configurations than others.

I did read that mixing user shares and disk shares is somehow unwise. Is that not true anymore?

 

3 hours ago, itimpi said:

To be honest those speeds look quite reasonable.    Do not forget the disk speed is being measured in bytes while things like network speed are measured in bits.  I often assume as a rule of thumb about a 10-1 ratio allowing for overheads.

For my stats shown in the first post I had adjusted the y-axis of the network diagram to show MB/s and not Mbit/s.... so it's directly comparable.

In the good old days of 1 Gbits/s network I could easily get a transfer speed of 110 MB/s. When I apply the same ratio to a 10 Gbit/s network I should get a transfer speed of 1100 MB/s. Especially when I'm using 4 NVME SSD's: even a single one of these should be able to saturate the network speed more than twice.

The poor performance of the SSD cache is what concerns me most.

[itimpi]

11 minutes ago, LennyNero said:

I did read that mixing user shares and disk shares is somehow unwise. Is that not true anymore?

The issue is still there, but the issue is about mixing disk and user shares in the SAME copy/move command.

[JorgeB]

25 minutes ago, LennyNero said:

Is that not true anymore?

Like mentioned perfectly fine to copy to disks shares from an outside source, like your desktop, enable them then and copy to \\tower\cache and \\tower\diskX

[Sinister]

20 minutes ago, LennyNero said:

I did read that mixing user shares and disk shares is somehow unwise. Is that not true anymore?

 

For my stats shown in the first post I had adjusted the y-axis of the network diagram to show MB/s and not Mbit/s.... so it's directly comparable.

In the good old days of 1 Gbits/s network I could easily get a transfer speed of 110 MB/s. When I apply the same ratio to a 10 Gbit/s network I should get a transfer speed of 1100 MB/s. Especially when I'm using 4 NVME SSD's: even a single one of these should be able to saturate the network speed more than twice.

The poor performance of the SSD cache is what concerns me most.

I was going completely insane a couple weeks ago when i added 10GB NIC's between my workstation and server, following alot of the advice here i ended up creating a ramdisk on both server and workstation and still not getting the 1Gb speeds i was expecting. After making a backup of my current install and reinstalling windows andinstalling things one at a time and testing a transfer with each thing i put back i came to find my firewall (avast premium Security) would slow my transfers down from 1Gb to like 100 - 300Mbs with firewall not installed i am able to transfer and achieve those speeds to ram disk and a sata SSD mounted by unassigned devices with no issues. But even transferring data to a cache enabled or cache only share im only able to get about 500 - 600 Mb/s which i think is how its suppose to be would love to know if my expectations are wrong here. You could also use iperf to test network throughput between server and workstation

[LennyNero]

Hello all,

I just did an iperf check to reconfirm the issue is not the network itself. Here are the results:

Looks pretty solid I think.

 

I'll check performance with different firewall settings next...

[Vr2Io]

As @johnnie.black reply, you will got better result with disk share ( or UD ) instead user share.

 

High performance always not Unraid selling point.

Edited April 18, 2020 by Benson

[LennyNero]

Update:

With the firewall disabled: at first it looked like an improvement - but as I re-tested I couldn't confirm the result.

The fluctuation it seems was just within the margin of error.

 

[Sinister]

2 hours ago, LennyNero said:

Update:

With the firewall disabled: at first it looked like an improvement - but as I re-tested I couldn't confirm the result.

The fluctuation it seems was just within the margin of error.

 

For me I had to completely remove the firewall component from the suite in order for it to work properly, but maybe whatever software you use is different if it functions fine with it installed I'd love to know what you are using as I myself am still searching for something secure that does not cause this complication 

Edited April 19, 2020 by Sinister

[LennyNero]

9 hours ago, Sinister said:

For me I had to completely remove the firewall component from the suite in order for it to work properly, but maybe whatever software you use is different if it functions fine with it installed I'd love to know what you are using as I myself am still searching for something secure that does not cause this complication 

Assuming that you're using Windows I'm afraid I can't help... as I'm just using the inbuilt firewall of macOS.