Jump to content

Slow read speeds from Unraid with direct IO on 10GBe


Recommended Posts

Well, THIS is fun. 

I've got an unRAID server that acts as our video vault at work. It's where we store our B-Roll, all previous YouTube episodes etc. It's got a 2 TB NVMe SSD acting as a cache drive. It's connected to our network switch, which is a 10GBe enabled Netgear XS712Tv2. Also connected to the switch is our iMac Pro edit machine with 10GBe networking, and when our part-timer is in the office, a MacBook Pro with a Sanlink3T1 offering 10GBe connections. 

We've made all of the usual tweaks to the SMB networking on OS X 10.13.6 so that all of the SMB signing is off. We're accessing the data on the server using SMB. 

Our SMB server settings are: 

[global]                                                                                                                                                               

ea support = yes
vfs objects = catia fruit streams_xattr
fruit:resource = file
fruit:metadata = netatalk
fruit:locking = none
fruit:encoding = native
max protocol = SMB2_02



If we turn direct IO off, we get read/write speeds using blackmagic speed test of around 750 MB/s - 820 MB/s read and 750 MB/s-820 MB/s write.  If I turn on Direct IO, write speed goes up to 1.1 GB/s but write drops way down to 320 MB/s and doesn't go any higher. 


iPerf is solid both ways. It gives us full saturation (so about 9.4  to 9.9 Gbit/s) to the iMac Pro and to the unRAID system (I've operated iPerf as both client and server on both machines to test the connection both ways). iPerf doesn't obviously care about Direct IO being enabled or not, as it isn't writing to the unRAID system. It's just checking the physical network speed. 

iPerf running as server on iMac Pro gives: 
 

[ ID] Interval           Transfer     Bandwidth

[  5]   0.00-1.00   sec  1.09 GBytes  9.38 Gbits/sec                  

[  5]   1.00-2.00   sec  1.15 GBytes  9.86 Gbits/sec                  

[  5]   2.00-3.00   sec  1.15 GBytes  9.85 Gbits/sec                  

[  5]   3.00-4.00   sec  1.15 GBytes  9.85 Gbits/sec                  

[  5]   4.00-5.00   sec  1.15 GBytes  9.86 Gbits/sec                  

[  5]   5.00-6.00   sec  1.15 GBytes  9.85 Gbits/sec                  

[  5]   6.00-7.00   sec  1.15 GBytes  9.85 Gbits/sec                  

[  5]   7.00-8.00   sec  1.15 GBytes  9.85 Gbits/sec                  

[  5]   8.00-9.00   sec  1.15 GBytes  9.86 Gbits/sec                  

[  5]   9.00-10.00  sec  1.15 GBytes  9.86 Gbits/sec                  

[  5]  10.00-10.04  sec  47.5 MBytes  9.88 Gbits/sec                  

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bandwidth

[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  sender

[  5]   0.00-10.04  sec  11.5 GBytes  9.81 Gbits/sec                  receiver

-----------------------------------------------------------

 

unRAID running as client on iPerf: 

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.14 GBytes  9.80 Gbits/sec    0   1.04 MBytes
[  4]   1.00-2.00   sec  1.15 GBytes  9.85 Gbits/sec    0   1.04 MBytes
[  4]   2.00-3.00   sec  1.15 GBytes  9.85 Gbits/sec    0   1.04 MBytes
[  4]   3.00-4.00   sec  1.15 GBytes  9.85 Gbits/sec    0   1.04 MBytes
[  4]   4.00-5.00   sec  1.15 GBytes  9.86 Gbits/sec    0   1.04 MBytes
[  4]   5.00-6.00   sec  1.15 GBytes  9.85 Gbits/sec    0   1.04 MBytes
[  4]   6.00-7.00   sec  1.15 GBytes  9.85 Gbits/sec    0   1.04 MBytes
[  4]   7.00-8.00   sec  1.15 GBytes  9.85 Gbits/sec    0   1.04 MBytes
[  4]   8.00-9.00   sec  1.15 GBytes  9.86 Gbits/sec    0   1.04 MBytes
[  4]   9.00-10.00  sec  1.15 GBytes  9.85 Gbits/sec    0   1.04 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  11.5 GBytes  9.85 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  11.5 GBytes  9.85 Gbits/sec                  receiver

 

unRAID running as iPerf server: 

[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  1.15 GBytes  9.88 Gbits/sec
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   3.00-4.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   4.00-5.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   5.00-6.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   7.00-8.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   9.00-10.00  sec  1.15 GBytes  9.89 Gbits/sec
[  5]  10.00-10.00  sec  1.35 MBytes  9.34 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver

 

iMac Pro running iPerf as client 

[  4]   0.00-1.00   sec  1.15 GBytes  9.89 Gbits/sec                  

[  4]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec                  

[  4]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec                  

[  4]   3.00-4.00   sec  1.15 GBytes  9.90 Gbits/sec                  

[  4]   4.00-5.00   sec  1.15 GBytes  9.89 Gbits/sec                  

[  4]   5.00-6.00   sec  1.15 GBytes  9.89 Gbits/sec                  

[  4]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec                  

[  4]   7.00-8.00   sec  1.15 GBytes  9.89 Gbits/sec                  

[  4]   8.00-9.00   sec  1.15 GBytes  9.89 Gbits/sec                  

[  4]   9.00-10.00  sec  1.15 GBytes  9.89 Gbits/sec                  

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bandwidth

[  4]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  sender

[  4]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver

 

In other words. Everything is just peachy with the network interface. 

So why in the blazes is read so slow when Direct IO is switched on? I wouldn't mind, but we use the 2TB NVMe SSD as our 'work' drive when editing video. When we're done working on the video, we write the data to the spinners using the mover. 

 

If it's just me in the studio, I'll happily settle with the 750-850 MB/s read and write, but i'd like to get that extra few hundred MB/s if I can. I rarely use that bandwidth, but with two people accessing the SSD at the same time, I'd like to get max performance to avoid lag. 

Attached: my system info, and thanks in advance ;) 


Nikki (currently perplexed of Portland) 

kranz-diagnostics-20180903-1051.zip

Link to comment
  • 1 year later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...