aminorjourney Posted September 3, 2018 Share Posted September 3, 2018 Well, THIS is fun. I've got an unRAID server that acts as our video vault at work. It's where we store our B-Roll, all previous YouTube episodes etc. It's got a 2 TB NVMe SSD acting as a cache drive. It's connected to our network switch, which is a 10GBe enabled Netgear XS712Tv2. Also connected to the switch is our iMac Pro edit machine with 10GBe networking, and when our part-timer is in the office, a MacBook Pro with a Sanlink3T1 offering 10GBe connections. We've made all of the usual tweaks to the SMB networking on OS X 10.13.6 so that all of the SMB signing is off. We're accessing the data on the server using SMB. Our SMB server settings are: [global] ea support = yes vfs objects = catia fruit streams_xattr fruit:resource = file fruit:metadata = netatalk fruit:locking = none fruit:encoding = native max protocol = SMB2_02 If we turn direct IO off, we get read/write speeds using blackmagic speed test of around 750 MB/s - 820 MB/s read and 750 MB/s-820 MB/s write. If I turn on Direct IO, write speed goes up to 1.1 GB/s but write drops way down to 320 MB/s and doesn't go any higher. iPerf is solid both ways. It gives us full saturation (so about 9.4 to 9.9 Gbit/s) to the iMac Pro and to the unRAID system (I've operated iPerf as both client and server on both machines to test the connection both ways). iPerf doesn't obviously care about Direct IO being enabled or not, as it isn't writing to the unRAID system. It's just checking the physical network speed. iPerf running as server on iMac Pro gives: [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 1.09 GBytes 9.38 Gbits/sec [ 5] 1.00-2.00 sec 1.15 GBytes 9.86 Gbits/sec [ 5] 2.00-3.00 sec 1.15 GBytes 9.85 Gbits/sec [ 5] 3.00-4.00 sec 1.15 GBytes 9.85 Gbits/sec [ 5] 4.00-5.00 sec 1.15 GBytes 9.86 Gbits/sec [ 5] 5.00-6.00 sec 1.15 GBytes 9.85 Gbits/sec [ 5] 6.00-7.00 sec 1.15 GBytes 9.85 Gbits/sec [ 5] 7.00-8.00 sec 1.15 GBytes 9.85 Gbits/sec [ 5] 8.00-9.00 sec 1.15 GBytes 9.86 Gbits/sec [ 5] 9.00-10.00 sec 1.15 GBytes 9.86 Gbits/sec [ 5] 10.00-10.04 sec 47.5 MBytes 9.88 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.04 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.04 sec 11.5 GBytes 9.81 Gbits/sec receiver ----------------------------------------------------------- unRAID running as client on iPerf: [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.14 GBytes 9.80 Gbits/sec 0 1.04 MBytes [ 4] 1.00-2.00 sec 1.15 GBytes 9.85 Gbits/sec 0 1.04 MBytes [ 4] 2.00-3.00 sec 1.15 GBytes 9.85 Gbits/sec 0 1.04 MBytes [ 4] 3.00-4.00 sec 1.15 GBytes 9.85 Gbits/sec 0 1.04 MBytes [ 4] 4.00-5.00 sec 1.15 GBytes 9.86 Gbits/sec 0 1.04 MBytes [ 4] 5.00-6.00 sec 1.15 GBytes 9.85 Gbits/sec 0 1.04 MBytes [ 4] 6.00-7.00 sec 1.15 GBytes 9.85 Gbits/sec 0 1.04 MBytes [ 4] 7.00-8.00 sec 1.15 GBytes 9.85 Gbits/sec 0 1.04 MBytes [ 4] 8.00-9.00 sec 1.15 GBytes 9.86 Gbits/sec 0 1.04 MBytes [ 4] 9.00-10.00 sec 1.15 GBytes 9.85 Gbits/sec 0 1.04 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 11.5 GBytes 9.85 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 11.5 GBytes 9.85 Gbits/sec receiver unRAID running as iPerf server: [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 1.15 GBytes 9.88 Gbits/sec [ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec [ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec [ 5] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec [ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 5.00-6.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec [ 5] 7.00-8.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec [ 5] 10.00-10.00 sec 1.35 MBytes 9.34 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec receiver iMac Pro running iPerf as client [ 4] 0.00-1.00 sec 1.15 GBytes 9.89 Gbits/sec [ 4] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec [ 4] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec [ 4] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec [ 4] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec [ 4] 5.00-6.00 sec 1.15 GBytes 9.89 Gbits/sec [ 4] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec [ 4] 7.00-8.00 sec 1.15 GBytes 9.89 Gbits/sec [ 4] 8.00-9.00 sec 1.15 GBytes 9.89 Gbits/sec [ 4] 9.00-10.00 sec 1.15 GBytes 9.89 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec sender [ 4] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec receiver In other words. Everything is just peachy with the network interface. So why in the blazes is read so slow when Direct IO is switched on? I wouldn't mind, but we use the 2TB NVMe SSD as our 'work' drive when editing video. When we're done working on the video, we write the data to the spinners using the mover. If it's just me in the studio, I'll happily settle with the 750-850 MB/s read and write, but i'd like to get that extra few hundred MB/s if I can. I rarely use that bandwidth, but with two people accessing the SSD at the same time, I'd like to get max performance to avoid lag. Attached: my system info, and thanks in advance ;) Nikki (currently perplexed of Portland) kranz-diagnostics-20180903-1051.zip Quote Link to comment
Fma965 Posted December 5, 2019 Share Posted December 5, 2019 did you ever figure this out? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.