Hi. I am currently on trial license and thinking to purchase a pro license after my trial expire. My unraid server is a custom built PC running on ASUS TUF GAMING X570-PLUS, AMD Ryzen 5 PRO 4650G, 32GB RAM, and a QNAP QXG-10G1TB, 10Gbe pcie card. The client machine is a Windows 10 with another QNAP QXG-10G1TB card with official driver installed. The client and server is linked directly with a CAT 7 cable without passing through any switch / router and jumbo frames are set on both sides. Basically I followed the following guide for setup:

 

I have 2 x 2TB NVME ssd setup as RAID 1 cache drive with the following specs:

1 x Viper VP4100 (pcie gen 4) (5000 MB/s read & 4400 MB/s write)

1 x Viper VPN100 (pcie gen 3) (3450 MB/s read & 3000 MB/s write)

 

Technically, I should be able to easily saturate the 10 Gbps link ( 1250 MB/s ) writing to cache but it seems that my write speed is capped at around 500 MB/s. Naturally, I started by investigating the network link between client and server using iperf3 and found out that most of the time the average speed is between 5 to 6 Gbps but occasionally it would hit 9 Gbps. I would say it hit 9 Gbps in 1 out of 10 repetitions. Please refer to attached screenshots. There was one test with -t 60 parameter and it was running consistently at 9 Gbps but as soon as I ended that test, the next one was back to 5 Gbps.

 

Have anybody experienced this and know what caused the issue?

Update: I found out that with the default's window size, iperf3 would underperform. I added the " -w 1M " flag and now I get consistent ~9 Gbit/s link speed between client and server. Now I know that link speed is fine but the problem of ~600 MB/s write speed cap is still there (update: after further test, the average write speed cap is ~600 MB/s, not ~500 MB/s as stated in my first post). Does anybody know where to tinker around?

Ok this is weird. Before tinkering, I can fully saturate my 10Gbps link when reading from unraid cache to windows 10 ( speed was ~1 GB/s) although write was at ~600 MB/s. Then I enabled the Direct IO in Global Share Setting and I got read ~800 MB/s and write ~700 MB/s. Then I disabled Direct IO and enabled Disk Share and I got full ~1 GB/s on both read and write. HOWEVER, after I reverted everything back to original setting, now I get read ~600 MB/s and write ~600 MB/s as opposed to the original read ~1 GB/s and write ~600 MB/s. Now I regret tinkering around with the setting but also found it really weird.

 

My conclusion is: it seems the slow write has to do with the overhead of using user share but now even the read is capped at ~600 MB/s. Anybody has any solution? At least I'd like the read speed to be back to ~1 GB/s while using user share.