Poor read speed over 10GBe from UNRAID Server

[fireb1adex]

Hi - I am running a P2P 10Gbe ethernet connection to my main PC, writing to the Array via (NVME) cache hits full speed (716MB/s), but reading from the cache or an unassigned NVME drive gives me full speed for a few seconds, dropping down to about 16MB/s, where it will then fluctuate up and down bouncing between 10 and 100MB/s.

 

 

I have set the MTU to 9014 on the server and client, this didn't make a difference, iperf gives me solid results too and from the server too so more inclined to think it's client side (Windows 10), however other PCs on the 1GBe network demonstrate similar behavior.I have tried Direct IO on and off and even messed with variations in NIC settings in Tips and Tweaks:

 

Flow Control: Off

NIC Offload: On

NIC Rx Buffer: 4096

NIC Tx Buffer: 4096

 

I have also bumped the receive buffer up to 4096 on the client PC, tried jumbo frames off and on etc....the client PC has an NVME drive and the write speed tested gives me 1800MB/s.

 

Any ideas on where else I can look please, it's bugging the hell out of me after investing heavily in getting these speeds up!

 

 

Edited December 31, 2019 by fireb1adex
extra info