I have finally taken the plunge to bring 10gbe into my network after many years of gigabit and multi-gigabit LAGG. All hardware is capable, but I am having issues with bandwidth.

BASIC HARDWARE SPECS:
UNRAID 6.11.5 - EPYCD8-2T w/ 7551 w/ built in x550
PFSENSE 2.7.0 - DELL 7010 i7-2600k + mellanox connectx
core switch - ubiquiti usw-pro-24-poe
10gbe addon switch - mikrotik CRS305-1G-4S+

DAC cable from pfsense to port 1 on ubiquiti
fiber between ubiquiti and mikrotik
cat7 cables w/ spf+/rj45 adapters to mikrotik

all connections report 10gb connectivity

 

My main PC is a VM on unraid w/ latest virtio drivers -- I passed through one of the 10gbe ports from my EPYCD8-2T to the VM (192.168.1.192)

VPN PC is a VM on unraid w/ updated (but not latest drivers-2021) running w virtio network driver on bridged vlan (192.168.40.253 @ br0.40)

PINGBOX is a vm on unraid w/ old drivers (2017) running w virti network driver on bridged main lan (192.168.1.10 @ br0)

 

When running iperf tests I am mostly not getting the 10gbe bandwidth I think should be there...

 

from main pc to iperf docker @ unraid

 

from main pc to VPN pc on other subnet

 

from main PC to PINGBOX 

 

from main pc to iperf @ pfsense

 

from PINGBOX VM to unraid iperf docker

 

I feel like somehow its not routing properly, but I have no idea why. I understand some bandwidth loss between vlans, but no way it should be that much (only pfblocker, no IDS or suricata). At the very least I should see much closer to line speed on same subnet.

For fun I set up L3 on ubiquiti and moved a speedtest docker to test. Theoretically the traffic should never route out of the switch, and i am not seeing anywhere near the speeds I should have

 

Am I missing something with 10gbe? All MTUs are 1500 but I should see faster even with that, no?

Any assistance would be hugely appreciated

adding diags

 

crunch-diagnostics-20240109-1609.zip

I've a quite similar setup and facing similar performances.

Did you find a solution ?