Recently, I bought a Intel T350-t4 NIC and tryed to use SR-IOV on it. I enable SR-IOV on eth2. And I also have a on board NIC(eth0). I bridged eth0 and eth2, and passthrough VF0 to VM.

The problem is when the bridge on, unraid cannot communicate to VM. If I cancel the bridge on eth2, everything works fine then.

 

I found many similar issue, it seems like there well be some wrong when enable bridge on PF.
https://forum.proxmox.com/threads/communication-issue-between-sriov-vm-vf-and-ct-on-pf-bridge.68638/
 

I tryed the solution in that links (type 'bridge fdb add 5c:b9:01:00:01:00 dev eth2' on unraid's terminal), but it didn't work for me.

Edited August 8, 20242 yr by bhiaibogf

Late, but the thread is unanswered and the command you tried is very close to

the right one - I think it was pointed at the wrong address.

What is happening: with SR-IOV on, the card runs an internal switch between the

PF, the VFs and the wire. It does not learn addresses; it delivers to the ones

it has been told about, which are the VFs' own and the PF's. Unraid's own

address lives behind br0, and the card has never heard of it. A frame from the

VM to Unraid is a miss, and a miss goes out of the physical port instead. That

is also why removing the bridge fixes it: without br0 in the way, the address

the VM talks to is the PF's own, which the card does know.

The address to register is the one behind the bridge - Unraid's, the one br0

holds - on the PF:

bridge fdb add <mac of br0> dev eth2 self permanent

ip -br link show br0 prints it.

If the MAC you used (5c:b9:01:00:01:00) was the VM's, that would explain the

result exactly, and it is worse than doing nothing: it tells the card that the

VM's address sits on the PF vport, so traffic arriving from the wire for the VM

stops reaching the VF. Worth removing that entry if it is still there:

bridge fdb del 5c:b9:01:00:01:00 dev eth2 self

Then the test that settles it in a minute:

1. from the VM, ping Unraid - should fail

2. bridge fdb add <mac of br0> dev eth2 self permanent

3. ping again - should work

4. bridge fdb del <same> dev eth2 self - should fail again

One caveat I would rather say than have you find out: I have reproduced this

failure and this fix on ConnectX-4 Lx, ConnectX-3 Pro, 82599ES and X710. I have

not tested an I350 (igb), so I cannot promise it behaves the same - the four

steps above are how you find out in a minute, and if step 3 changes nothing then

this whole approach is not the answer on your card.

If it does work, what is left is bookkeeping: that entry is static, and the set

of addresses behind a bridge is not - a second VM, a container, a device that

moves, an interface that goes down and comes back. I got tired of maintaining

them by hand on my own machines and wrote a daemon that watches the bridge's

forwarding database and keeps the card's filter in step (disclosure: mine, MIT):

https://github.com/Jimbambuli/sriov-mac-sync

It is a single static binary with no dependencies, so it runs on Unraid, but

mind that the packaging there is Debian and OpenWrt only - on Unraid you would

copy the binary somewhere persistent and start it from your go file yourself.

Before that is worth any effort, do the four steps.

Edited August 26Aug 26 by Jimbambuli