Update from my side:
I switched from a Fritzbox to Opnsense (n100 minipc) and the switches from Netgear to Unifi, since then I have no more dropped packages.
Since I noticed this only recently, i can't say if it was the switches or the Fritzbox, but it still runs on macvtap without any package loss.