I was curious if you ever found a solution to your dilemma? I plan to attempt something similar soon.
I have a PoE managed switch for my PoE cameras (each port is only 100mbps so I can't replace my gateway with it), I plan to plug the PoE switch into the second NIC on my server, completely isolated from my main network. I also intend for the Frigate docker to communicate with MQTT/Homebridge using the main network the rest of my server utilizes.