I too use Debian 11 Bullseye in a VM install method. I do so to run Home Assistant w/ a supervised using Docker. To remove Network-Manager breaks Home Assistant (HA) as HA requires Network-Manager as a dependency and not only complains but will refuse to install HA updates. I recommend setting the IP to static on your DHCP Server in your Server or Router, whichever is providing your DHCP Service.