Hmm, okay so the direct host of the card needs a whole X environment right?, because the DebianBuster-Nvidia should also contain a X environment.
I wanted to set a much higher memory clock for mining, yes and also use the card for transcoding if needed. Anyway if it is not possible I will try HiveOS then, thank you for the hint :D.
But I loved the idea that the miner is running within a container next to plex etc.