Not off base, it's a valid argument, but not a simple one.
I personally think the benefit of fail2ban far outweighs the potential risk. I mean, I heavily rely on fail2ban for ddos and brute force protection by using http auth with most of my proxied apps. Fail2ban reduces the likelihood of my server getting pawned. As long as I'm not pawned, I don't have to worry about net admin.
Also keep in mind that most web servers run nginx directly on the host OS, and it already has net admin capability plus a whole lot more. Nginx service runs as root (the workers run as unrpivilieged users). Here at least nginx is somewhat sandboxed inside a container.