I think you have tried running the command in qbittorrent console and not the unraid server console as I got the same error message when entering the command there
You want to run the command in the terminal that opens when you click this icon
After setting the debug variable to true for the docker contain I got the attached in my logs.
running the below in the unraid server terminal fixed the issue
/sbin/modprobe iptable_mangle