Disabling IOMMU is system wide. It's all or nothing.
If you don't need passthrough, disable IOMMU, some (if not all) USB 3.0 cards atleast, tend to work better, in my experience..
There should be no hypervisor drawbacks, your VMs should function like normal. A fast way to disable IOMMU for me, was to add "iommu=soft" to the startup parameters...