Responding to this thread because I just spend 2 days looking all over the internet for why my USB audio card was still crackling in my macOS Monteray VM. I do not have any additional PCIe USB cards in my system. I went to VM Manager and enabled "downstream" for PCIe ACS Override, then rebooted. (I turned off the auto-start on the array just because I wanted to be extra safe I don't accidentally pull out the SATA controller from my system for any reason..) Anyway, then when it came back up I found the IOMMU group that contained my motherboard's onboard USB 3.0 controller and set it to "bind to VFIO at boot" and rebooted. This IOMMU group contained my motherboards encryption controller as well as the motherboard audio ports. After the last reboot, I started up the array, and set newly isolated USB 3.0 controller to a pci device for my macOS VM, plugged the USB audio card into the port and it worked like a charm. I can now unplug usb devices from the VM while its running with no issue.
TL;DR - don't pass the USB Audio card directly to a macOS VM - it will still crackle. The only way to get around the crackle/popping/dropping out audio is to pass an entire USB controller to the VM.