Please have a look at the two screenshots. They show the editor windows of two existing VMs.
The Broadcom devices are marked gray (unavailable) if they are in use.
The USB devices stay available even if in use.
IMHO, the USB devices must be marked gray (unavailable) as well if they are in use.