Hi,
I haven't tried tinkering with the server yet but I think I got my answer. Thanks to meep for pointing out making the whole usb/pci controller passthrough the VM. Also this: http://lime-technology.com/forum/index.php?topic=36768.0. I'll post back if it works for me.
Edit:
I managed to make USB passthrough work with the above link. Except I used <qemu:arg value='vfio-pci,host=00:1a.0,bus=root.1,addr=00.0'/> address is .0 instead of .1 in the first post. Working fine now except my LAN connection suddenly dropped after a few hours. I had to physically remove and reconnect