Jump to content
SpaceInvaderOne

**VIDEO GUIDE** How to easily and safely pass through a USB controller **

17 posts in this topic Last Reply

Recommended Posts

Hi, Guys. I have made a tutorial showing how to easily and safely pass through a USB controller on your unRAID server.

You can see how to quickly identify all USB controllers in the server and see which devices and buses are connected to each controller and

also which iommu group each controller is in. So you can easily avoid accidentally passing through the controller used for the unRAID flash drive.

You will also see how to easily identify which USB controllers support resetting. I hope you find it useful :)

 

How to easily  pass though a USB controller

 

Share this post


Link to post

Nice guide! much thanks for simplifying the process.

 

Your text in the post isn't a link. had to click on your sig to get to your channel.

Share this post


Link to post

Hi Gridrunner,

 

Running the same USB pci card as you have. However, I keep running into problems when I restart my VM's. Either UNRAID freezes up or I am not able to start up the VM as the USB resource is busy. Are you experiencing the same? Is there anything I can do to check why my usb card is not resetting properly? Only a cold restart seems to resolve the problem. Running 6.3 rc6

 

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:03:00.0)) vs. 00000080 (ehci_hcd:usb1)

Dec 17 13:26:18 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3

Dec 17 13:26:19 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway

Dec 17 13:26:20 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:20 unRAID kernel: device vnet0 left promiscuous mode

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:21 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none

Share this post


Link to post

Hi Gridrunner,

 

Running the same USB pci card as you have. However, I keep running into problems when I restart my VM's. Either UNRAID freezes up or I am not able to start up the VM as the USB resource is busy. Are you experiencing the same? Is there anything I can do to check why my usb card is not resetting properly? Only a cold restart seems to resolve the problem. Running 6.3 rc6

 

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:03:00.0)) vs. 00000080 (ehci_hcd:usb1)

Dec 17 13:26:18 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3

Dec 17 13:26:19 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway

Dec 17 13:26:20 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:20 unRAID kernel: device vnet0 left promiscuous mode

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:21 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none

 

please ssh into your server and copy the result of this. Should say reset next to your usb card/controller if it supports reset

 

for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done

 

 

edit . if you are using  in syslinux maybe try the older pci-stub.ids= instead

so

vfio-pci.ids=1b73:1100

would be

pci-stub.ids=1b73:1100

 

may not solve your prob but worth a try as i dont think stub also it to go into d3

Share this post


Link to post

Can I pass through 2 USB controler to 2VMs? Where can I put an second Pci stub

 

Yes just be careful not to pass through unRAID's usb flash controller.

 

To stub 2 devices (well interesting point or maybe not lol ... its not stub anymore from linux 4.1 its vfio-pci.ids which is a newer version of 'stub' that allows devices to go into D3 state(low power state) when they are not in use)  :)

 

just put a comma between the ids heres my syslinux line below with 2 devices

 

append vfio-pci.ids=1b73:1100,8086:15a1 initrd=/bzroot 

  • Like 1

Share this post


Link to post

Hi Gridrunner,

 

Running the same USB pci card as you have. However, I keep running into problems when I restart my VM's. Either UNRAID freezes up or I am not able to start up the VM as the USB resource is busy. Are you experiencing the same? Is there anything I can do to check why my usb card is not resetting properly? Only a cold restart seems to resolve the problem. Running 6.3 rc6

 

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:03:00.0)) vs. 00000080 (ehci_hcd:usb1)

Dec 17 13:26:18 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3

Dec 17 13:26:19 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway

Dec 17 13:26:20 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:20 unRAID kernel: device vnet0 left promiscuous mode

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:21 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none

 

please ssh into your server and copy the result of this. Should say reset next to your usb card/controller if it supports reset

 

for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done

 

It should be able to reset itself:

 

IOMMU group 13

[RESET] 03:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev ff)

Share this post


Link to post

Hi Gridrunner,

 

Running the same USB pci card as you have. However, I keep running into problems when I restart my VM's. Either UNRAID freezes up or I am not able to start up the VM as the USB resource is busy. Are you experiencing the same? Is there anything I can do to check why my usb card is not resetting properly? Only a cold restart seems to resolve the problem. Running 6.3 rc6

 

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:03:00.0)) vs. 00000080 (ehci_hcd:usb1)

Dec 17 13:26:18 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3

Dec 17 13:26:19 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway

Dec 17 13:26:20 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:20 unRAID kernel: device vnet0 left promiscuous mode

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:21 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none

 

please ssh into your server and copy the result of this. Should say reset next to your usb card/controller if it supports reset

 

for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done

 

It should be able to reset itself:

 

IOMMU group 13

[RESET] 03:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev ff)

 

yeah thats strange.

not sure if you saw the edit i made on the last post to you but if using vfio-pci.ids= then try pci-stub.ids=  and vice versa

Share this post


Link to post

Problem persists using pci-stub, although stub is claiming the pci.

 

A cold boot into VM, everything works as it should. A VM reset, I can still login to the VM, but it is extremely slow. Third reboot and I cannot even start the VM and receive a pop up with the following message:

 

Execution error

internal error: qemu unexpectedly closed the monitor: 2016-12-17T15:42:06.737278Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: vfio: Error: Failed to setup INTx fd: Device or resource busy

2016-12-17T15:42:08.620251Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: Device initialization failed

 

Log VM:

2016-12-17T15:42:06.737278Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: vfio: Error: Failed to setup INTx fd: Device or resource busy

2016-12-17T15:42:08.620251Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: Device initialization failed

2016-12-17 15:42:08.664+0000: shutting down, reason=failed

 

Log unraid:

Dec 17 16:38:46 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway

Dec 17 16:38:47 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR

Dec 17 16:38:48 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none

Dec 17 16:38:48 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3

Dec 17 16:38:48 unRAID kernel: pci-stub 0000:03:00.0: claimed by stub

Share this post


Link to post

I dont know what else to suggest. I have not had any of the same problems with my fresco logic usb. Mine is  Inateck PCI-E to USB 3.0 5-Port PCI Express Card (KTU3FR-5O2I) you can see it here http://amzn.to/2f2OS5U?

I am using an x99 motherboard and dont have any acs overide enabled or anything else.

this is my syslinux file here.

default /syslinux/menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append vfio-pci.ids=1b73:1100,8086:15a1 initrd=/bzroot  
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Memtest86+
  kernel /memtest

 

Perhaps the card is just faulty??

Share this post


Link to post

Problem solved! I installed the drivers from the anateck website... should not have done that. Reverting back to the windows 10 drivers and everything is working accordingly.

Share this post


Link to post

Problem solved! I installed the drivers from the anateck website... should not have done that. Reverting back to the windows 10 drivers and everything is working accordingly.

 

Ah great. Glad you sorted it out  :)

Share this post


Link to post

What a great guide!  Thankyou!!  And perfect timing.  I'm going to do this over the Christmas break.  ????

 

Sent from my SM-N920I using Tapatalk

 

 

Share this post


Link to post

What a great guide!  Thankyou!!  And perfect timing.  I'm going to do this over the Christmas break.  ????

 

Sent from my SM-N920I using Tapatalk

Thanks glad you liked it

Share this post


Link to post

Thank you for the great videos.  I have been trying to pass through the exact same USB card to my windows VM.  However, it never shows up in "Other PCI Devices"

 

Below are the steps I have tried:

Confirmed that it is in its own IOMMU group (21) and shows RESET.  

Used the following string: vfio-pci.ids=1b73:1100

Added it to the flash drive syntax:  append vfio-pci.ids=1b73:1100 pcie_acs_override=downstream initrd=/bzroot

 

Note that unlike the tutorial mine has "pcie_acs_override=downstream" in it as well.  Was not sure how to handle that.

 

Assistance appricated!

 

 

 

 

Share this post


Link to post

Excellent guide, thank you gridrunner.

 

I needed to pasthru a USB controller and your guide made it very easy, including the steps to check where unRAID is conected to, great :) 

Share this post


Link to post
10 hours ago, bonienl said:

Excellent guide, thank you gridrunner.

 

I needed to pasthru a USB controller and your guide made it very easy, including the steps to check where unRAID is conected to, great :) 

 

Thanks. Glad it helped :)

  • Upvote 1

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now