SpaceInvaderOne Posted December 17, 2016 Share Posted December 17, 2016 Hi, Guys. I have made a tutorial showing how to easily and safely pass through a USB controller on your unRAID server. You can see how to quickly identify all USB controllers in the server and see which devices and buses are connected to each controller and also which iommu group each controller is in. So you can easily avoid accidentally passing through the controller used for the unRAID flash drive. You will also see how to easily identify which USB controllers support resetting. I hope you find it useful How to easily pass though a USB controller Quote Link to comment
masstic1es Posted December 17, 2016 Share Posted December 17, 2016 Nice guide! much thanks for simplifying the process. Your text in the post isn't a link. had to click on your sig to get to your channel. Quote Link to comment
bmdegraaf Posted December 17, 2016 Share Posted December 17, 2016 Hi Gridrunner, Running the same USB pci card as you have. However, I keep running into problems when I restart my VM's. Either UNRAID freezes up or I am not able to start up the VM as the USB resource is busy. Are you experiencing the same? Is there anything I can do to check why my usb card is not resetting properly? Only a cold restart seems to resolve the problem. Running 6.3 rc6 Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:03:00.0)) vs. 00000080 (ehci_hcd:usb1) Dec 17 13:26:18 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3 Dec 17 13:26:19 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway Dec 17 13:26:20 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state Dec 17 13:26:20 unRAID kernel: device vnet0 left promiscuous mode Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state Dec 17 13:26:21 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none Quote Link to comment
Siwat2545 Posted December 17, 2016 Share Posted December 17, 2016 Can I pass through 2 USB controler to 2VMs? Where can I put an second Pci stub Quote Link to comment
SpaceInvaderOne Posted December 17, 2016 Author Share Posted December 17, 2016 Hi Gridrunner, Running the same USB pci card as you have. However, I keep running into problems when I restart my VM's. Either UNRAID freezes up or I am not able to start up the VM as the USB resource is busy. Are you experiencing the same? Is there anything I can do to check why my usb card is not resetting properly? Only a cold restart seems to resolve the problem. Running 6.3 rc6 Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:03:00.0)) vs. 00000080 (ehci_hcd:usb1) Dec 17 13:26:18 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3 Dec 17 13:26:19 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway Dec 17 13:26:20 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state Dec 17 13:26:20 unRAID kernel: device vnet0 left promiscuous mode Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state Dec 17 13:26:21 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none please ssh into your server and copy the result of this. Should say reset next to your usb card/controller if it supports reset for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done edit . if you are using in syslinux maybe try the older pci-stub.ids= instead so vfio-pci.ids=1b73:1100 would be pci-stub.ids=1b73:1100 may not solve your prob but worth a try as i dont think stub also it to go into d3 Quote Link to comment
SpaceInvaderOne Posted December 17, 2016 Author Share Posted December 17, 2016 Can I pass through 2 USB controler to 2VMs? Where can I put an second Pci stub Yes just be careful not to pass through unRAID's usb flash controller. To stub 2 devices (well interesting point or maybe not lol ... its not stub anymore from linux 4.1 its vfio-pci.ids which is a newer version of 'stub' that allows devices to go into D3 state(low power state) when they are not in use) just put a comma between the ids heres my syslinux line below with 2 devices append vfio-pci.ids=1b73:1100,8086:15a1 initrd=/bzroot 1 Quote Link to comment
bmdegraaf Posted December 17, 2016 Share Posted December 17, 2016 Hi Gridrunner, Running the same USB pci card as you have. However, I keep running into problems when I restart my VM's. Either UNRAID freezes up or I am not able to start up the VM as the USB resource is busy. Are you experiencing the same? Is there anything I can do to check why my usb card is not resetting properly? Only a cold restart seems to resolve the problem. Running 6.3 rc6 Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:03:00.0)) vs. 00000080 (ehci_hcd:usb1) Dec 17 13:26:18 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3 Dec 17 13:26:19 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway Dec 17 13:26:20 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state Dec 17 13:26:20 unRAID kernel: device vnet0 left promiscuous mode Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state Dec 17 13:26:21 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none please ssh into your server and copy the result of this. Should say reset next to your usb card/controller if it supports reset for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done It should be able to reset itself: IOMMU group 13 [RESET] 03:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev ff) Quote Link to comment
SpaceInvaderOne Posted December 17, 2016 Author Share Posted December 17, 2016 Hi Gridrunner, Running the same USB pci card as you have. However, I keep running into problems when I restart my VM's. Either UNRAID freezes up or I am not able to start up the VM as the USB resource is busy. Are you experiencing the same? Is there anything I can do to check why my usb card is not resetting properly? Only a cold restart seems to resolve the problem. Running 6.3 rc6 Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc Dec 17 13:26:18 unRAID kernel: genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:03:00.0)) vs. 00000080 (ehci_hcd:usb1) Dec 17 13:26:18 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3 Dec 17 13:26:19 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway Dec 17 13:26:20 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state Dec 17 13:26:20 unRAID kernel: device vnet0 left promiscuous mode Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state Dec 17 13:26:21 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none please ssh into your server and copy the result of this. Should say reset next to your usb card/controller if it supports reset for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done It should be able to reset itself: IOMMU group 13 [RESET] 03:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev ff) yeah thats strange. not sure if you saw the edit i made on the last post to you but if using vfio-pci.ids= then try pci-stub.ids= and vice versa Quote Link to comment
bmdegraaf Posted December 17, 2016 Share Posted December 17, 2016 Problem persists using pci-stub, although stub is claiming the pci. A cold boot into VM, everything works as it should. A VM reset, I can still login to the VM, but it is extremely slow. Third reboot and I cannot even start the VM and receive a pop up with the following message: Execution error internal error: qemu unexpectedly closed the monitor: 2016-12-17T15:42:06.737278Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: vfio: Error: Failed to setup INTx fd: Device or resource busy 2016-12-17T15:42:08.620251Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: Device initialization failed Log VM: 2016-12-17T15:42:06.737278Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: vfio: Error: Failed to setup INTx fd: Device or resource busy 2016-12-17T15:42:08.620251Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: Device initialization failed 2016-12-17 15:42:08.664+0000: shutting down, reason=failed Log unraid: Dec 17 16:38:46 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway Dec 17 16:38:47 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR Dec 17 16:38:48 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none Dec 17 16:38:48 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3 Dec 17 16:38:48 unRAID kernel: pci-stub 0000:03:00.0: claimed by stub Quote Link to comment
SpaceInvaderOne Posted December 18, 2016 Author Share Posted December 18, 2016 I dont know what else to suggest. I have not had any of the same problems with my fresco logic usb. Mine is Inateck PCI-E to USB 3.0 5-Port PCI Express Card (KTU3FR-5O2I) you can see it here http://amzn.to/2f2OS5U? I am using an x99 motherboard and dont have any acs overide enabled or anything else. this is my syslinux file here. default /syslinux/menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label unRAID OS menu default kernel /bzimage append vfio-pci.ids=1b73:1100,8086:15a1 initrd=/bzroot label unRAID OS GUI Mode kernel /bzimage append initrd=/bzroot,/bzroot-gui label unRAID OS Safe Mode (no plugins, no GUI) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest Perhaps the card is just faulty?? Quote Link to comment
bmdegraaf Posted December 19, 2016 Share Posted December 19, 2016 Problem solved! I installed the drivers from the anateck website... should not have done that. Reverting back to the windows 10 drivers and everything is working accordingly. Quote Link to comment
SpaceInvaderOne Posted December 19, 2016 Author Share Posted December 19, 2016 Problem solved! I installed the drivers from the anateck website... should not have done that. Reverting back to the windows 10 drivers and everything is working accordingly. Ah great. Glad you sorted it out Quote Link to comment
matt_webb Posted December 19, 2016 Share Posted December 19, 2016 What a great guide! Thankyou!! And perfect timing. I'm going to do this over the Christmas break. ? Sent from my SM-N920I using Tapatalk Quote Link to comment
SpaceInvaderOne Posted December 20, 2016 Author Share Posted December 20, 2016 What a great guide! Thankyou!! And perfect timing. I'm going to do this over the Christmas break. ? Sent from my SM-N920I using Tapatalk Thanks glad you liked it Quote Link to comment
dubbly Posted April 30, 2017 Share Posted April 30, 2017 Thank you for the great videos. I have been trying to pass through the exact same USB card to my windows VM. However, it never shows up in "Other PCI Devices" Below are the steps I have tried: Confirmed that it is in its own IOMMU group (21) and shows RESET. Used the following string: vfio-pci.ids=1b73:1100 Added it to the flash drive syntax: append vfio-pci.ids=1b73:1100 pcie_acs_override=downstream initrd=/bzroot Note that unlike the tutorial mine has "pcie_acs_override=downstream" in it as well. Was not sure how to handle that. Assistance appricated! Quote Link to comment
bonienl Posted May 9, 2017 Share Posted May 9, 2017 Excellent guide, thank you gridrunner. I needed to pasthru a USB controller and your guide made it very easy, including the steps to check where unRAID is conected to, great Quote Link to comment
SpaceInvaderOne Posted May 10, 2017 Author Share Posted May 10, 2017 10 hours ago, bonienl said: Excellent guide, thank you gridrunner. I needed to pasthru a USB controller and your guide made it very easy, including the steps to check where unRAID is conected to, great Thanks. Glad it helped 1 Quote Link to comment
vvolfpack Posted February 15, 2020 Share Posted February 15, 2020 Thanks so much for your videos, Ed! As a newbie, I can't thank you enough for all your help! I followed this one to the T, but sadly, my Unraid won't boot unless it's booted in GUI Mode. I'm assuming it's because of the syslinux config default menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label Unraid OS menu default kernel /bzimage append vfio-pci.ids=1022:149c initrd=/bzroot label Unraid OS GUI Mode kernel /bzimage append initrd=/bzroot,/bzroot-gui label Unraid OS Safe Mode (no plugins, no GUI) kernel /bzimage append initrd=/bzroot unraidsafemode label Unraid OS GUI Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot,/bzroot-gui unraidsafemode label Memtest86+ kernel /memtest My USB Hub is in its own IOMMU group. However, it is occupying two devices on the Bus 005 and Bus 006 each. It's a TP-Link 7-port Hub with 2 charging ports. Bus 1 --> 0000:07:00.1 (IOMMU group 21) Bus 001 Device 002: ID 0781:5571 SanDisk Corp. Cruzer Fit Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 2 --> 0000:07:00.1 (IOMMU group 21) Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 3 --> 0000:07:00.3 (IOMMU group 21) Bus 003 Device 003: ID 0b05:18f3 ASUSTek Computer, Inc. AURA LED Controller Bus 003 Device 002: ID 046d:c31c Logitech, Inc. Keyboard K120 Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 4 --> 0000:07:00.3 (IOMMU group 21) Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 5 --> 0000:0c:00.3 (IOMMU group 31) Bus 005 Device 003: ID 0bda:5411 Realtek Semiconductor Corp. 4-Port USB 2.0 Hub Bus 005 Device 002: ID 0bda:5411 Realtek Semiconductor Corp. 4-Port USB 2.0 Hub Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 6 --> 0000:0c:00.3 (IOMMU group 31) Bus 006 Device 003: ID 0bda:0411 Realtek Semiconductor Corp. 4-Port USB 3.0 Hub Bus 006 Device 002: ID 0bda:0411 Realtek Semiconductor Corp. 4-Port USB 3.0 Hub Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub I think I'm passing the right ID too IOMMU group 31 [RESET] 0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] Any help would be greatly appreciated! 1 Quote Link to comment
vvolfpack Posted February 22, 2020 Share Posted February 22, 2020 BUMP! Any help would be greatly appreciated here! Quote Link to comment
vvolfpack Posted February 23, 2020 Share Posted February 23, 2020 In case others are wondering, I followed this excellent guide for my Asus Tuf Gaming X570 mobo and it worked like a charm. They also reference SpaceInvader's GPU passthrough tutorial there. Quote Link to comment
JasonK Posted March 16, 2020 Share Posted March 16, 2020 (edited) Followed the guide, see it in my pass through: Check the box, save the settings. Try to start the VM and get the following: edit: I've also tried all 3 different settings for ACS override and the error persists. Edited March 16, 2020 by JasonK Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.