March 31, 20251 yr Hi In order to get my thunderbolt controller passthrough fully working (incl VM reboot & start/stop cycles) I have to "hard-reset" the controller. Unraid's function level reset (FLR) is unfortunately not enough. root@Tower:~# lspci -k 7b:00.0 USB controller: ASMedia Technology Inc. Device 2425 (rev 01) Subsystem: ASMedia Technology Inc. Device 2425 Kernel driver in use: vfio-pci Kernel modules: thunderbolt root@Tower:~# The first thing I have to do is blacklist the thunderbolt driver. Yes Unraid binds vfio-pci to it at boot times but after the hard-reset it seems to load the thunderbolt driver if not blacklisted. Where in Unraid 7 is the correct config file to target and what do I have to configure? Is this correct: (Update: seems to be correct, at least it's working) touch /boot/config/modprobe.d/thunderbolt.conf echo "blacklist thunderbolt" > /boot/config/modprobe.d/thunderbolt.conf Where do I have to attach a script that executes a Thunderbolt controller hard-reset whenever the VM starts and/or reboots? Do I have to use libvirt hooks? If so where do I have to configure those in Unraid? This is the hard reset I need to do: echo "1" > /sys/bus/pci/devices/0000\:7b\:00.0/remove sleep 2 echo "1" > /sys/bus/pci/rescan Then I need to manually bind the Thunderbolt controller to VFIO again. How do I do that? I tried with echo 0000:7b:00.0 > /sys/bus/pci/drivers/vfio-pci/bind but this does not work. I get > echo: write error: No such device Thank you very much. Edited March 31, 20251 yr by Tom082 Update
March 31, 20251 yr Author root@Tower:~# cat /etc/modprobe.d/README # /etc/modprobe.d/README # # The monolithic "blacklist" file (and others) that used to be here in this # directory have been split into several more fine-grained files and moved # to the /lib/modprobe.d/ directory. Any file in /lib/modprobe.d/ will be # overridden by an identically named file in this directory (/etc/modprobe.d/) # or /run/modprobe.d/ (but since /run is on a tmpfs, it's not persistent # across reboots, so you probably don't want to use it). # # See "man modprobe.d" for more information. # root@Tower:~# This is very confusing, because .conf files in /etc/modprode.d/ do not seem to have any effect and are not persistent across reboots as suggested in Unraid's own README. /boot/config/modprobe.d/ works though. Edited March 31, 20251 yr by Tom082
April 1, 20251 yr /boot/config/modprobe.d is linked, so use that one always. 11 hours ago, Tom082 said: Where do I have to attach a script that executes a Thunderbolt controller hard-reset whenever the VM starts and/or reboots? You could use a user script, but would need to execute it when needed.
April 1, 20251 yr Author Thank you for confirming the correct location for blacklisting_ One more question regarding blacklisting. It looks like I also have to blacklist the USB4 controller that is part of the TBT (or vice versa - its one chip). How can I blacklist a driver only for a certain device and not globally? I do not want to loose all my other USB ports. Do I understand this correctly that Unraid does not have the capability to execute scripts on VM start/stop? At least not exposed to the user? What about libvirt hook-scripts, are they exposed to the user? If yes, where? Edited April 1, 20251 yr by Tom082
April 1, 20251 yr 34 minutes ago, Tom082 said: How can I blacklist a driver only for a certain device and not globally? AFAIK that is not possible, but maybe you can bind just that device to vfio-pci, and no driver will be loaded in that case. 35 minutes ago, Tom082 said: Do I understand this correctly that Unraid does not have the capability to execute scripts on VM start/stop? Not AFAIK.
April 1, 20251 yr Author 1 hour ago, JorgeB said: maybe you can bind just that device to vfio-pci, and no driver will be loaded in that case. How can I do that? Unraid binds it automatically to vfio-pci at booting. Unfortunately the reset method I have to use for my TBT controller removes that binding and when I do a rescan to reconnect the USB4 controller it will load the USB driver. Blacklisting the thunderbolt driver only works because there is only 1 TBT controller in my system. For USB4 I need to be able to blacklist a single device to load driver X or I need the ability to manually add USB4 with the vfio-pci driver. Mhhh, if I could only temporarily blacklist a driver this might also work. Do you know by chance if the blacklisting config in /boot/config/modprobe.d is reloaded everytime it changes or if it's consulted whenever the system needs to load a driver?
April 2, 20251 yr 12 hours ago, Tom082 said: Unraid binds it automatically to vfio-pci at booting. Not sure I follow, Unraid doesn't bind anything automatically.
April 2, 20251 yr Author 6 hours ago, JorgeB said: Not sure I follow, Unraid doesn't bind anything automatically. OK, let me try to show step by step what happens with my Thunderbolt Controller. In Unraid if have marked the TBT ctrl for passthrough. Unraid binds the controller correctly to VFIO. The same is true for the USB4 ctrl (both are part of the same chip but have different IOMMU groups): I have blacklisted the TBT driver but not the USB4 driver: root@Tower:~# ls /boot/config/modprobe.d/ thunderbolt.conf root@Tower:~# cat /boot/config/modprobe.d/thunderbolt.conf blacklist thunderbolt root@Tower:~# ls /etc/modprobe.d/ README cdc_ether.conf iommu_unsafe_interrupts.conf kvm.conf mpt3sas.conf scsi-sata-controllers.conf thunderbolt.conf zfs.conf I cannot blacklist the USB driver in the same way, otherwise I loose all USB ports on the host incl the port with the Unraid flash drive. This is the state after booting into Unraid: root@Tower:~# lspci -d 1b21:* -k 7a:00.0 USB controller: ASMedia Technology Inc. Device 2426 (rev 01) Subsystem: ASMedia Technology Inc. Device 2426 Kernel driver in use: vfio-pci 7b:00.0 USB controller: ASMedia Technology Inc. Device 2425 (rev 01) Subsystem: ASMedia Technology Inc. Device 2425 Kernel driver in use: vfio-pci Kernel modules: thunderbolt As expected both are bound to VFIO Resetting both ctrl with a FLR reset does not change this binding: root@Tower:~# echo "1" > /sys/bus/pci/devices/0000\:7b\:00.0/reset root@Tower:~# echo "1" > /sys/bus/pci/devices/0000\:7a\:00.0/reset root@Tower:~# lspci -d 1b21:* -k 7a:00.0 USB controller: ASMedia Technology Inc. Device 2426 (rev 01) Subsystem: ASMedia Technology Inc. Device 2426 Kernel driver in use: vfio-pci 7b:00.0 USB controller: ASMedia Technology Inc. Device 2425 (rev 01) Subsystem: ASMedia Technology Inc. Device 2425 Kernel driver in use: vfio-pci Kernel modules: thunderbolt Unfortunately this reset is not good enough. My monitor attached to USB4 remains black after a VM reboot or VM stop/start cycle. At this point the only way to revive it is to reboot the host. I was experimenting with a Fedora Rawhide live flash drive and there I could reproduce a similar issue by detaching/attaching the TBT monitor multiple times. At some point it remains black. A FLR as shown above does not change that. However removing the TBT ctrl from the PCI bus and rescanning the bus revived my TBT monitor after a few seconds. After that I could go through 20+ detach/attach cycles and the monitor came back each time. So I wanted to reproduce that on Unraid and use the "hard-reset" mode. Doing that has results in the following: root@Tower:~# echo "1" > /sys/bus/pci/devices/0000\:7a\:00.0/remove root@Tower:~# echo "1" > /sys/bus/pci/devices/0000\:7b\:00.0/remove root@Tower:~# lspci -d 1b21:* -k root@Tower:~# echo "1" > /sys/bus/pci/rescan root@Tower:~# lspci -d 1b21:* -k 7a:00.0 USB controller: ASMedia Technology Inc. Device 2426 (rev 01) Subsystem: ASMedia Technology Inc. Device 2426 Kernel driver in use: xhci_hcd 7b:00.0 USB controller: ASMedia Technology Inc. Device 2425 (rev 01) Subsystem: ASMedia Technology Inc. Device 2425 Kernel modules: thunderbolt As you can see, after the /rescan neither TBT nor USB ctrl are bound to VFIO anymore and I can not start my VM. My question is how can I change that. I can bind them back to VFIO but since both ctrl where already claimed by Unraid this makes the hard-rest questionable at best. root@Tower:~# lspci -n | grep -E "7a:00|7b:00" 7a:00.0 0c03: 1b21:2426 (rev 01) 7b:00.0 0c03: 1b21:2425 (rev 01) root@Tower:~# root@Tower:~# echo 1b21 2426 > /sys/bus/pci/drivers/vfio-pci/new_id root@Tower:~# echo 1b21 2425 > /sys/bus/pci/drivers/vfio-pci/new_id root@Tower:~# echo 0000:7a:00.0 > /sys/bus/pci/drivers/xhci_hcd/unbind root@Tower:~# echo 0000:7a:00.0 > /sys/bus/pci/drivers/vfio-pci/bind root@Tower:~# echo 0000:7b:00.0 > /sys/bus/pci/drivers/thunderbolt/unbind root@Tower:~# echo 0000:7b:00.0 > /sys/bus/pci/drivers/vfio-pci/bind root@Tower:~# lspci -d 1b21:* -k 7a:00.0 USB controller: ASMedia Technology Inc. Device 2426 (rev 01) Subsystem: ASMedia Technology Inc. Device 2426 Kernel driver in use: vfio-pci 7b:00.0 USB controller: ASMedia Technology Inc. Device 2425 (rev 01) Subsystem: ASMedia Technology Inc. Device 2425 Kernel driver in use: vfio-pci Kernel modules: thunderbolt I need Unraid to bind both ctrl to VIFO after a /rescan. I tried with "runtime" blacklisting: root@Tower:~# touch /etc/modprobe.d/xhci_hcd.conf root@Tower:~# echo "blacklist xhci_hcd" > /etc/modprobe.d/xhci_hcd.conf root@Tower:~# root@Tower:~# touch /etc/modprobe.d/thunderbolt.conf root@Tower:~# echo "blacklist thunderbolt" > /etc/modprobe.d/thunderbolt.conf root@Tower:~# root@Tower:~# ls /etc/modprobe.d/ README iommu_unsafe_interrupts.conf mpt3sas.conf thunderbolt.conf zfs.conf cdc_ether.conf kvm.conf scsi-sata-controllers.conf xhci_hcd.conf root@Tower:~# but somehow this works only for TBT but not USB4. I'm very confused! root@Tower:~# echo "1" > /sys/bus/pci/devices/0000\:7a\:00.0/remove root@Tower:~# echo "1" > /sys/bus/pci/devices/0000\:7b\:00.0/remove root@Tower:~# echo "1" > /sys/bus/pci/rescan root@Tower:~# lspci -d 1b21:* -k 7a:00.0 USB controller: ASMedia Technology Inc. Device 2426 (rev 01) Subsystem: ASMedia Technology Inc. Device 2426 Kernel driver in use: xhci_hcd 7b:00.0 USB controller: ASMedia Technology Inc. Device 2425 (rev 01) Subsystem: ASMedia Technology Inc. Device 2425 Kernel driver in use: vfio-pci Kernel modules: thunderbolt This doesn't make much sense to me, how and why do the two controllers behave differently and how can I fix this? The reason I want to also "hard-reset" the USB ctrl is because both controllers seem to share at least one memory block. Unfortunately I don't have this output anymore I also can't remember where I have seen it. I believe it was when I was booting the Rawhide live flash.
April 2, 20251 yr 16 minutes ago, Tom082 said: In Unraid if have marked the TBT ctrl for passthrough. If you mark it for pass through, it will be bound to vfio-pci, and no driver will be loaded for that device, you don't need to blacklist the driver.
April 2, 20251 yr Author 1 hour ago, JorgeB said: If you mark it for pass through, it will be bound to vfio-pci, and no driver will be loaded for that device, you don't need to blacklist the driver. I don't want to be rude, I really appreciate the help but did you actually read though my post or did you stop at "I blacklisted the TBT controller? I know that under normal circumstance that is not necessary. But as soon as you use /remove and /rescan things are different as you can see above. Probably I should have mentioned the blacklisting part further down in my post when it's actually necessary. Edited April 2, 20251 yr by Tom082
April 2, 20251 yr 21 minutes ago, Tom082 said: But as soon as you use /remove and /rescan things are different as you can see above. 1 hour ago, Tom082 said: I tried with "runtime" blacklisting: 1 hour ago, Tom082 said: but somehow this works only for TBT but not USB4. I'm very confused! Yeah, sorry, missed that, but that's not something I can help with, since I've never used that function, maybe someone else can.
April 2, 20251 yr Community Expert On 4/1/2025 at 5:29 PM, Tom082 said: Thank you for confirming the correct location for blacklisting_ One more question regarding blacklisting. It looks like I also have to blacklist the USB4 controller that is part of the TBT (or vice versa - its one chip). How can I blacklist a driver only for a certain device and not globally? I do not want to loose all my other USB ports. Do I understand this correctly that Unraid does not have the capability to execute scripts on VM start/stop? At least not exposed to the user? What about libvirt hook-scripts, are they exposed to the user? If yes, where? Yes you can create hooks scripts. Examples below. oot@computenode:~# ls /etc/libvirt/hooks/qemu.d/ ArcReset Notify* USB_Manager* VMTest root@computenode:~# cat /etc/libvirt/hooks/qemu.d/VMTest #!/bin/bash VM_NAME="$1" ACTION="$2" if [[ "$ACTION" == "prepare" ]]; then export LIBGL_ALWAYS_SOFTWARE=0 export DRI_PRIME=1 export VDPAU_DRIVER=nvidia export LIBVA_DRIVER_NAME=nvidia export __GLX_VENDOR_LIBRARY_NAME=nvidia export DISPLAY=:0 # Log the environment to verify it's applied env | grep -E 'LIBGL|DRI_PRIME|VDPAU|LIBVA|GLX' >> /tmp/qemu-hook.log fi root@computenode:~# cat /etc/libvirt/hooks/qemu.d/Notify #!/usr/bin/env php <?php if ($argv[2] == 'started' || $argv[2] == 'stopped'){ shell_exec("/usr/local/emhttp/webGui/scripts/notify -i normal -s \"VM {$argv[1]}\" -d \"".ucfirst($argv[2])."\" >/dev/null 2>&1 & disown") ; } ?> root@computenode:~# cat /etc/libvirt/hooks/qemu.d/USB_Manager #!/usr/bin/env php <?php #begin USB_MANAGER if ($argv[2] == 'prepare' || $argv[2] == 'stopped'){ shell_exec("/usr/local/emhttp/plugins/usb_manager/scripts/rc.usb_manager vm_action '{$argv[1]}' {$argv[2]} {$argv[3]} {$argv[4]} >/dev/null 2>&1 & disown") ; } #end USB_MANAGER ?> root@computenode:~#
April 2, 20251 yr Author 2 hours ago, JorgeB said: Yeah, sorry, missed that, but that's not something I can help with, since I've never used that function, maybe someone else can. NP, it was a long post 😅 Thank you very much!
April 2, 20251 yr Author 2 hours ago, SimonF said: Yes you can create hooks scripts. Examples below. Perfect! Thank you very much. Now I just have to figure this "runtime" reset out.
April 4, 20251 yr Author Solution By chance I found a solution that works for me even though the solution does not answer my question how to bind directly to VFIO without blacklisting drivers or only temporary "runtime blacklisting". So here is what happened: I got distracted by a phone call while my VM once again had the blackscreen issue. After the call I forgot that the VM was still running and was "hard-resetting" both the USB and TBT controller on the host. After that Unraid bound the USB controller to the USB driver and the Thunderbolt controller to VFIO as described above. Suddenly the monitor had a picture again (at this point I realized that the VM was still running). I would have expected at least a VM crash but all was running fine. Even more wired is the fact that neither VM nor the host had any drivers for Thunderbolt loaded but suddenly I could reboot start/stop the VM without any issues. This only broke after I power cycled the monitor (I guess detach/attach of the cable would have the same effect). This is when I realized that the only way this is possible is if Thunderbolt is only required for device authorization which happened in the VM when the controller was passed through. As long as the monitor was connected and powered on the authorization remained valid. If this is correct then passthrough of the Thunderbolt controller should also not be required. In the VM I used boltctl in order to authorize Thunderbolt devices. Unfortunately neither boltd nor boltctl or any other tool for thunderbolt exists in Unraid except the driver and I could not find any. All I found is one post in another forum where a guy was building the boltctl package for slackware but this I didn't want to try, this is way over my head at this point. In the end I found this admin guide. So my new setup does neither passthrough the Thunderbolt nor the USB4 controller, neither driver is blacklisted. The Thunderbolt monitor gets authorized by this udev rule: root@Tower:~# cat /boot/config/udev/50_enable_thunderbolt.rules ACTION=="add", SUBSYSTEM=="thunderbolt", \ ATTR{authorized}=="0", \ ATTR{authorized}="1", \ RUN+="/bin/sh -c 'echo 1 > /sys/bus/thunderbolt/devices/domain0/0-0/0-1/authorized && echo 1 > /sys/bus/pci/rescan'" root@Tower:~# I also wanted to use ATTR{iommu_dma_protection}=="1", \ ATTR{security}=="dponly", \ but unfortunately my system seems not to support these. Anyway, this solves my issue. I can now reboot/start/stop my VM with iGPU passthrough. It's running for 2 days now I once had to "hard-reset" the Thunderbolt controller and power cycle the monitor a 2nd time after the monitor was off over night. I' not sure why, but I could do the hard-reset without stopping the VM. I will investigate a bit more. Worst case I create a script in the VM that triggers at login and reset the Thunderbolt controller on the host or something similar. @SimonF and @JorgeB thank you very much for your time and help 💗 Edited April 7, 20251 yr by Tom082
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.