Tom082

I tried modifying in the runtime environment. This seems to work. In order not to loose the keyboard and mouse I left the rescan in for now. I will experiment with that later. Here are the dmesg log I get now:

POWER-OFF
[1486828.614255] thunderbolt 0-1: device disconnected [1486829.474582] pcieport 0000:1b:00.0: Unable to change power state from D3hot to D0, device inaccessible [1486829.476126] pcieport 0000:1b:04.0: Unable to change power state from D3hot to D0, device inaccessiblePOWER-ON
[1486847.852133] thunderbolt 0-1: new device found, vendor=0xd4 device=0xc045 [1486847.852139] thunderbolt 0-1: DELL U2724DE [1486848.018060] pcieport 0000:1a:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring [1486848.018159] pcieport 0000:1b:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring [1486848.018172] pcieport 0000:1b:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring [1486848.018184] pcieport 0000:1b:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring [1486848.018195] pcieport 0000:1b:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring [1486848.018216] pcieport 0000:1b:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring [1486848.018250] pci_bus 0000:1c: busn_res: [bus 1c] end is updated to 1c [1486848.018282] pci_bus 0000:1d: busn_res: [bus 1d-2b] end is updated to 2b [1486848.018312] pci_bus 0000:2c: busn_res: [bus 2c-3a] end is updated to 3a [1486848.018341] pci_bus 0000:3b: busn_res: [bus 3b-48] end is updated to 48 [1486848.018371] pci_bus 0000:49: busn_res: [bus 49] end is updated to 49 [1486848.018377] pci_bus 0000:1b: busn_res: [bus 1b-49] end is updated to 49 [1486849.138586] thunderbolt 0-1: device disconnected [1486849.740645] pcieport 0000:1b:04.0: Unable to change power state from D3hot to D0, device inaccessible [1486849.742187] pcieport 0000:1b:00.0: Unable to change power state from D3hot to D0, device inaccessible [1486852.641049] thunderbolt 0-1: new device found, vendor=0xd4 device=0xc045 [1486852.641054] thunderbolt 0-1: DELL U2724DE [1486852.804162] pcieport 0000:1a:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring [1486852.804260] pcieport 0000:1b:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring [1486852.804272] pcieport 0000:1b:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring [1486852.804284] pcieport 0000:1b:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring [1486852.804295] pcieport 0000:1b:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring [1486852.804315] pcieport 0000:1b:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring [1486852.804350] pci_bus 0000:1c: busn_res: [bus 1c] end is updated to 1c [1486852.804383] pci_bus 0000:1d: busn_res: [bus 1d-2b] end is updated to 2b [1486852.804414] pci_bus 0000:2c: busn_res: [bus 2c-3a] end is updated to 3a [1486852.804443] pci_bus 0000:3b: busn_res: [bus 3b-48] end is updated to 48 [1486852.804473] pci_bus 0000:49: busn_res: [bus 49] end is updated to 49 [1486852.804479] pci_bus 0000:1b: busn_res: [bus 1b-49] end is updated to 49My guess is that the disconnect reconnect cycle happens due to the rescan. Correct?

What are those power state errors I'm getting now and why did I not see them before? How can I fix those?
They are related to Thunderbolt:
root@blackbox:~# lspci | grep 1a: 1a:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen Ridge 2020] (rev 03) root@blackbox:~# lspci | grep 1b: 1b:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen Ridge 2020] (rev 03) 1b:01.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen Ridge 2020] (rev 03) 1b:02.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen Ridge 2020] (rev 03) 1b:03.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen Ridge 2020] (rev 03) 1b:04.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen Ridge 2020] (rev 03) root@blackbox:~#
Also I'm not sure if powering off the monitor will immediately invalidate the authorization or if there is some kind a timeout, meaning that if I power on before the timeout expired the monitor is still authorized and I do not test the udev rule. In case of a timeout I will have more updates tomorrow after the monitor was off over night. So far it looks much improved :-) Thank you so much @bmartino1

However, after running this:

 
Permissions look like this now:

 
Dolphin is reporting this:

 
So it's working now.
 
I just don't understand why. The only difference is the execute flag and still the read/write behavior has changed. Can someone please explain?

Thanks a lot for you help, it's highly appreciated!
 
Luckily I do not have data loss at least nothing important. I'm new to Unraid and Linux. I just set this server up last week and so far I'm mostly playing around, testing and learning.
I have dGPU passthrough working to my gaming VM and have setup my daily-driver VM which should autostart once all is running.
Before this share/pool problem happened I was installing some dockers and working on the iGPU passthrough to the daily VM. All my important data is still on the "old NAS". If I have lost something then its dockers or VMs and this is not a big deal.
 
In the meantime I deleted both the "cache" and "gaming" pool and recreated them as "cache" and "game" but this time with XFS. I learned today that "nested" COW is a bad idea and with my old setup I believe I had 3 levels of COW, 1st the host with ZFS then the vdisks with QCOW2 and last BTRFS in the guests. So recreating the pools also has its upsides.
With the new setup I will have XFS then RAW and the new VMs I will build with LVM partitions and EXT4.
My VMs are not critical so a weekly backup is good enough, I don't rely on a self-healing FS. The important data is on the array and I have the dynamics file integrity plugin installed to at least get bitrot warnings. The super important stuff gets backed up to an external drive and the idea is to setup an additional offsite backup via Tailscale to my brother's place (and vice versa). So I should be covered.
 
Currently I'm restoring the pools. The copying will take some time . The shares are already setup and this time no complaints from FCP. Tomorrow I will most likely have to fix or recreate my VMs. Let's see what works and what I can recover...

By chance I found a solution that works for me even though the solution does not answer my question how to bind directly to VFIO without blacklisting drivers or only temporary "runtime blacklisting".
 
So here is what happened:
I got distracted by a phone call while my VM once again had the blackscreen issue. After the call I forgot that the VM was still running and was "hard-resetting" both the USB and TBT controller on the host. After that Unraid bound the USB controller to the USB driver and the Thunderbolt controller to VFIO as described above. Suddenly  the monitor had a picture again (at this point I realized that the VM was still running). I would have expected at least a VM crash but all was running fine. Even more wired is the fact that neither VM nor the host had any drivers for Thunderbolt loaded but suddenly I could reboot start/stop the VM without any issues. This only broke after I power cycled the monitor (I guess detach/attach of the cable would have the same effect).
This is when I realized that the only way this is possible is if Thunderbolt is only required for device authorization which happened in the VM when the controller was passed through. As long as the monitor was connected and powered on the authorization remained valid. If this is correct then passthrough of the Thunderbolt controller should also not be required.
 
In the VM I used boltctl in order to authorize Thunderbolt devices. Unfortunately neither boltd nor boltctl or any other tool for thunderbolt exists in Unraid except the driver and I could not find any. All I found is one post in another forum where a guy was building the boltctl package for slackware but this I didn't want to try, this is way over my head at this point.
In the end I found this admin guide.
 
So my new setup does neither passthrough the Thunderbolt nor the USB4 controller, neither driver is blacklisted. The Thunderbolt monitor gets authorized by this udev rule:
 
root@Tower:~# cat /boot/config/udev/50_enable_thunderbolt.rules ACTION=="add", SUBSYSTEM=="thunderbolt", \ ATTR{authorized}=="0", \ ATTR{authorized}="1", \ RUN+="/bin/sh -c 'echo 1 > /sys/bus/thunderbolt/devices/domain0/0-0/0-1/authorized && echo 1 > /sys/bus/pci/rescan'" root@Tower:~#  
I also wanted to use
ATTR{iommu_dma_protection}=="1", \ ATTR{security}=="dponly", \ but unfortunately my system seems not to support these.
 
Anyway, this solves my issue. I can now reboot/start/stop my VM with iGPU passthrough. It's running for 2 days now I once had to "hard-reset" the Thunderbolt controller and power cycle the monitor a 2nd time after the monitor was off over night. I' not sure why, but I could do the hard-reset without stopping the VM. I will investigate a bit more. Worst case I create a script in the VM that triggers at login and reset the Thunderbolt controller on the host or something similar.
 
@SimonF and @JorgeB thank you very much for your time and help 💗
 
 

Except for the Thunderbolt issue the solution for blackscreen and login issues are the following:
 
Blackscreen:
- detach all video cables from the iGPU (this includes USB-C in case of Thunderbolt).
- boot into Unraid
- attach video cables to the iGPU
- start VM
 
Black screen until after login:
- install the VM with VNC plus iGPU otherwise the VM might not start the required drivers
 
For the Thunderbolt instability I opened another thread at Level1