blacklight

It turns out that it was a remnant docker network I created via Portainer.
I found this entry:
192.168.0.0/24 dev br-75ff69ec12b1 proto kernel scope link src 192.168.0.254 linkdown

with docker network ls
Which has the same exact subnet as the one on the bridge. From my understanding I thought the same subnets are possible in isolted docker networks (like the shim-networks),
so I can not explain why this error occured.

Deleting the network and restarting the service fixed the communcation error.

SOLVED ! See here: 
 
The problem was the SMB share !

Ok you never guessed what it was .... silly me thought the pools work like the array in terms of layout. Apparently you can not have empty slots in a pool ... 🤦‍♂️

Well I knew it was nothing complicated because I was able to mount and use the disks via console, which made them invisible from the UI.
After only using 2 slots it worked like a charm. I don't like that fact the the error message is very abitrary and I couldnt find the fact very easily but now I do undertand  
 
Also I am using this pool as a separate media storage, so I disabled all cahcing and moving settings, which completely cuts the pool of from my array (which is only for system data)

That beeing said, I am realy looking forward that Unraid adds "Array" as a pool type so one can setup multiple Arrays with potentially multiple cache drives. 
One main reason I would love that, is because my main storage is on a Truenas vm and I only have system data (docker, small VM images, plugins, etc.) on my array, which gurantees a small resilvering time when one of my 500gb drives fails. Also I could go with double parity on the system files, which I don't need on my movie collection.
 
Thanks anyway  

Hey there, I just wanted to let you know that I finally solved it after all the time. It was quite an odysse with the system because I unfortunately was also affected by the overvoltage bug from Intel Raptor-Lake. I dont know what caused the vfio dma map -22 error, but it was definietly something unresolved from qemu @SimonF.

Setp by step I fixed the problems over months:
 
vfio dma map -22:
I researched through qemu documentations and found a simple trick through try-and-error. My guess was that TrueNAS was causing a problem, if virtualized on Raptor-Lake, because it wouldn't run on the newest generation CPUs bare-metal (at least CORE wouldn't). By using older hardware with older CPUs I wouldn't have ran into that problem, but unfortunately when I began the project I did the mistake and avoided older Xeons and chose a newer i9 in the same price range with more performance. Not ideal but qemu allows for CPU emmulation so that is axactly what I did. I basically emulated an older CPU on top of the i9 with the <model name='Cascadelake-Server'/> entry. So my xml entry looks like this initally:
<cpu mode='custom' match='exact' check='none'> <model name='Cascadelake-Server'/> <topology sockets='1' dies='1' cores='6' threads='1'/> <cache mode='passthrough'/> <maxphysaddr mode='passthrough'/> </cpu> However this initial xml entry was changed automatically by Unraid to:
<cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>qemu64</model> <topology sockets='1' dies='1' cores='6' threads='1'/> <feature policy='require' name='x2apic'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='lahf_lm'/> <feature policy='disable' name='svm'/> </cpu>  
I don't know, if the automatic change was set by Unraid or the switch between XML and form view and I also don't know if the XML change reverted anything but it works for me right now and the system is now error free for 3 weeks and the log is clean since then 🥳
Definitely check out the qemu documentation and also their channels on the website. They are very helpful !
 
Freezing Unraid UI:
So this was automatically resolved with the change of the xml and since there is no VFIO_DMA_MAP error, the UI never bugged out. But from my point of view this is definitely a UNRAID/qemu problem and not a user error, because from what I saw/think, a bugged out VM should not have such effects on the hypervisor even more so when the VM is initally running normally. If the hypervisor crashes with the VM and logs the error or kills the VM to protect system stability, that would be a behavior I could interpret. But if such an unconnected UI bug happens I would never have guessed that this is connected to something like CPU incompatibility ... I am not a developer and I really appreciate the work put into Unraid but this simple bug did cost me months, because it forced me to restart the hypervisor and it made troubleshooting a VM so much more painful !
 
System not SHUTTING down:
So this was my fault, because I implemented ugly start & stop user scripts, that mount and unmount shares of the NAS vm and the NFS shares weren't actually unmounted properly ... oops well what helped here is correcting the user scripts and using the plugin "Tips and Tweaks" to shut down any opened consoles. Anyway it feels weird, that Unraid is not suceeding a force shutdown just because a NFS was buggy during unmount. Any tips here for a global unmount would be appreciated, I still add them manually into a user script that looks something like this:
#!/bin/bash sleep 150 # Wait for TrueNAS boot mkdir /mnt/remotes/mnt01... mkdir /mnt/remotes/mnt02... mkdir /mnt/remotes/mnt03... # # mount -t nfs 192.168.0... /mnt/remotes/mnt01... mount -t nfs 192.168.0... /mnt/remotes/mnt02... mount -t cifs -o username=...,password=... //192.168.0.... /mnt/remotes/mnt03... # # docker start *dockername* # delayed docker that depend on NAS Same goes for unmounting. The user scripts are then set to be either executed on startup or shutdown ... works the best for me.
 
Also ...:
-> ... I obvisouly updated the MB Bios for the new Intel microcode. Just wanted to mention it here. I think that wasn't the reason of any of the above mentioned probelms, but better safe than sorry.
-> ... I switched from a 9300 LSI HBA to a 9500 LSI HBA with a SAS-Expander to have all drives in one IOMMU group for the VM (Alle errors persitet afterwards, I double checked !). Same here: I don't think that should be a problem for Unraid, but it was one of the variables I also changed/eliminated.
 
Hope that helps someone.