Ok I tried with another PCI-E card (network card), Linux VM and xen-pciback.hide= method. Same problem.
When I start the VM the rounded arrows go round some time before VM actually starts, and the passthrough device isn't there
Edit. Whoa, I tried to passthrough internal USB3 gen1 controllers which are managed by CPU and it actually worked! (Mb: Gigabyte Aorus M and CPU 2200G). Yippee!