Sorry for late reply, but had to resolve some issues with New UnRAID update, but now that everything is back to normal, its working normally without changing anything, all i did was re-flash USB with Newest UnRAID and copy Config!
I still have no clue why it wasn't working before.
my setup:
Windows 10 VM:
with Both VNC and Nvidia.
Nvidia for sound.
UEFI boot mode for UnRAID
IOMMU: Enabled
PCIe ACS override: Downstream
Motherboard:
Motherboard doesn't have HDMI only DP and i don't have dummy DP plug.
Nvidia:
Plugged a dummy HDMI to the GTX1060
BIOS:
Set Initial Graphics Adapter: IGD (integrated graphics device) - setting it to PEG and enabling IGD multi-monitor didn't work.
this was my last setup before i gave up and started this Topic here, but then there was an update for UnRAID and it caused some issues and i had to re-flash USB drive and copy Config and when i booted the VM for the first time, everything was working normally, Nvidia drivers are installed correctly and there is no yellow exclamation mark on the device anymore.
I noticed something when having both VNC and Nvidia on a VM, the VNC screen only works while booting and then when VM reaches the windows login screen VNC stops Working, is that normal behavior ?