Did you check on the VMs tab that port 5900 was the one actually being used by the VM? The port is dynamically assigned (starting at 5900) to VMs so it could well be slightly different. Also, is the IP address you used that of the VM or of the Unraid server - you need to use the Unraid server address as it is the Unraid server providing the VNC server capability.
I have found you get much better performance and more functionality if you install something like noMachine server into the VM and then use a noMachine client to access the VM (this WOULD use the VMs IP address).