When you go to the VM's tab, select adn edit the VM you want to change the VNC port, and select edit, make sure the VM is turned off.
In the next screen, in the top right corner is a slider, called XML view, slide it.
The page will change in a custom edit screen.
Look for the following;
<graphics type='vnc' port='-1' autoport='yes' websocket='5703' listen='0.0.0.0' keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
You see that the port is set to -1, in other words "auto", this is repeated in the command after that, autoport=yes.
Changing these values, will change them.
eg.
<graphics type='vnc' port='5905' autoport='no' websocket='5703' listen='0.0.0.0' keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
In the Case above VNC will listen to port 5905, nothing else.
Keep in mind that you have to do this every time you make a change in the "normal" edit screen.
Good Luck