Display scaling on Linux guests

[Xaero]

So I've got an ArchLinux guest, I've also tried Manjaro and Antergos. I've installed and started+enabled the spice-vdagentd service; video is set to QXL.
If I set the builtin VNC Viewer to remote resize and resize the window or set fullscreen the display size never changes.
What am I missing for this setup?
I've read through the Archwiki page as well as what few posts I can find on the unraid website but nothing regarding this.