Thinking about it more and I definitely think point 1 was the solution.
When you make the request to go to your domain it will be https (port 443) and pretty sure pihole takes that right away before it can get to your router (which would forward to port 1443). So then the request is going to your server at port 443 (instead of 1443) and thus you cannot connect to nextcloud directly.
I couldn't find a way in pihole configs to forward 443 requests to 1443 with DNS records so setting swag to 443 fixed the issue if I recall correctly.