My current path for the server is (Unraid Server) -> (Dumb switch - Eth1) -> (EdgeRouter 4) -> (ONT)
I have hairpin/loopback NAT disabled
I do have a masquerade for outbound traffic to VLAN eth0.0, not sure if that qualifies as a double NAT.
Edit: I got my set up working without the masquerade, still no luck...