I would also like exactly this.
I tried using this example which uses caddy and docker-compose, but I couldn't even get that to work.
My current issue with this solution is getting the containers to be able to access each other. The AIO container was designed with the assumption that the user will access all services from the same base URL (e.g. nextcloud.whatever.com serves collabora from nextcloud.whatever.com/collabora, this is a made up example). But, if each container were separately added to tailscale, then the base URL would change. So, you have to bundle them all together and sidecar tailscale so that they are all served from the same tailscale URL. Anyway, I don't know enough about caddy and it's been painful to even get this far.