I installed the recommended tensorchord/pgvecto-rs docker container (if you pick the latest build, Immich will complain about it. It'll tell you what version to use in the logs).
The github install instructions says to enable the vector plugin by running the following command:
DROP EXTENSION IF EXISTS vectors;
CREATE EXTENSION vectors;