It seems there are quite a few things I don't know yet about how Unraid actually works, my apologies. I also removed my upvote.
This is the appdata share settings now, 'Use cache pool' as set to 'Prefer' before, I set it to 'Only' now.
I also checked in the user folder, and indeed I find the same folder structure there for Palworld, down to the config file, which has the same changes I made to it: max pals in base =20 and my custom password. So, if the custom password is on both locations (user and cache), should it not work either way?
After changing the settings like in the screenshot (with palserver not running) and then starting it up, nothing changed, I can still connect without a password. Or with a wrong password.
Oh, I also changed the port today, in the container settings, that worked as expected, I can now only connect under the new port.