strauberry Posted March 15 Share Posted March 15 Hello @Gazeley and first of all, thank you for taking the time to publish this guide here. I'm following your setup here now in March of 2023 and it seems as though some new requirements may have been added to the `homeserver.yaml` specification. Upon trying to execute: register_new_matrix_user http://localhost:8008 -c /data/homeserver.yaml We get: No 'registration_shared_secret' or 'registration_shared_secret_path' defined in config. I'm new to Matrix Synapse, but I'm assuming there has been some change in requirements from the server side since the time this guide was written. In order to get beyond this error you will now need to add a fourth secret key, like how Gazeley shows us during the `homeserver.yaml` generation. # The shared secret used for the Registration of new users registration_shared_secret: "yourrandomlygeneratedstring" I just added this to the bottom of my config file, using the same shell command we used to generate the other random strings as the key. This is where I'm at in the guide now, I will continue to post any new snags or stuck points as I continue following along here in order to preserve the relevance of this article and help others with their Matrix journeys. Cheers! Quote Link to comment
strauberry Posted March 15 Share Posted March 15 Hello it's me again, providing another update to a snag I hit in this guide. It appears that default permissions aren't working as expected. I have reached out to Unraid's package maintainer A75G and let him know what I found. The `matrix` package we use in Unraid's CA repo is going to need a little bit of a touchup moving forward in 2023. https://github.com/A75G/docker-templates/issues/77 Basically, if you look at the Container log you will see out of box permissions issues. The steps I took to resolve are attached inside that Github issue link. I had to manually modify some folder permissions to where the `/data` volume mapping was. And then I needed to update the `log.config` file to change the path where it attempts to create `homeserver.log` to exist in `/data/homeserver.log`. After restarting the container and making the modifications here, I was able to proceed beyond the step of running the following command: register_new_matrix_user http://localhost:8008 -c /data/homeserver.yaml Before doing this I would just get a massive traceback because the service running on 8008 was not even starting. Quote Link to comment
strauberry Posted March 15 Share Posted March 15 (edited) Me again, Everything was good up until this point, I got a positive test result from https://federationtester.matrix.org/ with no errors, however... The next snag I hit in the guide was when we got to the part about setting up `synapse-admin` and `element-web`. Neither of these worked for me, but luckily I got reproducible errors output in the JavaScript console for both services that indicated a CORS issue: Access to fetch at 'https://matrix.yourdomain.com/.well-known/matrix/client' from origin 'http://1.2.3.4:8075' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled. It has been a long time since I've diagnosed a CORS related issue. I'm not the greatest web host. But I *think* the right thing to do here is to amend the `matrix.subdomain.conf` file we added to our swag proxy-confs. I'm going to take a stab at this and report back. EDIT 01: - Tried all sorts of crazy things to the SWAG Nginx configs with no success yet. Just can't seem to get beyond CORS. WIll continue hacking... EDIT 02: - Tapping out for now possibly. I did all sorts of experimentation with CORS headers using `curl` requests and found that there is just something going on with my matrix subdomain specifically. CURL is reporting a (60) SSL certificate problem only on this subdomain. It is completely fine for the `element` subdomain and I can add headers there without issue in nginx. Hope to have better luck another day. EDIT 03: - So my server is working... kindof. I decided to test how things would behave outside my network on my LTE connection and I got way different results. Also just learned that according to https://github.com/vector-im/element-web/ they don't even recommend running this application on the same domain. Maybe will just kind of put my focus elsewhere for now. I was able to connect to my Matrix homeserver from my personal LTE device outside the network using the admin I created during setup. So the basic functionality is working... - It seems like (at least for me) this guide resulted in a situation where curl tests have SSL problems when they are performed inside the same network. When I curl from my LTE device for example, I have no issues. When I curl from inside the network to my matrix subdomain it complains about cert problems. At this point we are *could* be experiencing something at my firewall level. I do notice the pfsense trips a DNS Rebinding attack warning when I try to navigate directly to the matrix subdomain. I will continue to explore all these topics and hopefully make more progress tomorrow. Edited March 15 by strauberry update on all the things (gahhhhh) Quote Link to comment
Gazeley Posted March 15 Author Share Posted March 15 (edited) 14 hours ago, strauberry said: - It seems like (at least for me) this guide resulted in a situation where curl tests have SSL problems when they are performed inside the same network. When I curl from my LTE device for example, I have no issues. When I curl from inside the network to my matrix subdomain it complains about cert problems. At this point we are *could* be experiencing something at my firewall level. I do notice the pfsense trips a DNS Rebinding attack warning when I try to navigate directly to the matrix subdomain. I will continue to explore all these topics and hopefully make more progress tomorrow. Hi @strauberry, I seem to remember having a similar network issue and for me the magic solution was this setting in opnsense. Do you have something similar in pfsense and is it checked? Thank you for documenting your experience. I'm sorry it hasn't gone smoothly (I can relate) but if you manage to get all the kinks worked out I'll update the guide from your feedback. I've already added the part about generating a registration key. Edited March 15 by Gazeley 1 Quote Link to comment
Gazeley Posted March 15 Author Share Posted March 15 On 10/24/2022 at 9:31 PM, blaine07 said: I downloaded the Synapse Admin; I see that it lets me create registration tokens. How can I set this up to not allow any one to register but to require one of the tokens from Synapse Admin? I'm not aware of a way to do that from Synapse Admin, but you you can disable new user registration by adding this to your homerserver.yaml: enable_registration: false Quote Link to comment
strauberry Posted March 15 Share Posted March 15 (edited) 16 hours ago, Gazeley said: Hi @strauberry, I seem to remember having a similar network issue and for me the magic solution was this setting in opnsense. Do you have something similar in pfsense and is it checked? Thank you so much @Gazeley this was the missing piece. It was an enormous relief to wake up to your message here today. EDIT: ignore the 1:1 checkbox, that was not necessary I wound up removing that setting when I learned more about it Although admittedly I do not yet really have a full comprehension of why this worked, these settings just fixed all my problems: - Set NAT Reflection mode for Port Forwards to "Pure NAT" - Enable NAT Reflection for 1:1 NAT - Enable automatic outbound NAT for Reflection I can now access synapse-admin without issue, along with the element-ui subdomain in my browser. Everything just works now as expected. I will be able to move forward in the guide now long to the COTURN section and begin making customizations to my homeserver.yaml now. So much appreciated, have a great week. Edited March 16 by strauberry Correction Quote Link to comment
strauberry Posted March 16 Share Posted March 16 (edited) Alright! So we have arrived at the final step in the guide. I've hit a couple of problems in the COTURN section. The first one was very minor: - Guide doesn't mention that this `turnserver.conf` is actually a directory in the image (kind of odd, looked like a mistake to me). We say here to "edit the turnserver.conf" file but in my most recent run through when I created the coturn image there was no actual config file here. It was just a directory named `turnserver.conf` which I"m assuming the author intended on you creating the config file with the same name inside if that directory? - This actually kind of through me in a roundabout where I first deleted the directory because I thought it was a mistake, and then the container wouldn't restarted so I had to update the config volume mapping. I'm partially wondering if I need to just start over at this point and test again with `/turnserver.conf/turnserver.conf` which I think is what they expect by default. On 9/1/2022 at 5:50 PM, Gazeley said: Navigate to /appdata/coturn and edit turnserver.conf so it looks like this: So right, got through that hiccup and the server starts - I am getting back ping responses from the static server address I assigned to the COTURN container. I've updated the config file with all the things. We've got some certificate related warning in the logs, but otherwise nothing major sticks out to me here. 0: : WARNING: cannot find certificate file: turn_server_cert.pem (1) 0: : WARNING: cannot start TLS and DTLS listeners because certificate file is not set properly 0: : WARNING: cannot find private key file: turn_server_pkey.pem (1) 0: : WARNING: cannot start TLS and DTLS listeners because private key file is not set properly Here's where we're stuck now: Attempted a test call and we cannot actually make connections with my other users. Right not it just hangs at "Connecting....". We get a dial tone and the system does recognize that VOIP is enabled on my Matrix home server, but I suspect there is just some more firewall issues for me to work through. Using an external port checking utility, we observe 3478 as being closed for me at the moment. So my next course of action is to try to learn how to use the https://icetest.info/ tool you mentioned and see if I can get to the bottom of the final connectivity issue here. Getting so close now! Edited March 16 by strauberry Proof Quote Link to comment
strauberry Posted March 16 Share Posted March 16 A lot to unpack here with the COTURN server. Here are my findings thus far: The config file in the example has some stuff in it that is not really being parsed out well: The log is throwing out: CONFIGURATION ALERT: You specified --lt-cred-mech and --use-auth-secret in the same time. Be aware that you could not mix the username/password and the shared secret based auth methods. Shared secret overrides username/password based auth method. Check your configuration! So we should probably toss the `lt-cred-mech` option in the turnserver.conf it just seems to be throwing an error. I'm also not really clear on some of the reasoning behind the attribute settings in the turnserver.conf file such as "external-ip". It seems like this would expect the WAN address or the hostname instead of the internal static binding we gave the container. But there was probably a good reason you guys set it that way. I'm just confused. The good news: We were able to make a connection over LAN inside out network. So we can assume that the very basic functionality of COTURN is working and implemented now. The bad news: NAT traversal issues. We cannot establish connections with people outside of our firewall. I have done some pretty extensive basic probe testing through nc and ntop to verify that the main listener ports are open. Those looked fine. I couldn't really get any positive reading off the range we specified (49152-49172). I have no idea what I'm doing, oh well #hacktheplanet My journey continues... Quote Link to comment
strauberry Posted March 16 Share Posted March 16 Great news! I found it! This was one of those situations where I just had too much going on, too many moving parts and variables to see the simple thing right in front of me. It turns out that PFSense had automatically created it's firewall rules for the COTURN stuff under a default deny rule. I just didn't catch it right away. Once I moved the ordering of my firewall rules around, communication worked perfectly. So that brings us to the very end of this guide! Anyone who is considering implementing their own Matrix Synapse server in 2023 can be assured that this process here does work (there be dragons). It is a bit of a heavy lift in terms of implementation but ultimately worth the blood sweat and tears once you have your home server running. Matrix is amazing, Element is amazing. I couldn't be happier. Big shoutout to @Gazeley for the help and contribution of this guide. I think I'm still going to pursue working out the final console warning regarding the COTURN SSL certs just to get it out of the way. But otherwise all the basic functionality is running great now. Quote Link to comment
strauberry Posted March 16 Share Posted March 16 (edited) I have discovered one last thing worth mentioning while going through the effort to get my COTURN server working on SSL. It looks like the matrix synapse Docker package that we are using already contains a COTURN server in it by default: https://github.com/AVENTER-UG/docker-matrix This is why we see a whole bunch of warning and turn related output in the logs for the matrix container. It took me a while to realize what was actually happening. We probably don't want two different turn servers running like this? They do provide a way of changing this behavior via Environment Variable of COTURN_ENABLE="false" I'm testing out if everything is stable right now after adding this variable to my Matrix Container so we don't have two COTURN services running: Edited March 16 by strauberry Clarification Quote Link to comment
masterdot Posted April 28 Share Posted April 28 Maybe someone can help me a bit? I have everything installed now, but the server does start and stop. This here is the only output I get: Setting ownership on /data to 99:100 Config file '/data/homeserver.yaml' already exists. Generating any missing config files. Setting ownership on /data to 99:100 Config file '/data/homeserver.yaml' already exists. Generating any missing config files. Setting ownership on /data to 99:100 Config file '/data/homeserver.yaml' already exists. Generating any missing config files. Setting ownership on /data to 99:100 Config file '/data/homeserver.yaml' already exists. Generating any missing config files. ** Press ANY KEY to close this window ** Nothing else. My config looks like this here: server_name: "chatserver.domain.com" pid_file: /data/homeserver.pid web_client_location: https://element.yourdomain.com public_baseurl: https://chat.yourdomain.com listeners: - port: 8008 tls: false type: http x_forwarded: true bind_addresses: ['0.0.0.0'] resources: - names: [client,federation] compress: false database: name: psycopg2 args: user: matrix password: "passwd" database: matrix host: 192.168.178.22 port: 5432 cp_min: 5 cp_max: 10 log_config: "/data/chatserver.domain.com.log.config" media_store_path: "/data/media_store" suppress_key_server_warning: true report_stats: false macaroon_secret_key: "komplexKey" form_secret: "komplexKey" signing_key_path: "/data/chatserver.domain.com.signing.key" serve_server_wellknown: true trusted_key_servers: - server_name: "matrix.org" ## TURN ## # The public URIs of the TURN server to give to clients turn_uris: ["turn:chatserver.domain.com:3478?transport=udp", "turn:chatserver.domain.com:3478?transport=tcp"] # The shared secret used to compute passwords for the TURN server turn_shared_secret: "komplexKey" # New User Registration registration_shared_secret: "komplexKey" # vim:ft=yaml The firewall is configured with forwarding like this: Matrix Server TCP 10.10.10.234 8008 STUN Server TCP 10.10.10.234 3478 STUN Server UDP 10.10.10.234 3478 I did config a proxy with nginx proxy manager with ssl for the domain and a forward to: http://192.168.178.22:8008 I do get a 502 message if I try to access from outside. Any help appreciated Quote Link to comment
Moppen Posted May 12 Share Posted May 12 First of all, thank you very much for the comprehensive how-to. Anyhow I can't get the Matrix-Server off the ground: 12.05.2023 22:47:50 -=> generate turn config 12.05.2023 22:47:50 /start.sh: line 153: pwgen: command not found 12.05.2023 22:47:50 /start.sh: line 17: : No such file or directory 12.05.2023 22:47:50 /start.sh: line 18: : No such file or directory 12.05.2023 22:47:50 /start.sh: line 19: : No such file or directory 12.05.2023 22:47:50 /start.sh: line 20: : No such file or directory 12.05.2023 22:47:50 /start.sh: line 21: : No such file or directory 12.05.2023 22:47:50 /start.sh: line 22: : No such file or directory 12.05.2023 22:47:50 /start.sh: line 23: : No such file or directory 12.05.2023 22:47:50 -=> generate synapse config 12.05.2023 22:47:50 /start.sh: line 24: : No such file or directory 12.05.2023 22:47:53 Traceback (most recent call last): 12.05.2023 22:47:53 File "<frozen runpy>", line 198, in _run_module_as_main 12.05.2023 22:47:53 File "<frozen runpy>", line 88, in _run_code 12.05.2023 22:47:53 File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 398, in <module> 12.05.2023 22:47:53 main() 12.05.2023 22:47:53 File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 388, in main 12.05.2023 22:47:53 hs = setup(sys.argv[1:]) 12.05.2023 22:47:53 ^^^^^^^^^^^^^^^^^^^ 12.05.2023 22:47:53 File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 297, in setup 12.05.2023 22:47:53 config = HomeServerConfig.load_or_generate_config( 12.05.2023 22:47:53 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 12.05.2023 22:47:53 File "/matrix/venv/lib/python3.11/site-packages/synapse/config/_base.py", line 741, in load_or_generate_config 12.05.2023 22:47:53 if not path_exists(config_path): 12.05.2023 22:47:53 ^^^^^^^^^^^^^^^^^^^^^^^^ 12.05.2023 22:47:53 File "/matrix/venv/lib/python3.11/site-packages/synapse/config/_base.py", line 150, in path_exists 12.05.2023 22:47:53 raise e 12.05.2023 22:47:53 File "/matrix/venv/lib/python3.11/site-packages/synapse/config/_base.py", line 146, in path_exists 12.05.2023 22:47:53 os.stat(file_path) 12.05.2023 22:47:53 PermissionError: [Errno 13] Permission denied: '/data/homeserver.tmp' 12.05.2023 22:47:53 -=> configure some settings in homeserver.yaml 12.05.2023 22:47:53 mv: cannot move '/tmp/tmp.f2b7gI1MYB' to '': No such file or directory 12.05.2023 22:47:53 mv: cannot stat '/data/homeserver.tmp': Permission denied 12.05.2023 22:47:53 -=> configure some settings in chat.bingery.de.log.config 12.05.2023 22:47:53 sed: can't read /data/chat.bingery.de.log.config: Permission denied 12.05.2023 22:47:53 12.05.2023 22:47:53 -=> you have to review the generated configuration file homeserver.yaml 12.05.2023 23:11:24 -=> generate turn config 12.05.2023 23:11:24 /start.sh: line 153: pwgen: command not found 12.05.2023 23:11:24 /start.sh: line 17: : No such file or directory 12.05.2023 23:11:24 /start.sh: line 18: : No such file or directory 12.05.2023 23:11:24 /start.sh: line 19: : No such file or directory 12.05.2023 23:11:24 /start.sh: line 20: : No such file or directory 12.05.2023 23:11:24 /start.sh: line 21: : No such file or directory 12.05.2023 23:11:24 /start.sh: line 22: : No such file or directory 12.05.2023 23:11:24 /start.sh: line 23: : No such file or directory 12.05.2023 23:11:24 /start.sh: line 24: : No such file or directory 12.05.2023 23:11:24 -=> generate synapse config 12.05.2023 23:11:25 Traceback (most recent call last): 12.05.2023 23:11:25 File "<frozen runpy>", line 198, in _run_module_as_main 12.05.2023 23:11:25 File "<frozen runpy>", line 88, in _run_code 12.05.2023 23:11:25 File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 398, in <module> 12.05.2023 23:11:25 main() 12.05.2023 23:11:25 File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 388, in main 12.05.2023 23:11:25 hs = setup(sys.argv[1:]) 12.05.2023 23:11:25 ^^^^^^^^^^^^^^^^^^^ 12.05.2023 23:11:25 File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 297, in setup 12.05.2023 23:11:25 config = HomeServerConfig.load_or_generate_config( 12.05.2023 23:11:25 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 12.05.2023 23:11:25 File "/matrix/venv/lib/python3.11/site-packages/synapse/config/_base.py", line 741, in load_or_generate_config 12.05.2023 23:11:25 if not path_exists(config_path): 12.05.2023 23:11:25 ^^^^^^^^^^^^^^^^^^^^^^^^ 12.05.2023 23:11:25 File "/matrix/venv/lib/python3.11/site-packages/synapse/config/_base.py", line 150, in path_exists 12.05.2023 23:11:25 raise e 12.05.2023 23:11:25 File "/matrix/venv/lib/python3.11/site-packages/synapse/config/_base.py", line 146, in path_exists 12.05.2023 23:11:25 os.stat(file_path) 12.05.2023 23:11:25 PermissionError: [Errno 13] Permission denied: '/data/homeserver.tmp' 12.05.2023 23:11:25 -=> configure some settings in homeserver.yaml 12.05.2023 23:11:25 mv: cannot move '/tmp/tmp.M79aIw5ux3' to '': No such file or directory 12.05.2023 23:11:25 mv: cannot stat '/data/homeserver.tmp': Permission denied 12.05.2023 23:11:25 -=> configure some settings in chat.bingery.de.log.config 12.05.2023 23:11:25 sed: can't read /data/chat.bingery.de.log.config: Permission denied 12.05.2023 23:11:25 12.05.2023 23:11:25 -=> you have to review the generated configuration file homeserver.yaml Container stopped Because of the permission denied errors I thought of file permission problem, but I checked /appdata and the homeserver.yaml but nothing there I think. My config: server_name: "chat.bingery.de" pid_file: /data/homeserver.pid web_client_location: https://element.bingery.de public_baseurl: https://chat.bingery.de listeners: - port: 8008 tls: false type: http x_forwarded: true bind_addresses: ['0.0.0.0'] resources: - names: [client,federation] compress: false database: name: psycopg2 args: user: root password: "SecretKey" database: matrix host: hal9000 port: 5432 cp_min: 5 cp_max: 10 log_config: "/data/chat.bingery.de.log.config" media_store_path: "/data/media_store" suppress_key_server_warning: true report_stats: false macaroon_secret_key: "SecretKey" form_secret: "SecretKey" signing_key_path: "/data/chat.bingery.de.signing.key" serve_server_wellknown: true trusted_key_servers: - server_name: "matrix.org" ## TURN ## # The public URIs of the TURN server to give to clients turn_uris: ["turn:chat.bingery.de:3478?transport=udp", "turn:chat.bingery.de:3478?transport=tcp"] # The shared secret used to compute passwords for the TURN server turn_shared_secret: "SecretKey" # New User Registration registration_shared_secret: "SecretKey" # vim:ft=yaml (Only edited out the passwords) Any Ideas, anybody? Quote Link to comment
Lavoslav Posted June 22 Share Posted June 22 I'm stuck at install Matrix step. Directory matrix gets created in appdata folder but remains completely empty. No homeserver.yaml no directories, not a single file or directory inside. This is the log output of matrix docker upon first run: (SUBDOMAIN.MY_DOMAIN parts are sanitised) /start.sh: line 153: pwgen: command not found /start.sh: line 17: : No such file or directory /start.sh: line 18: : No such file or directory /start.sh: line 19: : No such file or directory /start.sh: line 20: : No such file or directory /start.sh: line 21: : No such file or directory /start.sh: line 22: : No such file or directory /start.sh: line 23: : No such file or directory /start.sh: line 24: : No such file or directory Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 398, in <module> main() File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 388, in main hs = setup(sys.argv[1:]) ^^^^^^^^^^^^^^^^^^^ File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 297, in setup config = HomeServerConfig.load_or_generate_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/matrix/venv/lib/python3.11/site-packages/synapse/config/_base.py", line 768, in load_or_generate_config with open(config_path, "w") as config_file: ^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied: '/data/homeserver.tmp' mv: cannot move '/tmp/tmp.BfJK0wnjWe' to '': No such file or directory mv: cannot stat '/data/homeserver.tmp': No such file or directory sed: can't read /data/SUBDOMAIN.MY_DOMAIN.com.log.config: No such file or directory -=> generate turn config -=> generate synapse config Generating config file /data/homeserver.tmp -=> configure some settings in homeserver.yaml -=> configure some settings in SUBDOMAIN.MY_DOMAIN.com.log.config -=> you have to review the generated configuration file homeserver.yaml ** Press ANY KEY to close this window ** Any pointers would be apprechiated.. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.