[GUIDE] Matrix-Synapse w/ postgres DB (chat server) + Element (web client) + Coturn (voice)

[strauberry]

Hello @Gazeley and first of all, thank you for taking the time to publish this guide here. I'm following your setup here now in March of 2023 and it seems as though some new requirements may have been added to the `homeserver.yaml` specification.

 

Upon trying to execute:

 

register_new_matrix_user http://localhost:8008 -c /data/homeserver.yaml

 

We get: 

 

No 'registration_shared_secret' or 'registration_shared_secret_path' defined in config.

 

I'm new to Matrix Synapse, but I'm assuming there has been some change in requirements from the server side since the time this guide was written. In order to get beyond this error you will now need to add a fourth secret key, like how Gazeley shows us during the `homeserver.yaml` generation. 

 

# The shared secret used for the Registration of new users
registration_shared_secret: "yourrandomlygeneratedstring"

 

I just added this to the bottom of my config file, using the same shell command we used to generate the other random strings as the key.

 

This is where I'm at in the guide now, I will continue to post any new snags or stuck points as I continue following along here in order to preserve the relevance of this article and help others with their Matrix journeys.

 

Cheers! 

[strauberry]

Hello it's me again, providing another update to a snag I hit in this guide.

 

It appears that default permissions aren't working as expected. I have reached out to Unraid's package maintainer A75G and let him know what I found. The `matrix` package we use in Unraid's CA repo is going to need a little bit of a touchup moving forward in 2023. 

 

https://github.com/A75G/docker-templates/issues/77

 

Basically, if you look at the Container log you will see out of box permissions issues. The steps I took to resolve are attached inside that Github issue link. I had to manually modify some folder permissions to where the `/data` volume mapping was. And then I needed to update the `log.config` file to change the path where it attempts to create `homeserver.log` to exist in `/data/homeserver.log`. 

 

After restarting the container and making the modifications here, I was able to proceed beyond the step of running the following command:

 

register_new_matrix_user http://localhost:8008 -c /data/homeserver.yaml

 

Before doing this I would just get a massive traceback because the service running on 8008 was not even starting. 

 

[strauberry]

Me again, 

 

Everything was good up until this point, I got a positive test result from https://federationtester.matrix.org/ with no errors, however...

 

The next snag I hit in the guide was when we got to the part about setting up `synapse-admin` and `element-web`.

 

Neither of these worked for me, but luckily I got reproducible errors output in the JavaScript console for both services that indicated a CORS issue:

 

Access to fetch at 'https://matrix.yourdomain.com/.well-known/matrix/client' from origin 'http://1.2.3.4:8075' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled. 

 

It has been a long time since I've diagnosed a CORS related issue. I'm not the greatest web host. But I *think* the right thing to do here is to amend the `matrix.subdomain.conf` file we added to our swag proxy-confs. I'm going to take a stab at this and report back.

EDIT 01:
- Tried all sorts of crazy things to the SWAG Nginx configs with no success yet. Just can't seem to get beyond CORS. WIll continue hacking...

 

EDIT 02:
- Tapping out for now possibly. I did all sorts of experimentation with CORS headers using `curl` requests and found that there is just something going on with my matrix subdomain specifically. CURL is reporting a (60) SSL certificate problem only on this subdomain. It is completely fine for the `element` subdomain and I can add headers there without issue in nginx. Hope to have better luck another day. 

 

EDIT 03:

- So my server is working... kindof. I decided to test how things would behave outside my network on my LTE connection and I got way different results. Also just learned that according to https://github.com/vector-im/element-web/ they don't even recommend running this application on the same domain. Maybe will just kind of put my focus elsewhere for now. I was able to connect to my Matrix homeserver from my personal LTE device outside the network using the admin I created during setup. So the basic functionality is working...

 

- It seems like (at least for me) this guide resulted in a situation where curl tests have SSL problems when they are performed inside the same network. When I curl from my LTE device for example, I have no issues. When I curl from inside the network to my matrix subdomain it complains about cert problems. At this point we are *could* be experiencing something at my firewall level. I do notice the pfsense trips a DNS Rebinding attack warning when I try to navigate directly to the matrix subdomain. I will continue to explore all these topics and hopefully make more progress tomorrow. 

Edited March 15, 2023 by strauberry
update on all the things (gahhhhh)

[Gazeley]

14 hours ago, strauberry said:

- It seems like (at least for me) this guide resulted in a situation where curl tests have SSL problems when they are performed inside the same network. When I curl from my LTE device for example, I have no issues. When I curl from inside the network to my matrix subdomain it complains about cert problems. At this point we are *could* be experiencing something at my firewall level. I do notice the pfsense trips a DNS Rebinding attack warning when I try to navigate directly to the matrix subdomain. I will continue to explore all these topics and hopefully make more progress tomorrow. 

 

Hi @strauberry, I seem to remember having a similar network issue and for me the magic solution was this setting in opnsense. Do you have something similar in pfsense and is it checked?

 

 

Thank you for documenting your experience. I'm sorry it hasn't gone smoothly (I can relate) but if you manage to get all the kinks worked out I'll update the guide from your feedback. I've already added the part about generating a registration key.

Edited March 15, 2023 by Gazeley

[Gazeley]

On 10/24/2022 at 9:31 PM, blaine07 said:

I downloaded the Synapse Admin; I see that it lets me create registration tokens. How can I set this up to not allow any one to register but to require one of the tokens from Synapse Admin?

 

I'm not aware of a way to do that from Synapse Admin, but you you can disable new user registration by adding this to your homerserver.yaml:

enable_registration: false

 

[strauberry]

16 hours ago, Gazeley said:

Hi @strauberry, I seem to remember having a similar network issue and for me the magic solution was this setting in opnsense. Do you have something similar in pfsense and is it checked?

 

Thank you so much @Gazeley this was the missing piece. It was an enormous relief to wake up to your message here today. 

 

 

EDIT: ignore the 1:1 checkbox, that was not necessary I wound up removing that setting when I learned more about it

 

 

Although admittedly I do not yet really have a full comprehension of why this worked, these settings just fixed all my problems:

 

- Set NAT Reflection mode for Port Forwards to "Pure NAT" 

- Enable NAT Reflection for 1:1 NAT

- Enable automatic outbound NAT for Reflection

 

I can now access synapse-admin without issue, along with the element-ui subdomain in my browser. Everything just works now as expected. I will be able to move forward in the guide now long to the COTURN section and begin making customizations to my homeserver.yaml now. 

 

So much appreciated, have a great week. 

Edited March 16, 2023 by strauberry
Correction

[strauberry]

Alright! So we have arrived at the final step in the guide. I've hit a couple of problems in the COTURN section. The first one was very minor:

 

- Guide doesn't mention that this `turnserver.conf` is actually a directory in the image (kind of odd, looked like a mistake to me). We say here to "edit the turnserver.conf" file but in my most recent run through when I created the coturn image there was no actual config file here. It was just a directory named `turnserver.conf` which I"m assuming the author intended on you creating the config file with the same name inside if that directory? 

- This actually kind of through me in a roundabout where I first deleted the directory because I thought it was a mistake, and then the container wouldn't restarted so I had to update the config volume mapping. I'm partially wondering if I need to just start over at this point and test again with `/turnserver.conf/turnserver.conf` which I think is what they expect by default. 

 

On 9/1/2022 at 5:50 PM, Gazeley said:

Navigate to /appdata/coturn and edit turnserver.conf so it looks like this:

 

So right, got through that hiccup and the server starts - I am getting back ping responses from the static server address I assigned to the COTURN container. I've updated the config file with all the things. We've got some certificate related warning in the logs, but otherwise nothing major sticks out to me here. 

 

0: : WARNING: cannot find certificate file: turn_server_cert.pem (1)
0: : WARNING: cannot start TLS and DTLS listeners because certificate file is not set properly
0: : WARNING: cannot find private key file: turn_server_pkey.pem (1)
0: : WARNING: cannot start TLS and DTLS listeners because private key file is not set properly

 

Here's where we're stuck now:

 

Attempted a test call and we cannot actually make connections with my other users. Right not it just hangs at "Connecting....". We get a dial tone and the system does recognize that VOIP is enabled on my Matrix home server, but I suspect there is just some more firewall issues for me to work through.

 

Using an external port checking utility, we observe 3478 as being closed for me at the moment. So my next course of action is to try to learn how to use the https://icetest.info/ tool you mentioned and see if I can get to the bottom of the final connectivity issue here.

 

Getting so close now!  

Edited March 16, 2023 by strauberry
Proof

[strauberry]

A lot to unpack here with the COTURN server. Here are my findings thus far:

 

The config file in the example has some stuff in it that is not really being parsed out well:

 

The log is throwing out:

 

CONFIGURATION ALERT: You specified --lt-cred-mech and --use-auth-secret in the same time. Be aware that you could not mix the username/password and the shared secret based auth methods. Shared secret overrides username/password based auth method. Check your configuration!

 

So we should probably toss the `lt-cred-mech` option in the turnserver.conf it just seems to be throwing an error. 

 

I'm also not really clear on some of the reasoning behind the attribute settings in the turnserver.conf file such as "external-ip". It seems like this would expect the WAN address or the hostname instead of the internal static binding we gave the container. But there was probably a good reason you guys set it that way. I'm just confused. 

 

The good news:

 

We were able to make a connection over LAN inside out network. So we can assume that the very basic functionality of COTURN is working and implemented now. 

The bad news:
 

NAT traversal issues. We cannot establish connections with people outside of our firewall. I have done some pretty extensive basic probe testing through nc and ntop to verify that the main listener ports are open. Those looked fine. I couldn't really get any positive reading off the range we specified (49152-49172).

 

I have no idea what I'm doing, oh well #hacktheplanet

 

My journey continues...

[strauberry]

Great news! I found it! 

 

This was one of those situations where I just had too much going on, too many moving parts and variables to see the simple thing right in front of me. 

 

It turns out that PFSense had automatically created it's firewall rules for the COTURN stuff under a default deny rule. I just didn't catch it right away. Once I moved the ordering of my firewall rules around, communication worked perfectly. 

 

So that brings us to the very end of this guide! Anyone who is considering implementing their own Matrix Synapse server in 2023 can be assured that this process here does work (there be dragons). It is a bit of a heavy lift in terms of implementation but ultimately worth the blood sweat and tears once you have your home server running. Matrix is amazing, Element is amazing. I couldn't be happier. 

 

Big shoutout to @Gazeley for the help and contribution of this guide. 

 

I think I'm still going to pursue working out the final console warning regarding the COTURN SSL certs just to get it out of the way. But otherwise all the basic functionality is running great now. 

[strauberry]

I have discovered one last thing worth mentioning while going through the effort to get my COTURN server working on SSL. 

 

It looks like the matrix synapse Docker package that we are using already contains a COTURN server in it by default: https://github.com/AVENTER-UG/docker-matrix

 

This is why we see a whole bunch of warning and turn related output in the logs for the matrix container. It took me a while to realize what was actually happening. We probably don't want two different turn servers running like this?

 

They do provide a way of changing this behavior via Environment Variable of COTURN_ENABLE="false"

 

I'm testing out if everything is stable right now after adding this variable to my Matrix Container so we don't have two COTURN services running:

 

Edited March 16, 2023 by strauberry
Clarification

[masterdot]

Maybe someone can help me a bit?

 

I have everything installed now, but the server does start and stop. This here is the only output I get:

Setting ownership on /data to 99:100
Config file '/data/homeserver.yaml' already exists. Generating any missing config files.
Setting ownership on /data to 99:100
Config file '/data/homeserver.yaml' already exists. Generating any missing config files.
Setting ownership on /data to 99:100
Config file '/data/homeserver.yaml' already exists. Generating any missing config files.
Setting ownership on /data to 99:100
Config file '/data/homeserver.yaml' already exists. Generating any missing config files.

** Press ANY KEY to close this window ** 

 

Nothing else. My config looks like this here:

server_name: "chatserver.domain.com"
pid_file: /data/homeserver.pid
web_client_location: https://element.yourdomain.com
public_baseurl: https://chat.yourdomain.com
listeners:
  - port: 8008
    tls: false
    type: http
    x_forwarded: true
    bind_addresses: ['0.0.0.0']
    resources:
      - names: [client,federation]
        compress: false
database:
  name: psycopg2
  args:
    user: matrix
    password: "passwd"
    database: matrix
    host: 192.168.178.22
    port: 5432
    cp_min: 5
    cp_max: 10
log_config: "/data/chatserver.domain.com.log.config"
media_store_path: "/data/media_store"
suppress_key_server_warning: true
report_stats: false
macaroon_secret_key: "komplexKey"
form_secret: "komplexKey"
signing_key_path: "/data/chatserver.domain.com.signing.key"
serve_server_wellknown: true
trusted_key_servers:
  - server_name: "matrix.org"

 ## TURN ##
# The public URIs of the TURN server to give to clients
turn_uris: ["turn:chatserver.domain.com:3478?transport=udp",  "turn:chatserver.domain.com:3478?transport=tcp"] 

# The shared secret used to compute passwords for the TURN server
turn_shared_secret: "komplexKey" 

# New User Registration
registration_shared_secret: "komplexKey"

# vim:ft=yaml

 

The firewall is configured with forwarding like this:

Matrix Server
TCP
10.10.10.234
8008

STUN Server
TCP
10.10.10.234
3478

STUN Server
UDP
10.10.10.234
3478

 

I did config a proxy with nginx proxy manager with ssl for the domain and a forward to:

http://192.168.178.22:8008

I do get a 502 message if I try to access from outside.

 

Any help appreciated

 

[Moppen]

First of all, thank you very much for the comprehensive how-to.

 

Anyhow I can't get the Matrix-Server off the ground:
 

12.05.2023 22:47:50
-=> generate turn config
12.05.2023 22:47:50
/start.sh: line 153: pwgen: command not found
12.05.2023 22:47:50
/start.sh: line 17: : No such file or directory
12.05.2023 22:47:50
/start.sh: line 18: : No such file or directory
12.05.2023 22:47:50
/start.sh: line 19: : No such file or directory
12.05.2023 22:47:50
/start.sh: line 20: : No such file or directory
12.05.2023 22:47:50
/start.sh: line 21: : No such file or directory
12.05.2023 22:47:50
/start.sh: line 22: : No such file or directory
12.05.2023 22:47:50
/start.sh: line 23: : No such file or directory
12.05.2023 22:47:50
-=> generate synapse config
12.05.2023 22:47:50
/start.sh: line 24: : No such file or directory
12.05.2023 22:47:53
Traceback (most recent call last):
12.05.2023 22:47:53
  File "<frozen runpy>", line 198, in _run_module_as_main
12.05.2023 22:47:53
  File "<frozen runpy>", line 88, in _run_code
12.05.2023 22:47:53
  File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 398, in <module>
12.05.2023 22:47:53
    main()
12.05.2023 22:47:53
  File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 388, in main
12.05.2023 22:47:53
    hs = setup(sys.argv[1:])
12.05.2023 22:47:53
         ^^^^^^^^^^^^^^^^^^^
12.05.2023 22:47:53
  File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 297, in setup
12.05.2023 22:47:53
    config = HomeServerConfig.load_or_generate_config(
12.05.2023 22:47:53
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
12.05.2023 22:47:53
  File "/matrix/venv/lib/python3.11/site-packages/synapse/config/_base.py", line 741, in load_or_generate_config
12.05.2023 22:47:53
    if not path_exists(config_path):
12.05.2023 22:47:53
           ^^^^^^^^^^^^^^^^^^^^^^^^
12.05.2023 22:47:53
  File "/matrix/venv/lib/python3.11/site-packages/synapse/config/_base.py", line 150, in path_exists
12.05.2023 22:47:53
    raise e
12.05.2023 22:47:53
  File "/matrix/venv/lib/python3.11/site-packages/synapse/config/_base.py", line 146, in path_exists
12.05.2023 22:47:53
    os.stat(file_path)
12.05.2023 22:47:53
PermissionError: [Errno 13] Permission denied: '/data/homeserver.tmp'
12.05.2023 22:47:53
-=> configure some settings in homeserver.yaml
12.05.2023 22:47:53
mv: cannot move '/tmp/tmp.f2b7gI1MYB' to '': No such file or directory
12.05.2023 22:47:53
mv: cannot stat '/data/homeserver.tmp': Permission denied
12.05.2023 22:47:53
-=> configure some settings in chat.bingery.de.log.config
12.05.2023 22:47:53
sed: can't read /data/chat.bingery.de.log.config: Permission denied
12.05.2023 22:47:53
12.05.2023 22:47:53
-=> you have to review the generated configuration file homeserver.yaml
12.05.2023 23:11:24
-=> generate turn config
12.05.2023 23:11:24
/start.sh: line 153: pwgen: command not found
12.05.2023 23:11:24
/start.sh: line 17: : No such file or directory
12.05.2023 23:11:24
/start.sh: line 18: : No such file or directory
12.05.2023 23:11:24
/start.sh: line 19: : No such file or directory
12.05.2023 23:11:24
/start.sh: line 20: : No such file or directory
12.05.2023 23:11:24
/start.sh: line 21: : No such file or directory
12.05.2023 23:11:24
/start.sh: line 22: : No such file or directory
12.05.2023 23:11:24
/start.sh: line 23: : No such file or directory
12.05.2023 23:11:24
/start.sh: line 24: : No such file or directory
12.05.2023 23:11:24
-=> generate synapse config
12.05.2023 23:11:25
Traceback (most recent call last):
12.05.2023 23:11:25
  File "<frozen runpy>", line 198, in _run_module_as_main
12.05.2023 23:11:25
  File "<frozen runpy>", line 88, in _run_code
12.05.2023 23:11:25
  File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 398, in <module>
12.05.2023 23:11:25
    main()
12.05.2023 23:11:25
  File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 388, in main
12.05.2023 23:11:25
    hs = setup(sys.argv[1:])
12.05.2023 23:11:25
         ^^^^^^^^^^^^^^^^^^^
12.05.2023 23:11:25
  File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 297, in setup
12.05.2023 23:11:25
    config = HomeServerConfig.load_or_generate_config(
12.05.2023 23:11:25
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
12.05.2023 23:11:25
  File "/matrix/venv/lib/python3.11/site-packages/synapse/config/_base.py", line 741, in load_or_generate_config
12.05.2023 23:11:25
    if not path_exists(config_path):
12.05.2023 23:11:25
           ^^^^^^^^^^^^^^^^^^^^^^^^
12.05.2023 23:11:25
  File "/matrix/venv/lib/python3.11/site-packages/synapse/config/_base.py", line 150, in path_exists
12.05.2023 23:11:25
    raise e
12.05.2023 23:11:25
  File "/matrix/venv/lib/python3.11/site-packages/synapse/config/_base.py", line 146, in path_exists
12.05.2023 23:11:25
    os.stat(file_path)
12.05.2023 23:11:25
PermissionError: [Errno 13] Permission denied: '/data/homeserver.tmp'
12.05.2023 23:11:25
-=> configure some settings in homeserver.yaml
12.05.2023 23:11:25
mv: cannot move '/tmp/tmp.M79aIw5ux3' to '': No such file or directory
12.05.2023 23:11:25
mv: cannot stat '/data/homeserver.tmp': Permission denied
12.05.2023 23:11:25
-=> configure some settings in chat.bingery.de.log.config
12.05.2023 23:11:25
sed: can't read /data/chat.bingery.de.log.config: Permission denied
12.05.2023 23:11:25
12.05.2023 23:11:25
-=> you have to review the generated configuration file homeserver.yaml
Container stopped

Because of the permission denied errors I thought of file permission problem, but I checked /appdata and the homeserver.yaml but nothing there I think.

My config:
 

server_name: "chat.bingery.de"
pid_file: /data/homeserver.pid
web_client_location: https://element.bingery.de
public_baseurl: https://chat.bingery.de
listeners:
  - port: 8008
    tls: false
    type: http
    x_forwarded: true
    bind_addresses: ['0.0.0.0']
    resources:
      - names: [client,federation]
        compress: false
database:
  name: psycopg2
  args:
    user: root
    password: "SecretKey"
    database: matrix
    host: hal9000
    port: 5432
    cp_min: 5
    cp_max: 10
log_config: "/data/chat.bingery.de.log.config"
media_store_path: "/data/media_store"
suppress_key_server_warning: true
report_stats: false
macaroon_secret_key: "SecretKey"
form_secret: "SecretKey"
signing_key_path: "/data/chat.bingery.de.signing.key"
serve_server_wellknown: true
trusted_key_servers:
  - server_name: "matrix.org"

 ## TURN ##
# The public URIs of the TURN server to give to clients
turn_uris: ["turn:chat.bingery.de:3478?transport=udp",  "turn:chat.bingery.de:3478?transport=tcp"]

# The shared secret used to compute passwords for the TURN server
turn_shared_secret: "SecretKey"

# New User Registration
registration_shared_secret: "SecretKey"

# vim:ft=yaml

(Only edited out the passwords)

Any Ideas, anybody?

 

 

 

[Lavoslav]

I'm stuck at install Matrix step.

 

Directory matrix gets created in appdata folder but remains completely empty. No homeserver.yaml no directories, not a single file or directory inside.

This is the log output of matrix docker upon first run:  (SUBDOMAIN.MY_DOMAIN parts are sanitised)

/start.sh: line 153: pwgen: command not found
/start.sh: line 17: : No such file or directory
/start.sh: line 18: : No such file or directory
/start.sh: line 19: : No such file or directory
/start.sh: line 20: : No such file or directory
/start.sh: line 21: : No such file or directory
/start.sh: line 22: : No such file or directory
/start.sh: line 23: : No such file or directory
/start.sh: line 24: : No such file or directory
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 398, in <module>
    main()
  File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 388, in main
    hs = setup(sys.argv[1:])
         ^^^^^^^^^^^^^^^^^^^
  File "/matrix/venv/lib/python3.11/site-packages/synapse/app/homeserver.py", line 297, in setup
    config = HomeServerConfig.load_or_generate_config(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/matrix/venv/lib/python3.11/site-packages/synapse/config/_base.py", line 768, in load_or_generate_config
    with open(config_path, "w") as config_file:
         ^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/data/homeserver.tmp'
mv: cannot move '/tmp/tmp.BfJK0wnjWe' to '': No such file or directory
mv: cannot stat '/data/homeserver.tmp': No such file or directory
sed: can't read /data/SUBDOMAIN.MY_DOMAIN.com.log.config: No such file or directory
-=> generate turn config
-=> generate synapse config
Generating config file /data/homeserver.tmp
-=> configure some settings in homeserver.yaml
-=> configure some settings in SUBDOMAIN.MY_DOMAIN.com.log.config

-=> you have to review the generated configuration file homeserver.yaml

** Press ANY KEY to close this window ** 

 

Any pointers would be apprechiated..

[JEZBRO]

I followed your guide and got it working, any chance you'd extend your guide to include Sliding Sync?

[JEZBRO]

16 hours ago, JEZBRO said:

I followed your guide and got it working, any chance you'd extend your guide to include Sliding Sync?

So from following this guide, some help online and ChatGPT I actually managed to get Sliding Sync working.

 

[JEZBRO]

For anyone looking for a guide on how I did this, here it is.
 

1. Login to pgAdmin4. On the left expand Server > Matrix > Databases, then right click Databases then go Create > Database...
   
   
   
   Then in the Database field type "syncv3" then click "Save"
   
    
2. Install Conpose.Manager from CA.
   
    
3. Navigate to the Docker tab, and then to the "Add New Stack". Put in "Sliding_Sync" in the stack_name field, then click "Advanced" and in the stack directory put /mnt/user/appdata/matrix/sliding-sync then click "OK"
   
   Now scroll down and below your dockers you should see Sliding_sync under Compose, click the COG > Edit Stack > Compose File Scroll down again and there should be a text editor. Copy and past the following into that.
    

   version: '3.8'
   services:
     slidingsync-proxy:
       container_name: slidingsync-proxy
       image: 'ghcr.io/matrix-org/sliding-sync:latest'
       restart: unless-stopped
       environment:
         - 'SYNCV3_SERVER=https://chat.yourdomain.com'
         - 'SYNCV3_SECRET=KEY'
         - 'SYNCV3_BINDADDR=:8009'
         - 'SYNCV3_DB=user=postgres-username dbname=syncv3 sslmode=disable host=slidingsync-db password=postgres-password'
       ports:
         - '8009:8009'
       depends_on:
         - slidingsync-db
   
     slidingsync-db:
       container_name: slidingsync-db
       image: postgres:15
       restart: unless-stopped
       volumes:
         - /mnt/user/appdata/matrix/sliding-sync/database:/var/lib/postgresql/data
       environment:
         - POSTGRES_PASSWORD=postgres-passowrd
         - POSTGRES_USER=postgres-username
         - POSTGRES_DB=syncv3

   
   Adjust `SYNCV3_SERVER`, `SYNCV3_DB`, `POSTGRES_PASSWOR`, `POSTGRES USER`, and 'POSTGRES_DB' to match your setup.
   
   You can use this command in the unraid console to generate a random string to put in 'SYNCV3_SECRET'
    

   echo -n $(openssl rand -hex 32)

   
    

4. Create a file called `client.json` with the following content. You can use any text editor for this.
    

   {
       "m.homeserver": {
           "base_url": "https://matrix.yourdomain.com"
       },
       "org.matrix.msc3575.proxy": {
           "url": "https://chat1.yourdomain.com"
       }
   }

   
   Place this file in the Nginx directory on your Unraid server, I put it in /mnt/user/appdata/Nginx-Proxy-Manager-Official/data/nginx/.well-known/matrix/client.json
   
    

5. Open the Nginx Proxy Manager UI in your browser.
   
   Add a New Proxy Host for Sliding Sync connections, I made mine chat1.yourdomain.com
   
   Set it up like normal and have it point to you Sliding Sync Proxy. I have my matrix server on port 8008 and Silding Proxy on 8009. 
   
   Before you hit save, go to the "Advanced" tab and add this to Custom Nginx Configuration
    

   location /.well-known/matrix/client {
       root /data/nginx/;
       try_files /.well-known/matrix/client.json =404;
       default_type application/json;
       add_header Access-Control-Allow-Origin *;
   }

   
    

6. Verify the Setup, go to https://chat1.yourdomain.com/.well-known/matrix/client
   It should return the contents of the .json file you made earlier
    

And that's it, it should all be up and running.

Edited February 11 by JEZBRO

[Nelinski]

@JEZBRO I'm trying to set this up but with SWAG. I believe I've followed this all to a T but I'm getting a 403. Any suggestions of what to check/look for to get this working?

Edited February 23 by Nelinski

[tekolote]

@Nelinski This is my config file for swag. I used "sliding" instead of "chat1" for my sliding-sync url. If I understand correctly, with this config you don't need to upload the JSON file becuase the well-known data is being returned by nginx. You'll have to change $upstream_app to point to your slidingsync-proxy,  you might be able to you the container name instead of IP. Also, add your correct domains the the location entry. I also added this same location entry into my swag config for the matrix-synapse container. Before I did that the Elementx app said the server doesn't have sliding-sync configured. after adding it to both I got no errors and it seems to be working

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name sliding.*;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    location / {

        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app UNRAIDIP;
        set $upstream_port 8009;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

}
    location /.well-known/matrix/client {
    add_header Access-Control-Allow-Origin *;
    default_type application/json;
    return 200 '{"m.homeserver": {"base_url": "https://chat.yourdomain.com"}, "org.matrix.msc3575.proxy": {"url": "https://sliding.yourdomain.com"}}';
}

}

 

Another thing I noticed is that the postgresql database created in the fist part of the sliding-sync guide through pgadmin on host of the matrix database is not being used. It looks like the compose file is creating a new postgresql15 host and database and using that instead. I wasn't able to get it to connect to the postgres host I made for matrix but I am not knowledgeable about docker-compose and networking. Someday maybe I'll figure it out

[JEZBRO]

On 2/24/2024 at 7:27 AM, Nelinski said:

@JEZBRO I'm trying to set this up but with SWAG. I believe I've followed this all to a T but I'm getting a 403. Any suggestions of what to check/look for to get this working?

Did you get it working following @tekolote advice? I'm not familiar with SWAG but I maybe able to help.

[Nelinski]

@tekolote Thanks, that was helpful. I think I've made progress but still not 100% there. When I go to my sliding URL/.well-known/matrix/client I get the return as expected:

{"m.homeserver": {"base_url": "https://bridge.domain.co.uk"}, "org.matrix.msc3575.proxy": {"url": "https://sliding.domain.co.uk"}}

 

If I go directly to https://sliding.domain.co.uk I get a 404 error.

 

When configuring Element X, I enter the URL as https://sliding.domain.co.uk and it detects I've got sliding enabled but doesn't get past that first configuration screen. Looking at the sliding logs it has the following:

21:14:48 INF Received connection from unknown access token, querying with homeserver
21:14:48 INF c= duration=0.000 path=//_matrix/client/r0/account/whoami size=0 status=301
21:14:48 INF c= duration=0.000 path=/_matrix/client/r0/account/whoami size=19 status=404
{"level":"warn","error":"/whoami returned HTTP 404","time":"2024-03-23T21:14:48Z","message":"failed to get user ID from device ID"}
21:14:48 ERR failed to get or create Conn error="HTTP 502 : /whoami returned HTTP 404"

 

Any ideas? ( @JEZBRO if you're able to help too, appreciate the offer!)

[Nelinski]

Fixed it, it's always the simplest things. I had a trailing / at the end of the home server URL in the compose file. Removed that and it starts fine.

[tekolote]

@JEZBRO I followed your guide for adding sliding-sync I noticed that you add a database to your matrix postgres server. Then, in the docker compose file it downloads and creates it's own postgresql15 server and database and uses that instead. You can check in pgadmin in the syncv3 database you created under schemas>tables. The tables dropdown will be empty if it's not being used because the sliding sync docker compose configuration is using it's own build in postgresql15 database instead of the one that was prepared for it on the matrix postgres server and places the database in the mounted volume at - /mnt/user/appdata/matrix/sliding-sync/database. I was able to get it to connect to the correct database by modifying the compose file. If you or anybody else following the guide would like to switch to the database you prepared Here's an example of my compose file. You'll have to add in your secret key and add the user, password, and server info to match yours

version: '3.8'
services:
  slidingsync-proxy:
    container_name: slidingsync-proxy
    image: 'ghcr.io/matrix-org/sliding-sync:latest'
    restart: unless-stopped
    environment:
      - 'SYNCV3_SERVER=https://chat.domain.com'
      - 'SYNCV3_SECRET=SECRETKEY'
      - 'SYNCV3_BINDADDR=:8009'
      - 'SYNCV3_DB=postgres://DATABASEUSER:DATABASEPASSWORD@DATABASEIP:DATABASEPORT/DATABASENAME'?sslmode=disable'
    ports:
      - '8009:8009'

 

[JEZBRO]

@tekolote Yeah, that's right. I wasn't sure if you could setup sliding sync in the same PG15 database as Matrix that's why I set it up like I did. Your way definitely seems cleaner.

When I set this up I was struggling to get any kind of help. Between a number of group chats about Matrix, Element, Sliding Sync and UNRAID, each one directed me to a different chat to get help. So I resorted to just using ChatGPT. I trained a GPT on all the documentation for each component then asked it how to set up Sliding Sync and my little tutorial is the result. I've only been using UNRAID for about 6 months and have no experience with Linux so I'm just teaching myself as I go.

[tekolote]

No worries. Getting support for things running on unraid when it's outside of the communities apps can be a real hassle since most developers understandably don't support it. I would have had a lot more trouble getting sliding-sync setup without your guide. so, Thanks for your contribution

[dave234ee]

got everything working for about 15mins then the database goes down with the bellow error 

 

024-04-13 22:00:09.231 BST [325] FATAL:  could not open file "global/pg_filenode.map": No such file or directory
2024-04-13 22:00:39.254 BST [326] FATAL:  could not open file "global/pg_filenode.map": No such file or directory
2024-04-13 22:01:09.258 BST [327] FATAL:  could not open file "global/pg_filenode.map": No such file or directory
2024-04-13 22:01:39.284 BST [328] FATAL:  could not open file "global/pg_filenode.map": No such file or directory
2024-04-13 22:02:09.287 BST [329] FATAL:  could not open file "global/pg_filenode.map": No such file or directory
2024-04-13 22:02:39.314 BST [330] FATAL:  could not open file "global/pg_filenode.map": No such file or directory
2024-04-13 22:03:09.291 BST [331] FATAL:  could not open file "global/pg_filenode.map": No such file or directory
2024-04-13 22:03:39.343 BST [332] FATAL:  could not open file "global/pg_filenode.map": No such file or directory
2024-04-13 22:03:43.221 BST [64] PANIC:  could not open file "pg_commit_ts": No such file or directory
2024-04-13 22:03:43.223 BST [1] LOG:  checkpointer process (PID 64) was terminated by signal 6: Aborted
2024-04-13 22:03:43.223 BST [1] LOG:  terminating any other active server processes
2024-04-13 22:03:43.230 BST [1] LOG:  all server processes terminated; reinitializing
2024-04-13 22:03:43.235 BST [1] PANIC:  could not open file "global/pg_control": No such file or directory
initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/data" or run initdb
with an argument other than "/var/lib/postgresql/data".
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.