[SOLVED] 6.8.3 Mellanox Network Problem, does not appear to work

Helmonder

By Helmonder
in General Support

Recommended Posts

Helmonder Proficient

Helmonder

Posted
Posted (edited)

Hi,

 

I currently have two unraid servers, one primary, one backup.

 

Both servers have Mellanox 10GB cards in them. The main server uses this connection to connect to my network-switch and this works fine (my desktop is also connected at 10GB and that gives me cool speeds between desktop and server.

 

I have added a similar card to my backupserver but my network switch only has two SFP+ ports so I am trying to do the following:

 

Directly connect backup and primary unraid server with an SFP DAC between two systems. The backup system is only used to receive SYNCTHING backups so if speeds between these servers are optimized this is fine and I do not need to upgrade my switch.

 

The backup is btw also connected with a regular 1GB RJ45 connection and the servers works fine (also syncthing over this connection works fine).

 

My regular (perfectly working) addresses are:

 

Primary: 192.168.1.5

Backup: 192.168.1.6

 

Both servers have secundary addresses on the Mellanox:

 

Primary: 192.168.10.5 (default gateway 192.168.10.6)

Backup: 192.168.10.6 (default gateway 192.168.10.5)

 

I have setup syncthing as dockers, the dockers are bridged so share the IP address of the server out of my regular pool:

 

Within the Syncthing dockers I have setup connections towards the Mellanox addresses:

 

Primary is connecting to 192.168.10.6

Backup is connecting to 192.168.10.5

 

Now for my issue:

 

The syncthing containers appear to be connecting fine but the transfers speeds are zero to a few kb...

 

So something is wrong.. 

 

I see no obvious errors in the logs:

 

Primary Logfile filtered on eth2 (which is the port for 192.168.10.5) 

Dec 28 08:49:07 Tower kernel: 8021q: adding VLAN 0 to HW filter on device eth2
Dec 28 08:49:09 Tower kernel: mlx4_en: eth2: Link Up
Dec 28 08:54:08 Tower ool www[14025]: /usr/local/emhttp/plugins/dynamix/scripts/netconfig 'eth2'
Dec 28 08:54:08 Tower rc.inet1: ip -4 addr flush dev eth2
Dec 28 08:54:08 Tower rc.inet1: ip link set eth2 down
Dec 28 08:54:08 Tower kernel: mlx4_en: eth2: Close port called
Dec 28 08:54:08 Tower kernel: mlx4_en: eth2: Link Down
Dec 28 08:54:09 Tower rc.inet1: ip -4 addr add 192.168.10.5/255.255.255.0 dev eth2
Dec 28 08:54:09 Tower rc.inet1: ip link set eth2 up
Dec 28 08:54:09 Tower kernel: mlx4_en: eth2: Steering Mode 1
Dec 28 08:54:09 Tower kernel: 8021q: adding VLAN 0 to HW filter on device eth2
Dec 28 08:54:09 Tower rc.inet1: ip -4 route add default via 192.168.1.250 dev eth2
Dec 28 08:54:09 Tower rc.inet1: ip -4 route add default via 192.168.1.250 dev eth2
Dec 28 08:54:12 Tower kernel: mlx4_en: eth2: Link Up
Dec 28 08:54:12 Tower kernel: mlx4_en: eth2: Link Down
Dec 28 08:54:12 Tower kernel: mlx4_en: eth2: Link Up
Dec 28 08:54:58 Tower kernel: mlx4_en: eth2: Close port called
Dec 28 08:54:58 Tower kernel: mlx4_en: eth2: Link Down
Dec 28 08:55:00 Tower kernel: mlx4_en: eth2: Steering Mode 1
Dec 28 08:55:00 Tower kernel: 8021q: adding VLAN 0 to HW filter on device eth2
Dec 28 08:55:02 Tower kernel: mlx4_en: eth2: Link Up
Dec 28 08:55:24 Tower kernel: docker0: port 2(veth274c512) entered blocking state
Dec 28 08:55:24 Tower kernel: docker0: port 2(veth274c512) entered disabled state
Dec 28 08:55:24 Tower kernel: device veth274c512 entered promiscuous mode
Dec 28 08:55:24 Tower kernel: IPv6: ADDRCONF(NETDEV_UP): veth274c512: link is not ready
Dec 28 08:55:24 Tower kernel: docker0: port 2(veth274c512) entered blocking state
Dec 28 08:55:24 Tower kernel: docker0: port 2(veth274c512) entered forwarding state
Dec 28 08:55:24 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth274c512: link becomes ready
Dec 28 08:56:53 Tower kernel: mlx4_en: eth2: Link Down
Dec 28 08:56:53 Tower kernel: mlx4_en: eth2: Link Up
Dec 28 08:58:54 Tower kernel: mlx4_en: eth2: Link Down
Dec 28 08:58:56 Tower kernel: mlx4_en: eth2: Link Up
Dec 28 09:58:00 Tower kernel: docker0: port 2(veth274c512) entered disabled state
Dec 28 09:58:00 Tower kernel: docker0: port 2(veth274c512) entered disabled state
Dec 28 09:58:00 Tower kernel: device veth274c512 left promiscuous mode
Dec 28 09:58:00 Tower kernel: docker0: port 2(veth274c512) entered disabled state
Dec 28 09:59:23 Tower ool www[21642]: /usr/local/emhttp/plugins/dynamix/scripts/netconfig 'eth2'
Dec 28 09:59:23 Tower rc.inet1: ip -4 addr flush dev eth2
Dec 28 09:59:23 Tower rc.inet1: ip link set eth2 down
Dec 28 09:59:23 Tower kernel: mlx4_en: eth2: Close port called
Dec 28 09:59:23 Tower kernel: mlx4_en: eth2: Link Down
Dec 28 09:59:23 Tower rc.inet1: ip -4 addr add 192.168.10.5/255.255.255.0 dev eth2
Dec 28 09:59:23 Tower rc.inet1: ip link set eth2 up
Dec 28 09:59:23 Tower kernel: mlx4_en: eth2: Steering Mode 1
Dec 28 09:59:23 Tower kernel: 8021q: adding VLAN 0 to HW filter on device eth2
Dec 28 09:59:23 Tower rc.inet1: ip -4 route add default via 192.168.10.6 dev eth2
Dec 28 09:59:26 Tower kernel: mlx4_en: eth2: Link Up
Dec 28 10:00:50 Tower kernel: mlx4_en: eth2: Link Down
Dec 28 10:00:52 Tower kernel: mlx4_en: eth2: Link Up

 

 Secundary logfile filtered on eth3 (which is the port for 192.168.10.6: 

Dec 28 08:49:09 Vault kernel: mlx4_en: eth3: Link Up
Dec 28 08:54:08 Vault kernel: mlx4_en: eth3: Link Down
Dec 28 08:54:12 Vault kernel: mlx4_en: eth3: Link Up
Dec 28 08:54:58 Vault kernel: mlx4_en: eth3: Link Down
Dec 28 08:55:02 Vault kernel: mlx4_en: eth3: Link Up
Dec 28 08:56:45 Vault kernel: mlx4_en: eth3: Steering Mode 1
Dec 28 08:56:45 Vault kernel: 8021q: adding VLAN 0 to HW filter on device eth3
Dec 28 08:56:46 Vault avahi-daemon[7894]: Joining mDNS multicast group on interface eth3.IPv6 with address fe80::202:c9ff:fe52:e976.
Dec 28 08:56:46 Vault avahi-daemon[7894]: New relevant interface eth3.IPv6 for mDNS.
Dec 28 08:56:46 Vault avahi-daemon[7894]: Registering new address record for fe80::202:c9ff:fe52:e976 on eth3.*.
Dec 28 08:56:53 Vault kernel: mlx4_en: eth3: Link Down
Dec 28 08:56:53 Vault kernel: mlx4_en: eth3: Link Up
Dec 28 08:58:54 Vault ool www[5867]: /usr/local/emhttp/plugins/dynamix/scripts/netconfig 'eth3'
Dec 28 08:58:54 Vault rc.inet1: ip -4 addr flush dev eth3
Dec 28 08:58:54 Vault rc.inet1: ip link set eth3 down
Dec 28 08:58:54 Vault kernel: mlx4_en: eth3: Close port called
Dec 28 08:58:54 Vault avahi-daemon[7894]: Interface eth3.IPv6 no longer relevant for mDNS.
Dec 28 08:58:54 Vault avahi-daemon[7894]: Leaving mDNS multicast group on interface eth3.IPv6 with address fe80::202:c9ff:fe52:e976.
Dec 28 08:58:54 Vault kernel: mlx4_en: eth3: Link Down
Dec 28 08:58:54 Vault avahi-daemon[7894]: Withdrawing address record for fe80::202:c9ff:fe52:e976 on eth3.
Dec 28 08:58:54 Vault rc.inet1: ip -4 addr add 192.168.10.6/255.255.255.0 dev eth3
Dec 28 08:58:54 Vault rc.inet1: ip link set eth3 up
Dec 28 08:58:54 Vault kernel: mlx4_en: eth3: Steering Mode 1
Dec 28 08:58:54 Vault avahi-daemon[7894]: Joining mDNS multicast group on interface eth3.IPv4 with address 192.168.10.6.
Dec 28 08:58:54 Vault kernel: 8021q: adding VLAN 0 to HW filter on device eth3
Dec 28 08:58:54 Vault avahi-daemon[7894]: New relevant interface eth3.IPv4 for mDNS.
Dec 28 08:58:54 Vault avahi-daemon[7894]: Registering new address record for 192.168.10.6 on eth3.IPv4.
Dec 28 08:58:54 Vault rc.inet1: ip -4 route add default via 192.168.1.250 dev eth3
Dec 28 08:58:54 Vault rc.inet1: ip -4 route add default via 192.168.1.250 dev eth3
Dec 28 08:58:56 Vault kernel: mlx4_en: eth3: Link Up
Dec 28 09:59:23 Vault kernel: mlx4_en: eth3: Link Down
Dec 28 09:59:26 Vault kernel: mlx4_en: eth3: Link Up
Dec 28 10:00:50 Vault ool www[21074]: /usr/local/emhttp/plugins/dynamix/scripts/netconfig 'eth3'
Dec 28 10:00:50 Vault rc.inet1: ip -4 addr flush dev eth3
Dec 28 10:00:50 Vault avahi-daemon[7894]: Withdrawing address record for 192.168.10.6 on eth3.
Dec 28 10:00:50 Vault avahi-daemon[7894]: Leaving mDNS multicast group on interface eth3.IPv4 with address 192.168.10.6.
Dec 28 10:00:50 Vault avahi-daemon[7894]: Interface eth3.IPv4 no longer relevant for mDNS.
Dec 28 10:00:50 Vault rc.inet1: ip link set eth3 down
Dec 28 10:00:50 Vault kernel: mlx4_en: eth3: Close port called
Dec 28 10:00:50 Vault kernel: mlx4_en: eth3: Link Down
Dec 28 10:00:50 Vault rc.inet1: ip -4 addr add 192.168.10.6/255.255.255.0 dev eth3
Dec 28 10:00:50 Vault rc.inet1: ip link set eth3 up
Dec 28 10:00:50 Vault kernel: mlx4_en: eth3: Steering Mode 1
Dec 28 10:00:50 Vault avahi-daemon[7894]: Joining mDNS multicast group on interface eth3.IPv4 with address 192.168.10.6.
Dec 28 10:00:50 Vault kernel: 8021q: adding VLAN 0 to HW filter on device eth3
Dec 28 10:00:50 Vault avahi-daemon[7894]: New relevant interface eth3.IPv4 for mDNS.
Dec 28 10:00:50 Vault avahi-daemon[7894]: Registering new address record for 192.168.10.6 on eth3.IPv4.
Dec 28 10:00:50 Vault rc.inet1: ip -4 route add default via 192.168.10.5 dev eth3
Dec 28 10:00:52 Vault kernel: mlx4_en: eth3: Link Up
Dec 28 10:00:52 Vault kernel: mlx4_en: eth3: Link Down
Dec 28 10:00:53 Vault kernel: mlx4_en: eth3: Link Up

I see nothing obvious going wrong here...

 

The syncthing log on the primary server: 

[services.d] starting services
[services.d] done.
[start] 10:11:20 INFO: syncthing v1.12.0 "Fermium Flea" (go1.15.6 linux-amd64) root@deea883fede0 2020-12-16 22:43:19 UTC [noupgrade]
[start] 10:11:20 INFO: Using large-database tuning
[NYD47] 10:11:20 INFO: My ID: NYD476I-JIDSNKM-GBFSVHP-F6SBLHR-XVQZM7N-F6IQYAK-TCXMNSC-P2N3IAJ
[NYD47] 10:11:20 INFO: Single thread SHA256 performance is 420 MB/s using minio/sha256-simd (361 MB/s using crypto/sha256).
[NYD47] 10:11:21 INFO: Hashing performance is 361.92 MB/s
[NYD47] 10:11:21 INFO: Overall send rate is unlimited, receive rate is unlimited
[NYD47] 10:11:21 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027
[NYD47] 10:11:21 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
[NYD47] 10:11:21 INFO: QUIC listener ([::]:22000) starting
[NYD47] 10:11:21 INFO: TCP listener ([::]:22000) starting
[NYD47] 10:11:21 INFO: Ready to synchronize "Tower-data" (rtecd-abbex) (sendonly)
[NYD47] 10:11:21 INFO: GUI and API listening on [::]:8384
[NYD47] 10:11:21 INFO: Access the GUI via the following URL: http://127.0.0.1:8384/
[NYD47] 10:11:21 INFO: My name is "Tower"
[NYD47] 10:11:21 INFO: Device 7CCSKMK-DCMAXVG-2XBD7HK-22XF4MR-2WKU36O-DJ3QZ52-CW5JEZV-JJHMKAN is "Vault" at [tcp://192.168.10.6:22000]
[NYD47] 10:11:21 INFO: Established secure connection to 7CCSKMK-DCMAXVG-2XBD7HK-22XF4MR-2WKU36O-DJ3QZ52-CW5JEZV-JJHMKAN at 172.17.0.4:50586-192.168.10.6:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
[NYD47] 10:11:21 INFO: Device 7CCSKMK-DCMAXVG-2XBD7HK-22XF4MR-2WKU36O-DJ3QZ52-CW5JEZV-JJHMKAN client is "syncthing v1.12.0" named "Vault" at 172.17.0.4:50586-192.168.10.6:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
[NYD47] 10:41:22 INFO: Sent usage report (version 3)

 

The syncthing log on the backup server: 

[services.d] starting services
[services.d] done.
[start] 10:11:15 INFO: syncthing v1.12.0 "Fermium Flea" (go1.15.6 linux-amd64) root@deea883fede0 2020-12-16 22:43:19 UTC [noupgrade]
[start] 10:11:15 INFO: Using large-database tuning
[7CCSK] 10:11:15 INFO: My ID: 7CCSKMK-DCMAXVG-2XBD7HK-22XF4MR-2WKU36O-DJ3QZ52-CW5JEZV-JJHMKAN
[7CCSK] 10:11:16 INFO: Single thread SHA256 performance is 305 MB/s using minio/sha256-simd (206 MB/s using crypto/sha256).
[7CCSK] 10:11:17 INFO: Hashing performance is 261.37 MB/s
[7CCSK] 10:11:17 INFO: Overall send rate is unlimited, receive rate is unlimited
[7CCSK] 10:11:17 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027
[7CCSK] 10:11:17 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
[7CCSK] 10:11:17 INFO: TCP listener ([::]:22000) starting
[7CCSK] 10:11:17 INFO: QUIC listener ([::]:22000) starting
[7CCSK] 10:11:17 INFO: GUI and API listening on [::]:8384
[7CCSK] 10:11:17 INFO: Access the GUI via the following URL: http://127.0.0.1:8384/
[7CCSK] 10:11:17 INFO: My name is "Vault"
[7CCSK] 10:11:17 INFO: Device NYD476I-JIDSNKM-GBFSVHP-F6SBLHR-XVQZM7N-F6IQYAK-TCXMNSC-P2N3IAJ is "Tower" at [tcp://192.168.10.5:22000]
[7CCSK] 10:11:17 INFO: Ready to synchronize "Tower-data" (rtecd-abbex) (receiveonly)
[7CCSK] 10:11:21 INFO: Established secure connection to NYD476I-JIDSNKM-GBFSVHP-F6SBLHR-XVQZM7N-F6IQYAK-TCXMNSC-P2N3IAJ at 172.17.0.2:22000-192.168.10.5:50586/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256
[7CCSK] 10:11:21 INFO: Device NYD476I-JIDSNKM-GBFSVHP-F6SBLHR-XVQZM7N-F6IQYAK-TCXMNSC-P2N3IAJ client is "syncthing v1.12.0" named "Tower" at 172.17.0.2:22000-192.168.10.5:50586/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256
[7CCSK] 10:41:18 INFO: Sent usage report (version 2)

Would really appreciate some help... I will cross-post in the syncthing forum to.. Since I do not really know where the error lies..

 

 

 

 

Edited by Helmonder
Link to comment
  • JorgeB changed the title to [SOLVED] 6.8.3 Mellanox Network Problem, does not appear to work

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing