Extremely slow downstream on remote smb shares

[blackfox]

Hey guys,

 

I set up a wireguard server-to-server connection to my friend's server, which works.

Then I mounted his smb shares via the "add remote smb/nfs share" and they show up, which is good.

I can access all shares and vice versa. Confusingly, some shares show up as USED 0 B FREE 0 B (see screenshots). Sometimes all show up as 0 B, sometimes a few of them, sometimes none. I don't get it, it's completly random.

BUT, the main problem is when I want to download something from his shares. I only get 0,5MB/s in average while downloading (he has 50Mbit upstream, so it should be around 6MB/s). When I upload something, I get my full upload bandwidth of about 5MB/s.

When he pushes something to my server we get the full bandwidth of his upstream. Also, he can download something from my shares and gets the full upload bandwidth from me. I'm really confused right now as I even don't know where to look for the problem.

Is it wireguard or Unraid or something else?

When I restart my server, for a short period of time (some minutes) I can download from his shares with full bandwidth. I tried to replicate the poblem, but couldn't figure out when the 'drop' happens and what triggered it.
The command I used (also tried rclone, same result):

rsync -rvhP /mnt/remotes/"his share" /mnt/user/"my destination"

I even downgraded from 6.9.0-rc2 to rc1 but it didn't help.

I think the issue that remote shares show as 0 B for me has something to do with my main problem, but it's just a guess.

Diagnostics and screenshots of the remote smb shares is attached!

 

I really hope someone can help me, I would appreciate it!

 

diagnostics-20210225-1333.zip

[weirdcrap]

I can't speak to the SMB shares not detecting space correctly, though your speed issues sounds kind of similar to mine.

 

I've tried copying files between my two servers using SSH & rsync, NFS shares, SSHFS, and they all suffer from severe performance loss somewhere mid transfer with no warning:

 

I have been unable to definitively nail down where the problem is, though the more I work at it the more it seems like something to do with WireGuard rather than my chosen transfer protocol, my servers, or my network/internet.

 

Sorry I don't have more help to offer, but I just wanted to chime in that you aren't alone in your performance issues with moving data over WireGuard.

 

Edited February 25, 2021 by weirdcrap

[blackfox]

Thanks for the reply and the link to you post @weirdcrap

What I don't understand is: why is my upload speed high and stable, but the download speed isn't?

Also, my friend uses wireguard on his unbuntu server as well to connect to me, of course.

He hasn't gotten any issues, neither with down- nor upload.

So if I might guess, I think it has to do with the implementation of wireguard in Unraid.

[weirdcrap]

 

2 minutes ago, blackfox said:

Thanks for the reply and the link to you post @weirdcrap

What I don't understand is: why is my upload speed high and stable, but the download speed isn't?

Also, my friend uses wireguard on his unbuntu server as well to connect to me, of course.

He hasn't gotten any issues, neither with down- nor upload.

So if I might guess, I think it has to do with the implementation of wireguard in Unraid.

I'm not sure why you can't seem to get a good speed only when downloading from his remote server. It could be an intermediate issue somewhere along your route? Do you see poor response times from any particular hop when you run a traceoute?

 

I've seen reports of WireGuard having poor performance when your MTU is to high and causes packet fragmentation, that could be another possible avenue of investigation. I just made a post about it over in my thread as that is where my investigation is at right now.

[itimpi]

It almost sounds as if the upload speed from the remote server is not as fast as it is expected to be and that is limiting the overall transfer speed.

 

have tests been done at the remote server to check that it can actually achieve the expected upload speed?

[blackfox]

2 hours ago, itimpi said:

It almost sounds as if the upload speed from the remote server is not as fast as it is expected to be and that is limiting the overall transfer speed.

 

have tests been done at the remote server to check that it can actually achieve the expected upload speed?

The upload speed of the remote server isn't an issue at all.

1.) when the remote server uploads to my server (so my download is in use), we achieve the max speed

2.) when I download from the remote server via https (nextcloud for instance), I also get the full speed

3.) for a short period of time I get the full speed, then it drops to a few KB/s

 

We did iperf with and without wireguard, speeds are good. Through wireguard slightly slower, but still quite good.

Also tracert is ok, no poor response times at any hop.

 

I will test some different settings with MTU as @weirdcrap said and report back.

Thanks for the hint so far!

[blackfox]

So, after days and weeks of trying I could not solve the problem.

Neither different settings with MTU @weirdcrap nor a reconfiguration of wireguard solved it.

 

What we tested:

- https connection (curl) to the remote server without wireguard -> no issues, full speed

- my friend came over to my location and connected back to his server (the remote server) to see if there may be a problem with my network -> but everything was fine

- after a restart of the server or the array, I get max. speed through wireguard, but after a couple of minutes it drops to 5-10% of the max. speed.

- Unraid 6.9.2 didn't solve my issue as well

 

It seems, something's wrong with wireguard implementation in Unraid, however unfortunately I cannot be more specific...

[weirdcrap]

On 4/19/2021 at 10:17 AM, blackfox said:

So, after days and weeks of trying I could not solve the problem.

Neither different settings with MTU @weirdcrap nor a reconfiguration of wireguard solved it.

 

What we tested:

- https connection (curl) to the remote server without wireguard -> no issues, full speed

- my friend came over to my location and connected back to his server (the remote server) to see if there may be a problem with my network -> but everything was fine

- after a restart of the server or the array, I get max. speed through wireguard, but after a couple of minutes it drops to 5-10% of the max. speed.

- Unraid 6.9.2 didn't solve my issue as well

 

It seems, something's wrong with wireguard implementation in Unraid, however unfortunately I cannot be more specific...

Yes I've written WireGuard off at this point, its nice for basic remote access but it's hot garbage for file transfers.

 

I don't know if it's UnRAID's implementation or if WireGuard just sucks at prolonged file transfers but I can't even get through a single 8GB file without my speed tanking to >100KB/s. Supposedly there are kernel level debug options to get some logs from WireGuard but I never got a response from LJM42 if those were present in UnRAID or not: 

 

 

At some point if I have tons of free time I might try to setup a WireGuard tunnel outside of UnRAID in Linux. But at this point I just go completely around the WireGuard tunnel.

Edited April 25, 2021 by weirdcrap