Help me speed up copies from SSHFS mount. I've tried everything and only getting 1.5MB/s!

[tmchow]

I'm copying files from an SSHFS mount (installed via Nerd Pack) on a remote server, and pulling my hair out why I'm only getting 1.5MB/s.

My server has a gigabit NIC and I've confirmed it's at a 1000tx/gigabit link speed. Dashboard also shows this:

 

 

 

The sshfs mount is at /mnt/cache/foo using this command:

(This was altered to hide my paths and server IP):

sshfs [email protected]:/home/owner/downloads /mnt/cache/foo -o IdentityFile=~/.ssh/id_rsa -o StrictHostKeyChecking=no,Compression=no,reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,allow_other

I've done a variety of things to try to diagnose WTF is going on.

 

First, I've done an iPerf3 test between my local unraid server and a remote server. iPerf is giving me 500Mbit/s.

 

Second, I've measured copy speeds between /mnt/cache/foo (the SSHFS mount) and /mnt/cache by running

pv /mnt/cache/foo/movie.mkv > /mnt/cache/movie.mkv

and this only gets 1.5MB/s.

 

Third, I've also copied between array and cache drive and get super fast speeds of 1.96GiB/s. This was just to verify if any local issue.

 

Fourth, I suspected it was a Cipher problem since I've read that can impact SSHFS copy speeds a lot. I've tried using arcfour, but couldn't get that to work even after I added a Ciphers statement in my ~/.ssh/config and /etc/ssh/ssh_config on my local server. I've verified with the following command that arcfour is supported on the remote server:

sshd -T | grep "\(ciphers\|macs\|kexalgorithms\)" 

Thanks to a tip on reddit, I got a pointer to this article on cipher performance. I then tried aes128-ctr and aes256-ctr, adding a Ciphers option at the end of the SSHFS command:

sshfs [email protected]:/home/owner/downloads /mnt/cache/foo -o IdentityFile=~/.ssh/id_rsa -o StrictHostKeyChecking=no,Compression=no,reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,allow_other,Ciphers=aes128-ctr

This completed successfully, but transfer speeds were exactly the same as when I didn't specify these Ciphers (so 1.5MB/s).

 

Fifth, for fun I mounted a SMB share to compare speeds and got 6-10MB/s (so 6 to 10 times faster). I mounted it like this:

mount -t cifs -o username=myUser,password=myPassword //remote-server.com/downloads /mnt/cache/foo2

What the heck is going on? How can I solve this? What am I missing with SSHFS? I've read reports that slower speeds are expected but read people are getting 40-50MB/s copying from remote servers. Obviously this is contingent on internet connection speeds, but recall I've got gigabit fiber so don't think that's the issue (and see my first point above where iperf is showing 500Mbit/s)

Edited July 17, 2019 by tmchow

[tmchow]

4 hours ago, BRiT said:

If you're getting faster speeds between the 2 points using different protocol then it's likely overhead in whatever sshfs does. Have you gotten similiar speeds between sshfs and smb using other servers or other files?

You're right it could be about overhead, but I've read that other people get 40-50MB/s with SSHFS.  This is obviously is dependent on internet connect speeds, but shouldn't be an issue for me.  Not sure why I'm so far off.

 

I haven't tested remote server, I'm don't have another one I can easily test with.

[ijuarez]

Have you tested using a different rig, just to rule out your connection

[tmchow]

2 hours ago, ijuarez said:

Have you tested using a different rig, just to rule out your connection

I was just about to reply I don't have another server on my LAN to test with,  but realized I could just use my macbook over wifi and mount a share to do the same test.  I'll try that tonight when I get home.

[qarl]

Did you ever get resolution to this problem?  I'm seeing a 20x slow down with SSFS vs rsyc over ssh with the same parameters.  Something is broke.

Same as you, I get 1.5MB/s with SSFS and 25MB/s with rsync.

 

K.