quicky Posted August 9 Share Posted August 9 Dear all, After going through all existing posts about "rsync transfers are too slow" I decided to add another post as none of the existing solutions seem to be working for me. Basically I'm connecting my Macbook via a 2.5Gb LAN adapter directly to my Unraid server. I'm using the internal IP to access the Unraid server. When testing the connection with a random SFTP transfer I get a speed with about 250MByte/s. However rsync won't get faster than 100MByte/s (the files are identical and they are all big (above 80mb each) I've tested a lot the proposed solution (like lowering the SSH overhead with a weaker Cipher "aes128-ctr") but I also checked the CPU load and it seems that it's not the bottleneck at all (in fact it's never above 10% when the transfer runs). Increase the SSH Buffer Size. etc. etc. but this all has no impact to the speed. It always stays at about 100MByte/sec. I thought it could be some Mac issue but SFTP transfers are as fast as expected with the same connection. My current command is pretty basic again: /usr/bin/rsync -avP --progress /Volumes/drive/folder/ [email protected]:/mnt/user/media/folder Any idea what else I should be looking into to solve this? Thanks a million for any idea... Cheers, Oliver Quote Link to comment
itimpi Posted August 9 Share Posted August 9 What are you writing the files to on the Unraid end? If to the main array then 100MBps is about the best you can expect due to the overheads if mai taunt real-time parity. 1 Quote Link to comment
quicky Posted August 9 Author Share Posted August 9 Hi itimpi, Thanks a lot for your feedback! In fact I'm writing it to the main array but I disabled parity by stopping the array first and then temporarily unassigning the partiy disk. Is there any other way I could do the sync? It's actually going to be a one off. So if there's a onetime solution to speed up that transfer I'd be happy to do that. Quote Link to comment
Frank1940 Posted August 9 Share Posted August 9 What are the disks (Manufacturer/Part Number) that you are writing to? SMR vs CMR makes a difference. Spindle speed makes a difference. Percentage full of the current target disk makes a difference. Writing to a User Share vs Disk Share makes a difference. Disk size makes a difference-- larger capacity disks tend to be faster. What was the speed of your last parity check? I would bet it was nowhere here 250MB/s. My guess would be that it was less than 150MB/s... Quote Link to comment
quicky Posted August 9 Author Share Posted August 9 (edited) I've just added the Paritiy drive again and the Parity check started... it's running with about 280.0 MB/sec constantly. It's 12TB Seagate IronWolf drives (ST12000VN0008) but my main point is that with the exact same setup I was able to get 250MByte/s with a SFTP transfer. I was accessing the same share and everything. I'm a complete noob so if you're telling me that 100MByte/sec is a good rate I would live with that (especially because it's an activity which won't happen too often) but I was wondering if there's anything I could do to improve rsync if a standard FTP transfer can be so much faster. Edited August 9 by quicky Quote Link to comment
itimpi Posted August 9 Share Posted August 9 18 minutes ago, quicky said: I've just added the Paritiy drive again and the Parity check started If you do not have a parity drive then you can get higher speeds. However once you add a parity drive then this slows down transfers significantly. You may find this section useful from the online documentation accessible via the Manual link at the bottom of the Unraid GUI to understand why. Quote Link to comment
Frank1940 Posted August 9 Share Posted August 9 32 minutes ago, quicky said: I've just added the Paritiy drive again and the Parity check started... it's running with about 280.0 MB/sec constantly. It's 12TB Seagate IronWolf drives (ST12000VN0008) but my main point is that with the exact same setup I was able to get 250MByte/s with a SFTP transfer. I was accessing the same share and everything. Unraid uses any unused RAM as a disk cache to buffer data before it is written to disk. The data is written from this RAM cache to the disk(s) involved as fast as the disk can write it. When the RAM cache is full the transfer rate will slow to that of the disk involved. Quote Link to comment
quicky Posted August 9 Author Share Posted August 9 Please bare with me, but doesn't this all add to my point that it's constantly transferring at about 100MB/sec. With or without Parity. And even when I start the transfer it's at 100MB. So the 64Gig of RAM can't be filled up at the beginning of the copy-process. Quote Link to comment
Frank1940 Posted August 9 Share Posted August 9 Please upload the diagnostics file in a new post. Are you using a MTU of 1500 bytes? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.