May 31, 20233 yr I've got a new Unraid server setup. 4 Seagate Ironwolf 8TB 7200rpm HDDs (1 is for parity) + 2 Crucial Gen4 1TB SSDs in a cache pool. Setting up my shares, I have a personal folder for random stuff configured to write to the cache first then move at default transfer time. The other share is a media folder which goes straight to the array. I have the "md_write_method" set to "reconstruct write" because my first job is to transfer everything that goes on the server off my PC, and I wanted to transfer as fast as possible. As I've been transferring files, I've noticed that the r/w speeds are virtually identical for each share. I get a sustained transfer speed for large files of ~75-80 MB/s for write and ~80-85 MB/s for read. I expected quite a bit more speed copying to the cache since my PC drive is also a Gen4 SSD. I verified that my personal share is writing to the cache. I'm doing all this by drag-copying folders in Windows Explorer to my network shares. My network setup is pretty simple. The server is directly connected to my Asus RT-AX88U router on the two ports supporting 802.3ad link-aggregation. On the server I have SMB multi-channel turned on. My PC is also directly connected to the router. All ports (PC/server/router) are 1 Gbps. I'm using UGREEN Cat8 cables for everything. I unplugged one cable from the server and tried the transfers again to see if aggregation was somehow messing things up -- no change in speed. Any suggestions? I'm not a network expert, but so far I haven't been able to pin this down. Or is this all expected behavior for the cache?
May 31, 20233 yr Community Expert 80/85MB/s is not far from Gigabit LAN max speed, and link aggregation won't help for a single file transfer session, you can run iperf to check LAN bandwidth.
May 31, 20233 yr Author So for reference, I was using these same HDDs in my old Synology 4-bay with no SSD cache. Used same ports/cables. The sustained write speed there was ~110-120 MB/s which is just shy of the hard upper limit of 125. I was expecting my SSD pool on the Unraid server to be a lot closer to that.
May 31, 20233 yr Author Here's the results of a test sending 2GB from my PC to the server: C:\Users\afbag\Downloads\iperf-3.1.3-win64>iperf3.exe -c 192.168.50.100 -n 2000M -bidir Connecting to host 192.168.50.100, port 5201 [ 4] local 192.168.50.200 port 63778 connected to 192.168.50.100 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 65.0 MBytes 543 Mbits/sec [ 4] 1.00-2.00 sec 28.8 MBytes 242 Mbits/sec [ 4] 2.00-3.00 sec 61.6 MBytes 518 Mbits/sec [ 4] 3.00-4.00 sec 102 MBytes 855 Mbits/sec [ 4] 4.00-5.00 sec 62.8 MBytes 526 Mbits/sec [ 4] 5.00-6.00 sec 95.1 MBytes 798 Mbits/sec [ 4] 6.00-7.00 sec 101 MBytes 851 Mbits/sec [ 4] 7.00-8.00 sec 103 MBytes 865 Mbits/sec [ 4] 8.00-9.00 sec 103 MBytes 863 Mbits/sec [ 4] 9.00-10.00 sec 105 MBytes 881 Mbits/sec [ 4] 10.00-11.00 sec 106 MBytes 885 Mbits/sec [ 4] 11.00-12.00 sec 106 MBytes 885 Mbits/sec [ 4] 12.00-13.00 sec 102 MBytes 859 Mbits/sec [ 4] 13.00-14.01 sec 84.8 MBytes 705 Mbits/sec [ 4] 14.01-15.01 sec 52.6 MBytes 443 Mbits/sec [ 4] 15.01-16.00 sec 59.2 MBytes 499 Mbits/sec [ 4] 16.00-17.01 sec 80.0 MBytes 665 Mbits/sec [ 4] 17.01-18.01 sec 106 MBytes 892 Mbits/sec [ 4] 18.01-19.01 sec 81.5 MBytes 682 Mbits/sec [ 4] 19.01-20.00 sec 87.4 MBytes 739 Mbits/sec [ 4] 20.00-21.01 sec 73.2 MBytes 610 Mbits/sec [ 4] 21.01-22.01 sec 22.6 MBytes 190 Mbits/sec [ 4] 22.01-23.00 sec 81.4 MBytes 686 Mbits/sec [ 4] 23.00-24.00 sec 107 MBytes 900 Mbits/sec [ 4] 24.00-24.21 sec 23.4 MBytes 930 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-24.21 sec 1.95 GBytes 693 Mbits/sec sender [ 4] 0.00-24.21 sec 1.95 GBytes 693 Mbits/sec receiver iperf Done.
May 31, 20233 yr Community Expert That is fluctuating a lot, you want to see that always close to line speed, I would start by removing the 802.3ad link and re-testing.
May 31, 20233 yr Author I'll report back on turning off the link stuff with my next post. Running the same test with an increased buffer of 32K was better. Bandwidth both directions jumped to ~825 Mbits/sec over several tests.
May 31, 20233 yr Author Link aggregation disabled on the router and bonding turned off for the server. Connected on eth0. Still a lot of fluctuation. C:\Users\afbag\Downloads\iperf-3.1.3-win64>iperf3.exe -c 192.168.50.100 -n 2000M -bidir Connecting to host 192.168.50.100, port 5201 [ 4] local 192.168.50.200 port 59322 connected to 192.168.50.100 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.01 sec 26.6 MBytes 222 Mbits/sec [ 4] 1.01-2.00 sec 27.6 MBytes 233 Mbits/sec [ 4] 2.00-3.00 sec 103 MBytes 864 Mbits/sec [ 4] 3.00-4.00 sec 106 MBytes 886 Mbits/sec [ 4] 4.00-5.00 sec 105 MBytes 883 Mbits/sec [ 4] 5.00-6.00 sec 103 MBytes 864 Mbits/sec [ 4] 6.00-7.00 sec 105 MBytes 885 Mbits/sec [ 4] 7.00-8.00 sec 102 MBytes 858 Mbits/sec [ 4] 8.00-9.00 sec 106 MBytes 887 Mbits/sec [ 4] 9.00-10.00 sec 55.2 MBytes 463 Mbits/sec [ 4] 10.00-11.00 sec 66.8 MBytes 560 Mbits/sec [ 4] 11.00-12.00 sec 59.6 MBytes 500 Mbits/sec [ 4] 12.00-13.00 sec 52.4 MBytes 438 Mbits/sec [ 4] 13.00-14.00 sec 67.8 MBytes 571 Mbits/sec [ 4] 14.00-15.00 sec 28.5 MBytes 239 Mbits/sec [ 4] 15.00-16.00 sec 48.5 MBytes 407 Mbits/sec [ 4] 16.00-17.01 sec 50.8 MBytes 420 Mbits/sec [ 4] 17.01-18.00 sec 92.0 MBytes 782 Mbits/sec [ 4] 18.00-19.00 sec 106 MBytes 888 Mbits/sec [ 4] 19.00-20.00 sec 108 MBytes 906 Mbits/sec [ 4] 20.00-21.00 sec 100 MBytes 843 Mbits/sec [ 4] 21.00-22.01 sec 102 MBytes 855 Mbits/sec [ 4] 22.01-23.00 sec 105 MBytes 887 Mbits/sec [ 4] 23.00-24.00 sec 109 MBytes 913 Mbits/sec [ 4] 24.00-24.60 sec 63.2 MBytes 883 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-24.60 sec 1.95 GBytes 682 Mbits/sec sender [ 4] 0.00-24.60 sec 1.95 GBytes 682 Mbits/sec receiver iperf Done. Edited May 31, 20233 yr by allenpdx
June 1, 20233 yr Author After more iperf3 testing. If I make my Unraid box the iperf client, the fluctuations don't show up. Tried this over 8 tests. Connection is solid. The fluctuations only happen when my server is the receiving end of data transfer. Any networking folks have an idea what that means?
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.