allenpdx

After more iperf3 testing. If I make my Unraid box the iperf client, the fluctuations don't show up. Tried this over 8 tests. Connection is solid. The fluctuations only happen when my server is the receiving end of data transfer. Any networking folks have an idea what that means?

Link aggregation disabled on the router and bonding turned off for the server. Connected on eth0. Still a lot of fluctuation. C:\Users\afbag\Downloads\iperf-3.1.3-win64>iperf3.exe -c 192.168.50.100 -n 2000M -bidir Connecting to host 192.168.50.100, port 5201 [ 4] local 192.168.50.200 port 59322 connected to 192.168.50.100 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.01 sec 26.6 MBytes 222 Mbits/sec [ 4] 1.01-2.00 sec 27.6 MBytes 233 Mbits/sec [ 4] 2.00-3.00 sec 103 MBytes 864 Mbits/sec [ 4] 3.00-4.00 sec 106 MBytes 886 Mbits/sec [ 4] 4.00-5.00 sec 105 MBytes 883 Mbits/sec [ 4] 5.00-6.00 sec 103 MBytes 864 Mbits/sec [ 4] 6.00-7.00 sec 105 MBytes 885 Mbits/sec [ 4] 7.00-8.00 sec 102 MBytes 858 Mbits/sec [ 4] 8.00-9.00 sec 106 MBytes 887 Mbits/sec [ 4] 9.00-10.00 sec 55.2 MBytes 463 Mbits/sec [ 4] 10.00-11.00 sec 66.8 MBytes 560 Mbits/sec [ 4] 11.00-12.00 sec 59.6 MBytes 500 Mbits/sec [ 4] 12.00-13.00 sec 52.4 MBytes 438 Mbits/sec [ 4] 13.00-14.00 sec 67.8 MBytes 571 Mbits/sec [ 4] 14.00-15.00 sec 28.5 MBytes 239 Mbits/sec [ 4] 15.00-16.00 sec 48.5 MBytes 407 Mbits/sec [ 4] 16.00-17.01 sec 50.8 MBytes 420 Mbits/sec [ 4] 17.01-18.00 sec 92.0 MBytes 782 Mbits/sec [ 4] 18.00-19.00 sec 106 MBytes 888 Mbits/sec [ 4] 19.00-20.00 sec 108 MBytes 906 Mbits/sec [ 4] 20.00-21.00 sec 100 MBytes 843 Mbits/sec [ 4] 21.00-22.01 sec 102 MBytes 855 Mbits/sec [ 4] 22.01-23.00 sec 105 MBytes 887 Mbits/sec [ 4] 23.00-24.00 sec 109 MBytes 913 Mbits/sec [ 4] 24.00-24.60 sec 63.2 MBytes 883 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-24.60 sec 1.95 GBytes 682 Mbits/sec sender [ 4] 0.00-24.60 sec 1.95 GBytes 682 Mbits/sec receiver iperf Done.

I'll report back on turning off the link stuff with my next post. Running the same test with an increased buffer of 32K was better. Bandwidth both directions jumped to ~825 Mbits/sec over several tests.

Here's the results of a test sending 2GB from my PC to the server: C:\Users\afbag\Downloads\iperf-3.1.3-win64>iperf3.exe -c 192.168.50.100 -n 2000M -bidir Connecting to host 192.168.50.100, port 5201 [ 4] local 192.168.50.200 port 63778 connected to 192.168.50.100 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 65.0 MBytes 543 Mbits/sec [ 4] 1.00-2.00 sec 28.8 MBytes 242 Mbits/sec [ 4] 2.00-3.00 sec 61.6 MBytes 518 Mbits/sec [ 4] 3.00-4.00 sec 102 MBytes 855 Mbits/sec [ 4] 4.00-5.00 sec 62.8 MBytes 526 Mbits/sec [ 4] 5.00-6.00 sec 95.1 MBytes 798 Mbits/sec [ 4] 6.00-7.00 sec 101 MBytes 851 Mbits/sec [ 4] 7.00-8.00 sec 103 MBytes 865 Mbits/sec [ 4] 8.00-9.00 sec 103 MBytes 863 Mbits/sec [ 4] 9.00-10.00 sec 105 MBytes 881 Mbits/sec [ 4] 10.00-11.00 sec 106 MBytes 885 Mbits/sec [ 4] 11.00-12.00 sec 106 MBytes 885 Mbits/sec [ 4] 12.00-13.00 sec 102 MBytes 859 Mbits/sec [ 4] 13.00-14.01 sec 84.8 MBytes 705 Mbits/sec [ 4] 14.01-15.01 sec 52.6 MBytes 443 Mbits/sec [ 4] 15.01-16.00 sec 59.2 MBytes 499 Mbits/sec [ 4] 16.00-17.01 sec 80.0 MBytes 665 Mbits/sec [ 4] 17.01-18.01 sec 106 MBytes 892 Mbits/sec [ 4] 18.01-19.01 sec 81.5 MBytes 682 Mbits/sec [ 4] 19.01-20.00 sec 87.4 MBytes 739 Mbits/sec [ 4] 20.00-21.01 sec 73.2 MBytes 610 Mbits/sec [ 4] 21.01-22.01 sec 22.6 MBytes 190 Mbits/sec [ 4] 22.01-23.00 sec 81.4 MBytes 686 Mbits/sec [ 4] 23.00-24.00 sec 107 MBytes 900 Mbits/sec [ 4] 24.00-24.21 sec 23.4 MBytes 930 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-24.21 sec 1.95 GBytes 693 Mbits/sec sender [ 4] 0.00-24.21 sec 1.95 GBytes 693 Mbits/sec receiver iperf Done.

So for reference, I was using these same HDDs in my old Synology 4-bay with no SSD cache. Used same ports/cables. The sustained write speed there was ~110-120 MB/s which is just shy of the hard upper limit of 125. I was expecting my SSD pool on the Unraid server to be a lot closer to that.

I've got a new Unraid server setup. 4 Seagate Ironwolf 8TB 7200rpm HDDs (1 is for parity) + 2 Crucial Gen4 1TB SSDs in a cache pool. Setting up my shares, I have a personal folder for random stuff configured to write to the cache first then move at default transfer time. The other share is a media folder which goes straight to the array. I have the "md_write_method" set to "reconstruct write" because my first job is to transfer everything that goes on the server off my PC, and I wanted to transfer as fast as possible. As I've been transferring files, I've noticed that the r/w speeds are virtually identical for each share. I get a sustained transfer speed for large files of ~75-80 MB/s for write and ~80-85 MB/s for read. I expected quite a bit more speed copying to the cache since my PC drive is also a Gen4 SSD. I verified that my personal share is writing to the cache. I'm doing all this by drag-copying folders in Windows Explorer to my network shares. My network setup is pretty simple. The server is directly connected to my Asus RT-AX88U router on the two ports supporting 802.3ad link-aggregation. On the server I have SMB multi-channel turned on. My PC is also directly connected to the router. All ports (PC/server/router) are 1 Gbps. I'm using UGREEN Cat8 cables for everything. I unplugged one cable from the server and tried the transfers again to see if aggregation was somehow messing things up -- no change in speed. Any suggestions? I'm not a network expert, but so far I haven't been able to pin this down. Or is this all expected behavior for the cache?

So I closed the loop on this earlier today. I got my bonded link re-established by changing the type to "802.3ad" instead of the "active-backup" that Unraid chose. Since my router was already configured to support link aggregation on the two eth ports I had it connected to, it found the correct DHCP info again.

Figured out at least part of the problem. My router did not like the bonded-pair link Unraid had configured. I turned off bonding in the Unraid settings, and it immediately got correct DHCP info. My router was just fine with a bonded link on my old Synology NAS.

Directly connected.

Attached my diagnostics file. cloudnine-diagnostics-20230529-0626.zip

I'm new to Unraid and trying to set up an instance for the first time (as a Trial, if it matters). I can boot up my system but network configuration is a mess. I tried DHCP first. Unraid chose an IP starting with 169.254 which is not a part of my home network address range set through my ASUS router (which like most starts with 192.168). In this DHCP mode I can't communicate with the server or if logged in via Unraid GUI mode it can't reach the outside. Next tried static IP and assigned to one in my DHCP address range. This time I was able to access the GUI from my Windows box using the IP as my URL (tower.local didn't work). Changed the root password and was presented with the setup page. However it cannot get to the Internet. I can open the Unraid console and ping my PC, but it can't ping the gateway (like my PC can). Thought to use this opportunity to bind the local IP address for the server on my router so it's always the same assignment. Switched Unraid back to DHCP. It went back to using the 169.254 IP. Weirdly enough my router sees the presence of the server using the IP address I bound to it, but I can't communicate with the server though either IP. I'm out of ideas at the moment. Any suggestions? EDIT: Trying to install v6.12.0-RC6. My AsRock motherboard has 2 - 1G/2.5G eth ports that Unraid automatically set as a link-bonded pair.