poto2 Posted September 6, 2022 Share Posted September 6, 2022 I'm seeing slower-than-expected copy speeds between UnRaid NVME cache drive and Truenas zfs array. Copies from UnRaid>>TrueNAS proceed at a steady 445MB/s, however copies from TrueNAS>>UnRaid start out around 700MB/s, then drop to steady 200MB/s. Iperf3 reflects same speed discrepancy - traffic to TrueNAS maxes out 10gb connection, but traffic to UnRaid peaks at 2.8Gbps. Direct benchmarks of UnRaid & TrueNAS perform as expected, leading me to believe I have a networking issue. I've attached screenshots of steps taken so far, as well as current diagnostics. I don't see any cpu cores at 100% during transfers. Network cables are Cat5e, 2m or less. If anyone with experience troubleshooting could take a look, I would appreciate it. Cheers, John iperf3.txt truenas-02 fio ssd array.txt unraid-one-diagnostics-20220906-1342.zip Quote Link to comment
JorgeB Posted September 7, 2022 Share Posted September 7, 2022 9 hours ago, poto2 said: Iperf3 reflects same speed discrepancy That indicates a network related issue, could be NIC, NIC driver, cables, switch, network config etc Quote Link to comment
poto2 Posted September 7, 2022 Author Share Posted September 7, 2022 Still trying to eliminate possible problems here. Tried the following: - changed nic port - changed switch port - changed network cable - removed SMB multi-channel command from "SMB Extras", and made sure "Enable SMB Multi Channel" checked "No" in SMB Settings - changed nic - reboot Each run of iperf3 showed UnRaid>>TrueNAS maxing out 10Gb connection, but TrueNAS to UnRaid peaking around 2.8Gbps. AFAIK I'm using default network configuration, and in-box nic driver, really stumped here. root@TrueNAS-02[~]# iperf3 -c 192.168.1.162 -P 4 -i 30 -t 30 -R Connecting to host 192.168.1.162, port 5201 Reverse mode, remote host 192.168.1.162 is sending [ 5] local 192.168.1.160 port 28353 connected to 192.168.1.162 port 5201 [ 7] local 192.168.1.160 port 17600 connected to 192.168.1.162 port 5201 [ 9] local 192.168.1.160 port 35904 connected to 192.168.1.162 port 5201 [ 11] local 192.168.1.160 port 34427 connected to 192.168.1.162 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-30.00 sec 2.49 GBytes 712 Mbits/sec [ 7] 0.00-30.00 sec 2.58 GBytes 737 Mbits/sec [ 9] 0.00-30.00 sec 2.61 GBytes 749 Mbits/sec [ 11] 0.00-30.00 sec 2.67 GBytes 764 Mbits/sec [SUM] 0.00-30.00 sec 10.3 GBytes 2.96 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.13 sec 2.49 GBytes 710 Mbits/sec 134864 sender [ 5] 0.00-30.00 sec 2.49 GBytes 712 Mbits/sec receiver [ 7] 0.00-30.13 sec 2.58 GBytes 735 Mbits/sec 131587 sender [ 7] 0.00-30.00 sec 2.58 GBytes 737 Mbits/sec receiver [ 9] 0.00-30.13 sec 2.62 GBytes 746 Mbits/sec 131321 sender [ 9] 0.00-30.00 sec 2.61 GBytes 749 Mbits/sec receiver [ 11] 0.00-30.13 sec 2.67 GBytes 761 Mbits/sec 131679 sender [ 11] 0.00-30.00 sec 2.67 GBytes 764 Mbits/sec receiver [SUM] 0.00-30.13 sec 10.4 GBytes 2.95 Gbits/sec 529451 sender [SUM] 0.00-30.00 sec 10.3 GBytes 2.96 Gbits/sec receiver iperf Done. root@TrueNAS-02[~]# iperf3 -c 192.168.1.162 -P 4 -i 30 -t 30 Connecting to host 192.168.1.162, port 5201 [ 5] local 192.168.1.160 port 21998 connected to 192.168.1.162 port 5201 [ 7] local 192.168.1.160 port 20626 connected to 192.168.1.162 port 5201 [ 9] local 192.168.1.160 port 50486 connected to 192.168.1.162 port 5201 [ 11] local 192.168.1.160 port 17773 connected to 192.168.1.162 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-30.00 sec 8.50 GBytes 2.43 Gbits/sec 160 113 KBytes [ 7] 0.00-30.00 sec 7.52 GBytes 2.15 Gbits/sec 149 142 KBytes [ 9] 0.00-30.00 sec 7.74 GBytes 2.22 Gbits/sec 145 162 KBytes [ 11] 0.00-30.00 sec 8.63 GBytes 2.47 Gbits/sec 143 174 KBytes [SUM] 0.00-30.00 sec 32.4 GBytes 9.28 Gbits/sec 597 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 8.50 GBytes 2.43 Gbits/sec 160 sender [ 5] 0.00-30.12 sec 8.50 GBytes 2.42 Gbits/sec receiver [ 7] 0.00-30.00 sec 7.52 GBytes 2.15 Gbits/sec 149 sender [ 7] 0.00-30.12 sec 7.52 GBytes 2.14 Gbits/sec receiver [ 9] 0.00-30.00 sec 7.74 GBytes 2.22 Gbits/sec 145 sender [ 9] 0.00-30.12 sec 7.74 GBytes 2.21 Gbits/sec receiver [ 11] 0.00-30.00 sec 8.63 GBytes 2.47 Gbits/sec 143 sender Quote Link to comment
JorgeB Posted September 8, 2022 Share Posted September 8, 2022 What are the iperf results between Unraid and the Windows PC? Quote Link to comment
poto2 Posted September 8, 2022 Author Share Posted September 8, 2022 Eliminating TrueNAS from the equation probably a good idea! Iperf3 bare-metal Win10 PC>>UnRaid much better, 9Gbps from UR, 6Gbps to UR w/4 streams. This is from bare-metal Win10 PC, 10GB nic, Intel NVME drive. perf313>iperf3 -c unraid-one -P 4 -R Connecting to host unraid-one, port 5201 Reverse mode, remote host unraid-one is sending [ 4] local 192.168.1.194 port 50573 connected to 192.168.1.162 port 5201 [ 6] local 192.168.1.194 port 50574 connected to 192.168.1.162 port 5201 [ 8] local 192.168.1.194 port 50575 connected to 192.168.1.162 port 5201 [ 10] local 192.168.1.194 port 50576 connected to 192.168.1.162 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 262 MBytes 2.20 Gbits/sec [ 6] 0.00-1.00 sec 255 MBytes 2.14 Gbits/sec [ 8] 0.00-1.00 sec 230 MBytes 1.93 Gbits/sec [ 10] 0.00-1.00 sec 240 MBytes 2.01 Gbits/sec [SUM] 0.00-1.00 sec 987 MBytes 8.28 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 1.00-2.00 sec 277 MBytes 2.33 Gbits/sec [ 6] 1.00-2.00 sec 277 MBytes 2.33 Gbits/sec [ 8] 1.00-2.00 sec 260 MBytes 2.18 Gbits/sec [ 10] 1.00-2.00 sec 270 MBytes 2.26 Gbits/sec [SUM] 1.00-2.00 sec 1.06 GBytes 9.10 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 2.00-3.00 sec 279 MBytes 2.34 Gbits/sec [ 6] 2.00-3.00 sec 253 MBytes 2.12 Gbits/sec [ 8] 2.00-3.00 sec 311 MBytes 2.61 Gbits/sec [ 10] 2.00-3.00 sec 282 MBytes 2.37 Gbits/sec [SUM] 2.00-3.00 sec 1.10 GBytes 9.44 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 3.00-4.00 sec 304 MBytes 2.55 Gbits/sec [ 6] 3.00-4.00 sec 286 MBytes 2.40 Gbits/sec [ 8] 3.00-4.00 sec 274 MBytes 2.30 Gbits/sec [ 10] 3.00-4.00 sec 262 MBytes 2.20 Gbits/sec [SUM] 3.00-4.00 sec 1.10 GBytes 9.44 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 4.00-5.00 sec 277 MBytes 2.32 Gbits/sec [ 6] 4.00-5.00 sec 284 MBytes 2.39 Gbits/sec [ 8] 4.00-5.00 sec 289 MBytes 2.42 Gbits/sec [ 10] 4.00-5.00 sec 275 MBytes 2.30 Gbits/sec [SUM] 4.00-5.00 sec 1.10 GBytes 9.43 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 5.00-6.00 sec 268 MBytes 2.25 Gbits/sec [ 6] 5.00-6.00 sec 270 MBytes 2.26 Gbits/sec [ 8] 5.00-6.00 sec 312 MBytes 2.62 Gbits/sec [ 10] 5.00-6.00 sec 275 MBytes 2.30 Gbits/sec [SUM] 5.00-6.00 sec 1.10 GBytes 9.44 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 6.00-7.00 sec 274 MBytes 2.30 Gbits/sec [ 6] 6.00-7.00 sec 274 MBytes 2.30 Gbits/sec [ 8] 6.00-7.00 sec 288 MBytes 2.42 Gbits/sec [ 10] 6.00-7.00 sec 289 MBytes 2.43 Gbits/sec [SUM] 6.00-7.00 sec 1.10 GBytes 9.44 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 7.00-8.00 sec 272 MBytes 2.28 Gbits/sec [ 6] 7.00-8.00 sec 300 MBytes 2.51 Gbits/sec [ 8] 7.00-8.00 sec 285 MBytes 2.39 Gbits/sec [ 10] 7.00-8.00 sec 269 MBytes 2.26 Gbits/sec [SUM] 7.00-8.00 sec 1.10 GBytes 9.44 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 8.00-9.00 sec 295 MBytes 2.47 Gbits/sec [ 6] 8.00-9.00 sec 284 MBytes 2.38 Gbits/sec [ 8] 8.00-9.00 sec 274 MBytes 2.30 Gbits/sec [ 10] 8.00-9.00 sec 273 MBytes 2.29 Gbits/sec [SUM] 8.00-9.00 sec 1.10 GBytes 9.44 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 9.00-10.00 sec 276 MBytes 2.32 Gbits/sec [ 6] 9.00-10.00 sec 290 MBytes 2.43 Gbits/sec [ 8] 9.00-10.00 sec 269 MBytes 2.26 Gbits/sec [ 10] 9.00-10.00 sec 290 MBytes 2.43 Gbits/sec [SUM] 9.00-10.00 sec 1.10 GBytes 9.44 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 2.72 GBytes 2.34 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 2.72 GBytes 2.34 Gbits/sec receiver [ 6] 0.00-10.00 sec 2.71 GBytes 2.33 Gbits/sec 0 sender [ 6] 0.00-10.00 sec 2.71 GBytes 2.33 Gbits/sec receiver [ 8] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec 0 sender [ 8] 0.00-10.00 sec 2.73 GBytes 2.34 Gbits/sec receiver [ 10] 0.00-10.00 sec 2.66 GBytes 2.29 Gbits/sec 0 sender [ 10] 0.00-10.00 sec 2.66 GBytes 2.29 Gbits/sec receiver [SUM] 0.00-10.00 sec 10.8 GBytes 9.30 Gbits/sec 0 sender [SUM] 0.00-10.00 sec 10.8 GBytes 9.29 Gbits/sec receiver iperf Done. C:\download\iperf313>iperf3 -c unraid-one -P 4 Connecting to host unraid-one, port 5201 [ 4] local 192.168.1.194 port 50578 connected to 192.168.1.162 port 5201 [ 6] local 192.168.1.194 port 50579 connected to 192.168.1.162 port 5201 [ 8] local 192.168.1.194 port 50580 connected to 192.168.1.162 port 5201 [ 10] local 192.168.1.194 port 50581 connected to 192.168.1.162 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 180 MBytes 1.51 Gbits/sec [ 6] 0.00-1.00 sec 180 MBytes 1.51 Gbits/sec [ 8] 0.00-1.00 sec 201 MBytes 1.69 Gbits/sec [ 10] 0.00-1.00 sec 174 MBytes 1.46 Gbits/sec [SUM] 0.00-1.00 sec 735 MBytes 6.17 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 1.00-2.00 sec 166 MBytes 1.39 Gbits/sec [ 6] 1.00-2.00 sec 185 MBytes 1.55 Gbits/sec [ 8] 1.00-2.00 sec 260 MBytes 2.18 Gbits/sec [ 10] 1.00-2.00 sec 208 MBytes 1.75 Gbits/sec [SUM] 1.00-2.00 sec 820 MBytes 6.88 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 2.00-3.00 sec 144 MBytes 1.21 Gbits/sec [ 6] 2.00-3.00 sec 180 MBytes 1.51 Gbits/sec [ 8] 2.00-3.00 sec 180 MBytes 1.51 Gbits/sec [ 10] 2.00-3.00 sec 180 MBytes 1.51 Gbits/sec [SUM] 2.00-3.00 sec 684 MBytes 5.74 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 3.00-4.00 sec 182 MBytes 1.53 Gbits/sec [ 6] 3.00-4.00 sec 209 MBytes 1.75 Gbits/sec [ 8] 3.00-4.00 sec 179 MBytes 1.50 Gbits/sec [ 10] 3.00-4.00 sec 164 MBytes 1.38 Gbits/sec [SUM] 3.00-4.00 sec 734 MBytes 6.16 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 4.00-5.00 sec 150 MBytes 1.26 Gbits/sec [ 6] 4.00-5.00 sec 162 MBytes 1.36 Gbits/sec [ 8] 4.00-5.00 sec 189 MBytes 1.59 Gbits/sec [ 10] 4.00-5.00 sec 222 MBytes 1.87 Gbits/sec [SUM] 4.00-5.00 sec 724 MBytes 6.07 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 5.00-6.00 sec 167 MBytes 1.40 Gbits/sec [ 6] 5.00-6.00 sec 174 MBytes 1.46 Gbits/sec [ 8] 5.00-6.00 sec 172 MBytes 1.45 Gbits/sec [ 10] 5.00-6.00 sec 205 MBytes 1.72 Gbits/sec [SUM] 5.00-6.00 sec 718 MBytes 6.03 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 6.00-7.00 sec 152 MBytes 1.28 Gbits/sec [ 6] 6.00-7.00 sec 190 MBytes 1.60 Gbits/sec [ 8] 6.00-7.00 sec 211 MBytes 1.77 Gbits/sec [ 10] 6.00-7.00 sec 196 MBytes 1.64 Gbits/sec [SUM] 6.00-7.00 sec 750 MBytes 6.29 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 7.00-8.00 sec 183 MBytes 1.53 Gbits/sec [ 6] 7.00-8.00 sec 175 MBytes 1.46 Gbits/sec [ 8] 7.00-8.00 sec 205 MBytes 1.72 Gbits/sec [ 10] 7.00-8.00 sec 196 MBytes 1.64 Gbits/sec [SUM] 7.00-8.00 sec 758 MBytes 6.36 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 8.00-9.00 sec 223 MBytes 1.87 Gbits/sec [ 6] 8.00-9.00 sec 211 MBytes 1.77 Gbits/sec [ 8] 8.00-9.00 sec 160 MBytes 1.35 Gbits/sec [ 10] 8.00-9.00 sec 162 MBytes 1.36 Gbits/sec [SUM] 8.00-9.00 sec 756 MBytes 6.35 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 9.00-10.00 sec 142 MBytes 1.19 Gbits/sec [ 6] 9.00-10.00 sec 167 MBytes 1.40 Gbits/sec [ 8] 9.00-10.00 sec 199 MBytes 1.67 Gbits/sec [ 10] 9.00-10.00 sec 169 MBytes 1.42 Gbits/sec [SUM] 9.00-10.00 sec 678 MBytes 5.69 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 1.65 GBytes 1.42 Gbits/sec sender [ 4] 0.00-10.00 sec 1.65 GBytes 1.42 Gbits/sec receiver [ 6] 0.00-10.00 sec 1.79 GBytes 1.54 Gbits/sec sender [ 6] 0.00-10.00 sec 1.79 GBytes 1.54 Gbits/sec receiver [ 8] 0.00-10.00 sec 1.91 GBytes 1.64 Gbits/sec sender [ 8] 0.00-10.00 sec 1.91 GBytes 1.64 Gbits/sec receiver [ 10] 0.00-10.00 sec 1.83 GBytes 1.57 Gbits/sec sender [ 10] 0.00-10.00 sec 1.83 GBytes 1.57 Gbits/sec receiver [SUM] 0.00-10.00 sec 7.19 GBytes 6.17 Gbits/sec sender [SUM] 0.00-10.00 sec 7.18 GBytes 6.17 Gbits/sec receiver iperf Done. C:\download\iperf313> File transfers are still puzzling, though. UR to WS is as expected 500-700MB/s, but WS to UR starts out around 600MB/s, then drops to steady 200MB/s for remainder of transfer. I tried some additonal steps to isolate problem: - reset UR network settings to "default" - enabled disk shares & copied direct to disk vs user share - added different NVME drive to UR as 2nd cache pool & tried file copies there All trials continued to show writes to UR cache drive/s being throttled to around 200MB/s. The only thing I haven't measured is internal write speed of NVME cache. I think the diskspeed docker only tests writes. Ran fio sequential write against current and added nvme cache drives. Both results seem reasonable. root@unraid-one:~# fio -direct=1 -iodepth=128 -rw=write -ioengine=libaio -bs=128k -numjobs=1 -time_based=1 -runtime=60 -group_reporting -filename=/dev/nvme1n1 -name=test test: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=128 fio-3.23 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][w=626MiB/s][w=5010 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=21979: Thu Sep 8 11:24:56 2022 write: IOPS=5028, BW=629MiB/s (659MB/s)(36.9GiB/60030msec); 0 zone resets slat (nsec): min=3327, max=60505, avg=9983.52, stdev=5930.32 clat (usec): min=333, max=58025, avg=25441.33, stdev=10088.11 lat (usec): min=340, max=58032, avg=25451.49, stdev=10088.09 clat percentiles (usec): | 1.00th=[ 6915], 5.00th=[ 9110], 10.00th=[12125], 20.00th=[15664], | 30.00th=[18482], 40.00th=[21365], 50.00th=[25297], 60.00th=[29230], | 70.00th=[32113], 80.00th=[35390], 90.00th=[39060], 95.00th=[41157], | 99.00th=[44827], 99.50th=[46400], 99.90th=[49546], 99.95th=[50594], | 99.99th=[52167] bw ( KiB/s): min=608512, max=690944, per=100.00%, avg=644709.46, stdev=18437.83, samples=119 iops : min= 4754, max= 5398, avg=5036.77, stdev=144.05, samples=119 lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.02%, 10=6.38%, 20=28.64%, 50=64.87% lat (msec) : 100=0.08% cpu : usr=4.09%, sys=5.33%, ctx=286419, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,301881,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): WRITE: bw=629MiB/s (659MB/s), 629MiB/s-629MiB/s (659MB/s-659MB/s), io=36.9GiB (39.6GB), run=60030-60030msec Disk stats (read/write): nvme1n1: ios=0/603433, merge=0/0, ticks=0/15330361, in_queue=15330361, util=100.00% root@unraid-one:~# fio -direct=1 -iodepth=128 -rw=write -ioengine=libaio -bs=128k -numjobs=1 -time_based=1 -runtime=60 -group_reporting -filename=/dev/nvme0n1 -name=test test: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=128 fio-3.23 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][w=1806MiB/s][w=14.4k IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=979: Thu Sep 8 11:27:07 2022 write: IOPS=12.8k, BW=1605MiB/s (1683MB/s)(94.1GiB/60006msec); 0 zone resets slat (nsec): min=3089, max=70086, avg=7303.86, stdev=4234.12 clat (usec): min=187, max=86843, avg=9958.33, stdev=7533.10 lat (usec): min=192, max=86848, avg=9965.76, stdev=7533.19 clat percentiles (usec): | 1.00th=[ 379], 5.00th=[ 603], 10.00th=[ 857], 20.00th=[ 3032], | 30.00th=[ 6390], 40.00th=[ 7832], 50.00th=[ 8848], 60.00th=[ 9896], | 70.00th=[12387], 80.00th=[16909], 90.00th=[18482], 95.00th=[19530], | 99.00th=[33817], 99.50th=[46924], 99.90th=[64226], 99.95th=[69731], | 99.99th=[76022] bw ( MiB/s): min= 577, max= 1808, per=100.00%, avg=1606.06, stdev=328.10, samples=119 iops : min= 4616, max=14466, avg=12848.50, stdev=2624.81, samples=119 lat (usec) : 250=0.08%, 500=2.80%, 750=4.74%, 1000=5.09% lat (msec) : 2=6.73%, 4=0.82%, 10=40.31%, 20=35.23%, 50=3.81% lat (msec) : 100=0.39% cpu : usr=7.38%, sys=8.44%, ctx=548065, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,770638,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128 Run status group 0 (all jobs): WRITE: bw=1605MiB/s (1683MB/s), 1605MiB/s-1605MiB/s (1683MB/s-1683MB/s), io=94.1GiB (101GB), run=60006-60006msec Disk stats (read/write): nvme0n1: ios=45/768933, merge=0/0, ticks=9/7655367, in_queue=7655375, util=99.99% root@unraid-one:~# I don't think the switch is bad, its an unmanaged netgear 8-port 10g copper, XS708. I could buy some cat6 cables, but hate to throw $$ at it without exhausting other options. Thanks for looking at this anyway, I'm open to any/all suggestions. Quote Link to comment
JorgeB Posted September 8, 2022 Share Posted September 8, 2022 24 minutes ago, poto2 said: but WS to UR starts out around 600MB/s, then drops to steady 200MB/s for remainder of transfer. Drop is kind of fast to be the effects of RAM caching, try enabling disk shares and copying directly to the cache share. Quote Link to comment
JorgeB Posted September 8, 2022 Share Posted September 8, 2022 P.S. a single stream iperf test is much better to give an idea of the actual transfer speed, when you copy using one windows explorer session it's a single transfer. Quote Link to comment
poto2 Posted September 8, 2022 Author Share Posted September 8, 2022 (edited) Understood on the iperf streams, just making sure total bandwidth was available. The Win10<<>>UR file copies were done using user share. I also cleared and reformatted cache drive, then tried copying again via user share. Same result: UR>>Win10 700MB/s, Win10>>UR 600MB/s for a couple seconds, then 200MB/s. Grasping for straws, also checked router for duplicate IP addresses, but nothing out of the ordinary there. And restarted switches. Edited September 8, 2022 by poto2 Quote Link to comment
JorgeB Posted September 8, 2022 Share Posted September 8, 2022 30 minutes ago, poto2 said: Understood on the iperf streams, just making sure total bandwidth was available. If all is well you'll get close to line speed with iperf and a single stream, I get around 9.5Gbits Quote Link to comment
poto2 Posted September 8, 2022 Author Share Posted September 8, 2022 Something is definitely wrong, nowhere near 10g with single-stream iperf3. C:\download>cd iperf313 C:\download\iperf313>iperf3 -c unraid-one Connecting to host unraid-one, port 5201 [ 4] local 192.168.1.194 port 62698 connected to 192.168.1.162 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 169 MBytes 1.42 Gbits/sec [ 4] 1.00-2.00 sec 161 MBytes 1.35 Gbits/sec [ 4] 2.00-3.00 sec 166 MBytes 1.39 Gbits/sec [ 4] 3.00-4.00 sec 176 MBytes 1.47 Gbits/sec [ 4] 4.00-5.00 sec 146 MBytes 1.23 Gbits/sec [ 4] 5.00-6.00 sec 169 MBytes 1.41 Gbits/sec [ 4] 6.00-7.00 sec 200 MBytes 1.67 Gbits/sec [ 4] 7.00-8.00 sec 176 MBytes 1.48 Gbits/sec [ 4] 8.00-9.00 sec 144 MBytes 1.20 Gbits/sec [ 4] 9.00-10.00 sec 170 MBytes 1.43 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 1.64 GBytes 1.41 Gbits/sec sender [ 4] 0.00-10.00 sec 1.64 GBytes 1.41 Gbits/sec receiver iperf Done. C:\download\iperf313>iperf3 -c unraid-one -R Connecting to host unraid-one, port 5201 Reverse mode, remote host unraid-one is sending [ 4] local 192.168.1.194 port 62700 connected to 192.168.1.162 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 552 MBytes 4.63 Gbits/sec [ 4] 1.00-2.00 sec 544 MBytes 4.56 Gbits/sec [ 4] 2.00-3.00 sec 547 MBytes 4.59 Gbits/sec [ 4] 3.00-4.00 sec 540 MBytes 4.53 Gbits/sec [ 4] 4.00-5.00 sec 544 MBytes 4.56 Gbits/sec [ 4] 5.00-6.00 sec 542 MBytes 4.55 Gbits/sec [ 4] 6.00-7.00 sec 541 MBytes 4.54 Gbits/sec [ 4] 7.00-8.00 sec 543 MBytes 4.56 Gbits/sec [ 4] 8.00-9.00 sec 546 MBytes 4.58 Gbits/sec [ 4] 9.00-10.00 sec 549 MBytes 4.60 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 5.32 GBytes 4.57 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 5.32 GBytes 4.57 Gbits/sec receiver iperf Done. Quote Link to comment
poto2 Posted September 8, 2022 Author Share Posted September 8, 2022 Update: Direct-connect two Win10 PC, iperf3 still around 4Gbps with single-stream. File copy, however is 700MB/s either direction, so problem is likely network cables or switch. I've ordered cat6 patch cables, will update after swapping those in. 1 Quote Link to comment
Solution poto2 Posted September 9, 2022 Author Solution Share Posted September 9, 2022 Did a bit more digging, and this forum already had the answer for the Windows PC network throttling. Windows nic driver setting for interrupt moderation "disabled" resolved this issue. For UnRaid, what worked for me was adjusting interrupt coalescing. My nic driver didn't allow tx-usecs adjustment, but rx-usecs increase did the trick. root@unraid-one:~# ethtool -C eth0 rx-usecs 200 root@unraid-one:~# ethtool -c eth0 Coalesce parameters for eth0: Adaptive RX: n/a TX: n/a stats-block-usecs: n/a sample-interval: n/a pkt-rate-low: n/a pkt-rate-high: n/a rx-usecs: 200 rx-frames: n/a rx-usecs-irq: n/a rx-frames-irq: n/a tx-usecs: 0 tx-frames: n/a tx-usecs-irq: n/a tx-frames-irq: n/a rx-usecs-low: n/a rx-frame-low: n/a tx-usecs-low: n/a tx-frame-low: n/a rx-usecs-high: n/a rx-frame-high: n/a tx-usecs-high: n/a tx-frame-high: n/a CQE mode RX: n/a TX: n/a root@unraid-one:~# That change increased file copy speed from Win PC or TrueNAS to UnRaid cache drive quite a bit. I've a bit more reading to figure out how to make change permanent (user scripts?), and equivalent adjustment for TrueNAS, but problem solved AFAIK. Thanks to @JorgeB for suggesting areas to investigate. 3 1 Quote Link to comment
mytime34 Posted January 24, 2023 Share Posted January 24, 2023 Does anyone know, where to get fio? Quote Link to comment
poto2 Posted January 24, 2023 Author Share Posted January 24, 2023 nerdpack plugin will allow installation of useful tools, including fio. There may be a newer version of nerdpack, slightly different name for latest version of unraid. Careful with fio options - some will write to drives & impact filesystem. Quote Link to comment
FQs19 Posted March 5, 2023 Share Posted March 5, 2023 On 9/9/2022 at 4:53 PM, poto2 said: Windows nic driver setting for interrupt moderation "disabled" resolved this issue. For UnRaid, what worked for me was adjusting interrupt coalescing. My nic driver didn't allow tx-usecs adjustment, but rx-usecs increase did the trick. For me, all I had to do was Disable Interrupt Moderation and I got my nvme speeds back. I did not have to adjust Interrupt coalescing. Thank you for posting the solution. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.