April 25, 20215 yr Hi, my setup : Version: 6.9.2 (initiated on Version: 6.9.1) 1 gig wired network connection 2x 480GB cache SSD in Raid 3x HDD (one parity) Seagate IronWolf 16GB ECC RAM Xeon X3470 all drives set to ahci in bios dual LAN connected to my supermicro MB set to balance-alb I also tried with single cable Right now I have it set to "reconstruct write". Though, I guess that should not matter with cache drives. I am getting max transfer speeds of 50MB/s when lucky. Majority of times it is going 5-10 MB/s. Troubleshooting: I can exclude the network as I can saturate that (tested wit istat, iperf, OpenSpeedTest) There is no difference, if I have cache or not enabled for a share. Both top around ~50 MB/s. CPU seems OK (only rarely 1 core spikes to 100%) RAM seems OK When mover runs HDD can reach ~150 MB/s I am desperate for any kind of help, tips and diagnostics that could pinpoint the bottleneck. Thank you. darwin-diagnostics-20210425-0827.zip
April 25, 20215 yr Community Expert 2 hours ago, MarekNosek said: Right now I have it set to "reconstruct write". Though, I guess that should not matter with cache drives. It doesn't, but do you get better speeds if you transfer directly to the array?
April 25, 20215 yr Author 6 minutes ago, JorgeB said: It doesn't, but do you get better speeds if you transfer directly to the array? no, there is no detectable difference tested with a 3GB dummy file
April 25, 20215 yr Community Expert A 3GB file would be mostly cached to RAM, so it should transfer close to line speed, you mentioned running iperf, please post single stream test result.
April 25, 20215 yr Author 1 hour ago, JorgeB said: A 3GB file would be mostly cached to RAM, so it should transfer close to line speed, you mentioned running iperf, please post single stream test result. root@Darwin:~# iperf3 -s -V iperf 3.9 Linux Darwin 5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021 x86_64 ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Time: Sun, 25 Apr 2021 10:26:52 GMT Accepted connection from 10.10.10.100, port 60807 Cookie: Mareks-MacBook-Pro.local.1619346412. TCP MSS: 0 (default) [ 5] local 10.10.10.10 port 5201 connected to 10.10.10.100 port 60808 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 59.4 MBytes 498 Mbits/sec [ 5] 1.00-2.00 sec 85.2 MBytes 715 Mbits/sec [ 5] 2.00-3.00 sec 86.7 MBytes 727 Mbits/sec [ 5] 3.00-4.00 sec 86.9 MBytes 729 Mbits/sec [ 5] 4.00-5.00 sec 85.9 MBytes 720 Mbits/sec [ 5] 5.00-6.00 sec 83.5 MBytes 701 Mbits/sec [ 5] 6.00-7.00 sec 84.6 MBytes 710 Mbits/sec [ 5] 7.00-8.00 sec 86.1 MBytes 723 Mbits/sec [ 5] 8.00-9.00 sec 85.5 MBytes 717 Mbits/sec [ 5] 9.00-10.00 sec 87.1 MBytes 731 Mbits/sec [ 5] 10.00-10.00 sec 32.5 KBytes 336 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate [ 5] (sender statistics not available) [ 5] 0.00-10.00 sec 831 MBytes 697 Mbits/sec receiver rcv_tcp_congestion bbr iperf 3.9 Linux Darwin 5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021 x86_64 ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- I ran "iperf3 -c 10.10.10.10 -P 1" on the client side, if that is what you meant. in the morning I had Bitrate at 900s Mbits/sec (iperf3 -c 10.10.10.10)
April 25, 20215 yr Community Expert That's not very good, iperf should give you very close to line speed, still faster than the speeds you're getting, try enabling disk shares (settings -> global share settings ) then transfer to a disk (e.g.: \\tower\disk1) and see if speeds are better.
April 25, 20215 yr Author speeds to Cache and disk1 are similarly around 55 MB/s (with the same 3GB dummy file) I also tested multiple transfers(3) at the same time of that file to different shares and that tops around 75 MB/s
April 25, 20215 yr Community Expert Sorry, no more ideas, likely some hardware bottleneck, CPU is kind of old and SMB is single threaded, but still should be faster than that.
April 25, 20215 yr Author Thank you very much nonetheless. I will keep trying. Maybe I stumble upon some solution eventually.
April 26, 20215 yr Author in the morning I am reaching higher: Time: Mon, 26 Apr 2021 05:29:35 GMT Accepted connection from 10.10.10.100, port 64020 Cookie: Mareks-MacBook-Pro.local.1619414975. TCP MSS: 0 (default) [ 5] local 10.10.10.10 port 5201 connected to 10.10.10.100 port 64021 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 112 MBytes 936 Mbits/sec [ 5] 1.00-2.00 sec 112 MBytes 940 Mbits/sec [ 5] 2.00-3.00 sec 112 MBytes 940 Mbits/sec [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec [ 5] 4.00-5.00 sec 111 MBytes 931 Mbits/sec [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec [ 5] 6.00-7.00 sec 112 MBytes 939 Mbits/sec [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec [ 5] 9.00-10.00 sec 112 MBytes 940 Mbits/sec [ 5] 10.00-10.00 sec 492 KBytes 902 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate [ 5] (sender statistics not available) [ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec receiver rcv_tcp_congestion bbr iperf 3.9 Linux Darwin 5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021 x86_64 ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- both ways : Connecting to host 10.10.10.100, port 5201 [ 5] local 10.10.10.10 port 60268 connected to 10.10.10.100 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 115 MBytes 964 Mbits/sec 0 305 KBytes [ 5] 1.00-2.00 sec 112 MBytes 939 Mbits/sec 0 308 KBytes [ 5] 2.00-3.00 sec 112 MBytes 940 Mbits/sec 0 311 KBytes [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 0 303 KBytes [ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 328 KBytes [ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 322 KBytes [ 5] 6.00-7.00 sec 112 MBytes 944 Mbits/sec 0 305 KBytes [ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 0 311 KBytes [ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 311 KBytes [ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec receiver iperf Done. So I m questioning to try to OC the CPU ( it is running very low temps all the time).
April 26, 20215 yr Author which leads me to a potential discovery... do I understand it right that the X3470 I bought is underclocked? Every 1.0s: grep MHz /proc/cpuinfo cpu MHz : 1237.918 cpu MHz : 1442.808 cpu MHz : 2354.694 cpu MHz : 2413.491 cpu MHz : 1297.061 cpu MHz : 1386.903 cpu MHz : 1753.300 cpu MHz : 1744.538
April 26, 20215 yr Community Expert 1 hour ago, MarekNosek said: which leads me to a potential discovery... do I understand it right that the X3470 I bought is underclocked? Every 1.0s: grep MHz /proc/cpuinfo cpu MHz : 1237.918 cpu MHz : 1442.808 cpu MHz : 2354.694 cpu MHz : 2413.491 cpu MHz : 1297.061 cpu MHz : 1386.903 cpu MHz : 1753.300 cpu MHz : 1744.538 Have you checked the CPU Governor there where reports of it being set to power save on 6.9 cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor Edited April 26, 20215 yr by SimonF
April 27, 20215 yr Author Thank you SimonF for the tip, The CPU is running in 3000s MHz when copying (often dipping to 1800-2500), but it seems to be a bit faster, at least. What does not make sense to me is the inconsistency in the speeds, the fact that the RAM is almost all the time full with cached files and that the ssd writes only occasionally. Is it correct that the RAM does not "clear the cache" after the transfer to ssd? Lame comment (maybe), but it feels that the speed is dropping when RAM hits the ceiling and all is halted. These are statistics while copying many streams at the same time (I get the fastest results this way - around 70 MB/s):
April 27, 20215 yr On 4/25/2021 at 6:35 PM, MarekNosek said: Cookie: Mareks-MacBook-Pro.local.1619346412. MacOS or Windows ? MacOS seems have performance issues wirh Unraud in file transfer. 8 hours ago, MarekNosek said: Is it correct that the RAM does not "clear the cache" after the transfer to ssd? This normal, no problem. Edited April 27, 20215 yr by Vr2Io
April 28, 20215 yr Author 13 hours ago, Vr2Io said: MacOS or Windows ? yes it is MacOS 13 hours ago, Vr2Io said: MacOS seems have performance issues is there any "setting" to make it better that you know of? Of topic - I have ordered a pre-built PC with better CPU specs to test whether it is bottlenecking because of the the single thread performance. I will report soon.
April 30, 20215 yr Author OK so a tower with a newer CPU (Intel® Core™ i5-7500 CPU @ 3.40GHz) has a very similar results. I copied the same dummy file to a test share onto the same cache drive. ~ % rsync -a --progress --stats --human-readable XXXXXX/your-file-name.zip /Volumes/test building file list ... 1 file to consider your-file-name.zip 3.22G 100% 53.48MB/s 0:00:57 (xfer#1, to-check=0/1) Number of files: 1 Number of files transferred: 1 Total file size: 3.22G bytes Total transferred file size: 3.22G bytes Literal data: 3.22G bytes Matched data: 0 bytes File list size: 86 File list generation time: 0.002 seconds File list transfer time: 0.000 seconds Total bytes sent: 3.22G Total bytes received: 42 sent 3.22G bytes received 42 bytes 54.14M bytes/sec total size is 3.22G speedup is 1.00 So I am guessing there must be an issue elsewhere.
Archived
This topic is now archived and is closed to further replies.