MarekNosek Posted April 25, 2021 Share Posted April 25, 2021 Hi, my setup : Version: 6.9.2 (initiated on Version: 6.9.1) 1 gig wired network connection 2x 480GB cache SSD in Raid 3x HDD (one parity) Seagate IronWolf 16GB ECC RAM Xeon X3470 all drives set to ahci in bios dual LAN connected to my supermicro MB set to balance-alb I also tried with single cable Right now I have it set to "reconstruct write". Though, I guess that should not matter with cache drives. I am getting max transfer speeds of 50MB/s when lucky. Majority of times it is going 5-10 MB/s. Troubleshooting: I can exclude the network as I can saturate that (tested wit istat, iperf, OpenSpeedTest) There is no difference, if I have cache or not enabled for a share. Both top around ~50 MB/s. CPU seems OK (only rarely 1 core spikes to 100%) RAM seems OK When mover runs HDD can reach ~150 MB/s I am desperate for any kind of help, tips and diagnostics that could pinpoint the bottleneck. Thank you. darwin-diagnostics-20210425-0827.zip Quote Link to comment
JorgeB Posted April 25, 2021 Share Posted April 25, 2021 2 hours ago, MarekNosek said: Right now I have it set to "reconstruct write". Though, I guess that should not matter with cache drives. It doesn't, but do you get better speeds if you transfer directly to the array? Quote Link to comment
MarekNosek Posted April 25, 2021 Author Share Posted April 25, 2021 6 minutes ago, JorgeB said: It doesn't, but do you get better speeds if you transfer directly to the array? no, there is no detectable difference tested with a 3GB dummy file Quote Link to comment
JorgeB Posted April 25, 2021 Share Posted April 25, 2021 A 3GB file would be mostly cached to RAM, so it should transfer close to line speed, you mentioned running iperf, please post single stream test result. Quote Link to comment
MarekNosek Posted April 25, 2021 Author Share Posted April 25, 2021 1 hour ago, JorgeB said: A 3GB file would be mostly cached to RAM, so it should transfer close to line speed, you mentioned running iperf, please post single stream test result. root@Darwin:~# iperf3 -s -V iperf 3.9 Linux Darwin 5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021 x86_64 ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Time: Sun, 25 Apr 2021 10:26:52 GMT Accepted connection from 10.10.10.100, port 60807 Cookie: Mareks-MacBook-Pro.local.1619346412. TCP MSS: 0 (default) [ 5] local 10.10.10.10 port 5201 connected to 10.10.10.100 port 60808 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 59.4 MBytes 498 Mbits/sec [ 5] 1.00-2.00 sec 85.2 MBytes 715 Mbits/sec [ 5] 2.00-3.00 sec 86.7 MBytes 727 Mbits/sec [ 5] 3.00-4.00 sec 86.9 MBytes 729 Mbits/sec [ 5] 4.00-5.00 sec 85.9 MBytes 720 Mbits/sec [ 5] 5.00-6.00 sec 83.5 MBytes 701 Mbits/sec [ 5] 6.00-7.00 sec 84.6 MBytes 710 Mbits/sec [ 5] 7.00-8.00 sec 86.1 MBytes 723 Mbits/sec [ 5] 8.00-9.00 sec 85.5 MBytes 717 Mbits/sec [ 5] 9.00-10.00 sec 87.1 MBytes 731 Mbits/sec [ 5] 10.00-10.00 sec 32.5 KBytes 336 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate [ 5] (sender statistics not available) [ 5] 0.00-10.00 sec 831 MBytes 697 Mbits/sec receiver rcv_tcp_congestion bbr iperf 3.9 Linux Darwin 5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021 x86_64 ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- I ran "iperf3 -c 10.10.10.10 -P 1" on the client side, if that is what you meant. in the morning I had Bitrate at 900s Mbits/sec (iperf3 -c 10.10.10.10) Quote Link to comment
JorgeB Posted April 25, 2021 Share Posted April 25, 2021 That's not very good, iperf should give you very close to line speed, still faster than the speeds you're getting, try enabling disk shares (settings -> global share settings ) then transfer to a disk (e.g.: \\tower\disk1) and see if speeds are better. Quote Link to comment
MarekNosek Posted April 25, 2021 Author Share Posted April 25, 2021 speeds to Cache and disk1 are similarly around 55 MB/s (with the same 3GB dummy file) I also tested multiple transfers(3) at the same time of that file to different shares and that tops around 75 MB/s Quote Link to comment
JorgeB Posted April 25, 2021 Share Posted April 25, 2021 Sorry, no more ideas, likely some hardware bottleneck, CPU is kind of old and SMB is single threaded, but still should be faster than that. Quote Link to comment
MarekNosek Posted April 25, 2021 Author Share Posted April 25, 2021 Thank you very much nonetheless. I will keep trying. Maybe I stumble upon some solution eventually. Quote Link to comment
MarekNosek Posted April 26, 2021 Author Share Posted April 26, 2021 in the morning I am reaching higher: Time: Mon, 26 Apr 2021 05:29:35 GMT Accepted connection from 10.10.10.100, port 64020 Cookie: Mareks-MacBook-Pro.local.1619414975. TCP MSS: 0 (default) [ 5] local 10.10.10.10 port 5201 connected to 10.10.10.100 port 64021 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 112 MBytes 936 Mbits/sec [ 5] 1.00-2.00 sec 112 MBytes 940 Mbits/sec [ 5] 2.00-3.00 sec 112 MBytes 940 Mbits/sec [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec [ 5] 4.00-5.00 sec 111 MBytes 931 Mbits/sec [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec [ 5] 6.00-7.00 sec 112 MBytes 939 Mbits/sec [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec [ 5] 9.00-10.00 sec 112 MBytes 940 Mbits/sec [ 5] 10.00-10.00 sec 492 KBytes 902 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate [ 5] (sender statistics not available) [ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec receiver rcv_tcp_congestion bbr iperf 3.9 Linux Darwin 5.10.28-Unraid #1 SMP Wed Apr 7 08:23:18 PDT 2021 x86_64 ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- both ways : Connecting to host 10.10.10.100, port 5201 [ 5] local 10.10.10.10 port 60268 connected to 10.10.10.100 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 115 MBytes 964 Mbits/sec 0 305 KBytes [ 5] 1.00-2.00 sec 112 MBytes 939 Mbits/sec 0 308 KBytes [ 5] 2.00-3.00 sec 112 MBytes 940 Mbits/sec 0 311 KBytes [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 0 303 KBytes [ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 328 KBytes [ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 322 KBytes [ 5] 6.00-7.00 sec 112 MBytes 944 Mbits/sec 0 305 KBytes [ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 0 311 KBytes [ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 311 KBytes [ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec receiver iperf Done. So I m questioning to try to OC the CPU ( it is running very low temps all the time). Quote Link to comment
MarekNosek Posted April 26, 2021 Author Share Posted April 26, 2021 which leads me to a potential discovery... do I understand it right that the X3470 I bought is underclocked? Every 1.0s: grep MHz /proc/cpuinfo cpu MHz : 1237.918 cpu MHz : 1442.808 cpu MHz : 2354.694 cpu MHz : 2413.491 cpu MHz : 1297.061 cpu MHz : 1386.903 cpu MHz : 1753.300 cpu MHz : 1744.538 Quote Link to comment
SimonF Posted April 26, 2021 Share Posted April 26, 2021 (edited) 1 hour ago, MarekNosek said: which leads me to a potential discovery... do I understand it right that the X3470 I bought is underclocked? Every 1.0s: grep MHz /proc/cpuinfo cpu MHz : 1237.918 cpu MHz : 1442.808 cpu MHz : 2354.694 cpu MHz : 2413.491 cpu MHz : 1297.061 cpu MHz : 1386.903 cpu MHz : 1753.300 cpu MHz : 1744.538 Have you checked the CPU Governor there where reports of it being set to power save on 6.9 cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor Edited April 26, 2021 by SimonF Quote Link to comment
MarekNosek Posted April 27, 2021 Author Share Posted April 27, 2021 Thank you SimonF for the tip, The CPU is running in 3000s MHz when copying (often dipping to 1800-2500), but it seems to be a bit faster, at least. What does not make sense to me is the inconsistency in the speeds, the fact that the RAM is almost all the time full with cached files and that the ssd writes only occasionally. Is it correct that the RAM does not "clear the cache" after the transfer to ssd? Lame comment (maybe), but it feels that the speed is dropping when RAM hits the ceiling and all is halted. These are statistics while copying many streams at the same time (I get the fastest results this way - around 70 MB/s): Quote Link to comment
Vr2Io Posted April 27, 2021 Share Posted April 27, 2021 (edited) On 4/25/2021 at 6:35 PM, MarekNosek said: Cookie: Mareks-MacBook-Pro.local.1619346412. MacOS or Windows ? MacOS seems have performance issues wirh Unraud in file transfer. 8 hours ago, MarekNosek said: Is it correct that the RAM does not "clear the cache" after the transfer to ssd? This normal, no problem. Edited April 27, 2021 by Vr2Io Quote Link to comment
MarekNosek Posted April 28, 2021 Author Share Posted April 28, 2021 13 hours ago, Vr2Io said: MacOS or Windows ? yes it is MacOS 13 hours ago, Vr2Io said: MacOS seems have performance issues is there any "setting" to make it better that you know of? Of topic - I have ordered a pre-built PC with better CPU specs to test whether it is bottlenecking because of the the single thread performance. I will report soon. Quote Link to comment
MarekNosek Posted April 30, 2021 Author Share Posted April 30, 2021 OK so a tower with a newer CPU (Intel® Core™ i5-7500 CPU @ 3.40GHz) has a very similar results. I copied the same dummy file to a test share onto the same cache drive. ~ % rsync -a --progress --stats --human-readable XXXXXX/your-file-name.zip /Volumes/test building file list ... 1 file to consider your-file-name.zip 3.22G 100% 53.48MB/s 0:00:57 (xfer#1, to-check=0/1) Number of files: 1 Number of files transferred: 1 Total file size: 3.22G bytes Total transferred file size: 3.22G bytes Literal data: 3.22G bytes Matched data: 0 bytes File list size: 86 File list generation time: 0.002 seconds File list transfer time: 0.000 seconds Total bytes sent: 3.22G Total bytes received: 42 sent 3.22G bytes received 42 bytes 54.14M bytes/sec total size is 3.22G speedup is 1.00 So I am guessing there must be an issue elsewhere. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.