nid Posted April 8, 2018 Share Posted April 8, 2018 That's why I bought the external Intel 82574L PCIe but I didn't see any greatness ever after I switched from the onboard (probably Realtek) NIC. Do you know where I can check and set the setting for "IRQ Coalescing" for the Intel NIC? I have Netdata installed and I captured two screenshots for you during receiving a large video file from the server. You can see this "brick" curve on the right. This was a steady 50mbyte/sec copy. Of course the WD Red could reach 100mbyte/sec as far as I know. mpstat during the copy is showing around 6000/second. According to the article anything over 5000/second is a lot. Also I found that IRQs 24 and 25 have the most requests and here is the list: root@Tower:~# cat /proc/interrupts CPU0 0: 143 IO-APIC 2-edge timer 8: 3 IO-APIC 8-edge rtc0 9: 0 IO-APIC 9-fasteoi acpi 14: 0 IO-APIC 14-edge pata_atiixp 15: 0 IO-APIC 15-edge pata_atiixp 17: 593202 IO-APIC 17-fasteoi ehci_hcd:usb1, ehci_hcd:usb2, ehci_hcd:usb3 18: 727490 IO-APIC 18-fasteoi ohci_hcd:usb4, ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7 19: 526037 IO-APIC 19-fasteoi ahci[0000:00:11.0] 24: 3410481 PCI-MSI 524288-edge eth0-rx-0 25: 2148133 PCI-MSI 524289-edge eth0-tx-0 26: 2 PCI-MSI 524290-edge eth0 NMI: 0 Non-maskable interrupts LOC: 53314583 Local timer interrupts SPU: 0 Spurious interrupts PMI: 0 Performance monitoring interrupts IWI: 21269020 IRQ work interrupts RTR: 0 APIC ICR read retries RES: 0 Rescheduling interrupts CAL: 0 Function call interrupts TLB: 0 TLB shootdowns TRM: 0 Thermal event interrupts THR: 0 Threshold APIC interrupts DFR: 0 Deferred Error APIC interrupts MCE: 0 Machine check exceptions MCP: 266 Machine check polls HYP: 0 Hypervisor callback interrupts ERR: 0 MIS: 0 PIN: 0 Posted-interrupt notification event NPI: 0 Nested posted-interrupt event PIW: 0 Posted-interrupt wakeup event So far so good but I don't know what these results exactly mean... Thanks so far for all the help Quote Link to comment
John_M Posted April 8, 2018 Share Posted April 8, 2018 You'll need to check the Intel Ark to see what your chipset is capable of and the man page for ethtool to see how to control it. It looks as though -C is the option required. Quote Link to comment
pwm Posted April 8, 2018 Share Posted April 8, 2018 Yes, ethtool -C allows a number of parameters to be set. Both number of packets to buffer before issuing an interrupt, and max number of microseconds to wait. -c shows current values. In the same way -k and -K relates to protocol offload, i.e. hardware acceleration. One issue here is that your machine spent 11% on softinterrupts and over 70% on system time. Is it the Samba processes that consumes much of the system time? Quote Link to comment
nid Posted April 8, 2018 Share Posted April 8, 2018 40 minutes ago, pwm said: One issue here is that your machine spent 11% on softinterrupts and over 70% on system time. Is it the Samba processes that consumes much of the system time? While copying this is the output of top: Quote Link to comment
pwm Posted April 8, 2018 Share Posted April 8, 2018 So lots of time spent by shfs. Have you tried to tweak the disk settings with: md_num_stripes: 4096 md_sync_window: 2048 md_sync_thresh: 2000 Quote Link to comment
nid Posted April 8, 2018 Share Posted April 8, 2018 18 minutes ago, pwm said: So lots of time spent by shfs. Have you tried to tweak the disk settings with: md_num_stripes: 4096 md_sync_window: 2048 md_sync_thresh: 2000 Just tried. It doesn't get better. Copy speed is now inconsistent (going below but also over 50mb/sec) and shfs has even higher %CPU. Quote Link to comment
nid Posted April 8, 2018 Share Posted April 8, 2018 (edited) I noticed that actually the higher the copy speed goes, the higher the %CPU of shfs is. Also the write (to Array) speed is faster (70mb/sec) than the read speed (50mb/sec). Test on the same file. Edited April 8, 2018 by nid Quote Link to comment
pwm Posted April 8, 2018 Share Posted April 8, 2018 1 minute ago, nid said: I noticed that actually the higher the copy speed goes, the higher the %CPU of shfs is. That's expected, since it needs to deliver more MB of file data / second. With 100% CPU, your machine hits the limits. So the only way to get more speed is to find tweaks that makes some of the operations consume less CPU. I have never compared if the tweaks I posted affects the CPU load of shfs since I haven't had any unRAID machine reach 100% on copy. The parameters are mostly posted for people who needs to get more speed out of the drives. Quote Link to comment
nid Posted April 8, 2018 Share Posted April 8, 2018 1 hour ago, pwm said: Yes, ethtool -C allows a number of parameters to be set. Both number of packets to buffer before issuing an interrupt, and max number of microseconds to wait. -c shows current values. In the same way -k and -K relates to protocol offload, i.e. hardware acceleration. I use the command to change some of the parameters but their status doesn't change. For example adaptive rx and tx remain off. Also tried other values for tx-usecs etc. but they don't change. Any ideas? root@Tower:~# ethtool -C eth0 adaptive-rx on root@Tower:~# ethtool -C eth0 adaptive-tx on root@Tower:~# ethtool -c eth0 Coalesce parameters for eth0: Adaptive RX: off TX: off stats-block-usecs: 0 sample-interval: 0 pkt-rate-low: 0 pkt-rate-high: 0 rx-usecs: 3 rx-frames: 0 rx-usecs-irq: 0 rx-frames-irq: 0 tx-usecs: 0 tx-frames: 0 tx-usecs-irq: 0 tx-frames-irq: 0 rx-usecs-low: 0 rx-frame-low: 0 tx-usecs-low: 0 tx-frame-low: 0 rx-usecs-high: 0 rx-frame-high: 0 tx-usecs-high: 0 tx-frame-high: 0 Quote Link to comment
pwm Posted April 8, 2018 Share Posted April 8, 2018 Maybe your card doesn't support it. But you probably don't really need it, since your goal isn't to get very low latency for a gaming computer but instead to get better bandwidth by reducing the CPU load. So you want non-zero values for rx-usecs, rx-frames, tx-usecs, tx-frames, ... What happens if you set rx-usecs 20 and tx-usecs 50? Quote Link to comment
nid Posted April 9, 2018 Share Posted April 9, 2018 8 hours ago, pwm said: But you probably don't really need it, since your goal isn't to get very low latency for a gaming computer but instead to get better bandwidth by reducing the CPU load. So you want non-zero values for rx-usecs, rx-frames, tx-usecs, tx-frames, ... What happens if you set rx-usecs 20 and tx-usecs 50? Exactly, now as you see almost everything is 0 and for some reason can't be changed. Quote Link to comment
pwm Posted April 9, 2018 Share Posted April 9, 2018 1 hour ago, nid said: Exactly, now as you see almost everything is 0 and for some reason can't be changed. Then it doesn't seem like you have so many options left. Quote Link to comment
nid Posted April 9, 2018 Share Posted April 9, 2018 3 minutes ago, pwm said: Then it doesn't seem like you have so many options left. Yeah either to leave it as is and not spend any more time troubleshooting or upgrade the hardware. Thanks for your time! Cheers Quote Link to comment
nid Posted May 1, 2018 Share Posted May 1, 2018 Hi. I did a test replacing the system with an older but much more powerful one that I had sitting around. Old system: Asrock e35lm1 (weak single core AMD E240) with 8GB DDR3 1066 Power consumption with all 4 disks spinning: 50W New system: Asus P5E with Intel Quad Core Q9650 @3GHZ and 4GB DDR2 800 Power consumption with all 4 disks spinning: 75W This processor is of course much faster and never hits 100% during copies, unlike the suffocating AMD. Maximum cpu % I've seen was around 55%. Now I can achieve copy speeds of up to 110MB/sec. But there is still no consistency. Sometimes the speed will drop down to 10MB/sec and even less and then slowly rise up. Another time just stays low throughout the whole copy. Another time yet reaches full speed throughout the whole copy. Even while copying the same large file. I'm talking about getting data from the array to the Windows PC. I'm using various random sample files from my collection to make this observation. Tried also the two NIC settings (offload etc) but didn't see any change. NIC is the same Intel 82574L. Don't know if it would make sense to try the onboard Marvell... I feel it is a software thing and currently not very trustful about Unraid performance until I see some solid performance, not with super expensive server grade hardware or even expensive workstation (say like an i7 machine). Just for copying data! Perhaps I should have just used some Linux distro for this but I've already purchased Unraid and it is also lovely for its simplicity. Moreover, I don't like this consumption increase. There is also an audible noise increase due to the higher demand for cooling of the Intel Core Quad. Who can recommend a modern cost effective and low consumption system (mb/cpu/ram) that can handle just data i/o with full gigabit speeds and Crashplan and Rutorrent running as dockers? I need it to run cool and silent 24/7, so it doesn't need to be a monster. The Asrock board was close but not there due to the extremely feeble performance. By the way I had taken advice by an older blog of someone to get that motherboard. Not very successful advice. Thanks!! Quote Link to comment
nid Posted May 2, 2018 Share Posted May 2, 2018 I attach some screenshots of the Windows copy dialog. I start with a just rebooted Unraid server and first two copies are reaching top speed 110MB/sec. Files tested are always single 1-4GB files. Subsequent copies start showing this kind of ramping. Copy to the array also shows the same. Maybe has to do with RAM getting filled up?... Quote Link to comment
Vr2Io Posted May 2, 2018 Share Posted May 2, 2018 unRAID array performance always not that good, I waiting a breakthrough for longtime ago. Quote Link to comment
John_M Posted May 2, 2018 Share Posted May 2, 2018 It depends what you're doing. Large file reads from unRAID are fast and easily saturate a gigabit Ethernet link. Writes to unRAID vary depending on circumstances. Initially they are fast because you're writing to a RAM buffer. After that it depends on whether you're writing to cache or directly to the array, and if the latter whether turbo write is enabled or not. Writing many small files is much slower than writing few large files because of the per-file overhead. Quote Link to comment
nid Posted May 2, 2018 Share Posted May 2, 2018 (edited) 1 hour ago, John_M said: It depends what you're doing. Large file reads from unRAID are fast and easily saturate a gigabit Ethernet link. Writes to unRAID vary depending on circumstances. Initially they are fast because you're writing to a RAM buffer. After that it depends on whether you're writing to cache or directly to the array, and if the latter whether turbo write is enabled or not. Writing many small files is much slower than writing few large files because of the per-file overhead. I forgot to mention that I don't use a cache drive. Let's say OK about the writes. But what about the reads? Why is it so inconsistent even if there is no other load on the system or the network? Between my Windows machines the copies of large files get steady maximum gigabit speed, when there is no other significant load. I tried just now. And also tried again with Unraid, managed to copy around 10GB of various large files pinned to 110MB/sec. After that again the same sloppiness. Edited May 2, 2018 by nid Quote Link to comment
John_M Posted May 2, 2018 Share Posted May 2, 2018 2 hours ago, nid said: Why is it so inconsistent I don't know. The only thing anyone knows about your new server is that it has a particular motherboard, processor and RAM. That's really not much to go on Quote Link to comment
nid Posted May 2, 2018 Share Posted May 2, 2018 3 hours ago, John_M said: I don't know. The only thing anyone knows about your new server is that it has a particular motherboard, processor and RAM. That's really not much to go on More info is in the previous posts (with the old hardware). I change m/b, cpu and ram, keeping everything else exactly the same. I notice that now I can reach full gigabit speeds for most of the time, just not steady, which is the annoying part. I upload new diagnostics as of today. As far as I can tell the system is perfectly healthy and is almost without any load. Rutorrent is rarely (almost never these days) doing any transfer and Crashplan is backing up 2 windows pcs every 24 hours and I know when that is, so I'm not testing at the same time. Fix Common Problems says everything is fine. As everyone knows, one could test thousand of different scenarios (turn off this and that and try again etc) but I don't have time nor patience anymore to do that. As a last resort I could try a fresh Unraid installation but it is so much work and hassle that I'm very skeptical about it yet. To be honest after about 3 years I'm just disappointed from Unraid. I should be happy of course because at last I have lots of moments with top performance... This shows to me that the hardware can do it. What about the (paid) software? Still don't take me wrong, I admire the amazing work that has been done with Unraid. 8 hours ago, Benson said: unRAID array performance always not that good, I waiting a breakthrough for longtime ago. Well perhaps that says it all... And I remember along the years I've been coming across many posts with people complaining about transfer rate issues. tower-diagnostics-20180502-2335.zip Quote Link to comment
John_M Posted May 2, 2018 Share Posted May 2, 2018 (edited) I recommend you run an extended SMART self-test on disk1. It's showing signs of not being as healthy as the other disks. EDIT I see from the previous page I recommended that before and that you have run an extended test and it passed. Edited May 2, 2018 by John_M Quote Link to comment
Vr2Io Posted May 3, 2018 Share Posted May 3, 2018 (edited) Quote Well perhaps that says it all... And I remember along the years I've been coming across many posts with people complaining about transfer rate issues. For pure read disk array with 1G network, read always could be ~100MB/s and no drop (may be just few and short). Of course, the file not in disk inner slow area. I mainly complaint the write speed or Network was 10G. Pls ensure the destination ( file copy ) also provide sufficient writing speed too. Edited May 3, 2018 by Benson Quote Link to comment
nid Posted May 5, 2018 Share Posted May 5, 2018 On 5/3/2018 at 1:34 PM, Benson said: For pure read disk array with 1G network, read always could be ~100MB/s and no drop (may be just few and short). Of course, the file not in disk inner slow area. I mainly complaint the write speed or Network was 10G. Copying between my two Windows 10 workstations I have this "just a few and short" drops. It is so infrequent and totally acceptable. But copying from the Unraid array the drops are huge and long, from 110MB/sec to 20MB/sec, going up and down like the screenshots I posted before. Even copying the same file (after some time has passed when it seems it's not anymore cached) shows a different copy curve, which leads me to believe it is not file based or where the data resides on the disk platter. I guess even in the innermost area the WD RED is faster than 20MB/sec. On 5/3/2018 at 1:34 PM, Benson said: Pls ensure the destination ( file copy ) also provide sufficient writing speed too. Destination drives are always SSD (windows 10 system) drives. Quote Link to comment
John_M Posted May 5, 2018 Share Posted May 5, 2018 You could try copying from one disk to another from within unRAID. That would help you to determine whether your problem is disk-related or network-related. Quote Link to comment
nid Posted May 5, 2018 Share Posted May 5, 2018 (edited) 15 hours ago, John_M said: You could try copying from one disk to another from within unRAID. That would help you to determine whether your problem is disk-related or network-related. Good idea! I tried some copies within mc and the rate is mediocre. 10-30MB/sec, mostly staying close to 15, but fluctuating. What could this mean? Is it a valid test copying from a disk to itself? There is great probability that's what's happening since I only have 2 disks + parity. Edited May 6, 2018 by nid Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.