Slow read speeds from the array

[nid]

That's why I bought the external Intel 82574L PCIe but I didn't see any greatness ever after I switched from the onboard (probably Realtek) NIC. Do you know where I can check and set the setting for "IRQ Coalescing" for the Intel NIC?

 

I have Netdata installed and I captured two screenshots for you during receiving a large video file from the server. You can see this "brick" curve on the right. This was a steady 50mbyte/sec copy. Of course the WD Red could reach 100mbyte/sec as far as I know.

 

mpstat during the copy is showing around 6000/second. According to the article anything over 5000/second is a lot.

 

Also I found that IRQs 24 and 25 have the most requests and here is the list:

 

root@Tower:~# cat /proc/interrupts
           CPU0
  0:        143   IO-APIC   2-edge      timer
  8:          3   IO-APIC   8-edge      rtc0
  9:          0   IO-APIC   9-fasteoi   acpi
 14:          0   IO-APIC  14-edge      pata_atiixp
 15:          0   IO-APIC  15-edge      pata_atiixp
 17:     593202   IO-APIC  17-fasteoi   ehci_hcd:usb1, ehci_hcd:usb2, ehci_hcd:usb3
 18:     727490   IO-APIC  18-fasteoi   ohci_hcd:usb4, ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7
 19:     526037   IO-APIC  19-fasteoi   ahci[0000:00:11.0]
 24:    3410481   PCI-MSI 524288-edge      eth0-rx-0
 25:    2148133   PCI-MSI 524289-edge      eth0-tx-0
 26:          2   PCI-MSI 524290-edge      eth0
NMI:          0   Non-maskable interrupts
LOC:   53314583   Local timer interrupts
SPU:          0   Spurious interrupts
PMI:          0   Performance monitoring interrupts
IWI:   21269020   IRQ work interrupts
RTR:          0   APIC ICR read retries
RES:          0   Rescheduling interrupts
CAL:          0   Function call interrupts
TLB:          0   TLB shootdowns
TRM:          0   Thermal event interrupts
THR:          0   Threshold APIC interrupts
DFR:          0   Deferred Error APIC interrupts
MCE:          0   Machine check exceptions
MCP:        266   Machine check polls
HYP:          0   Hypervisor callback interrupts
ERR:          0
MIS:          0
PIN:          0   Posted-interrupt notification event
NPI:          0   Nested posted-interrupt event
PIW:          0   Posted-interrupt wakeup event

 

 

So far so good but I don't know what these results exactly mean...

Thanks so far for all the help

[John_M]

You'll need to check the Intel Ark to see what your chipset is capable of and the man page for ethtool to see how to control it. It looks as though -C is the option required.

[pwm]

Yes, ethtool -C allows a number of parameters to be set. Both number of packets to buffer before issuing an interrupt, and max number of microseconds to wait.

-c shows current values.

 

In the same way -k and -K relates to protocol offload, i.e. hardware acceleration.

 

One issue here is that your machine spent 11% on softinterrupts and over 70% on system time. Is it the Samba processes that consumes much of the system time?

[nid]

40 minutes ago, pwm said:

One issue here is that your machine spent 11% on softinterrupts and over 70% on system time. Is it the Samba processes that consumes much of the system time?

While copying this is the output of top:

 

 

[pwm]

So lots of time spent by shfs.

 

Have you tried to tweak the disk settings with:

md_num_stripes:	4096
md_sync_window:	2048
md_sync_thresh:	2000

 

[nid]

18 minutes ago, pwm said:

So lots of time spent by shfs.

 

Have you tried to tweak the disk settings with:

md_num_stripes:	4096
md_sync_window:	2048
md_sync_thresh:	2000

 

Just tried. It doesn't get better. Copy speed is now inconsistent (going below but also over 50mb/sec) and shfs has even higher %CPU.

[nid]

I noticed that actually the higher the copy speed goes, the higher the %CPU of shfs is. 

 

Also the write (to Array) speed is faster (70mb/sec) than the read speed (50mb/sec). Test on the same file.

Edited April 8, 2018 by nid

[pwm]

1 minute ago, nid said:

I noticed that actually the higher the copy speed goes, the higher the %CPU of shfs is. 

 

That's expected, since it needs to deliver more MB of file data / second.

 

With 100% CPU, your machine hits the limits. So the only way to get more speed is to find tweaks that makes some of the operations consume less CPU. I have never compared if the tweaks I posted affects the CPU load of shfs since I haven't had any unRAID machine reach 100% on copy. The parameters are mostly posted for people who needs to get more speed out of the drives.

[nid]

1 hour ago, pwm said:

Yes, ethtool -C allows a number of parameters to be set. Both number of packets to buffer before issuing an interrupt, and max number of microseconds to wait.

-c shows current values.

 

In the same way -k and -K relates to protocol offload, i.e. hardware acceleration.

 

I use the command to change some of the parameters but their status doesn't change.

For example adaptive rx and tx remain off. Also tried other values for tx-usecs etc. but they don't change. Any ideas?

 

root@Tower:~# ethtool -C eth0 adaptive-rx on
root@Tower:~# ethtool -C eth0 adaptive-tx on
root@Tower:~# ethtool -c eth0
Coalesce parameters for eth0:
Adaptive RX: off  TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 3
rx-frames: 0
rx-usecs-irq: 0
rx-frames-irq: 0

tx-usecs: 0
tx-frames: 0
tx-usecs-irq: 0
tx-frames-irq: 0

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

[pwm]

Maybe your card doesn't support it.

 

But you probably don't really need it, since your goal isn't to get very low latency for a gaming computer but instead to get better bandwidth by reducing the CPU load.

So you want non-zero values for rx-usecs, rx-frames, tx-usecs, tx-frames, ...

 

What happens if you set rx-usecs 20 and tx-usecs 50?

[nid]

8 hours ago, pwm said:

But you probably don't really need it, since your goal isn't to get very low latency for a gaming computer but instead to get better bandwidth by reducing the CPU load.

So you want non-zero values for rx-usecs, rx-frames, tx-usecs, tx-frames, ...

 

What happens if you set rx-usecs 20 and tx-usecs 50?

 

Exactly, now as you see almost everything is 0 and for some reason can't be changed.

[pwm]

1 hour ago, nid said:

 

Exactly, now as you see almost everything is 0 and for some reason can't be changed.

 

Then it doesn't seem like you have so many options left.

[nid]

3 minutes ago, pwm said:

 

Then it doesn't seem like you have so many options left.

 

Yeah either to leave it as is and not spend any more time troubleshooting or upgrade the hardware. Thanks for your time! Cheers

[nid]

Hi. I did a test replacing the system with an older but much more powerful one that I had sitting around.

 

Old system:

Asrock e35lm1 (weak single core AMD E240) with 8GB DDR3 1066

Power consumption with all 4 disks spinning: 50W

 

New system:

Asus P5E with Intel Quad Core Q9650 @3GHZ and 4GB DDR2 800

Power consumption with all 4 disks spinning: 75W

 

This processor is of course much faster and never hits 100% during copies, unlike the suffocating AMD. Maximum cpu % I've seen was around 55%. Now I can achieve copy speeds of up to 110MB/sec. But there is still no consistency. Sometimes the speed will drop down to 10MB/sec and even less and then slowly rise up. Another time just stays low throughout the whole copy. Another time yet reaches full speed throughout the whole copy. Even while copying the same large file. I'm talking about getting data from the array to the Windows PC. I'm using various random sample files from my collection to make this observation. Tried also the two NIC settings (offload etc) but didn't see any change. NIC is the same Intel 82574L. Don't know if it would make sense to try the onboard Marvell... I feel it is a software thing and currently not very trustful about Unraid performance until I see some solid performance, not with super expensive server grade hardware or even expensive workstation (say like an i7 machine). Just for copying data! Perhaps I should have just used some Linux distro for this but I've already purchased Unraid and it is also lovely for its simplicity.

 

Moreover, I don't like this consumption increase. There is also an audible noise increase due to the higher demand for cooling of the Intel Core Quad. Who can recommend a modern cost effective and low consumption system (mb/cpu/ram) that can handle just data i/o with full gigabit speeds and Crashplan and Rutorrent running as dockers? I need it to run cool and silent 24/7, so it doesn't need to be a monster. The Asrock board was close but not there due to the extremely feeble performance. By the way I had taken advice by an older blog of someone to get that motherboard. Not very successful advice.

 

Thanks!!

 

 

 

[nid]

I attach some screenshots of the Windows copy dialog. I start with a just rebooted Unraid server and first two copies are reaching top speed 110MB/sec. Files tested are always single 1-4GB files. Subsequent copies start showing this kind of ramping. Copy to the array also shows the same. Maybe has to do with RAM getting filled up?...

[Vr2Io]

unRAID array performance always not that good, I waiting a breakthrough for longtime ago.

[John_M]

It depends what you're doing. Large file reads from unRAID are fast and easily saturate a gigabit Ethernet link. Writes to unRAID vary depending on circumstances. Initially they are fast because you're writing to a RAM buffer. After that it depends on whether you're writing to cache or directly to the array, and if the latter whether turbo write is enabled or not. Writing many small files is much slower than writing few large files because of the per-file overhead.

[nid]

1 hour ago, John_M said:

It depends what you're doing. Large file reads from unRAID are fast and easily saturate a gigabit Ethernet link. Writes to unRAID vary depending on circumstances. Initially they are fast because you're writing to a RAM buffer. After that it depends on whether you're writing to cache or directly to the array, and if the latter whether turbo write is enabled or not. Writing many small files is much slower than writing few large files because of the per-file overhead.

 

I forgot to mention that I don't use a cache drive. Let's say OK about the writes. But what about the reads? Why is it so inconsistent even if there is no other load on the system or the network? Between my Windows machines the copies of large files get steady maximum gigabit speed, when there is no other significant load. I tried just now. And also tried again with Unraid, managed to copy around 10GB of various large files pinned to 110MB/sec. After that again the same sloppiness.

Edited May 2, 2018 by nid

[John_M]

2 hours ago, nid said:

Why is it so inconsistent

 

I don't know. The only thing anyone knows about your new server is that it has a particular motherboard, processor and RAM. That's really not much to go on

[nid]

3 hours ago, John_M said:

 

I don't know. The only thing anyone knows about your new server is that it has a particular motherboard, processor and RAM. That's really not much to go on

 

More info is in the previous posts (with the old hardware). I change m/b, cpu and ram, keeping everything else exactly the same. I notice that now I can reach full gigabit speeds for most of the time, just not steady, which is the annoying part. I upload new diagnostics as of today. As far as I can tell the system is perfectly healthy and is almost without any load. Rutorrent is rarely (almost never these days) doing any transfer and Crashplan is backing up 2 windows pcs every 24 hours and I know when that is, so I'm not testing at the same time. Fix Common Problems says everything is fine. As everyone knows, one could test thousand of different scenarios (turn off this and that and try again etc) but I don't have time nor patience anymore to do that. As a last resort I could try a fresh Unraid installation but it is so much work and hassle that I'm very skeptical about it yet. To be honest after about 3 years I'm just disappointed from Unraid. I should be happy of course because at last I have lots of moments with top performance... This shows to me that the hardware can do it. What about the (paid) software? Still don't take me wrong, I admire the amazing work that has been done with Unraid.

 

8 hours ago, Benson said:

unRAID array performance always not that good, I waiting a breakthrough for longtime ago.

 

Well perhaps that says it all... And I remember along the years I've been coming across many posts with people complaining about transfer rate issues.

 

 

tower-diagnostics-20180502-2335.zip

[John_M]

I recommend you run an extended SMART self-test on disk1. It's showing signs of not being as healthy as the other disks.

 

EDIT

I see from the previous page I recommended that before and that you have run an extended test and it passed.

Edited May 2, 2018 by John_M

[Vr2Io]

Quote

Well perhaps that says it all... And I remember along the years I've been coming across many posts with people complaining about transfer rate issues.

 

For pure read disk array with 1G network, read always could be ~100MB/s and no drop (may be just few and short). Of course, the file not in disk inner slow area.

I mainly complaint the write speed or Network was 10G.

 

Pls ensure the destination ( file copy ) also provide sufficient writing speed too.

Edited May 3, 2018 by Benson

[nid]

On 5/3/2018 at 1:34 PM, Benson said:

For pure read disk array with 1G network, read always could be ~100MB/s and no drop (may be just few and short). Of course, the file not in disk inner slow area.

I mainly complaint the write speed or Network was 10G.

Copying between my two Windows 10 workstations I have this "just a few and short" drops. It is so infrequent and totally acceptable. But copying from the Unraid array the drops are huge and long, from 110MB/sec to 20MB/sec, going up and down like the screenshots I posted before. Even copying the same file (after some time has passed when it seems it's not anymore cached) shows a different copy curve, which leads me to believe it is not file based or where the data resides on the disk platter. I guess even in the innermost area the WD RED is faster than 20MB/sec.

 

On 5/3/2018 at 1:34 PM, Benson said:

Pls ensure the destination ( file copy ) also provide sufficient writing speed too.

Destination drives are always SSD (windows 10 system) drives.

[John_M]

You could try copying from one disk to another from within unRAID. That would help you to determine whether your problem is disk-related or network-related.

[nid]

15 hours ago, John_M said:

You could try copying from one disk to another from within unRAID. That would help you to determine whether your problem is disk-related or network-related.

Good idea! I tried some copies within mc and the rate is mediocre. 10-30MB/sec, mostly staying close to 15, but fluctuating. What could this mean? Is it a valid test copying from a disk to itself? There is great probability that's what's happening since I only have 2 disks + parity.

Edited May 6, 2018 by nid