unRAID Windows 10 Slow Read Speeds - High CPU Load? - Resolved - ESATA Array Problem

[coolspot]

Hi all,

 

I see that others have raised issues about slow read speeds with unRAID ... and I seem to be in a similar situation.

 

This is what's happening:

 

• First file, even if it is multi-gigabyte, transfers really quickly, at full wire speed (110MB/s)
• Second, third file, may or may not transfer at full speed, usually drops to 55MB/s
• By the fourth file, it's around 20MB/s - 25MB/s

 

I tried the extra SMB option as well as enabling Direct I/O... but this has not fixed the issue.

 

No other apps are running on the server itself.

 

What can cause such a drastic slowdown after the 1st file?

 

Running TOP on the server, I noticed that my load averages are *very* high...

 

 

top - 10:17:55 up 4 min,  1 user,  load average: 11.44, 4.02, 1.47
Tasks: 339 total,   1 running, 338 sleeping,   0 stopped,   0 zombie
%Cpu(s): 12.2 us, 15.1 sy,  0.8 ni, 64.2 id,  5.5 wa,  0.0 hi,  2.3 si,  0.0 st
KiB Mem :  7877704 total,  2851652 free,   474936 used,  4551116 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  6159348 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
19081 root      20   0   16736   3028   2280 R  11.8  0.0   0:00.02 top
 9426 root      22   2  173552  22208  17096 S   5.9  0.3   0:03.05 php
    1 root      20   0    4360    644    584 S   0.0  0.0   0:09.17 init
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.49 ksoftirqd/0
    4 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    6 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/u4:0
    7 root      20   0       0      0      0 S   0.0  0.0   0:00.58 rcu_preempt
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_sched
    9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
   10 root      rt   0       0      0      0 S   0.0  0.0   0:00.05 migration/0
   11 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 lru-add-drain
   12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/0
   13 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/1
   14 root      rt   0       0      0      0 S   0.0  0.0   0:00.05 migration/1
   15 root      20   0       0      0      0 S   0.0  0.0   0:00.61 ksoftirqd/1
   16 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0
   17 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
   18 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs
   19 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns
   20 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/u4:1
   87 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/u4:2
  280 root      20   0       0      0      0 S   0.0  0.0   0:00.00 oom_reaper
  281 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback
  283 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kcompactd0
  284 root      25   5       0      0      0 S   0.0  0.0   0:00.00 ksmd
  285 root      39  19       0      0      0 S   0.0  0.0   0:00.04 khugepaged
  286 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 crypto
  287 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kintegrityd
  288 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset
  290 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kblockd
  291 root      20   0       0      0      0 S   0.0  0.0   0:00.08 kworker/1:1
  292 root      20   0       0      0      0 S   0.0  0.0   0:00.04 kworker/0:1
  459 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 ata_sff
  477 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 devfreq_wq

 

I'm running unRAID on a Celeron G1610 - so it's quite a slow machine, but surely reading/writing files over SMB shouldn't take this much CPU?

[JorgeB]

1 hour ago, coolspot said:

• First file, even if it is multi-gigabyte, transfers really quickly, at full wire speed (110MB/s)
• Second, third file, may or may not transfer at full speed, usually drops to 55MB/s
• By the fourth file, it's around 20MB/s - 25MB/s

That's a very strange symptom, first time I see it.

 

1 hour ago, coolspot said:

I tried the extra SMB option as well as enabling Direct I/O... but this has not fixed the issue.

Would surprise me if they worked in this situation.

 

Load averages are extremely high, I have servers with similar CPUs and for basic NAS duties it's more than enough, try booting in safe mode and see if there's any difference.

[coolspot]

1 hour ago, johnnie.black said:

That's a very strange symptom, first time I see it.

 

I can easily recreate the issue too - all I need to do is stop the file transfer in Windows, restart it and the file will transfer at full speed.

 

I guess I never noticed the slowness of the server since I usually only copy a handful of files. But I'm trying to offload TB of data and this is a PITA right now.

 

Another thing I noticed is the Page In value for SMBD is very high. I'm not sure if that is normal or not? I'm assuming it is since SMBD needs to read disk data and send it over the wire, hence it's paging data in? I have minor page outs, but doesn't look to be anything serious:

 

      7877704 K total memory
       180512 K used memory
       547992 K active memory
      6820048 K inactive memory
       222796 K free memory
        10128 K buffer memory
      7464268 K swap cache
            0 K total swap
            0 K used swap
            0 K free swap
         8678 non-nice user cpu ticks
         5644 nice user cpu ticks
        35351 system cpu ticks
       470865 idle cpu ticks
       117521 IO-wait cpu ticks
            0 IRQ cpu ticks
         4311 softirq cpu ticks
            0 stolen cpu ticks
     57845899 pages paged in
       132002 pages paged out
            0 pages swapped in
            0 pages swapped out
      7771146 interrupts
     12677600 CPU context switches
   1514913184 boot time
       188674 forks

 

IO-wait time is very high and keeps incrementing. I don't think it's my SAS card (LSI since parity and rebuilds run fast), what about the network card? I'm using an AR8151 card. 

 

I'll try safe mode and get back to you.

[coolspot]

This is slightly embrassing ... it's no unRAID but my eSATA array. For some reason my eSATA RAID 5 array drops in speed significantly after the first file.

 

Writing to my local hard drive I can get pretty good speeds (70MB/s - 110MB/s).

 

So... I guess the problem is resolved