This Problem might persist since a couple of versions. The mover is painfully slow - most likely due to how rsync copies the files. First let me say that everything else works up to spec (Array = 4 x 8TB + 1 x 8TB parity + 1TB SSD cache):

 

Over network (1Gbit) from SSD on remote Computer ->  unRaid share (cache set to yes) ~ 110 mb/s

unraid share (both from array or cache drive) -> over network (Gbit) to remote computer (SSD) ~ 110mb/s

Over network from SSD -> unraid Share (cache set to no and reconstructive write) ~ 110mb/s

Over network from SSD -> unraid Share (cache set to no and read/modiify/write) ~ 60mb/s:

 

So far so normal, but the mover (using rsync) manages only ~30-35mb/s which is kind of insane, as i could move the files faster over the network. This left the unRaid server unusable this morning for 4 hours (moving a couple of 100GB) when infact it schould have been done in an hour (I have reconstructive write enabled - without it it is even slower ~ 25mb/s!)

 

I narrowed down the problem to this (using  a 21.5 GB File):

 

root@Tower:/mnt/cache# time sh -c "rsync 00274.m2ts /mnt/user0/share/; sync"

real    13m42.944s
user    3m8.050s
sys     1m50.574s
root@Tower:/mnt/cache# time sh -c "rsync 00274.m2ts /mnt/disk4/share/; sync"

real    4m28.327s
user    2m53.923s
sys     1m38.891s

 

it seems like unRaid is painfully low when rsyncing to user0 (which is basically what the mover does). Is there any tweak (read or write buffer or anything) to improve on this? How can it be that copying over the network is faster than an internal copy (Even my internet connection is faster and when rsyncing directly to one of the array disks it still only manages ~78mb/s, below the speed over the network)? Same thing happens with the "cp" command  and midnight commander. I hope there is a solution.

Edited April 4, 20179 yr by Videodr0me

Seems you have eliminated a problem with disk4, but not really eliminated a problem with any other specific array disk. Any idea which specific disk(s) were written with your other tests? What disk actually got written when you were writing to /mnt/user0?

 

 

2 hours ago, trurl said:

Seems you have eliminated a problem with disk4, but not really eliminated a problem with any other specific array disk. Any idea which specific disk(s) were written with your other tests? What disk actually got written when you were writing to /mnt/user0?

 

It was always disk4 - I checked by looking where the file landed and by monitoring the disk speeds in the gui. I also tried the same with all other disks with the same results. Also note that over the network (see initial post) i get full 110mb/s when copying to the exact same share on the array (cache set to no for share - and verified that file was directly copied to disk 4). I am pretty sure it is no disk problem or otherwise hardware related. It is most likely due to the fuse filesystem imposing some kind of overhead (speculation: to small write buffers, or some other inefficiency) on all copy or write operations going to user or user0. In addition rsync is not that efficient copying large files, but i could live with 78mb/s (when writing to the disks directly) - but as the mover script rsyncs to user0 its much slower than this. Feel free to try this on your system  (with reconstructed write and without), chances are you will come up with very similar results.

 

Edited April 4, 20179 yr by Videodr0me

Did you try with direct_IO enable? it decreases considerably FUSE overhead for 10GbE, never tested if it helps with rsync.

Rsync is not efficient / slow, but rock reliable, So I won't use cache disk ( mover ) for any share due to slow speed no gain overall.

Edited April 4, 20179 yr by Benson

7 hours ago, Benson said:

Rsync is not efficient / slow, but rock reliable, So I won't use cache disk ( mover ) for any share due to slow speed no gain overall.

I would like to continue use the cache disk, because it keeps all disks spinned down during scattered downloads over the day - but i agree that performance wise the slow mover performance kind of defeats the idea of a cache disk. I hope there is some sort of tweak to fix this.....

11 hours ago, johnnie.black said:

Did you try with direct_IO enable? it decreases considerably FUSE overhead for 10GbE, never tested if it helps with rsync.

Will try tommorrow - got a parity check running just now. Are there any other tweaks that might help?

8 minutes ago, Videodr0me said:

I would like to continue use the cache disk, because it keeps all disks spinned down during scattered downloads over the day - but i agree that performance wise the slow mover performance kind of defeats the idea of a cache disk. I hope there is some sort of tweak to fix this.....

 

Understand and agree, there are no reason to spinup disk during download over the day.

On 4.4.2017 at 8:04 PM, johnnie.black said:

Did you try with direct_IO enable? it decreases considerably FUSE overhead for 10GbE, never tested if it helps with rsync.

Tried it now and it does help:

 

root@Tower:/mnt/cache# time sh -c "rsync 00274.m2ts /mnt/disk4/share/; sync"

real    3m9.109s
user    2m49.969s
sys     1m23.213s
root@Tower:/mnt/cache# time sh -c "rsync 00274.m2ts /mnt/user0/share/; sync"

real    3m48.587s
user    2m56.219s
sys     1m3.262s
 

Compared with times  posted in initial post (unraid in default direct i/o auto) this is a 3-4x performance increase. There is one minor BUT though - reads, while still fast over the network (~110mb/s) seem to fluctuate much more (sometimes dipping to 60mb/s for a couple of seconds) - maybe raising the read-ahead buffer for direct-io would help there. I remember that way back there was a setting one could set in the go file:

 

echo "sleep 30; blockdev --setra 2048 /dev/md*" >>/boot/config/go

 

would this still be valid vor 6.3.3 and has anybody experience with this? Also maybe somebody could weigh in why direct i/o enabled is not the default for unraid - if it helps for internal mover transfers and 10Gbe network speeds, why not have it enabled by default?

Edited April 6, 20179 yr by Videodr0me