I'm using mc to move data from one disk to another, reconstruct (turbo) write is turned on, and I've also tried it in auto, the same behavior occurs. In all my years of experience doing this, the transfer speed should be consistent. Docker and all other possible sources that would interfere have been disable.

 

What I'm observing the following:

1. The file copy starts fast at about 130MB/sec.

2. At about 1/3 complete, it slows to about 40MB/sec.

3. Then at 2/3 complete it picks back up speed again to 130MB/sec

 

This same cycle repeats on every file in the copy. The logs (attached) aren't showing any errors I see that would explain it. Any ideas how to probe for more information?

tower-diagnostics-20170808-0613.zip

Linux disk I/O is buffered in memory and will write at memory speed until the buffer fills, then will slow to disk speed while the buffer is flushed to disk.

12 minutes ago, trurl said:

Linux disk I/O is buffered in memory and will write at memory speed until the buffer fills, then will slow to disk speed while the buffer is flushed to disk.

 

That was my suspicion but don't understand enough to know how to confirm it. This is my first time doing disk to disk transfer with this much RAM, 128GB. Previously I was used to the behavior when I only had 32GB, would that explain it? Perhaps with lower member the buffer was being flushed more often so therefore the transfer rate was more consistent for the whole duration?

 

The disk activity lights on the server are as follows

1. Start to 1/3 of the transfer at 130MB/sec show the the source disk activity led on, no other hdd leds are showing activity (as I assume the file is fully being read into memory?)

2. At 1/3 to 2/3 all other disk LED's light up, source hdd led is off, target disk led is solid

3. Target led is solid, both parity's are solid, source led is off

 

Is that the expected behavior?

Hi -

 

Are the files small or large, consistent in size or variable?

@tdallen Very large files, 8GB to 24GB in size.

 

Another thought I had is that it's related to the file system. All my drives are XFS, and a few were extremely full. Free space is <20GB and in some cases, <2GB remaining on 8TB drives.

 

I remember this was an issue with ReiserFS. Some reading I did last night, suggested that XFS's optimal free space should be >5% of total disk capacity. So I'm moving the data (experiencing this fast,slow,fast cycle during file copy transfer) in hopes once I get to >5% free on each disk, I can then check the filesystems , repair if needed and then defrag the drives to see if there's any improvement.

Edited August 8, 20178 yr by Lev

XFS should not experience the same sort of performance degradation when disks are nearly full as was experienced with XFS.

 

Your description of the way the disk activity lights are behaving does make it seem like this could be caused by very large amounts of cached data needing to be flushed to disk.    Not sure of the difference between stages 2 and 3 - they sound very similar from your description.

 

There are settings you can make at the Linux level to limit the amount of RAM that will be used for caching purposes.   Maybe experimenting with these might be a good idea?   I cannot remember offhand what they were but searching the forum should find them.

It isn't necessarily a full solution, but since you have a very large amount of memory you might try installing the "Tips and Tweaks" plugin and on the 'Tweaks' tab look at 

 

Disk Cache 'vm.dirty_background_ratio' (%)

Disk Cache 'vm.dirty_ratio' (%):

 

And here's a post from RobJ talking about the settings:

In your case I'm not sure whether you'd be better off setting aside more memory for caching, or less.  Using more memory for caching would mean longer times between pauses, but potentially longer pauses as the cache is flushed to disk.  Using less memory should result in more consistent overall performance, albeit slightly slower. 

Thanks for all the great pointers in the right direction! With your help I think I'm able to explain the cycle I described in my first post was caused by the setting:

 

vm.dirty_expire_centisecs = 3000

 

This is triggering every 30 seconds, and the files on average were 9 -12GB that I'm transferring. Around the right size such that it would take trigger one or twice during a copy.

 

In the attached screenshot it shows copy speeds go from ~160MB/s (the section on storage chart that shows only read activity, cause it's reading the source disk straight into the cache) then when the cache flushes to target disk (write activity occurs on the chart, and lots of read activity due to using reconstruct write), that same file copy speed goes down to ~30MB/s.

 

I suspect that this is because the when the cache flushes to the target disk it not available and forces the copy that's still in process to write directly to the target disk at the same time. The cache flush and the file copy in progress are then competing for writes at the same time and that's why I'm seeing ~30MB/s during that time. Is that theory anywhere close to being correct?

 

I've been tinkering with the other settings mentioned to find the optimal settings but so far only getting ~90MB/s sustained... I recall in my prior server that 8TB WD Red's would reconstruct write at ~130MB/s from disk to disk.

A point to remember is that every ‘write’ operation to a data drive under unRAID actually involves 4 I/O operations (reads from parity and target to establish the ‘before’ state followed by writes to parity and target).    Therefore there is always more than 1 write running in parallel.   That is why using a cache disk speeds up the perceived speed - it is a single write to cache (without parity being involved) and no need for the initial read.

Edited August 9, 20178 yr by itimpi

7 hours ago, itimpi said:

A point to remember is that every ‘write’ operation to a data drive under unRAID actually involves 4 I/O operations (reads from parity and target to establish the ‘before’ state followed by writes to parity and target).    Therefore there is always more than 1 write running in parallel.   That is why using a cache disk speeds up the perceived speed - it is a single write to cache (without parity being involved) and no need for the initial read.

 

Thanks. Yes very clear on that. This scenario does not involve cache. It is a disk to disk copy using mc on a system with dual parity running with 128GB.

I think you're on the right track, and that it's a combination of things.  Since you have a very large amount of memory and it is allocated for caching on a percentage basis, you have a pretty large amount set aside for caching.  After 30 seconds you've got a big chunk of cached data that needs to be written out. I think the theme behind RobJ's recommendations was to start reducing - less cache, write more frequently, etc. in order to achieve consistent performance.

 

I typically get line speed, 113MB/s when writing large files over the network with turbo write.  Not always, though - it can be less.  Note that my server only has 8GB of RAM, less then you probably have reserved for your cache... If you want to send me an extra 120GB of RAM I'll do some testing for you .