The following behaviour is roughly consistent whether I'm copying:

- over a network through a switch (from Vista with explorer)

- directly connected network cables  (from Vista with explorer)

- locally mounted USB disc (using cp -r)

 

When copying (roughly)

<20G of data: copy speed ~15MB/s

>50G, <100G of data: copy speed ~10MB/s

>100G of data: copy speed ~5MB/s

 

For example,

if I select and copy many files totaling ~100G

they will be copy at ~5MB/s,

if I select a small subset of the same files totaling 

say 10G they will copy at about ~15 MB/s.

 

In all cases the copies complete ok, and I usually do

these overnight so it isn't an issue.

 

Has anyone else noticed this performance drop with large copies?

 

 

My unRAID box:

unRAID: 4.4.2

MB: Asus M4A78-VM

CPU: AMD SEMPRON LE-1200 CPU, 2.1GHz (45W)

RAM: 2GB (2x1GB) Corsair DDR2 PC-6400/800

PCI SATA: Adaptec 1430SA - PCI E x4 (installed in M4A78-VM 16x "graphics" slot)

HD: 1x 2TB Hitachi 7K2000 parity)

        1x 2TB WD Green WD20EADS

        1x 2TB Seagate ST32000542AS

        1x 1.5 TB WD Green WD15EADS

        1x 1.5 TB Seagate Barracuda ST31500341AS

        2x 1.5TB Samsung EcoGreen HD154UI

Is the parity on the 1430sa or on the motherboard?

I have not seen the results you mention, but I usually use rsync and bwlimit to keep it between 8-16Mb/s. this way the cache does not get overloaded and cause network traffic to stall while the buffer flushes.

Is the parity on the 1430sa or on the motherboard?

I have not seen the results you mention, but I usually use rsync and bwlimit to keep it between 8-16Mb/s. this way the cache does not get overloaded and cause network traffic to stall while the buffer flushes.

 

Parity is on the motherboard.

Especially, curious part is that it also occurs when doing copies from locally mounted usb disks...

Also, all the copies are to disk, as I don't use user shares.

 

I would try two things when doing disk to disk copies.

 

use rsync --bwlimit= and some number from 8192 to 16384 instead of using cp.

 

I would try moving the parity drive to the 1430sa.

 

The 1430sa is in a x4 PCIe slot may provide more unencumbered bandwidth directly to the drive.

 

I know on my motherboard the 6 SATA chipset ports have a max bandwidth of 384Mb/s.

In comparison an X1 has a bandwidth of 250MB/s and a X4 has 1GB/s as a maximum bandwidth.

 

 

I have never heard or seen anything like this myself.  My first thought is, check into heat buildup.  If certain system components are getting too hot, they may be 'pausing', or causing drive errors and resets, with associated pauses.  It is possible that the related errors may show up in your syslog.  Please see my sig for the Troubleshooting link, for instructions on capturing and posting your syslog.

 

5MB/s is clearly too slow, something is not right.

My first thought is, check into heat buildup.  If certain system components are getting too hot, they may be 'pausing', or causing drive errors and resets, with associated pauses.  It is possible that the related errors may show up in your syslog.  Please see my sig for the Troubleshooting link, for instructions on capturing and posting your syslog.

 

I don't think heat is a problem.  The two drives involved (parity+data) are typically both in their

mid/high 20's.  The 100G copy starts of at ~5MB/s i.e. no time for heat build up, the 10G copy can

commence within seconds of the 100G copy finishing and be ok at ~15MB/s, i.e. after the

100G copy would have caused heat build up.

 

It seams to be tied to the volume of data being copied.

 

I've looked at the sys logs and they show no difference between the different copies, no

mention of drive resets etc.

 

I'll post a syslog when I get a chance on the weekend.

I would try two things when doing disk to disk copies.

 

use rsync --bwlimit= and some number from 8192 to 16384 instead of using cp.

 

This actually helped with copies from locally mounted usb disks.  Thanks for the tip

Looks like cp is exhibiting similar behaviour to the vista copy, while rsync --bwlimit

is making more efficient use of resources.

 

I'll need to dig up rsync for cygwin or windows and give that a try.

 

 

I would try moving the parity drive to the 1430sa.

 

The 1430sa is in a x4 PCIe slot may provide more unencumbered bandwidth directly to the drive.

 

I know on my motherboard the 6 SATA chipset ports have a max bandwidth of 384Mb/s.

In comparison an X1 has a bandwidth of 250MB/s and a X4 has 1GB/s as a maximum bandwidth.

 

I tried copying (from vista using explorer) to the disk which was on the 1430sa, with parity still

on the motherboard and got the same trend.  (Don't have time/energy to swap cables at the

moment)  I assume this is testing the same effect as moving the parity to the 1430sa, i.e. one

disk on the motherboard and one on the 1430sa.

Your thinking sounds correct, as to heat.  Your comment though about the 100GB transfer starting at 5MB/s reminded me of one difference with the Reiser file system, compared with FAT and NTFS.  WeeboTech has pointed out elsewhere that ReiserFs takes much longer allocating the space a file will use, and I have noticed the same effect when deleting large files, deallocating the space.  With Windows, deleting a 4K file takes essentially the same amount of time as deleting a 40GB file, but ReiserFS takes a much longer time, proportional to the size of the file.  If the 100GB copy starts off with very large multi-gigabyte files, then the initial transfer speed will be dramatically affected.  It will be correct later in the transfer though.  Try timing these various sized transfers, and calculate the overall MB/s.

I tried copying (from vista using explorer) to the disk which was on the 1430sa, with parity still

on the motherboard and got the same trend.  (Don't have time/energy to swap cables at the

moment)  I assume this is testing the same effect as moving the parity to the 1430sa, i.e. one

disk on the motherboard and one on the 1430sa.

 

Yes its the same thing.

 

If you can borrow another 2GB of ram and test it may improve the 5MB/s situation.

It really depends on how often the cache is full and causes everything else to wait until the cache is flushed.

With the rsync bwlimit you are moving data over at a slower timed pace so you end up having a rolling flush, rather then a full flush and wait cycle. (where you cause network traffic to pause and thus loose some time).

I did some additional testing using a a timer, which highlighted Vista's estimate of throughput is not very accurate.

 

Timer showed:

<20G of data:              copy speed ~15MB/s

>50G, <100G of data:  copy speed ~13MB/s

>100G of data:            copy speed ~11MB/s

 

Theses results are significantly closer than those reported by Vista, however there is still close to 30% decrease in performance with the large transfers.

Nevertheless I cant complain with this performance 

 

The reasons suggested by RobJ and WeeboTech in the previous two mails provide the most logical explanations.

 

Thanks for all your help

Mario

 

PS I've posted the syslog in: http://lime-technology.com/forum/index.php?topic=4568.0

What about using teracopy or one of the tools that allows a controlled throughput of transfer.

If you could balance the flushing of the cache to disk along with network transfer then the longer transfers would not go slower and slower.

extra ram does help in this situation, but only so much. I had 8GB of ram and it still would not be faster then 16MB/s.

It might burst at 22-30Mb/s, but then stall down to 0 for a short while, then burst back up again.

 

Try one of the other schedulers too.

 

 

There is an additional effect directly related to transfer size, if very large transfers are done, and that is due to hard drives being slower as you fill them up.  Your fastest access is at the beginning of the drive, and slows significantly as seeks reach the end of the drive.  For example, you can easily see this effect if you monitor your parity check speeds.  My setup includes 8 drives - 4 500Gb, a 320GB, and 3 250GB drives.  I usually see about 65MB/s at the beginning, which dips to under 40MB/s as it approaches the end of the 250GB drives, then climbs some and slowly diminishes as it finishes the 320GB drive, then rises significantly again, and finally tails down to about 40MB/s at the end.  From 65 to 40 is a huge drop in performance, across a 500 GB drive.

 

You can test and eliminate this effect by first doing the 100GB test, but don't remove the 100GB transferred, so that the next test (20GB) begins where the last test left off.  It should begin at the ending speed of the 100GB test.

 

A 20GB test would not move the heads far enough to see a difference between starting and ending speeds.  And even a 100GB test might not be enough, if the drive is very large.  Plus, there are a number of other factors here, that will moderate the differences, such as the network bottleneck, drive rotation speed, etc.  I don't expect this to completely explain the differences you see.  I would like to see more users try comparable tests.

There is an additional effect directly related to transfer size, if very large transfers are done, and that is due to hard drives being slower as you fill them up.  Your fastest access is at the beginning of the drive, and slows significantly as seeks reach the end of the drive.  For example, you can easily see this effect if you monitor your parity check speeds. 

 

Yes, I also noticed this.  My parity speeds start in the high 90MB/s range, and then fall to the low 60MB/s range, resulting in an overall average in the high 70MB/s range.

 

You can test and eliminate this effect by first doing the 100GB test, but don't remove the 100GB transferred, so that the next test (20GB) begins where the last test left off.  It should begin at the ending speed of the 100GB test.

 

I tried this test, however I didn't get the expected results.  The transfer speed seems to be tied to the volume more than anything else.  E.g. if I do a large transfer and achieve 11MB/s, the following small transfer still achieves 15+MB/s.  In doing a little research on the ReiserFS I wouldn't be surprised if the Tail Packing scheme used to reduce internal fragmentation is somehow influencing performance, maybe forcing large file transfers to the end of the disk where there is more space resulting in slower transfers (noting this is pure speculation in my part ).

 

Plus, there are a number of other factors here, that will moderate the differences, such as the network bottleneck, drive rotation speed, etc.

 

Very true! And, it is very easy to reach the point of diminishing returns, where significant effort results in very little gain...

 

 

Some of the performance improvements I have implemented (thanks to information found in the forum) are:

- using rsync --bwlimit does help smooth out transfer rates resulting in an overall higher rate, suggesting flushing of the cache plays a role

- choosing a fast parity disk (originally I used the WD Green as parity, moving to the Hitachi increased performance by 20+%)

- choosing on which disk to place data (movies/archive are ok on slow disk, small often accessed files are best on faster disks). Not to mention keeping unused disks spun down.

 

My next experiment will be with testing other schedulers.

 

 

PS

The test results presented below were writing to the 2TB WD Green, when writing to the 1.5TB Seagate Barracuda write speeds are about 30% faster.  Noting the 15MB/s on the WD may be on the slow areas of the disk, while the 20MB/s on the Seagate may be on the faster areas