I already posted about this in the RC15 thread. The following happens:

 

My array consists of 15 disks, 6 2TB and 9 3TB ones.

 

When starting a parity check, the speed during the first 2 TB is between 40 and 75 MB/s. When it crosses the 2 TB boundary, the speed instantly grows to over 100 MB/s. Can it be one disk that is causing this? If so, how to find that disk? Can I test the speed of each individual disk?

This is normal => the speed of your parity checks is limited by your slowest disk.  When you cross the 2TB threshold, all of the 2TB disks are no longer involved, so the speed ramps up to that of the 3TB disks, which are likely newer 1TB/platter disks with much higher areal densities.

 

The initial speed -- while all disks are on the outer cylinders, should be higher than the speed as you approach the 2TB point (where the 2TB units are on the slowest inner cylinder part of the drives) ... but it's still likely slower than the 3TB units can achieve ever when you're 2/3rd of the way through them.

 

If your speeds in the past were higher during the first 2TB, then one of your disks has probably developed a problem -- or its interface connection is marginal.  You might try reseating all the disks and/or cables (depending on whether they're connected via direct SATA cables or a hot-swap chassis.  [Do that with the system OFF of course]

 

One other thought:  If the speeds used to be faster, then run SMART checks on all your 2TB disks and look in particular for a high reallocated sector count.    These cause thrashing, which, if you have a very high number, could slow down disk access for that disk => which, of course, would slow down the entire parity check.    It seems VERY unlikely, however, that you could have enough of these to really make a notable difference.

 

One other thought:  If the speeds used to be faster, then run SMART checks on all your 2TB disks and look in particular for a high reallocated sector count.    These cause thrashing, which, if you have a very high number, could slow down disk access for that disk => which, of course, would slow down the entire parity check.    It seems VERY unlikely, however, that you could have enough of these to really make a notable difference.

 

I pulled smart reports on those disks and I don't see anything extraordinary. Can you have look? One disk shows some reallocated events.

2Tdisks.zip

Three of the drives (sdo, sdt, and sdu) show interesting numbers I've not seen before -- the Power Off Retract counts match the load cycles !!  This is very strange ... I don't know if it's just an anomaly in the way those drives are reporting the SMART data or if it indicates an actual problem.    Power Off Retract counts the number of times the drive has lost power without the heads being parked ... i.e. they weren't properly shut down.  Matching numbers with the load cycles would indicate they were NEVER shut down correctly .. but quite frankly the numbers are too small for this to really be the case.

 

Did the parity check times on this system use to be a lot better?

 

If the answer to that is yes, then I'd try to isolate which drive is having problems.  I'd remove the 2TB drives, one-at-a-time;  connect them to a SATA port on a Windows machine (an eSATA dock would be simplest, but you could just set them outside a machine and run SATA and power cables);  and run HDTune on the drive -- just the Benchmark, which is completely safe as there's nothing written to the drive. 

Check all disks for a non-zero Current_Pending_Sector RAW_VALUE.

I have a simple script I run from a telnet session that extracts the information about pending sectors into an easy to peruse file:

#!/bin/sh
#	Check for re-allocated sectors

REP=reallocated_sectors_report
echo "UNCORRECTABLE SECTORS REPORT"  >$REP
echo "============================"  >>$REP

echo "Creating UNCORRECTABLE SECTORS REPORT in " $REP
for d in /dev/sd?
do
  echo "  checking $d"
  echo "" >> $REP
  echo $d ":" >> $REP
  echo $d ":"  >>REP
  smartctl -A $d 2>&1 | grep "Sector\|Uncorrectable" | cut -c 5-28,85-95 >>$REP
done
echo "DONE"

 

It could probably be improved, but it does give you a file that you can quickly scan to see if any disk has pending sector issues.