I have 11 Western Digital Green drives in my system. I've done parity checks before and they usually take about 8 hours (52MB/s). However my monthly parity check came along and i'm only getting 2MB/s. Which will take 11 days.

 

No errors in syslog. System specs are in signature.

 

Ideas?

 

syslog.txt

Woh, that ain't right? Is the read/write performance on your unRAID server running slow in general? Maybe some dodgy interface or cabling?

Woh, that ain't right? Is the read/write performance on your unRAID server running slow in general? Maybe some dodgy interface or cabling?

 

Well my reads are 90+MB/s on all drives according to unMENU. I tried restarting the server, and that didn't help either. The only difference is I added my 10th data drive. The drive was precleared and has no SMART errors. The reads/writes are fine on it.

 

I'm using PCI-X 133MHz cards, and they are in PCI-X 133MHz slots. I highly doubt it's a limit of these, because it doesn't get much faster.

Was this 11 day parity check caused by a power outage or the server being powered down ungracefully? Even so, it is way too long to do any kind of parity check/fix. Do yo have another spare hard drive to use as a replacement Parity drive for your unRAID server?

Hardware specific sounds like you have a decent rig, but before replacing the Parity drive, I would do a simple test. The quick way to test the Parity Drive is to write some files to your share(s), and see if you detect a big drop in write speeds.

If your on a full Gigabit network (And by your earlier description, it sounds like you are), it could be a dodgy SATA lead going to your Parity Drive, a dodgy SATA interface port on the mobo or the drive itself (whether it is the internal onboard disk controller or read/write heads). Some food for thought. Good Luck!

Was this 11 day parity check caused by a power outage or the server being powered down ungracefully? Even so, it is way too long to do any kind of parity check/fix. Do yo have another spare hard drive to use as a replacement Parity drive for your unRAID server?

Hardware specific sounds like you have a decent rig, but before replacing the Parity drive, I would do a simple test. The quick way to test the Parity Drive is to write some files to your share(s), and see if you detect a big drop in write speeds.

If your on a full Gigabit network (And by your earlier description, it sounds like you are), it could be a dodgy SATA lead going to your Parity Drive, a dodgy SATA interface port on the mobo or the drive itself (whether it is the internal onboard disk controller or read/write heads). Some food for thought. Good Luck!

 

Write speeds are fine, around 28MB/s with parity enabled with no cache drive enabled. Reads are 90MB/s or higher. My server has never been shut down ungracefully. It's been working fine for a month, the monthly parity check just happened on 3/1 and I noticed it was taking forever. Everything tests fine, it seems like an unRAID issue. Only thing I can think of is that unRAID is having an issue with this many green hard drives. My only change since my last parity check was adding my 10th data drive and that drive was fully tested before getting put in.

 

I've heard that some people experience very slow Parity Checks until about 5% and then it speeds up. I'm just going to let it go overnight and hopefully I wake up to a finished (or near finished) parity check. It's been going for about 3 hours now (2% done) and still at 2,000-3,000KB/s.

 

Tom, have any info on this?

True to that. When I start my monthly Parity Check, it originally reports it'll take a staggering 700+ or more minutes and after a refresh of the web interface, it goes to 250+ minutes, just within just 1 minute! From there on after, it drops by one minute increments every minute or so.

Also, the more drives you have, the longer the parity will take, and the more data you have also adds the the parity check length too. I'm not sure if this is a problem too but your config file for the disks could be causing a problem too.

Another thing, refer to the link below:

 

http://lime-technology.com/forum/index.php?topic=5449.0;topicseen

 

It might shed some light on the problem for you. Hope it helps mate.

 

Another thing, refer to the link below:

 

http://lime-technology.com/forum/index.php?topic=5449.0;topicseen

 

It might shed some light on the problem for you. Hope it helps mate.

 

 

Problem is, unlike that guy, i'm seeing 0 errors in my syslog so I have no clue on what drive (if any) is causing the slowdown. They all test fine individually so I have a hard time blaming cables/drives. I have 11 drives in there - 8 of which are filled (1-2% free space remaining). unRAID supports up to 20 drives, if it's "normal" for me to be getting speeds like this with only 10 data drives than there's a serious flaw in unRAID. 2.8%... still going insanely slow. Been around 5 hours. At this point, even if it speeds up at 5% or so.. i'm looking at 20 hour parity checks. Unacceptable. Thats pretty much a day every month that I can't use my server because I can't watch movies during a parity check without skipping.

 

I found a few people with similiar issues as this, with 0 syslog errors, and they usually got a reply from Tom himself. Which he told them that it was an issue with unRAID and would be fixed soon. Most of these threads date back a year or more though.

 

My friend recently went to unRAID and he's also having a huge number of problems - when his system was running on Windows XP for the past 5 years with 0 issues. I'm really starting to doubt unRAID's ability... it hasn't exactly been reliable for either of us. I find myself visiting these forums on a nearly daily basis trying to fix issues. After I fix out, another one comes out.

What drives are connected to what controllers?  Maybe your bottleneck is there. 

 

I would also check smart... maybe you have a drive on the way out or one of your sata cables is giving you grief.

i'm looking at 20 hour parity checks. Unacceptable. Thats pretty much a day every month that I can't use my server because I can't watch movies during a parity check without skipping.

First... the estimated time is as if you has all the disks the same size.  Once you get past the smaller disks in your array, the speed will increase dramatically.  Second, the initial estimates are often grossly wrong as they start their timing before the  disks spin-up.  I think there is a thread around here where some of us posted a screen show showing initial estimated of months duration.

 

Second, 20 hours is not unusual at all, even if all your disks were 2TB.  Let's say you have enough bus bandwidth to read 75Mb/s concurrently from all your disks. (nearly nobody has this bandwidth, but let's assume absolute best case)

 

The largest of your disks are 2TB, or 2,000,000 MB.  Doing the math, dividing by 75 (estimated MB-per-second) I get

26,666.66 seconds or

444.44  minutes or

7.4 hours.

 

Most people with a modern motherboard average between 50 and 60 MB/s on a parity check.  My older PCI based array averages 12-14 MB/s with 11 drives attached.

 

At 50 MB/s, your 2 TB array would take about 11 hours.

 

Since the parity check is limited by the read speed of your slowest disk, your speeds could easily be much longer in the initial estimate. I have one disk in my array which is a much older, slower IDE drive.

using

hdparm -t /dev/hdj

I can see the raw read speed of the disk.  It is:

root@Tower:# hdparm -t /dev/hdj

 

/dev/hdj:

Timing buffered disk reads:  64 MB in  3.01 seconds =  21.28 MB/sec

 

No way for me to ever have any parity check get above 21MB/s if that drive is part of the mix, even if not constrained by bus bandwidth.

 

Since my array is limited to 133MB/s max throughput on the PCI bus, and I have 9 disks in the array on the PCI bus, best I can do is 133/9 MB/s.  (Roughly the 14 MB/s I'm getting for my parity check speed)

 

Lastly, you should be able to watch a movie while performing a parity check.  I know it does not impact my viewing of movies at all.  It does slow down the parity check, as the disk involved has to constantly seek to the movie and the parity check blocks alternately, but there is no problem with watching a movie.

 

Not too long ago I did a test of the CPU usage on my PC.  The results were posted here in answer to a question of how much CPU was needed.

 

I was playing 5 full bit-rate DVD ISO images to 5 different media players on my LAN while performing a parity check.

 

Joe L.

I found that old thread where we were comparing parity check sob-stories based on the initial display from unRAID:

 

It is here: http://lime-technology.com/forum/index.php?topic=4493.msg40433#msg40433

 

Worst estimate posted so far is: 10852.85 hours...  (just over 452 days  )

 

These screen images were all grabbed while the disks were spinning up...

The final numbers are far better than these humorous examples.

A parity check in my system takes just around 8 hours.  This is an all SATA system with 6 onboard, 2 on a PCIex1 card, and 2 more on another PCIex1 card.

 

I know I can watch DVD's and TV Show rips from my system while it is doing a check also.  My room mate does it a lot as I have the parity check set to kick off at just after midnight, and he is usually still up and watching stuff from the server.

 

The parity build I did took a longer amount of time:

Feb 19 10:23:49 Tower kernel: md: sync done. time=52131sec rate=37473K/sec
Feb 19 10:23:49 Tower kernel: md: recovery thread sync completion status: 0

 

But the parity checks I have done since have been much faster:

Feb 19 19:02:34 Tower kernel: md: sync done. time=29110sec rate=67108K/sec
Feb 19 19:02:35 Tower kernel: md: recovery thread sync completion status: 0

 

The last check I did was on the first (my monthly parity check) and here are the results:

Mar 1 09:38:54 Tower kernel: md: sync done. time=29333sec rate=66597K/sec
Mar 1 09:38:55 Tower kernel: md: recovery thread sync completion status: 0

I just looked up your Supermicro AOC-SAT2-MV8 disk controllers.  They are PCI-X

 

Do you have them in PCI-X slots? or standard PCI slots on your MB?  If in PCI slots you are limited to 133MB/s total throughput.

I'm using PCI-X 133MHz cards, and they are in PCI-X 133MHz slots. I highly doubt it's a limit of these, because it doesn't get much faster.

 

^^^

Joe, *cough* *cough*

I'm using PCI-X 133MHz cards, and they are in PCI-X 133MHz slots. I highly doubt it's a limit of these, because it doesn't get much faster.

 

^^^

Joe, *cough* *cough*

<embarrassed> Note to self:  read all the posts by the person, not just the last one.  </embarrassed>

 

You are right.  I wish I had that kind of potential bus speed available.

 

Joe L.

Can you describe your full hardware for us, so we can get a better idea of what exactly we are dealing with.  Outlining what SATA drives are attached where (motherboard or PCI-X card).

Try one of the cards in the PCI-X 133 slot and the other in the PCI-X 100 slot. Load the card in the 133 slot first. That way, they each have their own buss to work on.

 

Peter

 

Try one of the cards in the PCI-X 133 slot and the other in the PCI-X 100 slot. Load the card in the 133 slot first. That way, they each have their own buss to work on.

 

Peter

 

 

This. The 133mhz slots share the same pci-e lanes. When I first set up my X7SBE with two AOC-SAT2s I had horrible performance because I had both in the 133mhz slots.

 

With one controller in the 100mhz slot and the other in a 133mhz I get:

 

 

It's been a bullet proof build, and is now my backup server that duplicates everything on the network.

85.9MB/s is very very nice.

 

With 2TB it should take about 6.5 hours for a full parity check.

 

Joe L.

The parity check took 14,820 seconds (247 minutes or 4.17 hours) to complete at 65,908 K/sec for the 20 WD10EADS drives.

 

(Thanks Fry's for the $60 shipped black friday sale.)

 

 

I just looked up your Supermicro AOC-SAT2-MV8 disk controllers.  They are PCI-X

 

Do you have them in PCI-X slots? or standard PCI slots on your MB?   If in PCI slots you are limited to 133MB/s total throughput.

 

They are in PCI-X 133Mhz slots. After 16 hours I am at 7.5%, at this point I am cancelling and I will try putting one of the cards in one of the PCI-X 100Mhz slots, as suggested.

 

Try one of the cards in the PCI-X 133 slot and the other in the PCI-X 100 slot. Load the card in the 133 slot first. That way, they each have their own buss to work on.

 

Peter

 

 

This. The 133mhz slots share the same pci-e lanes. When I first set up my X7SBE with two AOC-SAT2s I had horrible performance because I had both in the 133mhz slots.

 

With one controller in the 100mhz slot and the other in a 133mhz I get:

 

It's been a bullet proof build, and is now my backup server that duplicates everything on the network.

 

My bios actually allows me to set the PCI-X 100mhz slots to 133mhz as well. I'm wondering why it would allow that, seems risky?

 

Also.. what drives are those? And what is your setup? AKA What drive is connected to where? 82MB/s would be a dream.

 

Can you describe your full hardware for us, so we can get a better idea of what exactly we are dealing with.  Outlining what SATA drives are attached where (motherboard or PCI-X card).

 

Parity and Data disks 1-3 are onboard SATA. I'm 95% sure the other 7 Data disks are only on the first PCI-X 133mhz card. I plan to have 20 drives in here, so I have to fill my SATA cards and use 4 onboard slots. I know onboard isn't the issue because I had 6 drives hooked up to it before I got my SATA controllers, and parity checks were 52MB/s.

All 20 drives are WD10EADS. The parity and the first 3 drives are the newer 2 500gb platter variant. The rest have 3 333gb platters.

 

PCI-X 100mhz doesn't bottleneck these add-in cards. The Marvell controller on them is only capable of something like 650-700 MB/s total throughput, like when running a parity check. At least that's what I read a while back when they were popular for linux software raid arrays.

 

Edit: I wasn't thinking at all about how you only had 7 drives total on the 133mhz link. That shouldn't be bottle necking you. Hmm... The one time I had single digit MB/s parity check speeds was because of one of the drives.

 

The intel onboard ports won't bottleneck any 7200rpm drives in parity check speeds by the way. An array of fast drives will finish at over 100 MB/s average speed.

 

All 20 drives are WD10EADS. The parity and the first 3 drives are the newer 2 500gb platter variant. The rest have 3 333gb platters.

 

PCI-X 100mhz doesn't bottleneck these add-in cards. The Marvell controller on them is only capable of something like 650-700 MB/s total throughput, like when running a parity check. At least that's what I read a while back when they were popular for linux software raid arrays.

 

Edit: I wasn't thinking at all about how you only had 7 drives total on the 133mhz link. That shouldn't be bottle necking you. Hmm... The one time I had single digit MB/s parity check speeds was because of one of the drives.

 

The intel onboard ports won't bottleneck any 7200rpm drives in parity check speeds by the way. An array of fast drives will finish at over 100 MB/s average speed.

 

 

Actually, I just looked again. I'm using 4 cards on 1 PCI-X card, and 3 on the other.

 

I'm gonna have to rewire my entire case to be able to move it down to the PCI-X 100mhz slot. So i'll post back with results in about an hour.

All 20 drives are WD10EADS. The parity and the first 3 drives are the newer 2 500gb platter variant. The rest have 3 333gb platters.

 

PCI-X 100mhz doesn't bottleneck these add-in cards. The Marvell controller on them is only capable of something like 650-700 MB/s total throughput, like when running a parity check. At least that's what I read a while back when they were popular for linux software raid arrays.

 

Edit: I wasn't thinking at all about how you only had 7 drives total on the 133mhz link. That shouldn't be bottle necking you. Hmm... The one time I had single digit MB/s parity check speeds was because of one of the drives.

 

The intel onboard ports won't bottleneck any 7200rpm drives in parity check speeds by the way. An array of fast drives will finish at over 100 MB/s average speed.

 

 

Actually, I just looked again. I'm using 4 cards on 1 PCI-X card, and 3 on the other.

 

I'm gonna have to rewire my entire case to be able to move it down to the PCI-X 100mhz slot. So i'll post back with results in about an hour.

Do not be surprised it the array will not start when you power up again after moving the cards around.  The move might cause them to be scanned by linux in a different order and assigned different device IDs.  To fix you just have to go to the "devices" assignment page and assign the disks to back to their respective slots in the array.

 

Joe L.

Thanks everyone. I did a "round robin" type of wiring this time, as the Wiki claims that can offer speed increases. With one card in the PCI-X 100mhz slot, and the other in the PCI-X 133mhz slot.. my parity check is going 87MB/s. I also set that bottom PCI-X 100MHz to operate at 133MHz (according to the mobo manual it won't cause issues? If someone knows that this is bad please let me know!)

 

Such a huge difference! Thanks everyone.