V6.1.6 - Parity starts somewhat fast and then SLOWS down to a crawl

[ajgoyt]

Guys I finally got all of my data backed up after an issue with an MB upgrade etc, I ran 2 preclear runs with the 5tb drive before swapping out with the 4tb i had in the system. 

 

3 different times i have started a parity sync and it starts out around 35mb/s sec and then slows after a few minutes to 30 -60 kb/s.

 

1st time it went to 1.5% then next time it went to 5% , now this time at 7.5% , I noticed disk 9 has a few sector errors and i could remove it from the system if need be, Do you think disk 9 is the reason for the slow parity check?

 

for the times it stalls like this the webui is not locked up but definetly is really slow and will let me cancel the parity sync, but it sits for ever after i try to shutdown, and a couple of times ive had to hold the power button!

 

I have attached a health report let me know what you think i should do?

 

thanks

[Squid]

Guys I finally got all of my data backed up after an issue with an MB upgrade etc, I ran 2 preclear runs with the 5tb drive before swapping out with the 4tb i had in the system. 

 

3 different times i have started a parity sync and it starts out around 35mb/s sec and then slows after a few minutes to 30 -60 kb/s.

 

1st time it went to 1.5% then next time it went to 5% , now this time at 7.5% , I noticed disk 9 has a few sector errors and i could remove it from the system if need be, Do you think disk 9 is the reason for the slow parity check?

 

for the times it stalls like this the webui is not locked up but definetly is really slow and will let me cancel the parity sync, but it sits for ever after i try to shutdown, and a couple of times ive had to hold the power button!

 

I have attached a health report let me know what you think i should do?

 

thanks

The system diagnostics will tell the story as to what's actually going on

[theecatman]

Guys I finally got all of my data backed up after an issue with an MB upgrade etc, I ran 2 preclear runs with the 5tb drive before swapping out with the 4tb i had in the system. 

 

3 different times i have started a parity sync and it starts out around 35mb/s sec and then slows after a few minutes to 30 -60 kb/s.

 

1st time it went to 1.5% then next time it went to 5% , now this time at 7.5% , I noticed disk 9 has a few sector errors and i could remove it from the system if need be, Do you think disk 9 is the reason for the slow parity check?

 

for the times it stalls like this the webui is not locked up but definetly is really slow and will let me cancel the parity sync, but it sits for ever after i try to shutdown, and a couple of times ive had to hold the power button!

 

I have attached a health report let me know what you think i should do?

 

thanks

The system diagnostics will tell the story as to what's actually going on

Not at home right now,  I should be able to get a sys log while running parity right?

[garycase]

A log may show more details, but what you posted likely gives enough info to isolate the issue ==>  You have a failed drive (Disk #9) that is causing the sync to slow to a crawl ... and you don't have good parity, so you're not going to be able to re-build that data.

 

I'd stop the parity sync;  see if you can at least read Disk #9 well enough to get a directory of what's on it [save this if you can get it];  then do a New Config, assigning all of the other drives and your parity disk -- but NOT #9; and then you should be able to do a successful parity sync, so at least the rest of the array will be protected.

 

Then you'll need to reload all of the data that was on Disk #9 from your backups ... or if you don't have current backups you can attempt to recover what you can from that disk -- preferably using another computer, although you may be able to mount it outside the array and do this on the UnRAID server.

 

[garycase]

... Guys I finally got all of my data backed up ...

 

Just noticed that you do indeed have good backups ... so the only real question is whether or not you can read a directory of #9 so you know just what you need to restore.

 

 

[ajgoyt]

Gary - cooking dinner right now , but stopped by the win 10 pc and grabbed the log , check it out yes i beleive your right on with disk 9 being the issue, I will do as you say...

 

let me know if you see anything else! 

 

thanks again...

 

syslog.zip

[ajgoyt]

So far so good ............

 

Total size: 5 TB

Elapsed time: 2 hours, 43 minutes

Current position: 489 GB (9.8 %)

Estimated speed: 58.0 MB/sec

Estimated finish: 21 hours, 36 minutes

[garycase]

Definitely looking much better.    Still seems a little bit slow ... but that may be due to the controllers you're using or the areal density of some of your older drives.    I didn't look at the density of all your drives, but with the mix of 1.5TB, 2TB, 3TB, 4TB, and 5TB drives you'll have several "inner cylinder slowdowns" as the check progresses.    If any of these older drives are 500GB/platter then the speed you're seeing is likely due to that -- and things should speed up as it passes the size thresholds [especially after it passes the 2TB point].

 

But you may also have an older controller that's causing bandwidth throttling due to the connected drives being capable of higher bandwidth than the controller can provide => this won't have any impact during normal use of the array; but when all drives are active at once (such as during a parity check or rebuild) it will slow things down.

 

In any event, the speed doesn't look bad at all ... I suspect your issue was, as I surmised before, a bad drive.    If you happen to notice it, the speed should jump up as it crosses each of the thresholds ... i.e. when it gets past 1.5TB, then 2TB, then 3TB, etc.    It will slow down quite a bit as it approaches each of those, since the drive(s) of that size will be on their slowest innermost cylinders ... but then jump back up once those drives are no longer involved in the process.

 

[ajgoyt]

Gary,

That makes sense about some bottle necking in my system still..

 

I tried the Dell perc expansion card that you recommended earlier on my other meltdown thread but there was definitely something wrong with it, per it made both my unraid and my win 10 machines be on a constant reboot loop, I sent it back to the guy on Ebay for a refund. 

 

I ended up going with a Supermicro AOC-SAS2LP-MV8 that was local and according to the owner was used for awhile, I added the card into the slot and it appears to be working fine no firmware flash etc...  The other drives our directly hooked to the MB six sata slots, Then I did have to use a PCI 4 port per I was still 4 drives short. I do also have this installed to make up for the the top 5.25 slots in my case -http://www.amazon.com/Rosewill-5-25-Inch-3-5-Inch-Hot-swap-SATAIII/dp/B00DGZ42SM for 4 hot swap drives...

 

I would like to get another Supermicro AOC-SAS2LP-MV8 and install it per i have the room, I believe this will help speeds/bottle necking etc as i eliminate the older drives smaller drives installed through the PCI slot, Adding another Shouldn't be an issue having (2) - Supermicro AOC-SAS2LP-MV8  correct?

 

 

AJ

[garycase]

There have been some issues with the SAS2LP cards, but I think those are resolved in the latest v6 release (6.1.6).

 

As long as your system is working fine with one, there won't be any problem adding another -> and it will be a BIG improvement over a 4-port PCI card.

 

r.e. the Dell H310 card => I don't have any of those, but many folks on the forum have been using them and recommend them.    HOWEVER ... in reading more, it seems there is a minor hardware modification you need to make to that card =>  you have to put a small bit of electrical tape on the card to cover two pins !!  (unless you're using it in a Dell system)    I was not aware of this until a couple days ago, when someone else also had a problem and I did a bit more research on it.

 

The mod is VERY simple -- it's outlined here:  http://yannickdekoeijer.blogspot.com/2012/04/modding-dell-perc-6-sas-raidcontroller.html

 

Since you've already returned the card, it's irrelevant, but just thought I'd note it in case anyone else is reading this thread and wondering about that same card.

 

[JorgeB]

I've used the h310 on Supermicro,  Asus and Asrock boards without any modification.

[garycase]

I've noticed that many folks are using it -- and I'd certainly have thought the modification would be more widely known if it was necessary on all boards => your experience clearly shows it is not.

 

I wonder if there are different rev's of the H310 ... some that do; some that don't require the mod ... or if it's just a function of the specific chipset on the various boards.

 

In any event, if you get an H310 and it does NOT work ... I'd certainly try a bit of electrical tape before sending it back 

[ajgoyt]

WOW Wish i would have known about that before i sent it back... oh well the supermicro seems to be working fine.  I ended restarting the parity again per Disk 2 had errors   , So i am back at it again trying to run a clean parity sync with another Disk..

 

Total size: 5 TB

Elapsed time: 6 hours, 51 minutes

Current position: 1.31 TB (26.2 %)

Estimated speed: 60.1 MB/sec

Estimated finish: 17 hours, 4 minutes

 

 

[garycase]

... WOW Wish i would have known about that before i sent it back...

 

I wish I'd known about it before I suggested the card 

 

There are a LOT of folks using H310's on this board, and I had not seen anyone comment on needing to make that modification until very recently (just a few days ago).    Not sure if there's a different version of the card, or if it only impacts certain chipsets, or what => but clearly there are some systems that require that those two pins do not make contact.  Clearly a very simple modification -- but without it the card simply won't work if you have a card/motherboard combo that exhibits the problem.

 

... I've never had that specific issue, since I don't have an H310, but I HAVE had a couple of cards (TV tuner cards) that would cause reboot loops if I tried them in a newer system.    It's been a couple years, and I no longer have the cards, but I have to wonder if this was the same issue ... and if simply taping those two pins might have been all that was needed for them as well !!    In any event, I'd still buy an H310 if I needed extra ports -- but I'll certainly keep the link to that modification handy [actually, I "printed" it to a PDF file already] just in case I need it.

 

 

[ajgoyt]

... WOW Wish i would have known about that before i sent it back...

 

I wish I'd known about it before I suggested the card 

 

There are a LOT of folks using H310's on this board, and I had not seen anyone comment on needing to make that modification until very recently (just a few days ago).    Not sure if there's a different version of the card, or if it only impacts certain chipsets, or what => but clearly there are some systems that require that those two pins do not make contact.  Clearly a very simple modification -- but without it the card simply won't work if you have a card/motherboard combo that exhibits the problem.

 

... I've never had that specific issue, since I don't have an H310, but I HAVE had a couple of cards (TV tuner cards) that would cause reboot loops if I tried them in a newer system.    It's been a couple years, and I no longer have the cards, but I have to wonder if this was the same issue ... and if simply taping those two pins might have been all that was needed for them as well !!    In any event, I'd still buy an H310 if I needed extra ports -- but I'll certainly keep the link to that modification handy [actually, I "printed" it to a PDF file already] just in case I need it.

 

No problem about the Dell I am sure it was a card issue -  well i bought another MV8 too replace the PCI 4 port card on ebay with SAS cables shipped for $39, Once received i might have to follow this article by Rajahal - http://lime-technology.com/forum/index.php?topic=12404.0

 

Like you said early the speeds have increased 

 

Total size: 5 TB

Elapsed time: 11 hours, 54 minutes

Current position: 2.54 TB (50.8 %)

Estimated speed: 82.1 MB/sec

Estimated finish: 8 hours, 19 minutes