Sync errors in parity check after power outage (edit) - 2nd power outage during parity check more problems

[FrozenGamer]

Thanks!

[trurl]

40 minutes ago, FrozenGamer said:

Thanks Trurls - I have installed a ups - TrippLite G1000UB.  I ran 1 pass of memtest and it passed, unassigned both disks and reassigned them with spares.  It is a long process of 5 days to do the both drives.  

 

The rebuilds will happen in parallel. Do your parity checks normally take multiple days? I've never tried to run so many disks so probably you have some controller bottlenecks I don't have. For comparison my 6TB parity check completes in just over 15 hours.

[FrozenGamer]

Yes, i think it might be a bottle neck in the SE3016 caddy's they are older.  Going from 6 to 8 tb parity really slowed it down.  Does having 2 parity drives slow it down versus 1?  

6 tb drives were more expensive than 8tb and i got tired of paying more for less.  

[trurl]

53 minutes ago, FrozenGamer said:

Going from 6 to 8 tb parity really slowed it down.  Does having 2 parity drives slow it down versus 1?  

 

Going from 6 to 8 should only take longer proportionally. So for example if 6TB took 18 hours (3 hrs per TB) then 8TB should have taken 24, more or less. Actually larger drives maybe perform a little better proportionally due to greater density.

 

Dual parity works in parallel so no slow down from that except for possibly the parity2 calculation and only if you have a very old processor.

[FrozenGamer]

On 4/11/2018 at 6:54 AM, FrozenGamer said:

On 4/11/2018 at 4:49 AM, johnnie.black said:

You'll also need to check filesystem on disk2:

 

Apr 10 06:59:55 Tower kernel: XFS (md2): xfs_dabuf_map: bno 0 dir: inode 8837623527
Apr 10 06:59:55 Tower kernel: XFS (md2): [00] br_startoff 0 br_startblock -2 br_blockcount 1 br_state 0
Apr 10 06:59:55 Tower kernel: XFS (md2): Internal error xfs_da_do_buf(1) at line 2525 of file fs/xfs/libxfs/xfs_da_btree.c.  Caller xfs_da_reada_buf+0x31/0x78 [xfs]

The xfs check has completed, and it looks like it fixed the issue in the first line above, i don't know what line 2 and 3 mean.  I am attaching the log from xfs repair.xfs_repair disk2 log 04-16-17.txt    Also i should mention that the data has been restored and parity is good etc.  5 days 17 hours to do it though.  Should i Mark this thread solved and go to this one to try and figure out what is it to be so slow.  I can't imagine that it was just that little xfs_repair issue on disk 2?

 

 

 

Edited April 17, 2018 by FrozenGamer

[JorgeB]

Filesystem on disk2 should be completely repaired now, as for the performance start a non correcting parity check, is the speed the same or faster? (you can cancel the check at any time time).

[Vr2Io]

On ‎2018‎年‎4‎月‎11‎日 at 11:23 PM, FrozenGamer said:

Yes, i think it might be a bottle neck in the SE3016 caddy's they are older.  Going from 6 to 8 tb parity really slowed it down.  Does having 2 parity drives slow it down versus 1?  

6 tb drives were more expensive than 8tb and i got tired of paying more for less.  

 

In general, 2 Parity won't cause slow down. But a lowend CPU will. I have a ATOM J1900, if run in 2 parity it slow down almost in half.

 

Yesterday, I start switch my parity from 6TB to 8TB, the reason was same - paying more for less.

 

About a year ago, buying a 6TB Toshiba (7200rpm) or WD MyBook 8TB (5400pm) was same 170usd at Amazon. But that time I choose 6TB becuase it faster a lot and I already have several drive work quite solid.

 

The reason for now I choose WD 8TB because it is 2nd gen and performance raise a lot, almost same level as Toshiba. In previous, my array parity check was 11.5hrs average 143.4MB/s and now 16.5hrs average 134.5MB/s. This is a drawback for change to a large disk / low performance disk.

 

I still waiting for Toshiba MG06/7 series come out, it's performance quite good. I may be switch back to 6TB depends on the price/cost.

 

PS : I first change from HGST to Toshiba, Toshiba already slow down the array ...... but it more cheaper and also reliable. Array slow down again and again.

 

For your case, I hope you shot out the slow cause and fix ASAP.

 

 

Edited April 17, 2018 by Benson

[FrozenGamer]

8 hours ago, johnnie.black said:

Filesystem on disk2 should be completely repaired now, as for the performance start a non correcting parity check, is the speed the same or faster? (you can cancel the check at any time time).

Parity check without correction speeds are about 90MB/s to start, which seems to be fixed.  When i am using mover i get 10 to 15 MB/s still while writing to parity.  I have considered splitting the parity drives - so that one is in one chassis and one in the other and doing a move to see if that changes things.

Any other suggestions?

[FrozenGamer]

4 hours ago, Benson said:

 

In general, 2 Parity won't cause slow down. But a lowend CPU will. I have a ATOM J1900, if run in 2 parity it slow down almost in half.

 

Yesterday, I start switch my parity from 6TB to 8TB, the reason was same - paying more for less.

 

About a year ago, buying a 6TB Toshiba (7200rpm) or WD MyBook 8TB (5400pm) was same 170usd at Amazon. But that time I choose 6TB becuase it faster a lot and I already have several drive work quite solid.

 

The reason for now I choose WD 8TB because it is 2nd gen and performance raise a lot, almost same level as Toshiba. In previous, my array parity check was 11.5hrs average 143.4MB/s and now 16.5hrs average 134.5MB/s. This is a drawback for change to a large disk / low performance disk.

 

I still waiting for Toshiba MG06/7 series come out, it's performance quite good. I may be switch back to 6TB depends on the price/cost.

 

PS : I first change from HGST to Toshiba, Toshiba already slow down the array ...... but it more cheaper and also reliable. Array slow down again and again.

 

For your case, I hope you shot out the slow cause and fix ASAP.

 

 

Ironically i  am seeing 6tb drives for 109 finally coming down in price!

[JorgeB]

5 minutes ago, FrozenGamer said:

I have considered splitting the parity drives - so that one is in one chassis and one in the other and doing a move to see if that changes things.

Doubt it will change anything, but it won't hurt to try.

 

5 minutes ago, FrozenGamer said:

Any other suggestions?

If you have the chance try a different model disk as parity(s).

[FrozenGamer]

Splitting the 2 parities to different Chassis improved speeds by 30 to 50 percent. Speeds around 20MB/s for mover now. (EDIT) - maybe it didn't but it went up higher than ever before initially...   I am thinking digging around for an old SLI capable motherboard and see if i can put 2 sas cards in instead of 1.

As far as switching out the Parity drives, if i can locate 2 8tb drives that are, or 6tb drives (since i haven't incorporated any 8tb into this array for data) ---   Can i try a parity check with write to some test drives without messing up my 2 8tb seagates?  just shut down, put originals back in for parity and start it back up?

Edited April 17, 2018 by FrozenGamer

[JorgeB]

23 minutes ago, FrozenGamer said:

Can i try a parity check with write to some test drives without messing up my 2 8tb seagates?  just shut down, put originals back in for parity and start it back up?

Any writes will make the old ones invalid, but if just a few you can run a correcting parity check, it should be much faster in your case since the problem are writes.

[FrozenGamer]

2 minutes ago, johnnie.black said:

Any writes will make the old ones invalid, but if just a few you can run a correcting parity check, it should be much faster in your case since the problem are writes.

Ok, if i understand correctly no dice on the quick check of 6tb drives

but since i have fast reads, my next parity check should be a lot faster.