Jump to content
FrozenGamer

Slow Parity Sync/Rebuild

11 posts in this topic Last Reply

Recommended Posts

I am in the process of rebuilding 1 of 2 parity drives and another drive - see attached screenshot for drive config and current speeds.  It is taking about 5.5 days to do the rebuild.  It seems that the time really slowed down a while back when i installed 2 8tb Seagate (non archive) drives from 2 6tb seagate drives.  I have all my dockers turned off, i have directory caching to memory on.  I am using 2 silicon graphics se3016 16 drive caddies to hold the drives. https://www.servethehome.com/sgi-rackable-3u-16-bay-se3016-sas-expander-chassis-forum-deal/ They are directly connected with 1 cable each (not daisy chained) to a SAS9207-8e card, which is in pci express on mother board for video card.  I watched the speeds and was thinking, maybe its a 4 tb drive thats slowing things down, but speeds still bad going towards the 5 tb drives.  I will have to wait a while to find out if it speeds up once both 4 and 5 tb drives are eliminated from the parity check. 

I have 4 drives not being used in the array that are in the Se3016,  - they don't really have to be there, would that make a difference in speed?

Note: i have been told to check filesystem on disk 2 with xfs_repair when the rebuild is done. - but we are past the 4tb drives so while it may be a problem i assume its not causing this problem specifically.

What can i check/do to improve speeds with  unraid settings?

How much does drive placement in each Chassis make a difference - like maybe putting 1 parity in each chassis?

I think that something is wrong here beyond the fact that i have so many drives, but i am not sure what.

Are there utilities to test speeds? - i am going to try diskspeed plugin once rebuild is done to look for a slow drive.

If i can find a motherboard with 2 pci express slots i could get a 2nd SAS9207-8e card.  I know getting rid of some smaller drives and replacing with 8tb is in my future.. Thats about all i know, but just get the feeling something else is wrong.

Any other observations/suggestions? 

 

 

Screen shot 1 - of rebuild5ad151877a071_Screenshot2018-04-1316_51_46.thumb.png.b746d02f1167db4db09735c2643c4fe9.png

 

Screenshot 2 - of rebuild 5ad151b249e1d_Screenshot2018-04-1316_48_22.thumb.png.ee4b88e65a153f5fc9cb754cc6083407.png

 

 

Current Diagnostics during rebuild.tower-diagnostics-20180413-1657.zip

Share this post


Link to post
Posted (edited)

Seems Se3016 was 3gbps expander, if plug all drive in and R/W in sametime, then it really slow.

Edited by Benson

Share this post


Link to post
5 minutes ago, Benson said:

Seems Se3016 was 3gbps expander, if plug all drive in and R/W in sametime, then it really slow.

This is a screenshot of another server i have building 2 parity drives with same hardware except computer is much slower (but just 10 drives in 1 chassis instead of having 2 chassis going at same time) - total speeds are about 4x higher.  -  

Screenshot 2018-04-13 17.38.54.png

Share this post


Link to post
Posted (edited)

Could you explain how a 3gbps expander will got 763MB/s bandwidth ?

So the first screen capture seems reasonable ( assume those drive in Se3016 )

 

 

123.png

Edited by Benson

Share this post


Link to post

Diagnostics are filled with xfs corruption errors so not easy to find anything more, you are using the default tunables and they are not optimal for larger arrays so you can try changing the tunables with these, you can change them immediately but doubt there will be a big change since these alone can't explain such slow speeds:


 

Settings -> Disk Settings

Tunable (md_num_stripes): 4096
Tunable (md_sync_window): 2048
Tunable (md_sync_thresh): 2000

 

The ST8000DM004 might also have something to do with it, unlike the Archive SMR drives which are well tested and perform great with unRAID I have some doubts on these, but since you're over half way through, recommend you let the current rebuild finish anyway and try to troubleshoot this later.

 

6 hours ago, Benson said:

Could you explain how a 3gbps expander will got 763MB/s bandwidth ?

Single linked SAS1 expander is good for 1200MB/s (1100MB/s usable).

Share this post


Link to post

Thanks, it is wide port (4 port), the bandwidth need times 4.

Share this post


Link to post
6 hours ago, Benson said:

Thanks, it is wide port (4 port), the bandwidth need times 4.

I have been looking for documentation to show that it is 1GB/s.  But i cant find it.  I believe that is what it is though.

Share this post


Link to post
7 hours ago, johnnie.black said:

Diagnostics are filled with xfs corruption errors so not easy to find anything more, you are using the default tunables and they are not optimal for larger arrays so you can try changing the tunables with these, you can change them immediately but doubt there will be a big change since these alone can't explain such slow speeds:


 


Settings -> Disk Settings

Tunable (md_num_stripes): 4096
Tunable (md_sync_window): 2048
Tunable (md_sync_thresh): 2000

 

The ST8000DM004 might also have something to do with it, unlike the Archive SMR drives which are well tested and perform great with unRAID I have some doubts on these, but since you're over half way through, recommend you let the current rebuild finish anyway and try to troubleshoot this later.

 

Single linked SAS1 expander is good for 1200MB/s (1100MB/s usable).

I will wait for it to finish in a few more days. then start with that xfs repair you recommended and the tunables.  Do you know why it continues to write parity drives past 6tb when the only drives larger than 6 are the parity drives?  is it just putting additional parity in to fill the space?

Share this post


Link to post
4 minutes ago, FrozenGamer said:

is it just putting additional parity in to fill the space?

That's normal, it's filing it with zeros.

 

4 minutes ago, FrozenGamer said:

I will wait for it to finish in a few more days. then start with that xfs repair you recommended and the tunables.

The tunables can be applied now, though like I said probably not going to make much or any difference.

Share this post


Link to post

My rebuild will be done in about 6 hours.  I will do the xfs repair on disk 2 as suggested.  Any other troubleshooting tips for tonight?

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now