Slow Parity Sync/Rebuild


Recommended Posts

I am in the process of rebuilding 1 of 2 parity drives and another drive - see attached screenshot for drive config and current speeds.  It is taking about 5.5 days to do the rebuild.  It seems that the time really slowed down a while back when i installed 2 8tb Seagate (non archive) drives from 2 6tb seagate drives.  I have all my dockers turned off, i have directory caching to memory on.  I am using 2 silicon graphics se3016 16 drive caddies to hold the drives. https://www.servethehome.com/sgi-rackable-3u-16-bay-se3016-sas-expander-chassis-forum-deal/ They are directly connected with 1 cable each (not daisy chained) to a SAS9207-8e card, which is in pci express on mother board for video card.  I watched the speeds and was thinking, maybe its a 4 tb drive thats slowing things down, but speeds still bad going towards the 5 tb drives.  I will have to wait a while to find out if it speeds up once both 4 and 5 tb drives are eliminated from the parity check. 

I have 4 drives not being used in the array that are in the Se3016,  - they don't really have to be there, would that make a difference in speed?

Note: i have been told to check filesystem on disk 2 with xfs_repair when the rebuild is done. - but we are past the 4tb drives so while it may be a problem i assume its not causing this problem specifically.

What can i check/do to improve speeds with  unraid settings?

How much does drive placement in each Chassis make a difference - like maybe putting 1 parity in each chassis?

I think that something is wrong here beyond the fact that i have so many drives, but i am not sure what.

Are there utilities to test speeds? - i am going to try diskspeed plugin once rebuild is done to look for a slow drive.

If i can find a motherboard with 2 pci express slots i could get a 2nd SAS9207-8e card.  I know getting rid of some smaller drives and replacing with 8tb is in my future.. Thats about all i know, but just get the feeling something else is wrong.

Any other observations/suggestions? 

 

 

Screen shot 1 - of rebuild5ad151877a071_Screenshot2018-04-1316_51_46.thumb.png.b746d02f1167db4db09735c2643c4fe9.png

 

Screenshot 2 - of rebuild 5ad151b249e1d_Screenshot2018-04-1316_48_22.thumb.png.ee4b88e65a153f5fc9cb754cc6083407.png

 

 

Current Diagnostics during rebuild.tower-diagnostics-20180413-1657.zip

Link to comment
5 minutes ago, Benson said:

Seems Se3016 was 3gbps expander, if plug all drive in and R/W in sametime, then it really slow.

This is a screenshot of another server i have building 2 parity drives with same hardware except computer is much slower (but just 10 drives in 1 chassis instead of having 2 chassis going at same time) - total speeds are about 4x higher.  -  

Screenshot 2018-04-13 17.38.54.png

Link to comment

Diagnostics are filled with xfs corruption errors so not easy to find anything more, you are using the default tunables and they are not optimal for larger arrays so you can try changing the tunables with these, you can change them immediately but doubt there will be a big change since these alone can't explain such slow speeds:


 

Settings -> Disk Settings

Tunable (md_num_stripes): 4096
Tunable (md_sync_window): 2048
Tunable (md_sync_thresh): 2000

 

The ST8000DM004 might also have something to do with it, unlike the Archive SMR drives which are well tested and perform great with unRAID I have some doubts on these, but since you're over half way through, recommend you let the current rebuild finish anyway and try to troubleshoot this later.

 

6 hours ago, Benson said:

Could you explain how a 3gbps expander will got 763MB/s bandwidth ?

Single linked SAS1 expander is good for 1200MB/s (1100MB/s usable).

Link to comment
7 hours ago, johnnie.black said:

Diagnostics are filled with xfs corruption errors so not easy to find anything more, you are using the default tunables and they are not optimal for larger arrays so you can try changing the tunables with these, you can change them immediately but doubt there will be a big change since these alone can't explain such slow speeds:


 


Settings -> Disk Settings

Tunable (md_num_stripes): 4096
Tunable (md_sync_window): 2048
Tunable (md_sync_thresh): 2000

 

The ST8000DM004 might also have something to do with it, unlike the Archive SMR drives which are well tested and perform great with unRAID I have some doubts on these, but since you're over half way through, recommend you let the current rebuild finish anyway and try to troubleshoot this later.

 

Single linked SAS1 expander is good for 1200MB/s (1100MB/s usable).

I will wait for it to finish in a few more days. then start with that xfs repair you recommended and the tunables.  Do you know why it continues to write parity drives past 6tb when the only drives larger than 6 are the parity drives?  is it just putting additional parity in to fill the space?

Link to comment
4 minutes ago, FrozenGamer said:

is it just putting additional parity in to fill the space?

That's normal, it's filing it with zeros.

 

4 minutes ago, FrozenGamer said:

I will wait for it to finish in a few more days. then start with that xfs repair you recommended and the tunables.

The tunables can be applied now, though like I said probably not going to make much or any difference.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.