FrozenGamer Posted April 14, 2018 Share Posted April 14, 2018 I am in the process of rebuilding 1 of 2 parity drives and another drive - see attached screenshot for drive config and current speeds. It is taking about 5.5 days to do the rebuild. It seems that the time really slowed down a while back when i installed 2 8tb Seagate (non archive) drives from 2 6tb seagate drives. I have all my dockers turned off, i have directory caching to memory on. I am using 2 silicon graphics se3016 16 drive caddies to hold the drives. https://www.servethehome.com/sgi-rackable-3u-16-bay-se3016-sas-expander-chassis-forum-deal/ They are directly connected with 1 cable each (not daisy chained) to a SAS9207-8e card, which is in pci express on mother board for video card. I watched the speeds and was thinking, maybe its a 4 tb drive thats slowing things down, but speeds still bad going towards the 5 tb drives. I will have to wait a while to find out if it speeds up once both 4 and 5 tb drives are eliminated from the parity check. I have 4 drives not being used in the array that are in the Se3016, - they don't really have to be there, would that make a difference in speed? Note: i have been told to check filesystem on disk 2 with xfs_repair when the rebuild is done. - but we are past the 4tb drives so while it may be a problem i assume its not causing this problem specifically. What can i check/do to improve speeds with unraid settings? How much does drive placement in each Chassis make a difference - like maybe putting 1 parity in each chassis? I think that something is wrong here beyond the fact that i have so many drives, but i am not sure what. Are there utilities to test speeds? - i am going to try diskspeed plugin once rebuild is done to look for a slow drive. If i can find a motherboard with 2 pci express slots i could get a 2nd SAS9207-8e card. I know getting rid of some smaller drives and replacing with 8tb is in my future.. Thats about all i know, but just get the feeling something else is wrong. Any other observations/suggestions? Screen shot 1 - of rebuild Screenshot 2 - of rebuild Current Diagnostics during rebuild.tower-diagnostics-20180413-1657.zip Link to comment
Vr2Io Posted April 14, 2018 Share Posted April 14, 2018 Seems Se3016 was 3gbps expander, if plug all drive in and R/W in sametime, then it really slow. Link to comment
FrozenGamer Posted April 14, 2018 Author Share Posted April 14, 2018 5 minutes ago, Benson said: Seems Se3016 was 3gbps expander, if plug all drive in and R/W in sametime, then it really slow. This is a screenshot of another server i have building 2 parity drives with same hardware except computer is much slower (but just 10 drives in 1 chassis instead of having 2 chassis going at same time) - total speeds are about 4x higher. - Link to comment
Vr2Io Posted April 14, 2018 Share Posted April 14, 2018 Could you explain how a 3gbps expander will got 763MB/s bandwidth ? So the first screen capture seems reasonable ( assume those drive in Se3016 ) Link to comment
JorgeB Posted April 14, 2018 Share Posted April 14, 2018 Diagnostics are filled with xfs corruption errors so not easy to find anything more, you are using the default tunables and they are not optimal for larger arrays so you can try changing the tunables with these, you can change them immediately but doubt there will be a big change since these alone can't explain such slow speeds: Settings -> Disk Settings Tunable (md_num_stripes): 4096 Tunable (md_sync_window): 2048 Tunable (md_sync_thresh): 2000 The ST8000DM004 might also have something to do with it, unlike the Archive SMR drives which are well tested and perform great with unRAID I have some doubts on these, but since you're over half way through, recommend you let the current rebuild finish anyway and try to troubleshoot this later. 6 hours ago, Benson said: Could you explain how a 3gbps expander will got 763MB/s bandwidth ? Single linked SAS1 expander is good for 1200MB/s (1100MB/s usable). Link to comment
Vr2Io Posted April 14, 2018 Share Posted April 14, 2018 Thanks, it is wide port (4 port), the bandwidth need times 4. Link to comment
FrozenGamer Posted April 14, 2018 Author Share Posted April 14, 2018 6 hours ago, Benson said: Thanks, it is wide port (4 port), the bandwidth need times 4. I have been looking for documentation to show that it is 1GB/s. But i cant find it. I believe that is what it is though. Link to comment
FrozenGamer Posted April 14, 2018 Author Share Posted April 14, 2018 7 hours ago, johnnie.black said: Diagnostics are filled with xfs corruption errors so not easy to find anything more, you are using the default tunables and they are not optimal for larger arrays so you can try changing the tunables with these, you can change them immediately but doubt there will be a big change since these alone can't explain such slow speeds: Settings -> Disk Settings Tunable (md_num_stripes): 4096 Tunable (md_sync_window): 2048 Tunable (md_sync_thresh): 2000 The ST8000DM004 might also have something to do with it, unlike the Archive SMR drives which are well tested and perform great with unRAID I have some doubts on these, but since you're over half way through, recommend you let the current rebuild finish anyway and try to troubleshoot this later. Single linked SAS1 expander is good for 1200MB/s (1100MB/s usable). I will wait for it to finish in a few more days. then start with that xfs repair you recommended and the tunables. Do you know why it continues to write parity drives past 6tb when the only drives larger than 6 are the parity drives? is it just putting additional parity in to fill the space? Link to comment
JorgeB Posted April 14, 2018 Share Posted April 14, 2018 4 minutes ago, FrozenGamer said: is it just putting additional parity in to fill the space? That's normal, it's filing it with zeros. 4 minutes ago, FrozenGamer said: I will wait for it to finish in a few more days. then start with that xfs repair you recommended and the tunables. The tunables can be applied now, though like I said probably not going to make much or any difference. Link to comment
FrozenGamer Posted April 16, 2018 Author Share Posted April 16, 2018 My rebuild will be done in about 6 hours. I will do the xfs repair on disk 2 as suggested. Any other troubleshooting tips for tonight? Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.