Parity check, many errors (6TB Red issues?)


Recommended Posts

Started a parity check (correcting) yesterday, come in to find it at 46% and 454 errors. I got 4 errors a week or so ago, but I wrote it off as an anomaly, my reading here seemed to indicate it happens, and was usually the parity drive.

 

I ran long smart tests after that, found no issues so I forgot about it, and then this. I have made some hardware changes lately, I added 2 more SS-500s, added another breakout cable to my SASLP, and added 2 6TB Reds, a third precleared and on standby.

 

Specs are in my sig, drives are:

6TB Red Parity

6TB Red Data

5x 2TB WD EARS/EADS

1x 2TBHitachi 5k3000

1x 3TB WD Red

Cache is 1 TB WD Blue

Spare 6 TB Red

 

Included syslog, and smart reports from today (each file contains all smart reports for all drives), and also from a week ago (right after the long smart tests I think).

 

20141119_smart_summary.txt

20141127_smart_summary.txt

syslog1127.zip

Link to comment

Started a parity check (correcting) yesterday, come in to find it at 46% and 454 errors. I got 4 errors a week or so ago, but I wrote it off as an anomaly, my reading here seemed to indicate it happens, and was usually the parity drive.

 

I ran long smart tests after that, found no issues so I forgot about it, and then this. I have made some hardware changes lately, I added 2 more SS-500s, added another breakout cable to my SASLP, and added 2 6TB Reds, a third precleared and on standby.

 

Specs are in my sig, drives are:

6TB Red Parity

6TB Red Data

5x 2TB WD EARS/EADS

1x 2TBHitachi 5k3000

1x 3TB WD Red

Cache is 1 TB WD Blue

Spare 6 TB Red

 

Included syslog, and smart reports from today, and also from a week ago (right after the long smart tests I think).

 

If you search for similar you will see that I and others have had similar issues with WD 6TB Red drives. Booting into Xen can help reduce the issue, but I've still seen it happen there as well. Others have suggested configuring the 6TB Red drives to not spin down, which may help. I haven't done a manual parity check to confirm for myself, but the monthly one will happen this weekend so I will see how it goes.

 

Link to comment

I upgraded my parity ~3-4 weeks ago to a 6TB WD Red Drive.  Ran 5 preclear passes on the drive before I added to my array.  Ran the initial parity check plus ~ 2 additional.  Have not seen any errors.  One thing I did notice is that my parity check speed dropped from ~100-110 MB/s to high 60 to low 70s.  It did that on every run.  The current run I just kicked of has the same speeds.

 

I just completed 5 preclear cycles on a new 6TB Red with no issues and added it as a data drive.  Running a parity check now. Will post the results when complete.  On the first, the auto parity check will run again for another check.

 

I was torn between leaving it out of the array as a ready to go spare/data add or putting it in and using it now.

 

I am hoping that there are no inherent issues with these 6TB Reds.  The first one I received from NE was DOA and would not even let the server start correctly when it was in the system.

 

 

 

Update: Complete a parity check with zero errors.  One thing I happened to notice is that when the parity check got beyond the 1.5 TB boundary it sped up to ~100MB.  So either one or more of my 1.5TB dives is the issues or something else is causing it.  The 6TB parity is on the MB sata port while the 1.5TB drives are on the super micro and I was having speed issues in this configuration.  The new 6TB data drive is also on the SM controller.

 

Link to comment

The parity check speed is determined by the speed of the slowest drive currently being used.  Disks get slower as they reach the inner tracks. 

 

When you have drives of mixed sizes it is normal for the speed to get slower as a particular size of drive reaches its inner tracks, and then pick up speed again when the parity gets beyond that point so that size drive is no longer part of the parity check.

Link to comment

That is not my issue.  Before I upgraded my parity from 4TB to 6TB, when the parity check first kicked off, it would be in the 100-110MB/s range and slow down as it progressed as expected.  After I upgrade the parity to 6TB and placed the previous parity into the array my initial speeds dropped to high 60/low 70 Mb/s speed with no other changes.  The initial speed should not have changed. 

 

One of my instructors years ago had a good statement.  It was working before you touched it, you touched it, now it s not working correctly.  It was probably something you did.

 

So either one, or more, of my 1.5TB drives picked that time to start having problems or something else is at play.  I ultimately wanted to get rid of the 1.5TB, so I will probably migrate all of the data over and remove them from the system and see if that helps.

Link to comment

I completed the parity check with no errors. 

 

I have migrated all the data from the three 1.5TB to the to the new 6TB  and rebuilt the array without the drives.  I am in the process of rebuilding parity now.  After about 3-4 minutes it jumped from ~50MB/s to ~110MB/s.

 

@Traxxus

Not trying to hijack your thread, just trying to keep the data about the 6TB reds together as they are so new.

Link to comment

I used WD RED 6TB drives for parity and data, no obvious problems in the first few months with regular parity checks (disk spin down set to 1 hour).

 

I did have slow parity speeds for a while with no obvious reason behind it, ultimately one of the RED drives failed and parity check speeds returned to normal.

Link to comment

@Traxxus

Not trying to hijack your thread, just trying to keep the data about the 6TB reds together as they are so new.

 

Np, it's in the right place, no need to have a bunch of threads about it.

 

I disabled spindown on both the 6 TB Red drives before the parity check I was talking about in the OP completed, so they haven't spun down since then. No errors on the scheduled parity check yesterday. I would like to see if it's related to spindown, so I am going to leave it as is, do another parity check in 2 weeks, if all is good, I will then enable normal spindown (2 hours for me) for those drives, and do another parity check 2 weeks after that.

 

I've also noticed slow parity checks, takes a little over 18 hours, and initial speeds are only ~80 MB/s. I understand having various sizes of drives can slow it down because it slows down as it gets to the slowest part of each different sized disk, but it still doesn't make sense that adding 2 larger, denser/faster drives would slow things down in that way. I'm fairly sure the 6TB reds are on the SASLP, just going by memory on how I ran everything. I may move them to the motherboard, but I don't want to change anything else just yet, I'd like to see if it's really the spindown causing the errors.

 

I completed the parity check with no errors. 

 

I have migrated all the data from the three 1.5TB to the to the new 6TB  and rebuilt the array without the drives.  I am in the process of rebuilding parity now.  After about 3-4 minutes it jumped from ~50MB/s to ~110MB/s.

 

Minus what is already on the 6TB Reds, I have 8 GB of data left (out of ~13 total), and that is all on the 2 TB drives. I've been thinking about phasing them out, and have been consolidating data onto as few drives as possible with that in mind, though I'm going to wait for a v6 RC at least, so I can format them using one of the new filesystems, maybe, I dunno.  I don't remember exactly how old they are, but most have ~50,000 hours, or almost 6 years on them according to power on hours in SMART. 

 

 

Link to comment

Mine kicked off and ran the monthly parity check with no issues.  I can say for sure if the drive was spun up or not when the parity check kicked off.  I can say that my spin-down timer is enabled and set for one hour.  I have not seen any issues other than the parity check slowdown that occurred when I migrated to the first 6TB drive. 

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.