I replaced one of my 2TB drives with a 4TB drive a few days ago.  The 4TB drive I precleared and had no errors.  After replacing the drive, the new drive was rebuilt.  I then ran a parity check and had a few sync errors.  I then ran parity again and had some more sync errors.  I have not written to the array yet and still have my 2TB drive.  I am concerned the new drive may not have been rebuilt properly.  I have had no errors in the past on this system and was only replacing the drive to increase size.  I did receive a bad drive from NEWEGG - I could see the drive in unRaid "Drive Management" but the drive was non-responsive.  NEWEGG set me a new drive which seemed to work perfectly and precleared without error. 

 

After the current parity completes (syslog attached showing back to back parity checks and sync errors - 15 on first parity run and 6 so far in the current parity, all different), I plan to shut down the server, clean everything, check all connections.  I think I should then reinstall the 2TB drive and rebuilt parity, redo the preclear on the 4TB drive, then replace the 2TB drive with the 4TB drive, rebuild and finally run a parity check again.  I have attached my current syslog.  Any suggestions?  Also, if I reinstall the 2TB drive, what steps do I need to take to have unRaid use the data drives to rebuild parity?

 

Thanks,

 

Gary

syslog-2014-05-22.txt

Yes reinstall the original disk. Run memtest overnight on reboot. Let it run for up to 24 hours unless errors are shown.

Have rechecked all connections cleaned the server.  Ran memory check for 15 hours, no errors.  Restarted array and began parity check again - sync errors noted within 15 minutes.  Shut server down, removed the new 4TB drive and replaced with the original 2TB drive.  Reassigned 2TB drive to the array.

 

Question - I assume I use the "NEW CONFIG" utility to force unRaid to rebuild parity with the current set of drives? 

 

Am using unRaid ver 5.04 Pro

 

Gary

 

Yes, you would have to do a new config to rebuild parity.

 

Since you had a 4T drive, the array should have refused to start if you tried to replace it with a 2T drive. But if the drives had been the same size it would have rebuilt over top of your original disk, which is not what you would want. Doing a new config will cause parity to be rebuilt which is your goal.

Thanks, have restarted parity.  When done, will restart parity check again to ensure no sync errors.  If no sync errors, will again try to reintroduce the new 4TB drive after preclearing again. 

 

 

 

Gary

Thanks, have restarted parity.  When done, will restart parity check again to ensure no sync errors.  If no sync errors, will again try to reintroduce the new 4TB drive after preclearing again. 

 

Gary

 

Preclearing the 4T drive should not be necessary, unless a second torture test is required. The rebuild does not benefit from the preclear signature.

I've reinstalled the original 2TB drive, rebuilt parity and then began running a second parity.  I stopped the second parity after 10 sync errors showed up - ~45 minutes.  I've shut down, disconnected and reseated all sata cables.  After reseating all sata cables, I have restarted a parity sync (1.5 hours in and 8 sync errors).

 

I don't believe in coincidences, this all started after I installed a 4TB drive that was non-responsive. The positions I installed the drive in are all attached to an Adaptec RAID 1430SA SATAII PC Express X4 card.  When I installed the drive in my desktop as a 2nd drive to verify it was not working, my desktop would not boot.  The Adaptec card has the Parity drive, two three data drives (all three in a Zalman modular 3-Bay) and a SYBA Mobile Rack that I use to pre-clear HDs or on occasion for a cache drive.  The 2nd 4TB drive I received was installed in the SYBA Mobile Rack and pre-cleared w/o issue (preclear report attached) 

 

Is there a chance something could have happened to the Adaptec Raid card due to an issue with the 1st 4TB drive?

 

Any other suggestions?

 

 

 

Gary

preclear_rpt_W300T5SY_2014-05-18.doc

The most common reason for parity sync errors as you are describing is bad memory. Have you run a lengthy memtest?

15 hours with no errors. 

 

 

Gary

15 hours with no errors. 

 

Gary

 

I'd run it again - 24 hours

 

Another thing I've heard in similar situation is that updating controller firmware.

 

As a test you might run a non-correcting check, note the locations of the parity issues, and then runs second parity check and not the locations. See if they are identical.

 

Definately do not write to the array until you get this resolved.

Thanks and will do.  Had the system running for two years without any issues and this all began with the introduction of a 4TB within the last couple of weeks.  Do have 2 other 4TB drives currently in the system that were installed last December.

 

 

 

Gary

Ran a correcting parity check and had 22 sync errors:

 

May 27 20:46:59 Tower kernel: mdcmd (47): check CORRECT

May 27 20:46:59 Tower kernel: md: recovery thread woken up ...

May 27 20:46:59 Tower kernel: md: recovery thread checking parity...

May 27 20:46:59 Tower kernel: md: using 1536k window, over a total of 3907018532 blocks.

May 27 20:48:17 Tower kernel: md: correcting parity, sector=15689720

May 27 20:48:17 Tower kernel: md: correcting parity, sector=15689728

May 27 20:55:59 Tower kernel: md: correcting parity, sector=126600328

May 27 20:55:59 Tower kernel: md: correcting parity, sector=126600336

May 27 20:58:50 Tower kernel: md: correcting parity, sector=167841304

May 27 21:29:53 Tower kernel: md: correcting parity, sector=616753520

May 27 21:29:53 Tower kernel: md: correcting parity, sector=616753528

May 27 21:30:23 Tower kernel: md: correcting parity, sector=624142928

May 27 21:41:19 Tower kernel: md: correcting parity, sector=781911520

May 27 22:12:51 Tower kernel: md: correcting parity, sector=1223916024

May 27 22:36:07 Tower kernel: md: correcting parity, sector=1538951312

May 27 22:49:20 Tower kernel: md: correcting parity, sector=1712918968

May 27 22:54:14 Tower kernel: md: correcting parity, sector=1778378520

May 27 22:59:05 Tower kernel: md: correcting parity, sector=1840236888

May 27 23:00:46 Tower kernel: md: correcting parity, sector=1862329664

May 27 23:04:58 Tower kernel: md: correcting parity, sector=1917282880

May 27 23:05:35 Tower kernel: md: correcting parity, sector=1925740424

May 27 23:55:04 Tower kernel: md: correcting parity, sector=2523585448

May 28 00:49:33 Tower kernel: md: correcting parity, sector=3115478256

May 28 00:53:58 Tower kernel: md: correcting parity, sector=3160759864

May 28 01:14:17 Tower kernel: md: correcting parity, sector=3359522968

May 28 01:14:17 Tower kernel: md: correcting parity, sector=3359522976

May 28 03:18:19 Tower kernel: mdcmd (48): spindown 2

May 28 03:18:22 Tower kernel: mdcmd (49): spindown 3

May 28 03:18:24 Tower kernel: mdcmd (50): spindown 4

May 28 03:18:27 Tower kernel: mdcmd (51): spindown 5

May 28 03:18:29 Tower kernel: mdcmd (52): spindown 8

May 28 07:09:21 Tower kernel: md: sync done. time=37342sec

 

Then ran a non-correcting parity check and had 24 sync errors:

 

May 28 12:50:12 Tower kernel: mdcmd (57): check NOCORRECT

May 28 12:50:12 Tower kernel: md: recovery thread woken up ...

May 28 12:50:12 Tower kernel: md: recovery thread checking parity...

May 28 12:50:12 Tower kernel: md: using 1536k window, over a total of 3907018532 blocks.

May 28 12:59:18 Tower kernel: md: parity incorrect, sector=126600328

May 28 12:59:18 Tower kernel: md: parity incorrect, sector=126600336

May 28 13:02:11 Tower kernel: md: parity incorrect, sector=167841304

May 28 13:24:14 Tower kernel: md: parity incorrect, sector=485817304

May 28 13:31:24 Tower kernel: md: parity incorrect, sector=589413160

May 28 13:33:20 Tower kernel: md: parity incorrect, sector=616753520

May 28 13:33:20 Tower kernel: md: parity incorrect, sector=616753528

May 28 13:33:50 Tower kernel: md: parity incorrect, sector=624142928

May 28 14:07:01 Tower kernel: md: parity incorrect, sector=1094021832

May 28 14:16:18 Tower kernel: md: parity incorrect, sector=1223916024

May 28 14:39:28 Tower kernel: md: parity incorrect, sector=1538951312

May 28 14:49:15 Tower kernel: md: parity incorrect, sector=1668852888

May 28 14:52:39 Tower kernel: md: parity incorrect, sector=1712918968

May 28 15:02:30 Tower kernel: md: parity incorrect, sector=1841940792

May 28 15:04:04 Tower kernel: md: parity incorrect, sector=1862329664

May 28 15:08:52 Tower kernel: md: parity incorrect, sector=1925740424

May 28 15:29:49 Tower kernel: md: parity incorrect, sector=2183602296

May 28 15:58:22 Tower kernel: md: parity incorrect, sector=2523585448

May 28 16:52:40 Tower kernel: md: parity incorrect, sector=3113540960

May 28 16:52:40 Tower kernel: md: parity incorrect, sector=3113540968

May 28 16:52:51 Tower kernel: md: parity incorrect, sector=3115478256

May 28 16:57:15 Tower kernel: md: parity incorrect, sector=3160759864

May 28 17:17:35 Tower kernel: md: parity incorrect, sector=3359522968

May 28 17:17:35 Tower kernel: md: parity incorrect, sector=3359522976

May 28 19:21:33 Tower kernel: mdcmd (58): spindown 4

May 28 19:21:35 Tower kernel: mdcmd (59): spindown 5

May 28 19:34:27 Tower dhcpcd[1160]: eth0: renewing lease of 192.168.1.208

May 28 19:34:27 Tower dhcpcd[1160]: eth0: acknowledged 192.168.1.208 from 192.168.1.1

May 28 19:34:27 Tower dhcpcd[1160]: eth0: leased 192.168.1.208 for 86400 seconds

May 28 20:20:39 Tower kernel: mdcmd (60): spindown 2

May 28 20:20:41 Tower kernel: mdcmd (61): spindown 8

May 28 20:23:23 Tower kernel: mdcmd (62): spindown 3

May 28 23:13:17 Tower kernel: md: sync done. time=37384sec

 

All errors occurred in the first 2 TB of the 4TB parity drive.  Many but not all of the errors repeat.

 

Began a memory check and am at 23 hours and counting with zero errors.

 

 

 

Gary

Bjp999 is much better at this than me...so listen to him.

 

You can see in that last post, that you ran a correcting parity check.

(look at the last line: "May 28 01:14:17 Tower kernel: md: correcting parity, sector=3359522976")

then you ran a NON-correcting parity check:

(look at the last line: "May 28 17:17:35 Tower kernel: md: parity incorrect, sector=3359522976")...same address.

That's just one example...there are more just like this.

 

In your situation, what I'd do while waiting for Brian's advice is this:

Run SMART on all the data drives...and then run Reiserfsck --check on all the data drives (not parity).

IMHO there's still one drive that's not working correctly...and your original problem *could* be a coincidence.

 

Neither SMART nor Reiserfsck --check will 'write' to the array, so you should be safe. And you might find an errant drive.

 

2¢

See THIS POST for a user having a similar symptom.

 

Sorry I can't help much on this one. It is a hardware / firmware issue with your system, not a bug in UnRaid. This is fairly uncommon but does happen from time to time. If the post above doesn't help you can search the forums further. Updating firmware seems a good place to start.

 

It is possible you have an errant drive as Dale suggests. Smart reports might help identify that situation. But drives have a ton of error detection / ECC features that are supposed to make that impossible. But you are very deep in the land of unlikely and anything is theoretically possible. But I still think firmware updates is the first step.

I have run a reiserfsck on all data drives and "No corruptions found" was the result.

 

I have run a short smartctl on all drives with the following results:

 

(a)  Reallocated_Sector_Ct:  0 on all drives

(b)  Current_Pending_Sector:  0 on all drives

©  UDMA_CRC_Error_Count:  0 on Data Drives 3 through 8.  51 on Parity drive, 32 on Data Drive 1 and 2 on Data Drive 2

(d)  Temperatures:  unremarkable - with lowest min of 16 and highest max of 39.

 

Attached find smartctl report for parity drive.

 

Parity and Data Drives 1 and 2 are connected to the Adaptec 1430SA card and have the longest cable runs.  These three cables are tied to together plus one going to the carrier less removable drive mount.

 

The Adaptec 1430SA firmware is build 2322 (23 Aug 2007).  There are two later versions - 2329 (6 June 2008) and 2507 (8 Nov 2010)

 

So....course of action is:

(a) Update 1420SA firmware to latest - 2507

(b) Separate the cables.

 

Any other recommendations?

 

 

Thanks,

 

Gary

smart00a.txt

Update the firmware. Cable crosstalk can be an issue but not likely to cause the same parity locations to be detected over and over.

I've flashed the Adaptec 1430SA firmware to the latest version, and am in process with a parity check.  Currently at 38 sync errors all in the first 2TBs, no errors since with the 0.9TB remaining.  Most but not all of the previous sync errors show up along with new ones.  (see extract from syslog below)

 

 

May 31 00:29:35 Tower kernel: mdcmd (47): check CORRECT (unRAID engine)

May 31 00:29:35 Tower kernel: md: recovery thread woken up ... (unRAID engine)

May 31 00:29:35 Tower kernel: md: recovery thread checking parity... (unRAID engine)

May 31 00:29:35 Tower kernel: md: using 1536k window, over a total of 3907018532 blocks. (unRAID engine)

May 31 00:31:44 Tower kernel: md: correcting parity, sector=28904960 (unRAID engine)

May 31 00:38:46 Tower kernel: md: correcting parity, sector=126600328 (unRAID engine)

May 31 00:38:46 Tower kernel: md: correcting parity, sector=126600336 (unRAID engine)

May 31 00:41:39 Tower kernel: md: correcting parity, sector=167841304 (unRAID engine)

May 31 00:49:35 Tower kernel: md: correcting parity, sector=281883672 (unRAID engine)

May 31 01:03:05 Tower kernel: md: correcting parity, sector=476060928 (unRAID engine)

May 31 01:10:12 Tower kernel: md: correcting parity, sector=579171760 (unRAID engine)

May 31 01:10:12 Tower kernel: md: correcting parity, sector=579171768 (unRAID engine)

May 31 01:12:47 Tower kernel: md: correcting parity, sector=616753520 (unRAID engine)

May 31 01:12:47 Tower kernel: md: correcting parity, sector=616753528 (unRAID engine)

May 31 01:13:17 Tower kernel: md: correcting parity, sector=624142928 (unRAID engine)

May 31 01:18:46 Tower kernel: md: correcting parity, sector=701845776 (unRAID engine)

May 31 01:18:46 Tower kernel: md: correcting parity, sector=701845784 (unRAID engine)

May 31 01:50:22 Tower kernel: md: correcting parity, sector=1147499176 (unRAID engine)

May 31 01:52:50 Tower kernel: md: correcting parity, sector=1182962696 (unRAID engine)

May 31 01:52:50 Tower kernel: md: correcting parity, sector=1182962704 (unRAID engine)

May 31 01:55:51 Tower kernel: md: correcting parity, sector=1223916024 (unRAID engine)

May 31 02:06:59 Tower kernel: md: correcting parity, sector=1377432328 (unRAID engine)

May 31 02:19:00 Tower kernel: md: correcting parity, sector=1538951312 (unRAID engine)

May 31 02:19:39 Tower kernel: md: correcting parity, sector=1547972592 (unRAID engine)

May 31 02:32:12 Tower kernel: md: correcting parity, sector=1712918968 (unRAID engine)

May 31 02:41:17 Tower kernel: md: correcting parity, sector=1831890176 (unRAID engine)

May 31 02:43:38 Tower kernel: md: correcting parity, sector=1862329664 (unRAID engine)

May 31 02:48:27 Tower kernel: md: correcting parity, sector=1925740424 (unRAID engine)

May 31 02:58:47 Tower kernel: md: correcting parity, sector=2054071048 (unRAID engine)

May 31 03:00:36 Tower kernel: md: correcting parity, sector=2076494232 (unRAID engine)

May 31 03:00:36 Tower kernel: md: correcting parity, sector=2076494240 (unRAID engine)

May 31 03:04:40 Tower kernel: md: correcting parity, sector=2128028792 (unRAID engine)

May 31 03:04:40 Tower kernel: md: correcting parity, sector=2128028800 (unRAID engine)

May 31 03:34:02 Tower kernel: md: correcting parity, sector=2478643320 (unRAID engine)

May 31 03:34:02 Tower kernel: md: correcting parity, sector=2478643328 (unRAID engine)

May 31 03:37:57 Tower kernel: md: correcting parity, sector=2523585448 (unRAID engine)

May 31 04:09:06 Tower kernel: md: correcting parity, sector=2870257608 (unRAID engine)

May 31 04:18:16 Tower kernel: md: correcting parity, sector=2967286032 (unRAID engine)

May 31 04:32:27 Tower kernel: md: correcting parity, sector=3115478256 (unRAID engine)

May 31 04:36:51 Tower kernel: md: correcting parity, sector=3160759864 (unRAID engine)

May 31 04:57:11 Tower kernel: md: correcting parity, sector=3359522968 (unRAID engine)

May 31 04:57:11 Tower kernel: md: correcting parity, sector=3359522976 (unRAID engine)

May 31 07:54:51 Tower kernel: NTFS driver 2.1.30 [Flags: R/W MODULE]. (System)

May 31 07:55:00 Tower kernel: mdcmd (48): spinup 0 (Routine)

May 31 07:55:00 Tower kernel:  (Routine)

May 31 07:56:07 Tower kernel: mdcmd (49): spinup 1 (Routine)

May 31 07:56:07 Tower kernel:  (Routine)

May 31 07:56:37 Tower kernel: mdcmd (50): spinup 2 (Routine)

May 31 07:56:37 Tower kernel:  (Routine)

May 31 07:58:01 Tower kernel: mdcmd (51): spinup 3 (Routine)

May 31 07:58:01 Tower kernel:  (Routine)

May 31 07:58:16 Tower kernel: mdcmd (52): spinup 4 (Routine)

May 31 07:58:16 Tower kernel:  (Routine)

May 31 07:58:45 Tower kernel: mdcmd (53): spinup 5 (Routine)

May 31 07:58:45 Tower kernel:  (Routine)

May 31 07:59:15 Tower kernel: mdcmd (54): spinup 6 (Routine)

May 31 07:59:15 Tower kernel:  (Routine)

May 31 07:59:34 Tower kernel: mdcmd (55): spinup 7 (Routine)

May 31 07:59:34 Tower kernel:  (Routine)

May 31 08:00:17 Tower kernel: mdcmd (56): spinup 8 (Routine)

May 31 08:00:17 Tower kernel:  (Routine)

 

Any suggestions?

 

I'm stumped but could do the following:

 

(1)  Remove disk 2-8 from the array and rebuild and recheck parity.  Would only have 2 4TB drives in the array then and with no errors showing up in the 2 to 4TB range might indicate the issue is not in the 4TB drives or drive slots.  (should note my previous post was incorrect - the Parity drive, data drives 1 & 2 are connected to the motherboard SATA controller.  The Adaptec 1430SA is connected to data drives 6-8 plus the carrierless mobile rack.)

(2)  Add back drives, rebuilding and rechecking until an error shows up which would either indicate a drive or slot problem. (likely would take a day per drive)

(3)  Haven't disconnected and reconnected power from the drive cages and power supply yet (modular SeaSonic 650) but could do that as well.

 

I have a spare 2TB and 4TB drive plus all the data backed up on external hard drives.

 

 

Thanks

 

Gary

Are you running a correcting check? If so, it's the next check that will tell the story. Should have zero parity hours on the second check.

Yes, running correcting first then non-correcting.  140 minutes to go.  Will have a good idea quickly as the first errors were occurring in 9 to 30 minutes.

 

 

Thanks

 

Gary

Completed parity rebuild and non-correcting parity check with no errors.  Adding back in two more drives.

 

Believe the issue is going to turn out to be a bad 1430 card.  Noticed during the last full parity checks two of the 2TB drives on the 1430 card did not spin down.  Since the 1430 card is no longer available, does anyone have a recommendation for a 4 port replacement card? 

 

 

Gary

The Supermicro SASLP (not SAS2LP) is a PCIe x4 card like the 1430. But it supports 8 drives instead of 4.

This is the recommended 8 port device as listed in the "Sata Controllers" forum I assume.  It is noted there that depending on the BIOS some additional steps would be required to use this card in UnRaid - the date on the post would indicate these instructions would be for UnRaid v4.X.  Is this still the case in UnRaid v5.0 or is it true "Plug and Play" in v5.0?  I only ask because this exercise has shown that my computer skills have atrophied over the years and I would like to know if this will be a 5 minute or 5 hour task.  When I had to burn the 1430 BIOS using DOS removable media, it took me hours before I had a USB drive configured properly after failing at burning a CD with the requisite Adaptec program and firmware.  If I had only kept all the computers and software I have donated over the years, it would have been a simple 5 minute task. 

 

 

 

Thanks

 

Gary

After adding back in drives 5 and 6, parity errors cropped up again.  To make a long story short, adding any drives after the first four, resulted in parity errors.  Didn't matter which rack or slot the 5th drive was added back into or whether it was connected to the MB Sata ports or the 1430 card.  New sata cables arrived and I replaced those.  Same result.  At one point, my parity was generating 48 sync errors.  My order of a Supermicro SASLP arrived and I installed that with breakout cable, installed all drives and reran parity - result 3 sync errors (at approx 980GB, 1.53TB, & 1.89TB).  Reran parity with only two sync errors (first two locations from previous run).  Per the Wiki instructions, I ran MD5 checks on a 10GB range around the first Sync error location on all drives - result, no issue.  Reran parity and now I was down to one error at a new location (approx 1.45TB), repeated with the same result.  Reran MD5 checks at that location, no issue.  Reran parity - NO SYNC errors.  So.....is the problem solved?  I don't believe so, but will mark this post SOLVED for now as the gremlins appear to have vacated my system for now.

 

 

Gary