I ran memory test for around 10 hrs, no errors.  I ran nocorrect parity again, 11 parity errors.  Syslog attached.  I am running additional parity checks now.

syslog_6_8_10_11errors.txt.zip

I ran memory test for around 10 hrs, no errors.  I ran nocorrect parity again, 11 parity errors.  Syslog attached.  I am running additional parity checks now.

If the number of parity errors changes each time you run with the nocorrect option then some hardware in your server is intermittently returning different values than expected.  The problem is... unless the syslog shows errors we have absolutely no way to know if it is any specific disk, or the disk controller, or the motherboard chipset itself. 

 

The problem is, only 11 of the bytes out of the 2TB were not as expected, and we don't know which 11, or, if it is a specific bit that is always bad in all 11 words. (and even if we knew, it would not help fix it, since we don't replace parts at the bit level)

It seems to me that the only indications of possible hardware problems would be the small number of smart crc errors and the fact that limetech states that ears drives should not be used without jumpers on 7-8.  It seems to me that a first step would be to replace the ears drives with ones that are jumpered, no?  I am not sure what the process would be given that I am receiving parity errors. Can someone please tell me the appropriate sequence to replace the unjumpered drive with incorrect parity?

 

Joe, I also am not sure if you anyone else knowledgeable has looked at the syslog.  Maybe there is a problem there that I am missing.  Is "Clocksource tsc unstable (delta = 62502548 ns)"  a problem or not.  There is also a duplicate directory .ds_store line in there that looks out of place to me.

With 11 bit errors out of 20 trillion bits, the amount of corruption you have is slight.  The only way that I could see determining which drive is returning flaky data would be to run some sort of disk-wide CRC calculation (not sure how you'd do it but probably not hard).  If you run it on each drive multiple times you might be able to determine which drive is the culprit.  If you could find the command you could run it on multiple drives at a time.  Once you ID the drive causing the problem, it would be a simple thing to rebuild it.  (Of course if the problem is the jumper, there could be multiple drives giving problem.  That makes life trickier!)

 

To add the jumper you'd need replace the disks - 1 at a time - and have unRAID rebuild them.  You could do it in place if desired. Procedure would be something like this ...

 

1 - Shut down unRAID

2 - Remove disk

3 - Add jumper

3 - Boot server

4 - Start array (should simulate the failed disk)

5 - Cleanly shutdown the server

6 - Reinstall disk

7 - Boot server

 

unRAID should offer to rebuild the data when the array is started.

 

Good luck!

Thanks for the instructions on replacing the drives.  The part of the process that I am unclear about is regaining parity. If parity is giving errors before I replace a drive should I do a parity check after replacement?

 

fwiw the last nocorrect parity check I just ran came up with 7 errors.  running another now.

 

I had read about someone running md5 commands to figure out which device was causing unraid errors, but I have no idea how to go about doing this.

I replaced the EARS drives...still getting errors.  I am pretty much out of ideas in terms of troubleshooting this.  The only thing I can really think of is replacing the seagate(parity) drive with a spare drive I have.  I'm afraid I am on the verge of giving up on this system.

Can you be more specific about what you did?  What motherboard are you using.

I will post hardware info shortly.  I replaced the parity drive and had 3 parity error after install.  This time there appears to be some error info in the syslog:

 

Jun 10 22:03:41 Tower shfs: duplicate object: /mnt/disk2/./.DS_Store

Jun 10 22:03:41 Tower emhttp: shcmd (31): killall -HUP smbd

Jun 10 22:03:41 Tower emhttp: shcmd (32): /etc/rc.d/rc.nfsd restart | logger

Jun 11 05:17:25 Tower kernel: md: sync done. time=26025sec rate=56297K/sec

Jun 11 05:17:25 Tower kernel: md: recovery thread sync completion status: 0

Jun 11 05:30:23 Tower kernel: ata5: exception Emask 0x10 SAct 0x0 SErr 0x1810000 action 0xe frozen

Jun 11 05:30:23 Tower kernel: ata5: SError: { PHYRdyChg LinkSeq TrStaTrns }

Jun 11 05:30:23 Tower kernel: ata5: hard resetting link

Jun 11 05:30:23 Tower kernel: ata5: nv: skipping hardreset on occupied port

Jun 11 05:30:23 Tower kernel: ata6: exception Emask 0x10 SAct 0x0 SErr 0x1810000 action 0xe frozen

Jun 11 05:30:23 Tower kernel: ata6: SError: { PHYRdyChg LinkSeq TrStaTrns }

Jun 11 05:30:23 Tower kernel: ata6: hard resetting link

Jun 11 05:30:23 Tower kernel: ata6: nv: skipping hardreset on occupied port

Jun 11 05:30:23 Tower kernel: ata2: exception Emask 0x10 SAct 0x0 SErr 0x1810000 action 0xe frozen

Jun 11 05:30:23 Tower kernel: ata2: SError: { PHYRdyChg LinkSeq TrStaTrns }

Jun 11 05:30:23 Tower kernel: ata2: hard resetting link

Jun 11 05:30:23 Tower kernel: ata2: nv: skipping hardreset on occupied port

Jun 11 05:30:29 Tower kernel: ata5: link is slow to respond, please be patient (ready=0)

Jun 11 05:30:29 Tower kernel: ata6: link is slow to respond, please be patient (ready=0)

Jun 11 05:30:29 Tower kernel: ata2: link is slow to respond, please be patient (ready=0)

Jun 11 05:30:29 Tower kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

Jun 11 05:30:29 Tower kernel: ata6.00: configured for UDMA/133

Jun 11 05:30:29 Tower kernel: ata6: EH complete

Jun 11 05:30:30 Tower kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

Jun 11 05:30:30 Tower kernel: ata2.00: configured for UDMA/133

Jun 11 05:30:30 Tower kernel: ata2: EH complete

Jun 11 05:30:30 Tower kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

Jun 11 05:30:30 Tower kernel: ata5.00: configured for UDMA/133

Jun 11 05:30:30 Tower kernel: ata5: EH complete

 

I think I will do another parity check now and see if those errors were remapped.

Those are errors in communicating with your disks, but they are not media errors.  They involved three different disks.

 

Also, they occurred 13 minutes after the parity check was completed.

 

According to this page

https://ata.wiki.kernel.org/index.php/Libata_error_messages

Libata error messages are usually affiliated with cabling or power supply problems.

 

  PHYRdyChg = PhyRdy signal changed state

  LinkSeq        = Link state machine error occurred

  TrStaTrns      = Transport layer state transition error occurred

 

I know you've replaced much of the hardware...  Did you ever replace the power supply?  What exact make/model power supply are you using?

 

Joe L.

I will post supply info after parity check.  I believe it's a 650w cooler master. 

Never had that error in the log before.

Tower kernel: Clocksource tsc unstable (delta = 62500681 ns)

 

I had these errors I turned off AMD cool n quiet and C1E wait states and the issue went away. Certainly worth a try.

I have been working away to isolate this problem.  I disabled cool n quiet and don't see a ce1 option in my bios...still have unstable clocksourse.

 

I also picked up a new corsair CMPSU650TX power supply so that I am using a top quality power supply...this did not resolve parity errors, but the power connections are nice and tight now with no splitters.

 

I also bought high end shielded Silverston cp03 sata cables...I am running a nocorrect sync right now and am showing 1 error so far.  I am not sure if this is due to the fact that bad parity checks were previously written and need to be corrected.

I added a rosewill pci sata card and after running a parity check, it looks like the errors are gone.  My transfer rates are now about half what they were before with the onboard sata.  I am really starting to question unraid as a system for a non programmer/i.t. person.  2 Weeks of hardware troubleshooting to resolve 8 bits of bad data is starting to seem a little impractical.  It would be great if there was some sort of diagnostic suite built into the user interface.

 

I will transfer some data and check parity again a few times and see if the problem is really resolved.

The other alternatives aren't any better, in normal windows you wouldn't even know about the errors. Dare I say in any file server situation (WHS or RAID-5) you'd be presented with the same errors without any further cause as to what was cuasing the issues. Your statement becomes "I am really starting to question any fileserver as a system for a non programmer/i.t. person". Which is quite true.

 

This is also why some on here opt for the server grade equipment instead of the cheapest path possible. Our time is worth more than the savings on normal consumer grade parts.

 

Also, for diagnostics, what you really need is ability to diagnose every portion of the chain separately. This is not readily available via software alone. It becomes quite costly. You need to diagnose the following parts isolated from others: CPU, RAM, PSU, HDD, Sata Cable, Sata Controller. They are all influenced by each other.

I definitely won't not skimp on components in the future.  What more can I do?  I bought a recommended power supply, hard drives, highest quality sata cables, supported sata card and motherboard.  All I can do at this point is pray.

 

Is the goal of Lime-tech to appeal to non-I.T. consumers?  If so, I think the main weakness is the lack of u.i. for troubleshooting.

It would be very attractive to the lay person if there was an interface that gave you options such as:

 

1)Test memory and give results(run memtest)

2)Test drive health(smart tests)

3)Check that data is being written correctly to each drive(MD5 check...I think)

 

 

It would also be helpful to create a more streamlined list of "Supported Hardware"  This should be located on the lime-tech/support/unraid server installation, not buried.

 

And for that matter a troubleshooting flow chart would be great.

I think that's a good suggestion.  It should be easy to be tackled by the community once unRAID 5.x is released if it has better plugin / module addon support for the management console.

 

In the meantime, I haven't checked the WIKI to see what's listed or shown in the troubleshooting section. Although when issues occur in the forums, the answers are nearly always running through the checklist.

 

I think Limetech appeals to the non-I.T. consumers via their pre-packaged systems.

 

As for the streamlined supported list of hardware, that's a bit of a huge burden given two factors -- the speed of linux developments and the speed of hardware developments or changes. For the most part the hardware supported boils down to "If Linux supports it, and it's in the kernel provided in the unRAID distro then it's supported."

 

As in all things in life, there's bound to be mistakes and defects. Even if you had built an exact mirror of the systems sold by Lime-tech, you should could have encountered this unfortunate issue. A more streamlined supported hardware list won't help with this issue.

 

 

 

 

I actually planned on buying the 12 bay package originally, but the website has been showing it to be sold out for months.

 

In terms of supported hardware list...  I think a lot of people would be saved some headache if they are warned about problems with certain gigabit mb's, WD ears drives, etc. before they build their system.

 

I think in terms of my own troubleshooting ability, the problem is in figuring out which drives/ports are experiencing problems.  With no evidence in the syslog or smart report, I need another way to find out which drives are getting parity errors.  It sounds like writing some data to individual drives and then doing md5 checks could answer that.  I have not been able to figure out how to do that from the wiki or forum.  Maybe I should start a new thread asking for a md5 check for each drive procedure.

 

All in all, I really think unraid is great...I just need to figure out a way to approach troubleshooting more effectively.  So far I have had problems with the usb drive, a bad sata cable and now a bad sata port.  I have been planning to build a new unraid to backup my current one, so I guess that will be a good test for me.

Well part of the problem could be if you use a program like md5deep, that the drive where you write the md5sums to could be the one corrupting the data, so then you need md5sum of the initial in-memory md5sum stored elsewhere ... 

 

It's a real pita, and to be honest with you, this issue would have taken me nearly as long to troubleshoot as well. The only aspect that would have helped me out is having a secondary system to swap all the parts individually into.

 

Giving this a bit more thought but not enough to see if this is feasible, this is one area where unRAID could be improved by Limetech. It might require a bit more memory per drive slot in use. When a parity mismatch occurs, you keep the original raw information read from all the drives. Then a reread of all the raw information occurs, perhaps upto 3 times to check for consistency. The new raw information is then compared to the old raw information. If the old and new raw information read is not identical, then an error message can be generated on the syslog and displayed on the management console indicating which drive's information mismatched. If the old and new raw information matches but the checksum information does not, then it depends if the parity check is correct or nocorrect as to how it handles it.

 

I hope the above would become standard in all unRAID versions and not just TEST versions.

Giving this a bit more thought but not enough to see if this is feasible, this is one area where unRAID could be improved by Limetech. It might require a bit more memory per drive slot in use. When a parity mismatch occurs, you keep the original raw information read from all the drives. Then a reread of all the raw information occurs, perhaps upto 3 times to check for consistency. The new raw information is then compared to the old raw information. If the old and new raw information read is not identical, then an error message can be generated on the syslog and displayed on the management console indicating which drive's information mismatched. If the old and new raw information matches but the checksum information does not, then it depends if the parity check is correct or nocorrect as to how it handles it.

Excellent idea.

 

As far as I know, it should not be terribly difficult either.

In the same way, a more "verbose" error output showing the bytes read from each of the drives involved at the address where the parity error was detected might be toggled, perhaps with something like

/root/mdcmd check debug

command (It does not exist, but it could, and if invoked this way, the more detailed output would be sent to the log)

 

Joe L.

That sounds like it would really help out with the troubleshooting process