June 17, 201313 yr Author Super strange that it's rock solid sans parity drive. Not necessarily all that strange. When you're computing and/or checking parity is probably the ONLY time you're actually using all 12 drives at once. [A drive rebuild also does this; but clearly you aren't doing any rebuilds without good parity] I guess what I mean is that if it were a cabling problem, the drives with poor cabling would have problems with or without the parity drive... right? I definitely use all drives on the share with regularity...
June 17, 201313 yr Super strange that it's rock solid sans parity drive. Not necessarily all that strange. When you're computing and/or checking parity is probably the ONLY time you're actually using all 12 drives at once. [A drive rebuild also does this; but clearly you aren't doing any rebuilds without good parity] I guess what I mean is that if it were a cabling problem, the drives with poor cabling would have problems with or without the parity drive... right? I definitely use all drives on the share with regularity... The only time that all drives are read concurrently is when parity is in place and you're computing and/or checking parity or during a disk rebuild.
June 17, 201313 yr I definitely use all drives on the share with regularity... Using all the drives is not the same as actively reading from all of them at once. Unless you had a LOT of simultaneous reads from multiple systems, each of which was from a different drive, you would NOT be stressing them nearly as much as a parity check. Doesn't mean that's absolutely the issue -- but it IS strange that that's the only time you see problems, which could easily mean you have a power-related issue. r.e. "... I guess what I mean is that if it were a cabling problem, the drives with poor cabling would have problems with or without the parity drive... right? ..." ==> Not necessarily. If the cables were bad, then you're correct. But if the problem is you're having a "noise" issue on the cables, then not necessarily. Do you have your SATA cables "bunched" (tied together) ?? If so, unbundle them !! And how many drives do you have on a single cable from the PSU?
June 17, 201313 yr Author I definitely use all drives on the share with regularity... Using all the drives is not the same as actively reading from all of them at once. Unless you had a LOT of simultaneous reads from multiple systems, each of which was from a different drive, you would NOT be stressing them nearly as much as a parity check. Doesn't mean that's absolutely the issue -- but it IS strange that that's the only time you see problems, which could easily mean you have a power-related issue. r.e. "... I guess what I mean is that if it were a cabling problem, the drives with poor cabling would have problems with or without the parity drive... right? ..." ==> Not necessarily. If the cables were bad, then you're correct. But if the problem is you're having a "noise" issue on the cables, then not necessarily. Do you have your SATA cables "bunched" (tied together) ?? If so, unbundle them !! And how many drives do you have on a single cable from the PSU? Interesting. I do have all of the SATA data cables bunched together. I'll unbundle them. PSU cables are spread out as evenly as possible. I think I have 3 per cable and maybe only one splitter. I'll have to look. Been about a month since I've opened it up.
June 17, 201313 yr Bunching all the SATA cables together can induce electrical crosstalk when a lot of them are active at the same time. This COULD be your issue. You'll find out soon enough -- just unbunch them and then try generating parity
June 17, 201313 yr Author Bunching all the SATA cables together can induce electrical crosstalk when a lot of them are active at the same time. This COULD be your issue. You'll find out soon enough -- just unbunch them and then try generating parity As you know, I'll try anything to get this sucker working. I've spent too much money and too many hours to let it sit unprotected forever.
June 17, 201313 yr Bunching all the SATA cables together can induce electrical crosstalk when a lot of them are active at the same time. This COULD be your issue. You'll find out soon enough -- just unbunch them and then try generating parity As you know, I'll try anything to get this sucker working. I've spent too much money and too many hours to let it sit unprotected forever. Agree. And I think there's a good chance that's the issue, as it's much more likely that a data "glitch" is what's happening rather than a power glitch. There shouldn't be any real doubt about power, since the system clearly has enough power to handle the spinup requirements, and the operating load is well below that. But bunched non-shielded SATA cables can definitely induce crosstalk, which can cause signaling issues that result in data errors. And it would only be likely to happen when a LOT of the drives were active at once. Sound familiar? Like EXACTLY what happens when you're computing parity
June 25, 201313 yr Author Hey y'all. I swapped out the parity drive, unbundled my SATA data cables, and re-seated every connection. It's currently computing parity. Very interesting behavior going on currently... the parity rebuild is running and is at 47% done. Everything was running smoothly with no errors on any of my disks until (and immediately after) I re-enabled a couple of VMWare guests that are running on the NFS datastore that unRAID is hosting. This is very suspect to me. One of my NFS share disks has 6429 errors (and climbing, rapidly) but is still "green ball" and the parity rebuild continues. Again, this didn't happen until I started using the NFS share... errors are as follows: Jun 24 19:35:48 hoarder kernel: md: disk8 read error, sector=61752 Jun 24 19:35:48 hoarder kernel: md: disk8 read error, sector=61760 Jun 24 19:35:48 hoarder kernel: md: disk8 read error, sector=61768 Jun 24 19:35:48 hoarder kernel: md: disk8 read error, sector=61776 Jun 24 19:35:48 hoarder kernel: md: disk8 read error, sector=61784 Jun 24 19:35:48 hoarder kernel: sd 10:0:0:0: [sdi] Unhandled error code Jun 24 19:35:48 hoarder kernel: sd 10:0:0:0: [sdi] Result: hostbyte=0x04 driverbyte=0x00 Jun 24 19:35:48 hoarder kernel: sd 10:0:0:0: [sdi] CDB: cdb[0]=0x28: 28 00 00 00 f1 a0 00 00 08 00 Jun 24 19:35:48 hoarder kernel: end_request: I/O error, dev sdi, sector 61856 Should I be concerned? (Just to re-iterate, these errors never occurred prior to computing parity and not until I started using the NFS share... albeit sparingly). Thanks!!
June 25, 201313 yr The disk is having read errors. The disk will not be taken out of service unless a write to it fails. The error count on unRAID main should indicate the errors. Do any disks show errors on unRAID main? Post a SMART report for the disk.
June 25, 201313 yr Author The second drive down, disk 8, was the one that started causing issues as soon as I started using the NFS share and got up to a quarter million errors before Disk 7 failed... Disk 7 is also a part of the NFS share and apparently is also having problems. Lastly, disk 9 decided to log 6 thousand+ errors. I'll post SMART reports on these three HDDs but I'm 99.9% sure that the HDDs are not the issue. I've run SMART tests against them before with no problems and the drives do not report a single error when parity isn't being computed. I guess I need to buckle down and run that memtest now. Server is headless in the basement so I've been avoiding it because it's a pain... I'm exhausted though. This is SO frustrating. I just want it to work. Been MONTHS now of nothing but issues. EDIT: To get the array back into a working state I stopped it and all three of the drives listed above (notice that two are "green ball") have dropped out of the system and are showing as missing. Same freaking jacked up behavior as always.
June 25, 201313 yr Author I don't know guys... thinking ahead I'm wondering if it is, in fact, my PCI Express x1 4 Port SATA cards: http://www.newegg.com/Product/Product.aspx?Item=N82E16816124064 Too good to be true? What is a decent replacement? I only have the following expansion ports: 1 - PCI Express 3.0 x16 2 - PCI Express x1 Doesn't seem to be very much out there in the way of 4 port PCI Express x1 Controller Cards. All of them seem to require x4 or x8. Is this not doable? I need to get 8 drives on those three expansion ports. I've been looking through the hardware compatibility but I can't seem to find anything that fits the bill... Please advise! Thanks.
June 25, 201313 yr I don't know guys... thinking ahead I'm wondering if it is, in fact, my PCI Express x1 4 Port SATA cards: http://www.newegg.com/Product/Product.aspx?Item=N82E16816124064 Too good to be true? What is a decent replacement? I only have the following expansion ports: 1 - PCI Express 3.0 x16 2 - PCI Express x1 Doesn't seem to be very much out there in the way of 4 port PCI Express x1 Controller Cards. All of them seem to require x4 or x8. Is this not doable? I need to get 8 drives on those three expansion ports. I've been looking through the hardware compatibility but I can't seem to find anything that fits the bill... Please advise! Thanks. Looks like you have an x16 listed above - is it available - nothing currently in it? If so then any of the 8 port cards commonly used here would work like the SAS2LP, M1015 or the older SASLP.
June 25, 201313 yr You have an x16 slot, so I'd simply use a SuperMicro SAS2LP-MV8 card in that. That will give you 8 ports on an add-in card in a slot with ample bandwidth for the ports. Connecting 4 modern drives to an x1 slot is a bottleneck to the performance of the drives. It should NOT cause errors consistently, like you're seeing ... it should just slow things down; but in any event it'd a good idea to reduce the #1 of drives you're connecting to that slot (2 drives would be fine). There is one fairly simple modification you could make to significantly reduce your drive count: Replace your last 5 drives with a 2TB drive. You'd actually gain a bit of space; and have 4 fewer drives to connect. Then you'd only need to connect 4 drives to your 3 expansion ports ... something you could easily do with 2 drives each on x1 slots; or all 4 on the x16 slot.
June 25, 201313 yr Author Thanks y'all. I'm going to run my memory test tonight but I thought I'd reach out and ask about this in advance (in case the memory test doesn't come back with issues). Thanks again.
June 26, 201313 yr Author Memory test came back clean... Updated MOBO BIOS (was acting strangely, taking forever to boot, etc.) and decided to run another parity re-sync while I wait for the SAS2LP-MV8 to show up today. We'll see!!
July 2, 201313 yr Author Well, thank you all for your help. Seems the new controller solved (at least temporarily) my problems. Parity Re-Sync and checks have worked flawlessly since replacing with the SAS2LP-MV8. Wonder if the old controllers just couldn't handle all of the simultaneous reads or something. Who knows. Regardless, it seems fixed now. Thanks again everyone.
Archived
This topic is now archived and is closed to further replies.