April 6, 201511 yr I then rebuild disk20 without issue onto a spare disk in a different slot in the Norco 4224 and the server runs fine until I do another parity check. 4 times this has happened. I've included a syslog that shows the last 2 times. Since a new disk is used, and a new slot, I can't understand the cause of this issue. Thankfully I have a backup of disk20, but what is going on here? (Unraid 5.05 on a Supermicro X9scm with 2 1015 disk controllers) syslog.zip
April 6, 201511 yr Author Wow you are fast. I just attached the syslog. Yes, I thought cabling too, but a different disk and a different slot for the rebuild should have eliminated that. Don't you think? I'll post a smart report tomorrow when I am at that location.
April 6, 201511 yr Yes, a different disk and different slot certainly eliminates cabling as the issue ... UNLESS the slots you're using are all on the same breakout cable you're using and it's going to more than one of the slots you're using in the Norco. Try a slot that's for sure on a different breakout cable.
April 6, 201511 yr It might also be related to sharing a common power cable, particularly if splitters are in use?
April 6, 201511 yr Author No power splitters are in use. The last rebuild moved to a completely different row on the 4224 so a different breakout would have been used.
April 6, 201511 yr Really strange ... is there anything unique about disk 20? [i.e. different size than the others; different rpm; hybrid vs. standard drive; SAS vs. SATA; etc.]
April 6, 201511 yr I'm going to vote on power too. What is the model number of your power supply? How many drives does one cable from your power supply power? How are your back panes wired for power? Are you using the secondary power plugs on the Norco? How many drives are you using? What is the model of the drives that have had write failures? How many drives does one cable from your power supply power? Everything is pointing to something that is in common with the two failed parity checks. The only thing in common is the power supply and eliminating that as a cause is fairly easy, so why not start there.
April 6, 201511 yr Certainly COULD be power => but seems unlikely, since it's always the same disk #, regardless of which slot it's in. I confess, however, that it's a STRANGE issue. Question: Did disk #20 have any pending reallocations when this all started? I know there can be some "interesting" issues when you try to rebuild a disk with pending sectors ... perhaps the disk is never getting correctly rebuilt.
April 6, 201511 yr In my experience, red balls like this are normally caused by a momentary loss of power or data signal that causes a drive to drop offline. I have seen systemic cabling issues that lead to such failures in my own servers. A data cable that does not snug to a drive just right is all it takes to generate a hiccup in the billions of I/Os. Understanding steps to isolate to date, I would hook the drive directly to a motherboard port with a factory fresh SATA cable and see if it continues to drop offline and red ball.
April 6, 201511 yr Brian => He's having this problem with DIFFERENT drives; connected to DIFFERENT ports; using DIFFERENT cables The only common feature of the problem is it's always Disk #20 that fails !!
April 6, 201511 yr Brian => He's having this problem with DIFFERENT drives; connected to DIFFERENT ports; using DIFFERENT cables The only common feature of the problem is it's always Disk #20 that fails !! I thought he moved disk20 from one row to another. I am pretty sure it is the same disk.
April 7, 201511 yr Brian => He's having this problem with DIFFERENT drives; connected to DIFFERENT ports; using DIFFERENT cables The only common feature of the problem is it's always Disk #20 that fails !! I thought he moved disk20 from one row to another. I am pretty sure it is the same disk. I gather you didn't read this ... :) ... Yes, I thought cabling too, but a different disk and a different slot for the rebuild should have eliminated that. Don't you think?
April 7, 201511 yr Author Brian => He's having this problem with DIFFERENT drives; connected to DIFFERENT ports; using DIFFERENT cables The only common feature of the problem is it's always Disk #20 that fails !! I thought he moved disk20 from one row to another. I am pretty sure it is the same disk. No, it is different disks. A total of 4 different 3tb drives have been used in this role and all have red balled. Power supply is an Antec Neo Eco 620 that has served well for over 2 years powering this array of 13 (2tb and 3tb) disks. I have had some other strange things going on with this server lately as well. Every once in a while the beeper on the motherboard will go crazy while the server is running with a DEE / DAH - DEE / DAH sound. This is the CPU overheat warning on the X9SCM motherboard. In the last 6 months, it will come on at boot every time, and randomly thereafter (the cpu fan is running normally and no dust is built up). The server has a Xeon 1220 and 4 gb RAM. I have just pulled all the plugins and add ins off the server and rebooted and am rebuilding onto the 5th disk now. I will then do a parity check tomorrow naked (no plugins) to eliminate that possibility.
April 7, 201511 yr Brian => He's having this problem with DIFFERENT drives; connected to DIFFERENT ports; using DIFFERENT cables The only common feature of the problem is it's always Disk #20 that fails !! I thought he moved disk20 from one row to another. I am pretty sure it is the same disk. No, it is different disks. A total of 4 different 3tb drives have been used in this role and all have red balled. Power supply is an Antec Neo Eco 620 that has served well for over 2 years powering this array of 13 (2tb and 3tb) disks. I have had some other strange things going on with this server lately as well. Every once in a while the beeper on the motherboard will go crazy while the server is running with a DEE / DAH - DEE / DAH sound. This is the CPU overheat warning on the X9SCM motherboard. In the last 6 months, it will come on at boot every time, and randomly thereafter (the cpu fan is running normally and no dust is built up). The server has a Xeon 1220 and 4 gb RAM. I have just pulled all the plugins and add ins off the server and rebooted and am rebuilding onto the 5th disk now. I will then do a parity check tomorrow naked (no plugins) to eliminate that possibility. Sorry skimmed over that. Seems like a hardware malfunction. Either that or a problem with 20 drive arrays. My backup server has that many disks but it is not parity protected.
April 7, 201511 yr If I did my sleuthing right, the Neo 620 has two stands with molex connectors placing up to 12 drives on each strand. I use a 650 watt Corsair on my Norco 4224 and modified it so that each cable from the power supply feeds 8 drives. Now I'm not convinced that that is your problem, but it is good practice to minimize voltage drop. With only 13 drives in use this should not be an issue unless most of the drives are on a single cable. I would reseat the molex connectors since your server has been in use for a long time. In fact I'd reseat the SFF-8087 cables too just for good measure. I'm betting on a hardware problem still. I once had a really strange problem that was caused by a faulty hard disk that was not showing any smart errors, but my server would lock up periodically when that drive was part of the array. That took a while to track down. Good luck
April 7, 201511 yr Author A new drive has successfully rebuilt as Drive 20, and now we try a parity sync and see what happens. Will it red ball? I will try your reseating suggestions, and will order a new power supply for good measure once we have a the results from the parity check.
Archived
This topic is now archived and is closed to further replies.