Hey all,

 

I upgraded my hardware yesterday from what's in my sig below to:  SuperMicro X10SLH-F, Xeon E3-1231v3, and 32GB RAM.

 

Everything came up fine and seems to be working. I decided to cruise the logs and found some errors I don't know what to do with. These seem to repeat a handful of times for another handful of drives (I believe the ones on the onboard SATA):

 

Jun  5 20:27:34 spock kernel: ata4.00: exception Emask 0x10 SAct 0x8000000 SErr 0x280100 action 0x6 frozen
Jun  5 20:27:34 spock kernel: ata4.00: irq_stat 0x09000000, interface fatal error
Jun  5 20:27:34 spock kernel: ata4: SError: { UnrecovData 10B8B BadCRC }
Jun  5 20:27:34 spock kernel: ata4.00: failed command: READ FPDMA QUEUED
Jun  5 20:27:34 spock kernel: ata4.00: cmd 60/08:d8:08:00:00/00:00:00:00:00/40 tag 27 ncq 4096 in
Jun  5 20:27:34 spock kernel:         res 40/00:dc:08:00:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Jun  5 20:27:34 spock kernel: ata4.00: status: { DRDY }
Jun  5 20:27:34 spock kernel: ata4: hard resetting link
Jun  5 20:27:34 spock kernel: ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jun  5 20:27:34 spock kernel: ata4.00: configured for UDMA/133
Jun  5 20:27:34 spock kernel: ata4: EH complete

 

In my Norco, the bottom row backplane is (and was) connected to the onboard SATA ports. My parity drive resides in that set. My cache drive is also on the onboard SATA. Using a total of 5 ports (0-3 & 5)

 

I've attached the zip file form the diagnostics.

 

Any help is greatly appreciated.

 

Thank you!

 

 

EDIT: Changed subject to include Norco RPC-4220 as it is looking like that is the culprit.

spock-diagnostics-20160606-1328.zip

CRC errors are usually caused by a bad SATA cable, it could also be a bad enclosure/SATA port, check these for disks 18, 19 and parity.

Thanks for the quick reply.

 

Hmm.. ok.. I can check the seating. I suppose I'll have to order a mini-sas reverse breakout. Those 3 drives are all on the onboard SATA going to one of the Norco backplanes.  I wish it was as simple as swapping SATA cables. I have a ton of those.

 

Any thoughts on why that would creep up during a hardware swap? That cable was only muddled with at the SATA end and just to unplug and plug post board swap. How likely could it be the motherboard is faulty rather than the cable? That is the biggest variable change since yesterday. 

 

EDIT: Doh, I completely missed where you said " it could also be a bad enclosure/SATA port". Well.. hell..  Do I get a new cable or RMA the board? Bah.

 

Also, what is the danger of these errors until I get a new cable to test? Should I take the sever down?

 

Thanks again!

 

 

Start by replacing the SATA cables as they are the most likely cause.

 

A few crc errors like you're getting are usually not a big deal, in more extreme cases they will considerably slow down the array and can even make disks drop offline.

I'll order a replacement reverse breakout.

 

I had a thought that I wanna bounce in here. I have a 4 port SATA PCIe controller sitting in a box doing nothing. I also have a spare PCIe slot.  Would it make sense to slap it in, put drives 17-19 and parity? My thinking is this would test if the cable or the backplane is bad. However, I don't have a good understanding of how intermittent these kinds of errors are and when they might show up. Worth trying to isolate the issue?

 

Thanks again.

It worth trying, then do a parity check, these type of errors almost always show up during one.

Added in the 4 port controller and swapped the drives to that. Parity check finished without any errors popping up in the log file.

 

Is there more testing I could do? Or am I safe to assume it's the motherboard and need to RMA it? Is there a chance I've mucked up something in the BIOS?

 

Thanks!

Before you go down that route, it's definitely a good idea to first reseat all your sata cables with the onboard controller since you figured out the cables aren't bad. It's much easier to do that than go down the RMA route

Easily done..  ok. But do I run another parity check at that point? Is there something else I can do to test?

 

I appreciate all the feedback.

 

 

It could be bios related too. What mode are the sata ports set?

dmacias,

 

SATA Controller(s): Enabled

SATA Mode: AHCI

SATA Frozen: Enabled    <-- No idea what that means

SATA RAID Option ROM/UEFI Driver: Enabled  <-- Same

 

So.. I took the card out and hooked the drives back up to the MoBo. When I was looking at those values in the BIOS I saw that some of the ports were listed as "Empty" while others had the correct drive identification in them. I checked the new logs after boot (which I've attached here), and sure enough those errors are back.

 

I've put the controller back in the meantime, but I'm guessing this points to bad ports, yes? I'm guessing I should start the RMA process.

 

Any other ideas before I take that step?

 

Thanks so much.

spock-diagnostics-20160607-1843.zip

SATA RAID Option ROM/UEFI Driver: Enabled  <-- Same

My guess on this option is that it provides RAID capabilities to the MB ports.  The UEFI driver reference probably means it is UEFI compatible and would allow BOOTING from the RAID if you were booting with UEFI.

Back again. So I got the replacement board and slapped it in earlier today.

 

Upon checking the BIOS the first time, the drive I had connected to Port 3 was not being recognized. What's the likelihood that I would get another board with bad SATA ports? 0, 1, 2, 4 & 5 are all reporting fine.

 

Is there something I'm missing, or is this just bad luck again?

 

Also, I'm getting a message (before unRAID tries to boot) that the drive in Port 2 has failed SMART, however the SMART status is showing as fine when I run a test in unRAID.

 

Any additional thoughts much appreciated.

Hey all,

 

Ok.. well I RMAd the 2nd board and slapped in a new one. Again, port 3 in the bios shows no drive connected. The log shows 1 drive with CRC errors (I'll attach it here), however..  because it is port 3, and I don't like coincidences, I'm thinking board 2 wasn't bad at all.

 

But what I'm confused about is (as I said earlier in the thread): I slapped in a 4 port SATA controller and hooked drives 17-20 to it. Did a parity check and had no errors. That led me to believe that the backplane and the reverse breakout cable were fine and as a result I never ordered a replacement cable.

 

However, now I'm thinking either the cable or the backplane (or I guess the drive?) is actually bad or flaky. But why didn't that show when I had it on a separate controller?

 

Any help and guidance appreciated as always.

 

Thanks!

 

spock-diagnostics-20160717-1833.zip

Use the process of elimination, I'd guess backplane (or cable) as the most likely.

Yeah.. Doing that now. Port is definitely good.. a myriad of other drives show up fine when connected to it via a separate cable.

 

I've ordered a replacement reverse breakout cable which should show tomorrow. I hope that's it. I have no idea where to get a replacement Norco backplane.

 

Still confused why the existing cable/backplane work fine when connected to a PCIe SATA card..  but hey, if the new cable solves it, I'll let it go.

 

Thanks!

 

 

Yeah.. Doing that now. Port is definitely good.. a myriad of other drives show up fine when connected to it via a separate cable.

 

I've ordered a replacement reverse breakout cable which should show tomorrow. I hope that's it. I have no idea where to get a replacement Norco backplane.

 

Still confused why the existing cable/backplane work fine when connected to a PCIe SATA card..  but hey, if the new cable solves it, I'll let it go.

 

Thanks!

I got back planes for my 4224 from here: http://www.ipcdirect.net/4224-rpc-4224-sas-or-sata-to-mini-sas-storage-enclosure-backplane-bp-001/

Hi again,

 

I can't seem to figure this out. To recap:

 

Norco 4220, Supermicro X10SLH-F.

 

When connecting the bottom backplane to the motherboard using a Norco reverse breakout cable (plugging cable ports 0-3 into motherboard ports 0-3) port 3 shows empty in the BIOS and receive CRC errors in the unRAID log.

 

If I plug cable port 3 into any other motherboard port, the port reports empty and errors in unRAID.

If I replace the cable, port 3 (or whichever cable port 3 is connected to) reports empty.

If I connect either cable to a different backplane, port 3 reports empty.

If I move drives around in the backplanes, port 3 reports empty.

If I connect a drive directly to any motherboard port, the drives are reported correctly.

 

I do not believe the motherboard to have bad ports as directly connected drives are visible.

It is unlikely that I have two cables with a bad port 3 portion.

It is unlikely that I have multiple backplanes with a bad 4th slot port.

It is unlikely that all my drives are bad.

 

Does anyone have an idea on what I can do here? Am I missing something?

 

Thanks

Well.

 

The plot thickens. Hopefully this will be helpful to someone down the line.

 

I've been swapping drives and cables and going all mad scientist.

 

It turns out that if I put a lower capacity/power drive in the "bad" slot, the BIOS recognizes the drive just fine. I moved a 4TB WD Red out and put a 1.5TB Green in. I checked with Norco, and I apparently have a really old version of their backplanes in my 4220. (SAS400PM V16). I'm considering taking this opportunity to upgrade to a 4224 rather than buy new backplanes.

 

Whichever happens, I'll update this and mark as solved when I'm back running at full.

Received new case (4224). All drives visible. Seemed to be the backplanes in the 4220