I have 2 servers, and added 5 4TB Western Digital Reds to each. I did 2 preclear passes on them and they passed with no errors of any kind. I then added them to array, and did a parity rebuild, and everything was fine. Now today I decided to do a parity check. One server had one of them red ball about 3% in, the other about 6% in. They were write errors (caused by SCSI being frozen or something like that) and the drives were taken offline. I rebooted both servers, tried again. Same thing. I checked the SMART on these drives, there were no issues.

 

Here's the kicker: The drives had no critical data on them that I couldn't replace, so I initiated new drive config, re-assigned all drives in both servers, and did a parity rebuild. No problems or errors. Then I started a parity check again, both servers errored out again.

 

Could this possibly be related to my customized tunables?

(md_num_stripes): 3328

(md_write_limit): 1536

(md_sync_window): 1536

 

This did not happen with these settings before adding these drives though. Sadly, I didn't grab a log (i'll do that tomorrow). My theory is, these settings were fine with less drives, but with more drives low memory is running out causing the server to hiccup. Would this be a reasonable conclusion?

I'm not sure what may be causing this, but I'm certain it is NOT the tunables.    Those settings use very little RAM, and you have FAR more than you need to support that.    My settings are very close to that -- indeed mine actually use a bit more memory than yours do.

 

Sounds more like loose or insecure cables, or perhaps a poorly seated drive if it's in a hot-swap cage.    But it's really strange you're having the same error on two different servers with two different drives.    It's also possible, of course, that you actually have defective drives.

 

Do you have a spare drive you could try?    If not, swap the drives between the systems and then do another parity sync on each.

 

I'm not sure what may be causing this, but I'm certain it is NOT the tunables.    Those settings use very little RAM, and you have FAR more than you need to support that.    My settings are very close to that -- indeed mine actually use a bit more memory than yours do.

 

Sounds more like loose or insecure cables, or perhaps a poorly seated drive if it's in a hot-swap cage.    But it's really strange you're having the same error on two different servers with two different drives.    It's also possible, of course, that you actually have defective drives.

 

Do you have a spare drive you could try?    If not, swap the drives between the systems and then do another parity sync on each.

 

It's a full SAS to SAS setup, the cables lock securely in place. The power cables power bays of 4 drives each, so i'd only assume it would randomly affect other drives too. I would also think if it was cable related it'd also happen during rebuilds and not just checks, but i've seen stranger.  I have no spares, but I can always rotate the drives to a different bays and see if the errors follow the drives or the bays.

 

I see my SAS cards got a firmware update 7 months ago, about the same time 4TB drives arrived. Maybe I should try that?

 

EDIT:

Here's some more with SAS2LPs that experienced roughly the same until firmware updating:

http://lime-technology.com/forum/index.php?topic=29052.0

http://lime-technology.com/forum/index.php?topic=26719.0

Flashed them, was painless and fast. Seems to have fixed it, will report back if it happens again.

 

EDIT: See below.

I was just typing a reply suggesting you try the firmware update when your post appeared with the results => glad it fixed it 

 

Interesting that it doesn't seem to impact the other drives ... but it may have had something to do with writes when a lot of drives were involved -- and a parity sync is about the only case where that really happens (and all of the writes are to the parity drive).    Those writes may have somehow "gone wrong" ... thus resulting in the read errors when you did a subsequent check.

 

But that's purely a guess -- in reality it doesn't matter, since it's fixed !!

 

Happened again. Drive18 on both servers. Any ideas? Had to zip it since it was too big. The fact it happens on both servers at the same time, and the same drive, says it's software related. Other people reported the firmware update fixed it...

 

I'm going to go back to default tunables and see if it fixes it, however sync is going to be slower so it's not really an accurate test. It seems like my SAS cards are timing out to me.

syslog.zip

Agree it sounds like a timing issue.

 

At least it's not (or at least certainly doesn't seem to be) the drives 

 

Any suggestions? I tried flashing again and it still happens. It almost seems like their not fully compatible with the 4TB reds. This time it was drive16.

 

I called Supermicro and they simply said they don't support 4TB drives. This is a load of bull, many people have them around here.

 

I called Supermicro and they simply said they don't support 4TB drives.

 

!!!  Definitely a load of BS  ==> for one thing, the limiting factor is whether or not they support > 2TB

Given that the answer to that is Yes (which it clearly is), then they'll work with 3TB, 4TB, 5TB, 6TB, ...

 

Is this always happening on drives connected to the same card?    It's possible (although it seems unlikely) that you have an issue with one of your cards.  Since you have quite a few, you could swap it with a different card and see if the problem goes away.

 

I called Supermicro and they simply said they don't support 4TB drives.

 

!!!  Definitely a load of BS  ==> for one thing, the limiting factor is whether or not they support > 2TB

Given that the answer to that is Yes (which it clearly is), then they'll work with 3TB, 4TB, 5TB, 6TB, ...

 

Is this always happening on drives connected to the same card?    It's possible (although it seems unlikely) that you have an issue with one of your cards.  Since you have quite a few, you could swap it with a different card and see if the problem goes away.

 

I told him exactly the same, if it supports 3TB, then it supports 4TB. It's happening to drives on the card connected to the x4 slot on each server. The other 2 cards in each server are on x8 slots. It would have to be 2 faulty cards, or there's some conflict with running 3 of them at different speeds?

 

I set my tunables back to default and running checks.. however it is VERY slow with default unRAID tunables. I'm going 37MB/s and 49MB/s, looking at almost a day and a half on the slower one. I'm not really sure what this proves, as there's much lower load on the card. I'm just out of ideas and really stressed at the moment, so much money spent on these servers just to increase parity check speed and now it looks like I might need to throttle them down due to DEFINITE firmware issue on these cards.

 

The fact there are people saying they flashed their card with the same BIOS twice to fix it means Supermicros flashing process is flaky. Sadly, no matter how many times I flash it doesn't fix it.

It's happening to drives on the card connected to the x4 slot on each server.

 

Bingo!  That at least gives a fairly good clue about what's happening here.    The cards are SUPPOSED to run just fine (albeit slower) on an x4 slot; but I'm not surprised that you're getting a timing-related issue when doing so.

 

Since your motherboard plus 2 of the MV8's has enough ports that you shouldn't even need the 3rd MV8, I'd do exactly that => i.e. remove the 3rd MV8 from the x4 slot; and connect the drives that were connected to it to the motherboard ports.    Then try your parity check again.

 

It's happening to drives on the card connected to the x4 slot on each server.

 

Bingo!  That at least gives a fairly good clue about what's happening here.    The cards are SUPPOSED to run just fine (albeit slower) on an x4 slot; but I'm not surprised that you're getting a timing-related issue when doing so.

 

Since your motherboard plus 2 of the MV8's has enough ports that you shouldn't even need the 3rd MV8, I'd do exactly that => i.e. remove the 3rd MV8 from the x4 slot; and connect the drives that were connected to it to the motherboard ports.    Then try your parity check again.

 

I don't have any SAS to SATA cables... I would need to buy some to test that. My motherboard also only has 6 SATA ports.

 

EDIT: It just happened again on default tunables.

I don't have any SAS to SATA cables... I would need to buy some to test that.

 

Or just pop 4 drives out of the front and set them where you can get power & data cables to them.

 

 

My motherboard also only has 6 SATA ports.

 

 

Yes, but 6 plus 2 8-port cards = 22 ports ... and your config only shows 20 drives per server (18 data plus parity plus cache)

Is the current config different than that?

 

I don't have any SAS to SATA cables... I would need to buy some to test that.

 

Or just pop 4 drives out of the front and set them where you can get power & data cables to them.

 

 

My motherboard also only has 6 SATA ports.

 

 

Yes, but 6 plus 2 8-port cards = 22 ports ... and your config only shows 20 drives per server (18 data plus parity plus cache)

Is the current config different than that?

 

I plan on adding more, is there anyway to just force them all to run in x4 mode? It's still fast enough for what I need.

I'm not aware of any way to do that.  You could always replace one of the cards with the x4 version of the card, and use it in the x4 slot.

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16816101358

I'm not aware of any way to do that.  You could always replace one of the cards with the x4 version of the card, and use it in the x4 slot.

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16816101358

 

I really have no funds at the moment

 

I'm not sure that's the cause as it seems other people are running 3x of these cards on these motherboards. It also doesn't happen during fresh parity rebuilds, which is also very high I/O usage. There's got to be a way to force the cards or the PCI-E slots to all run in x4 mode, or i'd think anyway.

 

Going to see if I can borrow a friend's SAS to SATA cable this weekend... but honestly Supermicro needs to fix this because that would mean it's a firmware issue.

Borrowing some cables is an excellent idea =>  that will let you confirm that this is limited to an issue with the card running at x4.    What's not clear is whether it's because the card is running at x4; or because it's running at a different speed than the other cards.

 

I doubt that there's any such option => but look carefully in the system BIOS to see if there might in fact be some capability to throttle the slots to x4.

 

The only other way I can think of to force that would be to somehow cover the extra pins with insulation (e.g. electrical tape) and plug the card in so it "thinks" it's in an x4 slot 

I may have fixed it, or at least i'm 25 minutes further into a parity check than i've ever gotten.

 

Looking at this picture the PCI-E slots are:

x8 (shared?)

x8 (shared?)

x4 (not shared?)

x4 (shared?)

 

I had them in the top 3 slots, and I moved the card down to the 2nd x4 slot. I'm trying to figure out how this bandwidth is shared. I found one person say the x8 slots are shared with the bottom x4 slot, and the top x4 slot (which is exclusive to the X9SCM) has it's own bandwidth. This could be why my parity syncs were having timing issues, however it should still work correctly, and is a major selling part of this motherboard over it's inferior version that doesn't have that slot.

 

One thing i've noticed after moving the card down, is my parity checks are about 10% slower. If the bandwidth is shared with the x8 slots it kind of makes sense, but still not sure if that's the case. Anyone know?

 

Either way, i'm going to call Supermicro tomorrow, and tell them the top x4 slot has some issues on both my Motherboards. My guess is it's a problem with all X9SCMs with 3x SAS2LPs, and would probably require a BIOS update or some setting to be changed. This hardware is all from the same company, so you'd think they'd test this.

Looking at Intel's block diagram for the C204/6 chipset (your board uses the C204), it shows that the chipset supports 28 PCIe lanes, 20 from the CPU and 8 from the chipset PCH.    I suspect that on your board the "shared" slots all come from the CPU lanes; and the "not shared" are from the PCH.    Apparently the timing differences are enough to cause problems -- but by moving your card so that all 3 cards are on CPU-based PCIe lanes, everything's working fine now.

 

I'm surprised the hub-based lanes have this problem -- but at least you've isolated this and can (apparently) resolve it by simply using the other x4 slot.

 

Confirming that moving the card to the bottom x4 slot fixed it in both of my servers. The parity checks completed overnight. I'm very surprised no one else has experienced this around here. These motherboards and SAS cards are rather common.

These motherboards and SAS cards are rather common.

 

They are indeed commonly used -- but I'm not sure that many folks have 3 of them.    And if any of those that do already had one of the PCIe x4 cards, I'm sure they used it in the x4 slot; which probably also mitigates the issue.