3 M1015's in ESXi not showing all drives in unRAID

[BobPhoenix]

Just thought of something else.  You might try downloading ESXi again it is possible your ISO was corrupt.  Doubt this will make a difference but is something else to try.  If re downloading the ISO and re installing ESXi get the Cougar/Sandy Bridge back in ESXi passthrough then that will at least take care of that problem.

[RockDawg]

I don't have another mobo with more than one PCI-E slot and I only had one other PCI-E card to try so I tried it in each of the four slots and only when it is installed in the last slot does ESXi recognoize the PCI-E bus.  In every other slot, it shows as "Generic PCI/PCI Bridge".  As a last ditch effort, I am going to go pick up another X9SCM-F-O mobo and see it it helps.  I have my doubts that it is the problem since it works fine in straight unRAID, but I don't know what else to try.

 

Bob - I did try re-downloading and the re-installing ESXi with no luck and days ago I tried a "clean" unRAID install and it didn't help either.  Thanks for all your help.  Hopefully the new mobo fixes the issue.

[BobPhoenix]

I would think a new MB would fix the ESXi display problem since your tests prove that your Sandy Bridge ports don't display in ESXi correctly.  It might also fix the M1015 boot problem if you are lucky.  Hope it works.

[RockDawg]

As my luck would have it, it appears as though my n ew board has a 1.x BIOS.  Whn I power it on I get no display and 4 beeps.  A quick internet search says that means incompatible processor.  Man I'm striking out big time here.

[BobPhoenix]

I can still trade you mine if you want.

[RockDawg]

Sure.  If you want.  That was your X9SCM that showed the Sandy Bridge in ESXi, right?

 

EDIT:  I'm going to have to wait to talk to SuperBiiz on Monday.  That's where I purchased the first board and I just their return policy and I may be a bit screwed.  It says you can only request an exchange for refund within 5 days after the delivery date.  After that they say they will only do a replacement.  It's been 8 days today, so if they are rigid on that I'll have to send my newest board to the place where I bought it to get it's refund.  Hopefully SuperBizz if more understanding after I talk to them Monday.  Not a very liberal return policy they have!

[BobPhoenix]

Sure.  If you want.  That was your X9SCM that showed the Sandy Bridge in ESXi, right?

Correct. 

 

One thing I've noticed since I switched boxes to use the Tyan board I had is that the IPMI port appears to not remain connected unless a monitor is plugged in to the VGA port - not sure why yet as I just noticed this.  It was connected to a KVM switch in the basement before so I never noticed this before and I haven't found a bios setting that corrects it yet.  It is weird because when connected it works flawlessly.  The frames per second increase instead of being at zero with a frozen screen when monitor isn't connected.  I never would have noticed this but I have it sitting in my living room - was hoping to install the CPU and memory from it on my new Tyan board.  If you still want to knowing this (maybe you can find a setting to fix or maybe it is just my network - switch or cable) then I will swap with you.  Just PM where to send the board and I will package it up and send it out on Monday.  I think you should keep everything you got except the MB and I will do the same.  So I wouldn't be sending the IO shield for instance.

[RockDawg]

The monitor issue shouldn't be a problem for me as I imagine I will always have a monitor or KVM attached to it, but I appreciate the disclosure.  I am awaiting an RMA from SuperBiiz, but it looks like the will issue me a refund on the original 2.0a board.  In that case I should be able to trade boards with you. 

 

Was it your Supermicro board that showed the PCI-E slots correctly with the 3 M1015's in ESXi?

[BobPhoenix]

The monitor issue shouldn't be a problem for me as I imagine I will always have a monitor or KVM attached to it, but I appreciate the disclosure.  I am awaiting an RMA from SuperBiiz, but it looks like the will issue me a refund on the original 2.0a board.  In that case I should be able to trade boards with you. 

 

Was it your Supermicro board that showed the PCI-E slots correctly with the 3 M1015's in ESXi?

Yes.  The board above came with 2.0 bios that I upgraded to 2.0a.  It was the one I showed in ESXi and has 3 M1015s connected and unRAID VM installed on it.  It was the one the log came from above. 

 

I'm not going to need the trade for a while (2 months at least) so you can test your M1015s out and see if you want to keep it or not before you would send yours to me.  I'm going to be using the CPU and memory off of it on the Tyan I have RMA'd to Newegg.  I may end up retiring it permanently as I may go with an i5/i7 CPU and desktop board in place of the X9SCM board and another Xeon.

[RockDawg]

I appreciate the gesture.  As soon as I get a refund RMA from SuperBiiz, I can send you my board.  I would just love a board that works with ESXi.

[BobPhoenix]

I appreciate the gesture.  As soon as I get a refund RMA from SuperBiiz, I can send you my board.  I would just love a board that works with ESXi.

What ever you want to do.  I'll just leave it assembled for now and take it apart when I need to send it.

[RockDawg]

Someone on HardOCP has the exact same problem with the same hardware and he speculates that it isn't the motherboard, but rather the processor.  He and I are both running the E3-1230 V2 and he is assuming that ESXi is saying "Unknown PCI/PCI Bridge" because our processors are Ivy Bridge, not Sandy Bridge and he thinks it's too new for ESXi.

 

Bob - It looks like you are running The Sandy Bridge E3-1230 and your screenshot of the ESXi passthrough devices shows the PCI-E ports as Sandy Bridge.  Maybe that's why yours works and mine doesn't?

 

I also notice that Graywolf's setup that works appears to be running a Sandy Bridge processor as well.  Maybe the poster is on to something?

[BobPhoenix]

Someone on HardOCP has the exact same problem with the same hardware and he speculates that it isn't the motherboard, but rather the processor.  He and I are both running the E3-1230 V2 and he is assuming that ESXi is saying "Unknown PCI/PCI Bridge" because our processors are Ivy Bridge, not Sandy Bridge and he thinks it's too new for ESXi.

 

Bob - It looks like you are running The Sandy Bridge E3-1230 and your screenshot of the ESXi passthrough devices shows the PCI-E ports as Sandy Bridge.  Maybe that's why yours works and mine doesn't?

 

I also notice that Graywolf's setup that works appears to be running a Sandy Bridge processor as well.  Maybe the poster is on to something?

Sounds reasonable to me because I do have a V1 E3-1230.  Especially since it was showing as working in unRAID stand alone for you.  That leads me to think you have at least one BAD M1015.  You might try reflashing the firmware again as the card itself might be ok since turning off the M1015s boot roms in the X9SCM bios seems to let you boot without problems.  As you saw I have them all turned on and ESXi works fine for me.

[BobPhoenix]

Also in 1-2+ months I might be able to test the theory as I might get an Ivy Bridge i7 to replace my q6600 HTPC in the living room.  I can put it on the X9SCM to test first.  Not quite the same as the E3-1230 V2 but it might show the same results.  If it does that would be confirmation.  My other option is a E3-1230 V2 on X9SCM in the Q6600 box in living room to do recording (and couple more VMs) and a Foxcon A3700 for playback.  That would be a valid test.  Still wavering on what I want.

[RockDawg]

Someone on HardOCP has the exact same problem with the same hardware and he speculates that it isn't the motherboard, but rather the processor.  He and I are both running the E3-1230 V2 and he is assuming that ESXi is saying "Unknown PCI/PCI Bridge" because our processors are Ivy Bridge, not Sandy Bridge and he thinks it's too new for ESXi.

 

Bob - It looks like you are running The Sandy Bridge E3-1230 and your screenshot of the ESXi passthrough devices shows the PCI-E ports as Sandy Bridge.  Maybe that's why yours works and mine doesn't?

 

I also notice that Graywolf's setup that works appears to be running a Sandy Bridge processor as well.  Maybe the poster is on to something?

Sounds reasonable to me because I do have a V1 E3-1230.  Especially since it was showing as working in unRAID stand alone for you.  That leads me to think you have at least one BAD M1015.  You might try reflashing the firmware again as the card itself might be ok since turning off the M1015s boot roms in the X9SCM bios seems to let you boot without problems.  As you saw I have them all turned on and ESXi works fine for me.

 

You could be right about the bad card.  I'll play around with that when I get home tonight.

[RockDawg]

Also in 1-2+ months I might be able to test the theory as I might get an Ivy Bridge i7 to replace my q6600 HTPC in the living room.  I can put it on the X9SCM to test first.  Not quite the same as the E3-1230 V2 but it might show the same results.  If it does that would be confirmation.  My other option is a E3-1230 V2 on X9SCM in the Q6600 box in living room to do recording (and couple more VMs) and a Foxcon A3700 for playback.  That would be a valid test.  Still wavering on what I want.

 

I really hope that I have my problems solved within 1-2 months, but you never know.

[BobPhoenix]

Also in 1-2+ months I might be able to test the theory as I might get an Ivy Bridge i7 to replace my q6600 HTPC in the living room.  I can put it on the X9SCM to test first.  Not quite the same as the E3-1230 V2 but it might show the same results.  If it does that would be confirmation.  My other option is a E3-1230 V2 on X9SCM in the Q6600 box in living room to do recording (and couple more VMs) and a Foxcon A3700 for playback.  That would be a valid test.  Still wavering on what I want.

I really hope that I have my problems solved within 1-2 months, but you never know.

Yah.  I would do it now if money allowed me to.  Need to win lottery.

 

[BetaQuasi]

If it truly is an ivy bridge issue, maybe try the just-released vSphere 5.1 and see how that goes?

[RockDawg]

Unreal! - Beta, thanks for the heads up.  After installing 5.1 ESXi now recognizes the PCI-E ports as Ivy Bridge.  But of course that didn't solve my problem.  I am still getting the handshake/doorbell fault!  I just don't understand it.  I have a hard time getting two cards to work.  The only way they work is if they are both in the 8x slots.  If they are both in 4x slots or one is in the 8x and the other in the 4x, I get the error.  If I install only one card in either 4x slot, it works.  I haven't managed to try every combination, but I've tried a bunch.

 

I have no idea where to go from here.  I suppose I could still replace the mobo, but I just don't feel like it's the problem.  In fact I don't feel like there is any hardware problem since it all works in bare metal unRAID.  At the same time, others have the same hardware working with ESXi.  >

[RockDawg]

I'm thinking I am just going to purchase an Intel RES2SV240 SAS expander and run it off one of my M1015's and sell 2 of my M1015's to offset the cost.  I see others using this card with a M1015 in ESXi without issue so I am hoping it will work for me too. The luck I've been having lately though makes me less than 100% confident. We shall see.

[BobPhoenix]

I'm thinking I am just going to purchase an Intel RES2SV240 SAS expander and run it off one of my M1015's and sell 2 of my M1015's to offset the cost.  I see others using this card with a M1015 in ESXi without issue so I am hoping it will work for me too. The luck I've been having lately though makes me less than 100% confident. We shall see.

The only problem I had with my RES2SV240 SAS expander was when I connected two cables from my M1015 to it.  When I went to one cable only I haven't had any problems anymore.  That also had the advantage of giving me 24 drives from one M1015 instead of just 16.  For me I wanted this because I wanted more PCIe slots available for passthrough to other VMs.  I know others have posted they have two cables from a M1015 connected successfully to a RES2SV240.  I never experimented with seeing if a different port on the RES2SV240 would work.  And before you ask I don't remember exactly what the errors were anymore.  I believe I was getting failed writes when writing to drives.  I don't think I had problems with parity checks just writing data.  It may also have been only when transfering data from one drive to another in the array.  I do remember it crashing unRAID or at least crashing MC (Midnight Commander) on the ESXi display of the VM console.  I certainly like mine (RES2SV240) for the reasons stated.  I may even get a third next year sometime.

[RockDawg]

I PM'd you.

[RockDawg]

I am about ready to give up here!  So I decided to make the trek to my not-so-local computer store as they said they had one Intel RES2SV240NC on their shelf.  I get home, connect everything and all my drives now show up in unRAID  , BUT... my parity checks are atrociously slow.  I'm seeing like 2-3 MB/sec whereas before I was getting ~70 MB/sec.  I get these slow speeds whether I'm running unRAID through ESXi or bare metal.  I switched between my three M1015's to see if that made a difference and it didn't.

 

I'm starting to think that the god's do not want me to build this system.  I have never had so many issues with one system in my life!

 

I've attached the log.  Hopefully one of you wizards can point me in the right direction.

syslog.txt

[BobPhoenix]

Not really sure what to look for in your log.  Didn't see anything when I looked but that isn't saying much.

 

I have my RES2SV240 expanders on my Tyan S5512 and a Supermicro X7SBE currently.  And yes the parity check speeds are lower than running direct to controllers but certainly not that low.  My check speeds on my X7SBE are slow but that is most likely because I fried (60c temps for a parity check) over half the drives.  They still average > 30MB/s after they were cooked.

 

Since I am retiring my X9SCM - I have a Tyan S5512 coming.  I could send you mine to test with and see if you have a bad MB or if it is something else.  When I had my X9SCM installed and operating with my RES2SV240 I got 60+MB/s as I remember it.  That was also in a VM on ESXi 5.0 with the modified driver for the Intel 82579LM nic.

 

[RockDawg]

Thanks for loaning me your X9SCM-F Bob!  I only had a few moments to install it and play with it, but so far it acts exactly the same as mine.  With one M1015 and the Intel SAS expander, I get dog slow parity rebuild (~4MB/sec).  Tonight I'll try throwing in my three M1015's without the SAS expander and see what happens.  If it still acts the same as my board then I don't what to do.