3 M1015's in ESXi not showing all drives in unRAID

[Johnm]

"NC" means it was a No Cable version. it was shipped without cables.

although... I have an "NC" that game retail with cables. I'm guessing intel has placed NC versions into retail boxes when stock is low. Cards are identical.

[BobPhoenix]

I'm begining to think your other problems all stem from your Ivy Bridge CPU. 

 

Here is my odyssey with Ivy Bridge over the past week:

 

I have MBs from 3 different makers.  2x SuperMicro X9SCM-F (1 on loan other still bricked), Tyan S5512GM4NR, Tyan S5512GM2NR, Intel DH77EB.  The first X9SCM came with 1.0 bios and worked with AVer Media Duet card.  Upgraded the bios to 2.0 (Ivy Bridge compatible) and now Duet was not recognized.  The S5512GM2NR has 1.05 bios and is NOT Ivy Bridge compatible without a bios upgrade and currently has two Duet's in it working fine.  The S5512GM4NR has 2.02 bios which IS compatible with Ivy Bridge CPUs.  The Duet is not recognized as even being plugged into the MB.  The DH77EB has an i7 3770S Ivy Bridge CPU in it.  In order to install an OS on it I had to upgrade the bios to the latest.  The DH77EB would boot but the OS installation would fail.  I tried ESXi and Windows neither would install although the MB made it through post just fine.  Upgraded the bios to the latest and then either OS would install.  ESXi 5.0 did not recognize much after the install.  The Cougar chipset was not recognized nor the USB or video chipsets so just installed Windows and went with plan B - so far - still haven't installed Virtual Box yet.  The Duet is not recognized on the DH77EB either.  One nice thing about the DH77EB is that it shows you which slots are occupied in the bios screen and the Duet was not recognized in any of the 1x PCIe slots (don't know about the x16 but that is reserved for M1015).  The X9SCM and the Tyan are more cryptic about it but I believe you can infer which slots are occupied when you see the OPROMs listed in the bios.  When the bios is Ivy Bridge compatible the Duet does not show up with an OPROM.  I think that there are problems with the Ivy Bridge implementation since I've also had problems with the PCI slot on my S5512GM4NR that don't show up on the S5512GM2NR.  I have a USB 2.0 card and a Hauppauge HVR-1600 which were giving me problems in the PCI slot but work elsewhere.  If it wasn't for the power savings over the Sandy Bridge versions I would say stay away from Ivy Bridge CPUs and bios's.  Don't think it is the CPU particularly but the bios implementations that are at fault here since I am having problems on boards that have SandyBridge CPUs but bios's that are compatible with Ivy Bridge.

[RockDawg]

Bob - I take it you were running a Sandy Bridge processor in all those scenarios?  I don't where to go from here at this point.  I don't see SuperBiiz allowing me return the CPU and exchange it for a V1 due to an incompatibility.  I only have a few days left to return anything to them as it is.  I hate to go spend more money on a Xeon 1230 V1 CPU when that may not even fix the problem either.  :'(

[BobPhoenix]

Bob - I take it you were running a Sandy Bridge processor in all those scenarios?  I don't where to go from here at this point.  I don't see SuperBiiz allowing me return the CPU and exchange it for a V1 due to an incompatibility.  I only have a few days left to return anything to them as it is.  I hate to go spend more money on a Xeon 1230 V1 CPU when that may not even fix the problem either.  :'(

Yes I was. I agree you've spent plenty now. 

 

And for completeness to thread - PM's back and forth have further info. 

[Ford Prefect]

I'm begining to think your other problems all stem from your Ivy Bridge CPU. 

 

...FWIW, I was just about to post the same...re-reading the thread, besides a software problem with ESXi the only difference is the CPU.

 

I am also planning for a new build with lots of items on passthrough. This thread convinced me that IvyBridge is not for me.

Although the next option is even more expensive...I see a X9SRL-F  and E5 coming up....

[BobPhoenix]

I'm begining to think your other problems all stem from your Ivy Bridge CPU. 

 

...FWIW, I was just about to post the same...re-reading the thread, besides a software problem with ESXi the only difference is the CPU.

 

I am also planning for a new build with lots of items on passthrough. This thread convinced me that IvyBridge is not for me.

Although the next option is even more expensive...I see a X9SRL-F  and E5 coming up....

Actually I think it is the bios that is the problem.  If the bios is Ivy Bridge compatible regardless of the CPU used weird things start happening.  Hopefully that can be cured by a bios upgrade.  Sounds reasonable to me anyway since a bios upgrade is what started my problems.  I did nothing else but upgraded the bios from 1.0 to 2.0 on my X9SCM-F and it no longer recognized my AVer Media Duet tuner card.  That same card is not recognized on any Ivy Bridge compatible bios I've tried so far.  If I could downgrade without bricking the MB I would have no problem buying a 1155 board.  But since all of the MB manufactures are making there boards Ivy Bridge compatible I'm with you on waiting for the next option.

[Ford Prefect]

....did you contact Supermicro with that issue / your findings regarding BIOS?

 

...also, maybe Supermicro has a way on how to downgrade the BIOS.

[BobPhoenix]

....did you contact Supermicro with that issue / your findings regarding BIOS?

 

...also, maybe Supermicro has a way on how to downgrade the BIOS.

Not yet - it wasn't until yesterday evening I really thought about it at length.  I have asked for help from Tyan to downgrade the bios but - haven't had a CPU to try it with the SuperMicro.  I have an Ivy Bridge CPU on my Intel DH77EB so I CAN'T change the bios on that (need the cooler CPU).  I'm going to be doing further testing on the PCI slots on the S5512GM4NR and the  S5512GM2NR.  The S5512GM4NR has had problems in passthrough with a USB card and a Tuner card with it's Ivy Bridge bios so far.  I have a PCI disk controller (for it's port multiplier) on order that I will be trying for my Windows VM at the end of the week and I hope to try all three cards on the PCI port of the S5512GM2NR with it's Sandy Bridge bios then as well.  Then I can report my finding to Tyan.  I am expecting no problems on the S5512GM2NR and I'm hoping for no problems with the disk controller with the S5512GM4NR.  If I can unbrick my X9SCM to a 1.0 bios and I get my 2.0a bios board back from loan I can test with actual SuperMicro boards and the AVer Media Duet again for SuperMicro.

[RockDawg]

I'll have to look again, but I thought a few of the people who had working systems with three M1015's were running the Supermicro board with the 2.0a BIOS, but with the Sandy Bridge processor.  I could be wrong on that though.

[BobPhoenix]

I'll have to look again, but I thought a few of the people who had working systems with three M1015's were running the Supermicro board with the 2.0a BIOS, but with the Sandy Bridge processor.  I could be wrong on that though.

Yes three M1015's and Sandy Bridge CPU's worked on the X9SCM I loaned you.  My problem was the Duet tuner card just dropped out of site with the 2.0a bios.  unRAID worked fine even with 3 M1015's on passthrough.  I had two M1015's passed through.  One to a Windows VM and another to unRAID VM.  The unRAID VM was connected to a SAS expander.  What I'm curious about is whether that is still true with an actual Ivy Bridge CPU based on the problems your having.  My Ivy Bridge CPU still drops the Duet but I don't have a spare that I can test on my other boards.  If I get another Sandy Bridge CPU and I unbrick my older X9SCM with a 1.0 bios I can run lots of tests to confirm that the Ivy Bridge bios drops the Duet.  When I get the PCI port multiplier I can test the PCI slot on my Tyan boards to see if the Ivy Bridge bios affects it.  I'm confident it DOES affect a PCI tuner and PCI USB 2.0 card but I will test that too when my recording schedules permit.

[BetaQuasi]

My build has 3 m1015's but is the older 1.1a BIOS (the geek in me wants to upgrade it but if it ain't broke....)

 

Sandy bridge CPU though so yeah...

 

Only other difference is that I don't use plop (boot straight from vmdk) but I struggle to see how that'd make a difference tbh.

[Johnm]

Honestly. From day one of this topic, I was scared it was an "ivy bridge" thing since it is pcie 3x. Every additional post I see starts to reinforce that.., 

 

I hope that is not the case and it's just a bad m1015 (not that I wish the op broken hardware). I say that because that would limit a lot of functionality in the system.

 

We ran into a strange ivy bridge issue at work. One of our high end users has dual apple thunderbolt monitors on his rig (non apple pc) The only way to get both monitors to work was put a thunderbolt hard drive between them on the chain.  After we found the solution, other said that worked for them too.

 

 

Sent from my iPhone using Tapatalk

[BobPhoenix]

Honestly. From day one of this topic, I was scared it was an "ivy bridge" thing since it is pcie 3x. Every additional post I see starts to reinforce that.., 

 

I hope that is not the case and it's just a bad m1015 (not that I wish the op broken hardware). I say that because that would limit a lot of functionality in the system.

 

We ran into a strange ivy bridge issue at work. One of our high end users has dual apple thunderbolt monitors on his rig (non apple pc) The only way to get both monitors to work was put a thunderbolt hard drive between them on the chain.  After we found the solution, other said that worked for them too.

 

 

Sent from my iPhone using Tapatalk

Just remember my problems are not with an Ivy Bridge CPU but with an Ivy Bridge compatible bios.  I have Sandy Bridge CPUs on a 2.0x PCIe bus boards that have an Ivy Bridge bios firmware and have cards dropping out that worked with earlier Sandy Bridge only compatible bios firmwares.  That is from Tyan and SuperMicro so not just a single manufacturer.  The bios may have code in it for 3.0x PCIe that is causing my problem I suppose.  But it certainly isn't needed since the boards are not 3.0x PCIe compatible themselves.

[RockDawg]

Honestly. From day one of this topic, I was scared it was an "ivy bridge" thing since it is pcie 3x. Every additional post I see starts to reinforce that.., 

 

I hope that is not the case and it's just a bad m1015 (not that I wish the op broken hardware). I say that because that would limit a lot of functionality in the system.

 

We ran into a strange ivy bridge issue at work. One of our high end users has dual apple thunderbolt monitors on his rig (non apple pc) The only way to get both monitors to work was put a thunderbolt hard drive between them on the chain.  After we found the solution, other said that worked for them too.

 

 

Sent from my iPhone using Tapatalk

 

I do have one questionable M1015 that will only detect 7 drives even when 8 are connected.  It's BIOS always shows slot 5 as empty.  From the moment I noticed that I thought for sure that was my problem.  I have since changed my mind even though it's an obvious red flag.  The reason I've ruled it out is that if I use that card in a two card configuration, it works fine sans one drive missing.  I have also witnessed on more than one occasion the handshake/doorbell error in a two card configuration that did not include the questionable card. Also, all three cards are running just fine in the exact same system with bare metal unRAID and I'm running and using it every day.  My particular problem seems to be isolated to ESXi.

 

[RockDawg]

Finally success!!!  I was able to borrow a Xeon E3-1230 Sandy Bridge processor from a coworker and it worked!  So it would seem that the Xeon E3-1230 V2 Ivy Bridge processor was the source of all my problems.  The unRAID VM now boots up without error using all three of my M1015's and parity checks so far have been averaging ~75 MB/sec which is exactly what I experienced in bare metal unRAID.  Also, I no longer get the MPT BIOS fault during boot if I enable OPROM on my PCI-E slots.  So everything seems solved now.  Thank goodness!!  Anyone want to buy a 1230 V2 Ivy Bridge processor  ?

[BetaQuasi]

Great news mate, glad you are having the experience that most of the rest of us are!  Enjoy.

[prtsmgr]

Rockdawg,

 

Glad to hear you got it working.

In reference to an earlier part of this thread, has this changed?

The reason I ask is because my syslog looks the same as you had posted earlier.

Thanks

 

I have noticed one difference in the syslogs between straight unRAID and unRAID through ESXi:

 

Straight unRAID

Sep  3 09:17:29 Tower kernel: mpt2sas2: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (16608336 kB)

 

unRAID through ESXi

Sep  2 02:18:43 Tower kernel: mpt2sas2: 32 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (2074648 kB)

 

Why wouldn't they both be 64 BIT?

[RockDawg]

It's still showing as 32 bit.  No matter what I tried it always showed as 32-bit in in the unRAID VM.  Conversely, everytime I booted bare metal unRAID it shows as 64-bit.  Not sure if it really matters.  So far, now that I've swapped the CPU, everything is performing as I would expect.

[Johnm]

That was my fear.. the new PCIe structure/commands in the ivy bridge.

 

Bad news and good news all in one.

I'm glad you solved the issue.

 

[RockDawg]

Lesson learned.  The next time I venture into uncharted territory with software picky on hardware like ESXi is, I'll be sure to build with the exact hardware of other systems running it successfully.  That's what I should have done, but I didn't figure the CPU would make much difference.  I assumed the motherboard and RAM would be the ones that I need to be sure about.

 

Hopefully I'll be able to sell the Ivy Bridge processor without taking too big of a loss on it, but right now I'm just glad my system is now running as I had hoped.

[BobPhoenix]

Lesson learned.  The next time I venture into uncharted territory with software picky on hardware like ESXi is, I'll be sure to build with the exact hardware of other systems running it successfully.  That's what I should have done, but I didn't figure the CPU would make much difference.  I assumed the motherboard and RAM would be the ones that I need to be sure about.

 

Hopefully I'll be able to sell the Ivy Bridge processor without taking too big of a loss on it, but right now I'm just glad my system is now running as I had hoped.

I would have figured that the PCIe 3.0 spec would account for difference in order to be backwards compatible.  Usually when something changes (like PCI going from 33Mhz to 66Mhz) and requires a change that invalidates earlier specs it is rendered impossible to plug in an incompatible card (a 33Mhz card will only plug into a 33Mhz slot but a 66Mhz card can usually plug into both) into the slot.  Obviously the PCIe 3.0 spec designers didn't learn from Intels early Sandy Bridge chipset mistake that caused a recall.  This certainly won't do that but I still think they should have done a better job on backwards compatiblity.  Maybe our problems can be solved with a bios update but from the looks of it I'm going to be stuck on an OLD bios that I can't update since the MB manufacturers are not going to want to maintain two different bios paths - one for Ivy Bridge and one for Sandy Bridge.

[RockDawg]

Now that I'm running and playing with everything, I have a couple questions:

 

1.  I have one drive that I'm running ESXi on and storing my VM's on.  I would like to add a second drive and make it accessible to Ubuntu just for SAB to initially download files to.  If I do "add storage" under Configuration>Storage will that format the drive and allow me to use it for RDM or will that make it only for use as a datastore?

 

EDIT: I should mention that I've been assuming the the datastore is just for VM storage.  Is that the case?  All I want is to add the second drive just like if I were to add it to a physical Ubuntu machine.

 

2.  I'm wondering if my Norco 4224 has a bad backplane.  MY top backplae I have connected to the motherboard's SATA controller via a reverse breakout cable.  Initially I purchased a Seagate Momentus XT 750GB hybrid drive for ESXi and the datastore.  Having read how the newer 4224's had a history of bad backplanes, I tried the drive in all 4 slots of the top backplane and it worked in all of them.  I took it that the backplane was good.  But now that is the only drive that works in that backplane.  If I install any of my 3 spare spinners oin any of the four slots, they are never detected by my mobo's BIOS.  If I connect them straight to the motherboard, all three work fine.  Strangely though, the hybrid is always detected in any of the slots.  It sure seems like the backplane, but why would it always have no problem with the hybrid drive?

[BobPhoenix]

Now that I'm running and playing with everything, I have a couple questions:

 

1.  I have one drive that I'm running ESXi on and storing my VM's on.  I would like to add a second drive and make it accessible to Ubuntu just for SAB to initially download files to.  If I do "add storage" under Configuration>Storage will that format the drive and allow me to use it for RDM or will that make it only for use as a datastore?

 

EDIT: I should mention that I've been assuming the the datastore is just for VM storage.  Is that the case?  All I want is to add the second drive just like if I were to add it to a physical Ubuntu machine.

You don't add it to storage that makes it a datastore drive.  Putty into ESXi and use Johnm's instructions to create the RDM drive.  Basically find it with the "ls" command he mentions.  Then grab the VML file name for the DISK not the partition.  Then use the vmkfstools command he mentions to create the RDM vmdk file.  I originally had my WHSv1 data drives that way.  The boot drive was a regular virtual drive stored from space from the datastore drive but the other data drives were 4 1TB laptop drives in a 4 in 1 2.5 drive cage.

 

2.  I'm wondering if my Norco 4224 has a bad backplane.  MY top backplae I have connected to the motherboard's SATA controller via a reverse breakout cable.  Initially I purchased a Seagate Momentus XT 750GB hybrid drive for ESXi and the datastore.  Having read how the newer 4224's had a history of bad backplanes, I tried the drive in all 4 slots of the top backplane and it worked in all of them.  I took it that the backplane was good.  But now that is the only drive that works in that backplane.  If I install any of my 3 spare spinners oin any of the four slots, they are never detected by my mobo's BIOS.  If I connect them straight to the motherboard, all three work fine.  Strangely though, the hybrid is always detected in any of the slots.  It sure seems like the backplane, but why would it always have no problem with the hybrid drive?

Sorry have no idea on this.  Someone else may have an idea.

[jbrodriguez]

i have a x9scm-f, an ivy bridge e3-1240 chip running esxi 5.0u1 and when i installed a third m1015, had the same 01h error at boot ... it wouldn't recognize an intel nic i put in the 4th pciex slot either.

 

i was just about to upgrade to bios 2.0a, after much reading ... when i remembered also reading something about changing bios settings ... what i specifically did was disable the com ports (i think there were 4 of them).

 

after booting .. voila! ... the third m1015 was recognized AND the intel nic was recognized as well.

 

hope it helps someone else down the road.

[RockDawg]

Glad you got it working.  I never had a problem with ESXi recognizing the M1015's.  It always did.  My problem was that the second or third M1015 would always throw an error during the unRAID boot causing the connected drives to not show up in unRAID.