3 M1015's in ESXi not showing all drives in unRAID

[graywolf]

Not sure. How do I find out without rebooting?

 

what version of unRaid are you running? 

I'm running 5b11

 

[RockDawg]

I don't know of any way to check the BIOS without rebooting.  I'm running 5b12a.  I suppose I could try downgrading when I get home tonight.

 

If it's not too much trouble, could you check your syslog and see if you mpt2sas0, mpt2sas1 or mpt2sas2 are showing as 64 or 32 bit like in my example above?

[graywolf]

 

 

root@Tower:/root -> cat /var/log/syslog | grep mpt2sas

Sep  6 11:30:24 Tower kernel: mpt2sas version 06.100.00.00 loaded

Sep  6 11:30:24 Tower kernel: mpt2sas 0000:0b:00.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19

Sep  6 11:30:24 Tower kernel: mpt2sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (6227608 kB)

Sep  6 11:30:24 Tower kernel: mpt2sas 0000:0b:00.0: irq 72 for MSI/MSI-X

Sep  6 11:30:24 Tower kernel: mpt2sas0: PCI-MSI-X enabled: IRQ 72

Sep  6 11:30:24 Tower kernel: mpt2sas0: iomem(0x00000000d2500000), mapped(0xf8558000), size(16384)

Sep  6 11:30:24 Tower kernel: mpt2sas0: ioport(0x0000000000005000), size(256)

Sep  6 11:30:24 Tower kernel: mpt2sas0: Allocated physical memory: size(1103 kB)

Sep  6 11:30:24 Tower kernel: mpt2sas0: Current Controller Queue Depth(473), Max Controller Queue Depth(3432)

Sep  6 11:30:24 Tower kernel: mpt2sas0: Scatter Gather Elements per IO(128)

Sep  6 11:30:24 Tower kernel: mpt2sas0: LSISAS2008: FWVersion(14.00.00.00), ChipRevision(0x03), BiosVersion(07.27.00.00)

Sep  6 11:30:24 Tower kernel: mpt2sas0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)

Sep  6 11:30:24 Tower kernel: mpt2sas0: sending port enable !!

Sep  6 11:30:24 Tower kernel: mpt2sas0: port enable: SUCCESS

Sep  6 11:30:24 Tower kernel: mpt2sas 0000:13:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16

Sep  6 11:30:24 Tower kernel: mpt2sas1: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (6227608 kB)

Sep  6 11:30:24 Tower kernel: mpt2sas 0000:13:00.0: irq 73 for MSI/MSI-X

Sep  6 11:30:24 Tower kernel: mpt2sas1: PCI-MSI-X enabled: IRQ 73

Sep  6 11:30:24 Tower kernel: mpt2sas1: iomem(0x00000000d2600000), mapped(0xf85b0000), size(16384)

Sep  6 11:30:24 Tower kernel: mpt2sas1: ioport(0x0000000000006000), size(256)

Sep  6 11:30:24 Tower kernel: mpt2sas0: host_add: handle(0x0004), sas_addr(0x500605b002c5dce3), phys(

Sep  6 11:30:24 Tower kernel: mpt2sas1: Allocated physical memory: size(1103 kB)

Sep  6 11:30:24 Tower kernel: mpt2sas1: Current Controller Queue Depth(473), Max Controller Queue Depth(3432)

Sep  6 11:30:24 Tower kernel: mpt2sas1: Scatter Gather Elements per IO(128)

Sep  6 11:30:24 Tower kernel: mpt2sas1: LSISAS2008: FWVersion(14.00.00.00), ChipRevision(0x03), BiosVersion(07.27.00.00)

Sep  6 11:30:24 Tower kernel: mpt2sas1: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)

Sep  6 11:30:24 Tower kernel: mpt2sas1: sending port enable !!

Sep  6 11:30:24 Tower kernel: mpt2sas1: port enable: SUCCESS

Sep  6 11:30:24 Tower kernel: mpt2sas 0000:1b:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17

Sep  6 11:30:24 Tower kernel: mpt2sas2: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (6227608 kB)

Sep  6 11:30:24 Tower kernel: mpt2sas 0000:1b:00.0: irq 74 for MSI/MSI-X

Sep  6 11:30:24 Tower kernel: mpt2sas2: PCI-MSI-X enabled: IRQ 74

Sep  6 11:30:24 Tower kernel: mpt2sas2: iomem(0x00000000d2700000), mapped(0xf85d0000), size(16384)

Sep  6 11:30:24 Tower kernel: mpt2sas2: ioport(0x0000000000007000), size(256)

Sep  6 11:30:24 Tower kernel: mpt2sas1: host_add: handle(0x0001), sas_addr(0x500605b003eed420), phys(

Sep  6 11:30:24 Tower kernel: mpt2sas2: Allocated physical memory: size(1103 kB)

Sep  6 11:30:24 Tower kernel: mpt2sas2: Current Controller Queue Depth(473), Max Controller Queue Depth(3432)

Sep  6 11:30:24 Tower kernel: mpt2sas2: Scatter Gather Elements per IO(128)

Sep  6 11:30:24 Tower kernel: mpt2sas2: LSISAS2008: FWVersion(14.00.00.00), ChipRevision(0x03), BiosVersion(07.27.00.00)

Sep  6 11:30:24 Tower kernel: mpt2sas2: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)

Sep  6 11:30:24 Tower kernel: mpt2sas2: sending port enable !!

Sep  6 11:30:24 Tower kernel: mpt2sas2: port enable: SUCCESS

Sep  6 11:30:24 Tower kernel: mpt2sas2: host_add: handle(0x0001), sas_addr(0x500605b003e77200), phys(

root@Tower:/root ->

 

[RockDawg]

Thanks a lot!  I'm not sure why mine is showing 32 bit through ESXi.  I have no idea if that's part of the problem, but it's weird it's only happening on my unRAID/ESXi setup and not yours or my straight unRAID which both work.

[RockDawg]

Someone over on HardOCP viewed my config and noticed something that could be causing a problem.  Where you select devices for passthrough it shows the PCI-E bus for each M1015 as "Unknown PCI/PCI bridge" (see pic below).  Does anybody else with a X9SCM-F-O show the same thing or should it be recognized?

 

Untitled by rockdawg2232, on Flickr

[Johnm]

I'll try a d remote in tomorrow and look. I won't physically see my server for another week.

 

 

Sent from my iPhone using Tapatalk

[RockDawg]

Johnm - Thanks.  I'd really appreciate it.

[BobPhoenix]

Here is a quick ESXi setup with 3 LSI cards installed.  I've switched out my X9SCM for my Tyan board so I don't have unRAID booted in a VM but I could set that up without drives if it would help. This also wasn't with my modified ESXi 5.0 install CD for the incompatible nic - sorry couldn't find it.

[RockDawg]

I wonder if  the X9SCM-F should also show "Intel Corporation Sandy Bridge PCI Express Root Port" as the parent for each of the M1015's?  You didn't install any kind of VMWare tools or anything, did you?  I don't want you to go through too much trouble, but anything that might shed some light on my situation would be greatly appreciated.

[BobPhoenix]

I wonder if  the X9SCM-F should also show "Intel Corporation Sandy Bridge PCI Express Root Port" as the parent for each of the M1015's?  You didn't install any kind of VMWare tools or anything, did you?  I don't want you to go through too much trouble, but anything that might shed some light on my situation would be greatly appreciated.

Nope.  This has NO VM setup currently.  I will have to setup a datastore and copy a unRAID VM from another ESXi setup.  So would be tomorrow some time but can be done if you want.  After tomorrow it won't be because my new Tyan board is coming in and the CPU and memory will be going to that on Saturday.  Then the X9SCM will not have a CPU until I can afford to purchase one for my Windows install I'm planing on it.

 

The "Sandy Bridge PCI Express Root Port"s are the X8 slots and first x4 slot in my graphic and have 2  M1015s and a HVR-2250 tuner (in x4 slot).  The "Cougar Point PCI Express Root Port 1" is the last x4 slot with third M1015.  If you check the schematic in the manual on page 1-9 you will see that this is showing exactly what the schematic shows.  One x4 is connected to the South Bridge Cougar point chip and the other 3 slots are connected straight to the CPU.

[RockDawg]

If you wouldn't mind I would sure appreciate it.  I'm not sure if it will help me get mine running, but it may and hopefully it will at least show me what should or shouldn't be.  I appreciate your help either way.

[BobPhoenix]

Not a problem.  I should get it done sometime tomorrow.  I'll see if I can get graphics of my Bios setup screens also.  May have another week available if UPS fails to deliver the Tyan board tomorrow.

[RockDawg]

Thanks.  I know what it's like waiting on hardware so I hope you get yours tomorrow. 

[graywolf]

Here is mine.

shows "Intel Corporation Sandy Bridge PCI Express Root Port"

 

[RockDawg]

That's go to be the problem then.  I'll have to try to figure out why they are showing up like that.  Thanks!

[prtsmgr]

At this point it may be a good idea to double check all bios settings.

I think you mentioned you had the latest bios.

Did you re-enable VT-d after the bios upgrade?

http://lime-technology.com/forum/index.php?topic=20892.msg186042#msg186042

[RockDawg]

My board was shipped with 2.0a so I didn't have to upgrade it.  I've made sure VT-d is neabled and I have set the PCI-E slots OPROM to both enabled and disabled.  When I enable it on all of them I get the following error on boot up:

 

MPT BIOS Fault 01h encountered at adapter PCI(03h,00h,00h)

 

So I've been running them all disabled.  Beyond that, I'm not sure what to change in the BIOS that would have an affect on my problem.

[BobPhoenix]

Attached is a log from a temp unRAID VM I setup with 3 M1015's and 20 drives in array.  They were junk drives that show as unformatted except for one drive.  Most probably have windows partitions.

 

Also attached is the first part of my bios settings.  Part 2 in next post was too big even zipped.

X9SCM_Bios_Screens_1.zip

Tower_System_Log.zip

[BobPhoenix]

Here is part 2 of bios settings.

X9SCM_Bios_Screens_2.zip

[RockDawg]

Thanks a lot Bob.  I appreciate the time and effort you've spent on this.  Unfortunately, nothing seems to help.  Your BIOS settings were a little different than mine and I set mine exactly as yours are and it still didn't work.  Looking at your syslog I see your M1015's are showing up as 32-bit like mine, but other users are showing as 64-bit as does mine if I boot straight into unRAID.  I was thinking that was maybe the problem, but maybe it isn't since yours works like that.  I also noticed you have the OPROM for the PCI-E slots enabled.  Like I posted earlier, when I enable that on all my slots, I get the MPT BIOS error at boot up on one of M1015's that I posted above.

 

I even tried reinstalling ESXi and it still doesn't work, but now one of the cards shows up under "Intel Corporation Cougar Point PCI Express Root Port 1" instead of "Generic PCI/PCI Bridge" like other users systems, but the other two cards are still showing under "Generic PCI/PCI Bridge" like before".

 

[prtsmgr]

Could the M1015's have a "corrupted" flash or bios? Did you flash them yourself or did you get them already flashed. I originally got mine flashed, but have since flashed them to P14 firmware.

 

Sent from my DROID RAZR using Tapatalk 2

[RockDawg]

I flashed them myself.  How can I tell if either is corrupt?  I can get into the BIOS of all three.

[prtsmgr]

I'm not sure, it was just a thought. Maybe some body here will know?

 

Sent from my DROID RAZR using Tapatalk 2

[BobPhoenix]

One other thing to note.  My M1015's are P11 bios I haven't updated them to P14 yet.

Sep  7 16:50:36 Tower kernel: mpt2sas0: LSISAS2008: FWVersion(11.00.00.00), ChipRevision(0x03), BiosVersion(07.21.00.00)
Sep  7 16:50:36 Tower kernel: mpt2sas1: LSISAS2008: FWVersion(11.00.00.00), ChipRevision(0x03), BiosVersion(07.21.00.00)
Sep  7 16:50:36 Tower kernel: mpt2sas2: LSISAS2008: FWVersion(11.00.00.00), ChipRevision(0x03), BiosVersion(07.21.00.00)

Also I have at least a week before I get my Tyan MB so I can try anything else you want.  I wasn't lucky.  The Tyan board I got delivered today either has a manufacturing defect or I got an open box item even though I purchased new.  It had what resembles a scratch near the CPU that goes through a bank of diodes/resistors/capacitors.  Whether it was used or a manufacturing defect doesn't matter.  It could have shorts with the bare metal traces showing and then I would get mysterious reboots or lockups so I RMA'd it back to Newegg.

[BobPhoenix]

Thanks a lot Bob.  I appreciate the time and effort you've spent on this.  Unfortunately, nothing seems to help.  Your BIOS settings were a little different than mine and I set mine exactly as yours are and it still didn't work.  Looking at your syslog I see your M1015's are showing up as 32-bit like mine, but other users are showing as 64-bit as does mine if I boot straight into unRAID.  I was thinking that was maybe the problem, but maybe it isn't since yours works like that.  I also noticed you have the OPROM for the PCI-E slots enabled.  Like I posted earlier, when I enable that on all my slots, I get the MPT BIOS error at boot up on one of M1015's that I posted above.

 

I even tried reinstalling ESXi and it still doesn't work, but now one of the cards shows up under "Intel Corporation Cougar Point PCI Express Root Port 1" instead of "Generic PCI/PCI Bridge" like other users systems, but the other two cards are still showing under "Generic PCI/PCI Bridge" like before".

If you have another board (3 x4 or greater) slots you might try it on that board to see if you get the same problem.  You could also remove all the M1015s and plugin anything else in them and see if the ESXi display changes to show the Couger and Sandy Bridge port messages.  If you have the same problem on a different MB or the Couger/Sandy Bridge ports come back normal I think you would be able to assume it is the M1015s.  If the opposite is true it could be a bad mother board and time to get SuperMicro support in the mix.

 

Another thing I just thought of.  I had problems when I setup this test I had to reinstall unRAID on my flash drive.  I figured my VMDK boot HDD had 5.0 unRAID but the flash drive had 4.7 so I cleared the flash drive and put 5.0 on it and it booted.  It could also have been a plugin since I also had those starting originally but when I re installed I used a vanilla go script.  It locked up before it got to the plugins as far as I could tell so it probably wasn't a plugin.  In any case it boots fine now.