Notes about Supermicro X11SCA-F

[Tybio]

This new Supermicro board is turning out to be a little odd...so I thought I'd start a thread to note interesting information about it that come up as I refine my new deployment.

 

1> The VGA port is NOT tied to the intel processor, it is tied to a "BMC" video adapter

2> On first boot, I got no video from HDMI or Display port with no addon cards.  Had to force the board to use the iGPU by changing from "Auto" to "IGFX" in the bios

3> The fans don't follow the normal SM standard.  There are 2 zones, but rather than "FAN#" and "FANA" you get 2 CPU headers and 3 FAN headers.  This is being looked at in the IPMI plugin thread to see if it can be supported.

4> My LSI card would not allow drives to mount when in the PCI-E slot nearest the CPU.  I moved it and everything worked fine.  I'm fairly sure this was one of those transient errors you get when building a new system, but I wanted to mention it.

5> IPMI seems to be losing remote console access, it starts up, but never shows the console screen.  It has worked intermittently and having an HTML5 version is HEAVEN...when it works.

6> The fan thresholds are crap, I'll post my settings to precent the "Fan Cycling" condition in a reply.

7> I'm going to test an NVMe drive soon, the docs read that I should be able to us /1/ of the NVMe slots while using the 4x slot (10GB Ethernet card)...but I want to check that.

8> IOMMU groups are not "Bad" but not great.  Audio is burried and there is a real lack of USB ports exposed. 

[Tybio]

IOMMU group 0:	[8086:3ec6] 00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
IOMMU group 1:	[8086:1901] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07)
[8086:1905] 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 07)
[1000:0087] 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
IOMMU group 2:	[8086:3e96] 00:02.0 VGA compatible controller: Intel Corporation Device 3e96
IOMMU group 3:	[8086:1911] 00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model
IOMMU group 4:	[8086:a379] 00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
IOMMU group 5:	[8086:a36d] 00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
[8086:a36f] 00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
IOMMU group 6:	[8086:a368] 00:15.0 Serial bus controller [0c80]: Intel Corporation Device a368 (rev 10)
[8086:a369] 00:15.1 Serial bus controller [0c80]: Intel Corporation Device a369 (rev 10)
IOMMU group 7:	[8086:a360] 00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
IOMMU group 8:	[8086:a352] 00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
IOMMU group 9:	[8086:a340] 00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 (rev f0)
IOMMU group 10:	[8086:a32c] 00:1b.4 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 (rev f0)
IOMMU group 11:	[8086:a338] 00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0)
IOMMU group 12:	[8086:a33d] 00:1c.5 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #6 (rev f0)
IOMMU group 13:	[8086:a33e] 00:1c.6 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #7 (rev f0)
IOMMU group 14:	[8086:a33f] 00:1c.7 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #8 (rev f0)
IOMMU group 15:	[8086:a328] 00:1e.0 Communication controller: Intel Corporation Device a328 (rev 10)
IOMMU group 16:	[8086:a309] 00:1f.0 ISA bridge: Intel Corporation Device a309 (rev 10)
[8086:a348] 00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
[8086:a323] 00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
[8086:a324] 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
[8086:15bb] 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10)
IOMMU group 17:	[1d6a:d107] 04:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02)
IOMMU group 18:	[8086:1533] 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
IOMMU group 19:	[1a03:1150] 07:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
IOMMU group 20:	[10e3:8113] 09:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)

 

[Tybio]

Same setting for all fans:

 

Section 808_CPU_FAN1
	## Possible values: Yes/No
	Enable_All_Event_Messages                                                   Yes
	## Possible values: Yes/No
	Enable_Scanning_On_This_Sensor                                              Yes
	## Possible values: Yes/No
	Enable_Assertion_Event_Lower_Critical_Going_Low                             Yes
	## Possible values: Yes/No
	Enable_Assertion_Event_Lower_Non_Recoverable_Going_Low                      Yes
	## Possible values: Yes/No
	Enable_Assertion_Event_Upper_Critical_Going_High                            Yes
	## Possible values: Yes/No
	Enable_Assertion_Event_Upper_Non_Recoverable_Going_High                     Yes
	## Possible values: Yes/No
	Enable_Deassertion_Event_Lower_Critical_Going_Low                           Yes
	## Possible values: Yes/No
	Enable_Deassertion_Event_Lower_Non_Recoverable_Going_Low                    Yes
	## Possible values: Yes/No
	Enable_Deassertion_Event_Upper_Critical_Going_High                          Yes
	## Possible values: Yes/No
	Enable_Deassertion_Event_Upper_Non_Recoverable_Going_High                   Yes
	## Give valid input for sensor type = Fan; units = RPM
	Lower_Non_Critical_Threshold                                                100.000000
	## Give valid input for sensor type = Fan; units = RPM
	Lower_Critical_Threshold                                                    100.000000
	## Give valid input for sensor type = Fan; units = RPM
	Lower_Non_Recoverable_Threshold                                             100.000000
	## Give valid input for sensor type = Fan; units = RPM
	Upper_Non_Critical_Threshold                                                25300.000000
	## Give valid input for sensor type = Fan; units = RPM
	Upper_Critical_Threshold                                                    25400.000000
	## Give valid input for sensor type = Fan; units = RPM
	Upper_Non_Recoverable_Threshold                                             25500.000000
	## Give valid input for sensor type = Fan; units = RPM; 'None' to not use hysteresis
	Positive_Going_Threshold_Hysteresis                                         100.000000
	## Give valid input for sensor type = Fan; units = RPM; 'None' to not use hysteresis
	Negative_Going_Threshold_Hysteresis                                         100.000000
EndSection

 

[uldise]

according https://www.supermicro.com/products/motherboard/X11/X11SCA-F.cfm

"M.2#1~2 are shared with PCI-Ex4 slot, M.2#2 is shared with U.2"

if i understand correctly, if you use any M.2 slot, then PCI-Ex4 is disabled..

[Tybio]

18 minutes ago, uldise said:

according https://www.supermicro.com/products/motherboard/X11/X11SCA-F.cfm

"M.2#1~2 are shared with PCI-Ex4 slot, M.2#2 is shared with U.2"

if i understand correctly, if you use any M.2 slot, then PCI-Ex4 is disabled..

There is some question as to if that "~" is an or.  As each M.2 slot IS 4x PCI-E..it wouldn't make sense to disable 8 channels because you are using a 4x slot.

 

I believe this means that if you use PCI-E 4x slot then you can use ONE of the M.2 slots.  If you use U.2 (4x PCI-E lanes) you can only use M.2#1 if you use BOTH U.2 and PCI-E 4x then you can't use either.  I'm not sure about this, which is why I want to test it...but it would make no sense to disable both of them to use half the lanes for an expansion card

[Tybio]

This diagram from the manual says it better than I could

[uldise]

@Tybio how about your test with M.2 drive? i just saw another thread that you finished your build..

[Tybio]

1 minute ago, uldise said:

@Tybio how about your test with M.2 drive? i just saw another thread that you finished your build..

Waiting on the M.2 to come in...I don't have a handy spare one at the moment.  Should have it this weekend.

[Tybio]

Confirmed!  I've got a 250G NVME drive in the top slot by the CPU and a 4x card installed.  There is also a drive on every on-board SATA port, though that shouldn't matter as it is an NVMe.

 

As a note, I tried it in the other M.2 port and it DID knock out the 4x slot (So I had no net)

 

 

Edited December 29, 2018 by Tybio

[Tybio]

Thanks to the IPMI plugin I now have solid fan control, the update makes this board 100% functional for a quit unraid server!

 

The only outstanding issue left is with the KVM setup.  I can't get any console redirection to work via HTML5 or Java applet.  THink I'm going to open a case with supermicro.

[Tybio]

So I've been working with Supermicro on the IPMI problem, and bad news.  Right now it seems you have a choice...use the BMC's Video adapter and get remote console /OR/ use the iGPU.  If you use the IPMI video adapter, you don't seem to have access to the iGPU once the OS is booted...

 

This is a REAL issue for me, one I'm working with their support team to sort out...but at this point, it looks like remote console OR hardware transcoding...but not both.  I hope there is a way around that or this board was freaking pointless :).

[Hoopster]

4 minutes ago, Tybio said:

I hope there is a way around that or this board was freaking pointless :).

Has that always been the case with Supermicro boards with IPMI or just this one?  Fortunately, both of my past and current ASRock boards that have IPMI have allowed both; remote console via BMC AND transcoding with iGPU.  That is certainly something I will be checking on with the board manufacturer before my next upgrade.

 

I have seen your comments about HDR transcoding with iGPUs/Plex and hope that gets fixed, or; alternatively, I hope the efforts to include nVidia GPU drivers in unRAID so discrete GPUs can be used for Plex transcoding are successful.  I prefer to run Plex as a docker and don't want a full-time VM just for Plex transcoding.

[Tybio]

I'll keep you updated on the IPMI...and I also agree on the VM solution, it has some down-sides, but honestly...it is looking like a VM with a P2000 would have been a much better solution for this moment in time...then I could have remote console, and gone with any CPU I might like down the road

[Tybio]

Confirmed, Supermicro has stated this is the expected behavior...you can't have a remote console and iGPU.  I think I'll likely be in the market for a P2000 at some point later this year, looking toward the next release of Ryzen or TR.  If I'm going to have all these limitations, I might as well go for PCI-E lanes so I can work around them....sad to say.

[uldise]

i will add my two cents too  

i have an older SM X8 board with older quadro in it, and i'm experienced a very similar problem - if i choose quadro as my primary video in bios, then no remote console - just a blank screen. then i choose BMC integrated video as primary in BIOS and started experiment. first i installed Ubuntu desktop, and with all latest Nvidia drivers i cant use quadro as main video - simple blank screen. next, i installed a fresh Windows 10, with all latest Nvidia drivers i can successfully use quadro as primary video, and have access to remote console via integrated BMC.

so, looks like it's an old SM problem, and i'm not sure, even if you ever have a nvidia drivers in unraid, will quadro work correctly?

looks like a proven variant with VM for transcoding is a way to go.. 

[Tybio]

Yea, the P2000 would be in a windows VM and passedthrough so it has no impact at the unraid level.  

 

It looks like with an iGPU and BMC, you have either/or.  With a PCI-E vid card and BMC you can have both but it is driver dependent, but I've not tested this and have no intention too.  If I can't get iGPU and BMC then I only have two paths:

 

1> accept IPMI is mostly pointless for me

2> I have to move away from iGPU

[Output]

Sounds bad with the ipmi issue. When you run igpu and no ipmi do you have issues with fan control?

[Tybio]

Nope, the only thing you lose is remote console...all other functions of IPMI seem to work fine.

 

[Output]

cool!

 

[Melwin]

Hi Tybio,

I've been watching your posts closely for the last 2 months as i (was) heading for a light-version of your build (X11SCL-IF & i3-8300 in a Supermicro SC721 case). However, what i first didnt notice is that the C242 chipset wont work at all with the iGPU(Thx Intel for your (non-existent) docs..). After talking to the Supermicro support they told me that the X11SCH is the only C246 mainboard that supports iGPU used for VHD (virtual hosted desktop) at the moment, which according to him was the technical feature we were looking for.

 

Now i'm going for the Asrock Rack E3C246D2I. It's the only MB i can find with C246 in ITX format. Lets hope that VHD is supported

 

[smeep2k4]

This was incredibly useful information. I'm building a 20-bay server with this mobo, and I wasn't sure which lanes the m.2's used. I don't need the 4x PCIe slot, so losing that is no biggie. Same with the U.2 connector. 

[burg3]

On 1/28/2019 at 1:31 AM, Tybio said:

Nope, the only thing you lose is remote console...all other functions of IPMI seem to work fine.

 

Hi,

can you check comments below this review: https://www.servethehome.com/supermicro-x11sca-f-review-for-intel-xeon-e-2100-and-core-i3-8300/3/

 

There is a comment about an option in BIOS, which could allow BMC graphics for KVM and IGPU for transcoding or passthrough to VM.

What is not clear if BMC KVM remains accessible when this is option us set. I think BMC KVM will not work once IGPU is used in unraid (but should work during boot process?), but maybe it can?

Can you please check and report your findings?

[Tybio]

9 hours ago, burg3 said:

Hi,

can you check comments below this review: https://www.servethehome.com/supermicro-x11sca-f-review-for-intel-xeon-e-2100-and-core-i3-8300/3/

 

There is a comment about an option in BIOS, which could allow BMC graphics for KVM and IGPU for transcoding or passthrough to VM.

What is not clear if BMC KVM remains accessible when this is option us set. I think BMC KVM will not work once IGPU is used in unraid (but should work during boot process?), but maybe it can?

Can you please check and report your findings?

Very interesting, 

 

Going to do some testing today, I have a small monitor I can connect for the moment as I don't have a display emulator plug handy...but I should be able to replicate the setup.  Will report back.

[Hoopster]

On 1/31/2019 at 3:58 AM, burg3 said:

What is not clear if BMC KVM remains accessible when this is option us set. I think BMC KVM will not work once IGPU is used in unraid (but should work during boot process?), but maybe it can?

 

I would be interested to see how it works on this board.  The ASRock board in my backup server allows both.

 

It boots with the AST BMC and full IPMI remote console functions remain available after boot, while simultaneously allowing the iGPU to be used for Plex/Handbrake hardware transcodes.

 

For this to work, Onboard (which in this case is the AST BMC) is set as the primary graphics adapter.  The BIOS has another option "Intel VGA Device Control" which, when enabled, loads the iGPU drivers on boot so the host/dockers/VMs can use it.

 

I don't know whether or not the current ASRock Rack server boards with IPMI and C246 chipsets have that capability.

Edited February 2, 2019 by Hoopster

[rinseaid]

So I just got one of these boards, and am also interested in being able to retain IPMI KVM along with using the iGPU for Plex transcoding. I do not have a discrete GPU connected. When I switched the graphics option to 'SG', two additional options appeared: SG delay after power enable (default 300) and SG delay after reset (default 100). I'm really not sure what those values indicate.

 

My experience after setting SG as the graphics option and reboot is that I see the POST screens, but then once it gets to the step of enumerating the PCI bus, the IPMI VGA disconnects. This occurs prior to having the opportunity to enter the BIOS. The OS boots, and I can then SSH in and load the driver module for the iGPU successfully. But no VGA via IPMI. I don't currently have a monitor handy to see what's being output on the iGPU.

 

I shorted the CMOS reset jumper so I could get back into the BIOS, and then tried changing the two SG value settings to double (600 and 200) but it didn't seem to change anything for me. Interested if anyone has a different experience, or perhaps further insight into what these options do.