Tybio

Waiting on the M.2 to come in...I don't have a handy spare one at the moment. Should have it this weekend.

Ok, the build is done! I've updated my post in the compulsive design forum for people who want details and pictures. I just wanted to drop this in here, tested Plex hardware transcoding today and got it working...the iGPU works out of the box with unraid 6.6.5 (Standard steps to enable hardware transcoding in Plex). It is /throtteled/ when transcoding from 4k to 1080p (Meaning it can transcode well faster than the playback). Obviously only the video stream is hardware assisted...so I'm bouncing from 5-10% CPU when playing. That is with 2 VMs running and a boatload of dockers...my normal CPU is hovering at the 5% or less level. The normal temp of this server is ~29C, with transcoding it jumps up to ~35C.

This diagram from the manual says it better than I could

There is some question as to if that "~" is an or. As each M.2 slot IS 4x PCI-E..it wouldn't make sense to disable 8 channels because you are using a 4x slot. I believe this means that if you use PCI-E 4x slot then you can use ONE of the M.2 slots. If you use U.2 (4x PCI-E lanes) you can only use M.2#1 if you use BOTH U.2 and PCI-E 4x then you can't use either. I'm not sure about this, which is why I want to test it...but it would make no sense to disable both of them to use half the lanes for an expansion card

rebuild finished, all seems well! Thanks for the help.

Same setting for all fans: Section 808_CPU_FAN1 ## Possible values: Yes/No Enable_All_Event_Messages Yes ## Possible values: Yes/No Enable_Scanning_On_This_Sensor Yes ## Possible values: Yes/No Enable_Assertion_Event_Lower_Critical_Going_Low Yes ## Possible values: Yes/No Enable_Assertion_Event_Lower_Non_Recoverable_Going_Low Yes ## Possible values: Yes/No Enable_Assertion_Event_Upper_Critical_Going_High Yes ## Possible values: Yes/No Enable_Assertion_Event_Upper_Non_Recoverable_Going_High Yes ## Possible values: Yes/No Enable_Deassertion_Event_Lower_Critical_Going_Low Yes ## Possible values: Yes/No Enable_Deassertion_Event_Lower_Non_Recoverable_Going_Low Yes ## Possible values: Yes/No Enable_Deassertion_Event_Upper_Critical_Going_High Yes ## Possible values: Yes/No Enable_Deassertion_Event_Upper_Non_Recoverable_Going_High Yes ## Give valid input for sensor type = Fan; units = RPM Lower_Non_Critical_Threshold 100.000000 ## Give valid input for sensor type = Fan; units = RPM Lower_Critical_Threshold 100.000000 ## Give valid input for sensor type = Fan; units = RPM Lower_Non_Recoverable_Threshold 100.000000 ## Give valid input for sensor type = Fan; units = RPM Upper_Non_Critical_Threshold 25300.000000 ## Give valid input for sensor type = Fan; units = RPM Upper_Critical_Threshold 25400.000000 ## Give valid input for sensor type = Fan; units = RPM Upper_Non_Recoverable_Threshold 25500.000000 ## Give valid input for sensor type = Fan; units = RPM; 'None' to not use hysteresis Positive_Going_Threshold_Hysteresis 100.000000 ## Give valid input for sensor type = Fan; units = RPM; 'None' to not use hysteresis Negative_Going_Threshold_Hysteresis 100.000000 EndSection

IOMMU group 0: [8086:3ec6] 00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07) IOMMU group 1: [8086:1901] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07) [8086:1905] 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 07) [1000:0087] 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05) IOMMU group 2: [8086:3e96] 00:02.0 VGA compatible controller: Intel Corporation Device 3e96 IOMMU group 3: [8086:1911] 00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model IOMMU group 4: [8086:a379] 00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10) IOMMU group 5: [8086:a36d] 00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10) [8086:a36f] 00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10) IOMMU group 6: [8086:a368] 00:15.0 Serial bus controller [0c80]: Intel Corporation Device a368 (rev 10) [8086:a369] 00:15.1 Serial bus controller [0c80]: Intel Corporation Device a369 (rev 10) IOMMU group 7: [8086:a360] 00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10) IOMMU group 8: [8086:a352] 00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10) IOMMU group 9: [8086:a340] 00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 (rev f0) IOMMU group 10: [8086:a32c] 00:1b.4 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 (rev f0) IOMMU group 11: [8086:a338] 00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0) IOMMU group 12: [8086:a33d] 00:1c.5 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #6 (rev f0) IOMMU group 13: [8086:a33e] 00:1c.6 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #7 (rev f0) IOMMU group 14: [8086:a33f] 00:1c.7 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #8 (rev f0) IOMMU group 15: [8086:a328] 00:1e.0 Communication controller: Intel Corporation Device a328 (rev 10) IOMMU group 16: [8086:a309] 00:1f.0 ISA bridge: Intel Corporation Device a309 (rev 10) [8086:a348] 00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10) [8086:a323] 00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10) [8086:a324] 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10) [8086:15bb] 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10) IOMMU group 17: [1d6a:d107] 04:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02) IOMMU group 18: [8086:1533] 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) IOMMU group 19: [1a03:1150] 07:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04) IOMMU group 20: [10e3:8113] 09:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)

This new Supermicro board is turning out to be a little odd...so I thought I'd start a thread to note interesting information about it that come up as I refine my new deployment. 1> The VGA port is NOT tied to the intel processor, it is tied to a "BMC" video adapter 2> On first boot, I got no video from HDMI or Display port with no addon cards. Had to force the board to use the iGPU by changing from "Auto" to "IGFX" in the bios 3> The fans don't follow the normal SM standard. There are 2 zones, but rather than "FAN#" and "FANA" you get 2 CPU headers and 3 FAN headers. This is being looked at in the IPMI plugin thread to see if it can be supported. 4> My LSI card would not allow drives to mount when in the PCI-E slot nearest the CPU. I moved it and everything worked fine. I'm fairly sure this was one of those transient errors you get when building a new system, but I wanted to mention it. 5> IPMI seems to be losing remote console access, it starts up, but never shows the console screen. It has worked intermittently and having an HTML5 version is HEAVEN...when it works. 6> The fan thresholds are crap, I'll post my settings to precent the "Fan Cycling" condition in a reply. 7> I'm going to test an NVMe drive soon, the docs read that I should be able to us /1/ of the NVMe slots while using the 4x slot (10GB Ethernet card)...but I want to check that. 8> IOMMU groups are not "Bad" but not great. Audio is burried and there is a real lack of USB ports exposed.

That would be great, I can use the CPU temp/fans for the system side and the disk temps for the drive side....should be perfect if it is possible in the code :). Thanks for looking into this!

Ugh, had some cut-n-paste issues with the hex values, so ended up having to type them all out. Here you go, looks like it works as expected...though I can't confirm CPU_FAN2 will bundle with CPU_FAN1, but it seems to be likely. root@Tower:~# ipmi-raw 00 30 45 01 01 rcvd: 45 00 root@Tower:~# ipmisensors -t fan ID | Name | Type | Reading | Units | Event 808 | CPU_FAN1 | Fan | 1500.00 | RPM | 'OK' 875 | CPU_FAN2 | Fan | N/A | RPM | N/A 942 | SYS_FAN1 | Fan | 1800.00 | RPM | 'OK' 1009 | SYS_FAN2 | Fan | 2200.00 | RPM | 'OK' 1076 | SYS_FAN3 | Fan | 2400.00 | RPM | 'OK' root@Tower:~# ipmi-raw 00 30 70 66 01 00 32 rcvd: 70 00 root@Tower:~# ipmisensors -t fan ID | Name | Type | Reading | Units | Event 808 | CPU_FAN1 | Fan | 800.00 | RPM | 'OK' 875 | CPU_FAN2 | Fan | N/A | RPM | N/A 942 | SYS_FAN1 | Fan | 1800.00 | RPM | 'OK' 1009 | SYS_FAN2 | Fan | 2200.00 | RPM | 'OK' 1076 | SYS_FAN3 | Fan | 2400.00 | RPM | 'OK' root@Tower:~# ipmi-raw 00 30 70 66 01 00 64 rcvd: 70 00 root@Tower:~# ipmi-raw 00 30 70 66 01 01 32 rcvd: 70 00 root@Tower:~# ipmisensors -t fan ID | Name | Type | Reading | Units | Event 808 | CPU_FAN1 | Fan | 1500.00 | RPM | 'OK' 875 | CPU_FAN2 | Fan | N/A | RPM | N/A 942 | SYS_FAN1 | Fan | 1000.00 | RPM | 'OK' 1009 | SYS_FAN2 | Fan | 1100.00 | RPM | 'OK' 1076 | SYS_FAN3 | Fan | 1200.00 | RPM | 'OK' root@Tower:~# ipmi-raw 00 30 70 66 01 01 64 rcvd: 70 00 root@Tower:~# ipmi-raw 00 30 45 01 02 rcvd: 45 CC

Oddly, they show as expected from the documentation! That might be a first eh? root@Tower:~# ipmisensors -t fan ID | Name | Type | Reading | Units | Event 808 | CPU_FAN1 | Fan | 1000.00 | RPM | 'OK' 875 | CPU_FAN2 | Fan | N/A | RPM | N/A 942 | SYS_FAN1 | Fan | 1200.00 | RPM | 'OK' 1009 | SYS_FAN2 | Fan | 1400.00 | RPM | 'OK' 1076 | SYS_FAN3 | Fan | 1600.00 | RPM | 'OK' root@Tower:~# In this case I have 1 PWM hub in SYS_FAN3 for the Drive side of the case, and 1 PWM hub in SYS_FAN2 for the system side. SYS_FAN1 is the single fan at the back behind the CPU and CPU_FAN1 is obviously the fan on the cooler. My real goal is to be able to set a profile for SYS_FAN3, I can either let the MB control the rest, or have another profile for them...I'm more than willing to rejigger my hubs/ports/whatever....but without 2+ profiles I'm going to have issues balancing the air flow properly.

For reference, these are the default IOMMU groups on the Supermicro X11SCA-F: IOMMU group 0: [8086:3ec6] 00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07) IOMMU group 1: [8086:1901] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07) [8086:1905] 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 07) [1000:0087] 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05) IOMMU group 2: [8086:3e96] 00:02.0 VGA compatible controller: Intel Corporation Device 3e96 IOMMU group 3: [8086:1911] 00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model IOMMU group 4: [8086:a379] 00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10) IOMMU group 5: [8086:a36d] 00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10) [8086:a36f] 00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10) IOMMU group 6: [8086:a368] 00:15.0 Serial bus controller [0c80]: Intel Corporation Device a368 (rev 10) [8086:a369] 00:15.1 Serial bus controller [0c80]: Intel Corporation Device a369 (rev 10) IOMMU group 7: [8086:a360] 00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10) IOMMU group 8: [8086:a352] 00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10) IOMMU group 9: [8086:a340] 00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 (rev f0) IOMMU group 10: [8086:a32c] 00:1b.4 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 (rev f0) IOMMU group 11: [8086:a338] 00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0) IOMMU group 12: [8086:a33d] 00:1c.5 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #6 (rev f0) IOMMU group 13: [8086:a33e] 00:1c.6 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #7 (rev f0) IOMMU group 14: [8086:a33f] 00:1c.7 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #8 (rev f0) IOMMU group 15: [8086:a328] 00:1e.0 Communication controller: Intel Corporation Device a328 (rev 10) IOMMU group 16: [8086:a309] 00:1f.0 ISA bridge: Intel Corporation Device a309 (rev 10) [8086:a348] 00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10) [8086:a323] 00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10) [8086:a324] 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10) [8086:15bb] 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10) IOMMU group 17: [1d6a:d107] 04:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02) IOMMU group 18: [8086:1533] 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) IOMMU group 19: [1a03:1150] 07:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04) IOMMU group 20: [10e3:8113] 09:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01) It looks good, other than having a shocking lack of USB controllers to pass through, the only one I see is linked to the shared sram item.

That would be fine! I have one connection for the Drive side anyway. I think the issue is that this BMC is only exposing one group, or the plugin doesn't know how this new board is splitting them. Unless I'm missing a configuration somewhere?

Thanks for the advice Johnnie, I setup the new server with 1xLSI card and using 8xon-board. It is rebuilding that failed disk a good clip and I have one less power draw in the system!

After reading a great deal of the thread...but not all I admit. I'm not sure that th X11SCA follows the normal Supermicro setup It looks like IPMI exposes FAN1-5, but they break down like this: FAN1&2: CPU FAN3-5: System I wouldn't mind doing all fan control via IPMI tool, but with my case (D600) having two sides...one for the system and one for drives, I have to be able to setup two fan profiles. Is there any way to do that with this plugin?

Hey there, just upgraded to the X11SCA in my sig and I only see fan control for FAN1234, this board seems to have 5 fans and they show up in the list as FAN1-FAN5 with 1 and 2 being CPU fans and the other 3 case fans. I'd really like to have the MB control all the headers (CPU Side) and have this plugin control one header (I have 6 fans on the drive side of the chassis via a PWM hub). Is that possible? Any way to map one of the fan headers to the FANA structure I've read about (But don't see in my settings)?

The parts are in and the system is rebuilt. I had two issues: 1> Had to force the IGPU to be used by the BIOS. In Auto mode even without another GPU in the system it would not bring up video on the on-board ports 2> I had an LSI card in the first PCI-E slot (Closest to the CPU) and was unable to mount any drives...though the card ROM loaded fine and the drives were detected by the LSI card. I moved it to the second PCI-E slot and it is working fine ATM. Going to update the main post now.

Ok, loaded up fine and is showing the files on the bad disk. Going to re-add it now and start the rebuild Thanks for the help!

Johnnie, Sorry for being annoying here, but what will happen if I hit start and for some reason it can't mount the disks? Will it just fail, or try to initialize them?

Er, if it doesn't know the filesystem to mount with, wouldn't it just format the drive?

Ok, ran xfs_repair as the docs said from command line and didn't see anything obviously wrong. This is the output for the "Disabled" disk: root@Tower:/boot/config# xfs_repair -nv /dev/md8 Phase 1 - find and verify superblock... - block cache size set to 1460776 entries Phase 2 - using internal log - zero log... zero_log: head block 1389819 tail block 1389819 - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 2 - agno = 5 - agno = 0 - agno = 3 - agno = 4 - agno = 7 - agno = 6 - agno = 8 - agno = 1 - agno = 9 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting. XFS_REPAIR Summary Fri Dec 21 16:09:54 2018 Phase Start End Duration Phase 1: 12/21 16:09:49 12/21 16:09:49 Phase 2: 12/21 16:09:49 12/21 16:09:49 Phase 3: 12/21 16:09:49 12/21 16:09:52 3 seconds Phase 4: 12/21 16:09:52 12/21 16:09:52 Phase 5: Skipped Phase 6: 12/21 16:09:52 12/21 16:09:54 2 seconds Phase 7: 12/21 16:09:54 12/21 16:09:54 Total run time: 5 seconds root@Tower:/boot/config# Advice on next steps?

Well, it is in and up! I had some issues with my LSI card so I'm working in the support forum at the moment trying to recover my array....but the processor is in and running just fine after forcing the BIOS to use the internal GPU...leaving it to auto got me no video on the HDMI or DP ports. Once I get things up and running I'll report back, but it is idle with nothing running other than the OS at ~27c which is about 15c cooler than the old E3 was running...though that's likely down to the new cooler vs the old E3's included intel cooler.

In reading the docs, I'm starting to worry. It should know what file system the disks are, and have a file system check section...but it doesn't. Tell me I didn' tlose 50+TB of data please?

I'm not seeing that option, it doesn't even know the FS on the drive for some reason.

What's the best way to do an FS check on an xfs filesystem?