Daniel15

The MicroATX version of this motherboard ahas onboard IPMI, so I'm surprised the ATX version doesn't... You generally don't get XMP profiles for ECC RAM, as the ECC functionality is only tested with standard (non-XMP) memory speeds.

I used Supermicro's ipmicfg to reset the password. It works well with other brands too. ipmicfg -user list to get a list of all the users and their IDs, then ipmicfg -user setpwd 1 new_password to set user ID 1's password to new_password. The default admin user is either ID 1 or 2; I can't remember.

I couldn't find an out-of-the-box way, so I just set the speed of the fans closest to the HDDs to a constant speed. They're Noctua fans so they're not too loud even if I run them at 60% constantly (plus my server is in a closet away from all the bedrooms, so noise is less of an issue) The fan speeds are controllable via IPMI since it's visible in the web interface, so there should be a way to do it programmatically too. You could see if the fan speeds can be controlled via ipmitool or ipmicfg at the command line, and if so, it should be possible to script it or update the IPMI Tools plugin to handle it.

Yes. With DDR you should always have pairs, so either two or four is fine. There's two slots on the motherboard labeled "A" and two labeled "B" - just make sure you use the A slots before the B slots. This is documented in the manual too It depends on the game and how heavy its RAM usage is. LTT has this video: I usually prefer linking to Gamers Nexus instead (their testing is way more thorough) but they don't seem to have a video about RAM speeds, just about DDR4 vs 5. Generally you'd only use ECC RAM at "standard" RAM speeds, meaning no XMP profiles for the RAM. Overclocking the CPU is probably still possible but I haven't tried overclocking a CPU in maybe 15 years now. The turbo speeds are usually more than sufficient for me, and you can bump the maximum wattage for non-turbo modes in the BIOS.

I've got the Micro ATX version which only has two M.2 slots. They're two separate controllers (one is on the CPU and the other is on the chipset) so they're each in a separate IOMMU group. I'm not sure about the full ATX version that has another M.2 slot that goes to the chipset - it's possible the two that go to the chipset are in the same IOMMU group but I don't really have a way to check The person in the comment directly above yours did exactly this and got it working.

I don't have a discrete GPU though - just the Intel integrated GPU. Sounds like you solved it though

This is common across most IPMI systems. The issue is that the IPMI only sees content rendered using its onboard graphics chip - there's physically no way for it to get the video output from the GPU. If the GPU is your primary video output, nothing will be rendered via the IPMI, and thus the remote console will be blank. You need to either configure the BIOS to use the IPMI's display adapter as the default screen, or enable the dual screen mode so that it outputs to both the GPU and the onboard graphics at the same time. I can't remember exactly where this is in the BIOS, but search around and you should find it. Note that if you're intending to use the graphics card for video transcoding, Intel QuickSync will do the same thing with much less power consumption.

That's right. The plugin could check for the existence of energy_performance_available_preferences (and the values it contains) to determine if it's supported. I've got the other ones too. In my case at least, I've found "performance" focuses too much on performance (CPUs turbo a lot even when not needed - maybe good for gaming PCs where power saving isn't as important), and "power" focuses too much on power savings. "balance_performance" and "balance_power" are pretty well balanced. Interesting that performance is the default for you... it was balance_performance for me. It might depend on BIOS settings maybe? I have some of the higher performance options disabled in the BIOS.

I do see the bifurcation option in the BIOS! What I don't understand is how I'd connect the network card to a riser and still have the port available at the back of the computer. Wouldn't the riser mean the card is oriented the wrong way?

Another thing you should add to the list is configuring Intel HWP to prefer lower power consumption over performance: for x in /sys/devices/system/cpu/cpu*/cpufreq/; do echo balance_power > $x/energy_performance_preference; done Here's the documentation on intel_pstate for Linux 6.1: https://www.kernel.org/doc/html/v6.1/admin-guide/pm/intel_pstate.html

Thanks for the info @eicar. Very useful! It sounds like I don't really need L2ARC since I'm happy with the performance of the hard drives. I'm actually moving from a storage VPS "in the cloud", where dozens of people share the same hardware, so even just the hard drives with no caching is quite a bit faster than what I've gotten used to using for storage. I've got the MicroATX version of the Asus board, so I've only got two M.2 slots. I definitely want to have two drives in a mirror just in case one of them dies. One of the M.2 slots is directly below the CPU cooler, so it's noticeably warmer (right now one M.2 drive is 46C while the other one is 41C). Both are well within the operating temperatures for the drives I'm using (70C max) but I wonder if they'll wear unevenly as a result of the temperature difference? I didn't think to install the supplied heatsink onto the drive, and now it's too hard to attach it since I'd have to remove the whole CPU cooler to be able to reach it. <_< I've currently got a 10Gbps Ethernet card in the single CPU-attached PCIe 5.0 slot, but that only needs PCIe 2.0 x4. I think I can somehow bifurcate the x16 slot so I can plug in two x8 devices (in case I want more NVMe drives in the future for example) but I'm not quite sure how that works.

The "passive" mode for intel_pstate is mostly for older processors that don't support HWP (hardware-managed P-states). For processors that do support HWP, it's better to keep it in "active" mode and instead set energy_performance_preference to balance_power: for x in /sys/devices/system/cpu/cpu*/cpufreq/; do echo balance_power > $x/energy_performance_preference; done

Setting the CPU scaling governor to "power save" but keeping turbo boost enabled still seems to cap the CPU at the non-turbo speeds. Is that expected? I had to manually set the `scaling_freq_max` to the maximum frequency (cpuinfo_max_freq) per core: for x in /sys/devices/system/cpu/cpu*/cpufreq/; do cat $x/cpuinfo_max_freq > $x/scaling_max_freq; done It'd also be useful to have an option to control the energy/performance preference for the hardware-managed P-states (HWP) of Intel CPUs: The `energy_performance_preference` setting (`/sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference`) defaults to `balance_performance` but it can be set to `balance_power` to tell the CPU to focus on energy saving instead of performance. (the CPU scaling governor affects the software governor in Linux, while `energy_performance_preference` affects the CPU's internal logic) with energy_performance_preference set to balance_power, the CPU cores won't turbo unless they really needs to (if CPU usage is high enough to need it).

I'm in the BIOS today and PL1 is labelled as "Long Duration Package Power Limit" and PL2 is labelled as "Short Duration Package Power Limit". Both are under AI Tweaker -> Internal CPU Power Management.

What is a "metadata-only L2ARC"? I'm very new to ZFS so I'm still learning - my only knowledge of it is what is shown in the Unraid UI when setting up a pool. My current configuration is that I have two 20TB hard drives in a ZFS mirror, two 2TB NVMe drives in a ZFS mirror, 64GB RAM and 32GB of it dedicated to ZFS.

That's my understanding - it's just the throughput, which is around 16 GB/s for PCIe 4.0 x8. It's a lot of bandwidth and you're extremely unlikely to be using every single PCIe device at 100% capacity at the same time. Having said that, note that on the Pro WS W680-ACE IPMI, the two PCIe 5.0 x16 and the first M.2 slot are connected directly to the processor, so they're not subject to the DMI speed limit. You can look at the "Expansion Slots" and "Storage" sections on the specs page: https://www.asus.com/us/motherboards-components/motherboards/workstation/pro-ws-w680-ace/techspec/ . They have headers labelled "Intel® 13th & 12th Gen Processors" and "Intel® W680 Chipset" so you can tell how the slots are connected. Unfortunately they don't provide block diagrams, but this is good enough. If you're ever worried about PCIe bandwidth, even some older AMD EPYC processors have 128 PCIe 4.0 lanes that all go directly to the processor.

Can you please add bluez? It's needed for Bluetooth support in the Home Assistant Docker container. Thanks!

I wouldn't buy a T variant, as the non-T variant is usually the same price. You can see here that both the i9-13900T and i9-13900 both have a recommended retail price of $549: https://www.intel.com/content/www/us/en/products/sku/230498/intel-core-i913900t-processor-36m-cache-up-to-5-30-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/230499/intel-core-i913900-processor-36m-cache-up-to-5-60-ghz/specifications.html The T variant is essentially just a lower binned version with a lower power limit. If you want to reduce power usage, I'd buy a regular i9-13900 (not the T variant) and power limit it in the BIOS. Setting the PL1 limit to 35W and PL2 limit to 106W will make it equivalent to the T variant, except you can always increase the limits a bit if you want more performance in the future. These settings are labelled something different in the Asus BIOS, but I can't remember off the top of my head. You can also disable turbo and set the governor to powersave mode to save more power. The latter two can be done in Unraid using the "Tips and Tweaks" plugin. Intel don't officially sell the T variant to retail customers, so it's also a bit more difficult to find, and you'd receive it just in a little plastic container without any official Intel packaging. Honestly, that CPU is overkill. I bought an i5-13500 for mine (paid $249 for it) and even that's overkill if you aren't running a lot of VMs. It supports ECC RAM too. Up to you though - both are good choices

Is there a way to have two separate configs? I want to back up into two different repos: One for my SSDs and one for larger data on my hard drives. I can bind mount multiple things under /mnt/user, but it'd all still go to the same repo. Edit: I just realised that it can take multiple config files, they just have to have different names. I'll try it out.

I tried this with a Windows Server 2022 VM that I'm trying to migrate across. When I tried booting the VM on Unraid, it failed with an "inaccessible boot device" error. For some reason, the VirtIO driver was working fine with Proxmox, but the VM can no longer see my VirtIO drives when running under Unraid. I had to switch the drives from VirtIO to SATA to get them to appear. Now it's booting fine. For testing, I added a new VirtIO vdisk, but don't see it at all in Device Manager. I've tried reinstalling the virtio Windows drivers but it didn't fix it. Needs more investigation. Edit: I solved this by reinstalling the viostor driver via DISM: https://superuser.com/a/1200899

You normally need to edit files other than the netdata.conf file to enable extra graphs - each collector has its own separate config file. Netdata has a script called edit-config that'll copy the default config from the Netdata installation into /etc/netdata, after which you'd edit it. I don't know how to get the default configs to edit when it's running in Docker, though. I guess I could copy them from another server I've got that's running Netdata. Maybe getting a shell in the Docker container itself then running edit-data will work, as long as /etc/netdata is mapped properly.

By the way, you'll want to disable "fast boot" in the BIOS, otherwise booting from USB drives will fail unless you do it via the BIOS. Took me a while to figure that out. On the positive side, having IPMI is great since I can get into the BIOS remotely, without having to plug a screen and keyboard into the system itself. It's sitting on a shelf in a narrow closet which makes it a pain to physically plug stuff into it. The IPMI web interface is pretty decent and has all the basic features you'd expect (power on/off, shut down, remote control, sensors, fan control). If you're having trouble logging into the IPMI (e.g. it says the password is wrong), ipmitool or ipmicfg can reset the password. In a way, I could justify the higher cost of the motherboard because it saved me having to buy something like a PiKVM for remote control. The IPMI has its own basic GPU, so you can use both the IPMI remote control and the intel iGPU (e.g. for hardware accelerated encoding/decoding via QuickSync) at the same time. That's not the case on some boards.

How do you edit the configs when using this Docker container? For example, I want to enable more plugins. With regular Netdata, you'd run `sudo ./edit-config python.d/foo.conf`. What's the equivalent here?

I bought it directly from the Asus website. I'm in the USA but I'm not sure which other regions their store operates in. Newegg has them in stock too. For the RAM, I'm using two sticks of Kingston server DDR5 ECC RAM. Model number KSM48E40BD8KM-32HM. I got them for $100 each using a employer discount with a supplier we use at work. It's also the only ECC RAM on Asus' compatibility list that seems to be easily obtainable - I couldn't figure out where to buy the others. I don't see the RAM temperature anywhere, but I did see it when running memtest86, so there's probably just some sensor config I need to do? I'm not sure as sensors-detect didn't detect it... For the CPU, I'm using a Core i5-13500. I usually stick to the ...500 CPUs since they're often the best balance of price and performance. My previous PC had an i5-6500 and the small form factor PC I'm replacing with this new server has an i5-9500. If you want to do anything with the integrated graphics (Plex or Jellyfin server, security cameras, etc), the iGPU in the 13500 (UHD 770) is more powerful than the one in the 13400 and below (UHD 730) and can support more concurrent video encoding/decoding jobs. Both support SR-IOV which lets you use one GPU in multiple VMs and Docker containers at the same time (you'll need the SR-IOV plugin)

Thanks! I tried grepping for it and couldn't find it, but I must have done something wrong. I also didn't realise the code's on Github, but it makes sense since that code is licensed under GPL. I just totally deleted the blacklist regex check, and now I see the Coral: If I select it, then revert libvirt_helpers.php back so that the blacklist is in effect again, will modifying the VM in the web UI remove the device? Since the list only shows devices bound to vfio, I wonder if the blacklist on the VM page is even needed. I deleted the blacklist check and only see the Coral, since it's the only IOMMU group I've selected for vfio on the system devices page. The system devices page already doesn't seem to allow me to select various "core" system devices or devices in use by Unraid anyways.