T0a Posted February 25, 2020 Share Posted February 25, 2020 (edited) Hello Unraid community, back in October 2019 I was up to build/buy a NAS for data storage that could also host services for my network. I found UnRaid and this great community via a Linus Tech Tips video just before I was going to purchase a pre-build Synology NAS. Glad I didn't! This forum was really helpful for avoiding pitfalls regarding hardware selection and configuration such as backup, plugins and docker containers. My goal was to build a quiet low-power NAS as it runs in my living room and electricity costs are quite high where I am living. Today, I want to present you my final low-power server build. It runs since Nov'19 like a charm and consumes 15W in idle. When both disks are spin-up the server consumes 21W. Feel free to ask questions! 1. Hardware I purchased most of the parts used from Ebay. In total, I invested roughly 500 EUR including new WD Red drives for obvious reasons. Case: Fractal design define r6 USB-C Mainboard: Fujitsu D3644-B Intel C246 CPU: Intel Core i3-8100 RAM: 16GB Samsung M391A2K43BB1 ECC DIM Hard drives: 2x4TB WD Red and 512GB Samsung 860 Pro 2.5" (cache drive) Power supply: 400 Watt be quiet! Pure Power 11 CPU cooler: be quiet! Pure Rock Tower cooler Case fans: 2x Noctua NF-A12x25 PWM Power consumption meter: Intertek JGQ02S-01 Power consumption: 15W idle and 21W all disks spin-up (no powertop optimization yet) 1.1 Upgraded configuration for VMs (~ +400 EUR) Replaced the Intel Core i3-8100 with a Intel XEON-2146G Added additional 16GB Samsung M391A2K43BB1 ECC DIM RAM Added Inatek KT4006 2 x USB-3.0 PCI-e card for VM passthrough Added Samsung 860 EVO M.2 500 GB for VM disk passthrough Power consumption: 18,3W idle with all disks spin-down (powertop optimization) 1.2 IOMMU Groups IOMMU group 0: [8086:3e1f] 00:00.0 Host bridge: Intel Corporation 8th Gen Core 4-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 08) IOMMU group 1: [8086:3e91] 00:02.0 VGA compatible controller: Intel Corporation 8th Gen Core Processor Gaussian Mixture Model IOMMU group 2: [8086:a379] 00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10) IOMMU group 3: [8086:a36d] 00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10) [8086:a36f] 00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10) IOMMU group 4: [8086:a360] 00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10) IOMMU group 5: [8086:a352] 00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10) IOMMU group 6: [8086:a309] 00:1f.0 ISA bridge: Intel Corporation Cannon Point-LP LPC Controller (rev 10) [8086:a348] 00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10) [8086:a323] 00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10) [8086:a324] 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10) [8086:15bb] 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10) 1.3 Improvements and lesson learned Next time, I would probably buy two 8 GB ECC DIMs instead of a single 16 GB DIM. That way I could utilize the dual-channel mode. However, more memory DIMs consume also more power! When you plan using VMs a lot go with two 16 GB modules. I like the low-power profile of the MB, but it offers only a single Ethernet port. Next time, I would probably buy the Gigabyte C246M-WU4. Can't tell if that one would be as efficient as the Fujitsu though. I can also not pass through the onboard sound device as it is in the same IOMMU group as the single Ethernet controller. Note, ACS override patch does not split up the groups. Here, the Gigabyte C246M-WU4 would have been the better choice due to its additional Ethernet controller in a separate IOMMU group. 2. Software & Configuration 2.1 Docker Photoprism Cyberchef vm_custom_icons Borgmatic I do my off-site backups via Borg to a Hetzner storage box Guacamole for quick access to several VMs utilizing libvirtwol for WOL Zerotier for remote administration Because my phone provider only offers IPv4 and my DSL provider uses DS-Lite tunnels with a public IPv6 address, I cannot simply utilize a VPN connection. Paperless-ng container for my paperless office Redis for paperless-ng Custom WebDAV Container based on NGINX I build to send PDF documents to my paperless-ng instance I do not like the full-blown solutions with WebDAV integrations such as Nextcloud or Seafile. These applications usually do write a lot to the (cache) drive. 2.2 Plugins and Apps Dynamix SSD trim Dynamix Cache Directories (Causes higher CPU load) CA Appdata Backup / Restore (v2) CA Fix Common Problems User Scripts Schedules my incremental backups to an external drives using this script (thank you mgutt!) Dynamix System Stats Rclone for offsite backups to Google Drive (I switched to borgmatic and Hetzner storage box) Community Applications Unassigned Devices CA Cleanup Appdata CA Config Editor ControlR (Waiting for an official Unraid API) Nerd Tools VFIO-PCI CFG (No longer required after upgrade to 6.9) Libvirtwol 2.3 VMs Ubuntu as my daily driver with IGD passthrough (srvlx001) Windows 10 Pro with IGD passthrough for playing casual games (srvms001) Kali Linux for cyber security applications and CTFs (srvlx002) Tip: See this thread on how I worked out the IGD passthrough (also take a look at this) 2.4 Workflows SBM share phone for backing up my mobile phone media files. I use the iOS app PhotoSync for this. With Scanner Pro for iOS, I send my scanned PDF documents to my webDAV docker container that functions as input for paperless-ng 3. What's next Wait for native ZFS support? Update: - 05/14/20 Added IOMMU groups, information about IGD passthrough - 11/16/20 Added Workflows section, updated docker usage and added new VMs, added upgraded hardware - 01/19/21 Updated docker container, fixed spelling issues, updated backup strategy Edited August 28, 2021 by T0a updated 01/19/21 1 Quote Link to comment
Chess Posted February 25, 2020 Share Posted February 25, 2020 10 minutes ago, T0a said: 3. What's next 3.1 Buy a GPU for HD gaming in a Windows VM In future, I would like to have a Windows VM for playing games such as WoW. The main challenge will be preserving the low power consumption in idle. Haven't done a lot of research yet. i3 8100 is a pretty weak CPU (4 cores no HT). I think you'd want to upgrade that before trying to run a VM for gaming. It does not even have turbo boost. Looks like your mainboard can take an i7, see if you can find a CPU that have at least 4 cores with HT (8 logical cpus) as you'll want that extra horsepower for a gaming VM with everything else going on in the background. Quote Link to comment
witalit Posted March 3, 2020 Share Posted March 3, 2020 I'm not sure Gaming/UnRaid and low power go together. It will be a bit of a battle trying to get the right balance here. Quote Link to comment
testdasi Posted March 3, 2020 Share Posted March 3, 2020 10 minutes ago, witalit said: I'm not sure Gaming/UnRaid and low power go together. It will be a bit of a battle trying to get the right balance here. Unraid and lower power can go together fine. Gaming + low power is like trying to do an F1 race with a Fiat 500. Quote Link to comment
witalit Posted March 3, 2020 Share Posted March 3, 2020 4 minutes ago, testdasi said: Unraid and lower power can go together fine. Gaming + low power is like trying to do an F1 race with a Fiat 500. That's what I meant - Gaming+unraid+low power = Unlikely. My UnRaid box runs around 30w on idle so I know about lower power 🙂 Quote Link to comment
T0a Posted April 4, 2020 Author Share Posted April 4, 2020 (edited) On 2/25/2020 at 10:20 PM, Chess said: i3 8100 is a pretty weak CPU (4 cores no HT). I think you'd want to upgrade that before trying to run a VM for gaming. It does not even have turbo boost. Looks like your mainboard can take an i7, see if you can find a CPU that have at least 4 cores with HT (8 logical cpus) as you'll want that extra horsepower for a gaming VM with everything else going on in the background. I was hoping to be able to play games like WoW or Diablo 3 with minimum graphics configuration with the system. Considering the VM overhead you might be right. After your input @witalit, I plan to upgrade my rick with a core i7 9700. Do you think the idle power consumption (i.e. the state in which the system is most of the time) is comparable to my current i3 8100? I would then also buy a decent graphics card, which I only physically add to the system, when I want to play a game. That way, I could maintain the low-power profile of the server while being able to play games when I'm up to. However, this still depends on the idle consumption of the i7 9700, because I cannot remove that one this easy What do you think guys @witalit @testdasi? On 3/3/2020 at 1:38 PM, witalit said: That's what I meant - Gaming+unraid+low power = Unlikely. My UnRaid box runs around 30w on idle so I know about lower power 🙂 True and I'm fine with higher consumption when having a gaming session. But when the server is in idle, I would like the GPU to consume as less power as possible. However from what I recall from the forum, some GPUs have problems with entering the low power mode with unRaid. Therefore, I would be fine with physically removing the GPU when not needed. Edited May 9, 2020 by T0a updated Quote Link to comment
testdasi Posted May 10, 2020 Share Posted May 10, 2020 You are overthinking it with regards to CPU idle power consumption. The diff in idle consumption between an 8100 and 9700 is so small that if the electricity cost diff is a concern to you then you probably should be more concerned about necessities such as food and water instead. The reason dedicated graphics card used by Unraid doesn't idle properly is because Unraid doesn't contain proper AMD/Nvidia drivers. If you pass through the card to a VM then the card would run with the right drivers (in the VM) and thus would idle properly. (that is assuming there's always a VM using the card at all times, which should be the case - there's no point shutting down the VM while Unraid is running) Your CPU choices have iGPU so you should configure your BIOS to boot Unraid with the iGPU, so back to my first point about CPU idle power consumption being a non-concern. Quote Link to comment
T0a Posted May 16, 2020 Author Share Posted May 16, 2020 Thanks @testdasifor your input. I got my hands on a used Xeon-2146G for about 200 Euro on Ebay - waiting for it to arrive. For all that are interested in a comparison: Intel Xeon E-2146G Benchmarks (Idle is around 32W and maximum power consumption hits just under 109W) Intel Core i3-9100F Benchmarks (Idle Power (Performance Mode): 31W, STH 70% Load: 75W, STH 100% Load: 92W) Note that I ruled out the Core i7 option, because it does not support ECC RAM. Quote Link to comment
jang430 Posted August 14, 2020 Share Posted August 14, 2020 On 5/10/2020 at 7:19 PM, testdasi said: You are overthinking it with regards to CPU idle power consumption. The diff in idle consumption between an 8100 and 9700 is so small that if the electricity cost diff is a concern to you then you probably should be more concerned about necessities such as food and water instead. The reason dedicated graphics card used by Unraid doesn't idle properly is because Unraid doesn't contain proper AMD/Nvidia drivers. If you pass through the card to a VM then the card would run with the right drivers (in the VM) and thus would idle properly. (that is assuming there's always a VM using the card at all times, which should be the case - there's no point shutting down the VM while Unraid is running) Your CPU choices have iGPU so you should configure your BIOS to boot Unraid with the iGPU, so back to my first point about CPU idle power consumption being a non-concern. @testdasi, you mean Unraid will consume less power with VM started (GPU passed through) than VM turned off? If this is the case, what are the steps to make sure passed through GPU idles properly? I have graphics driver installed properly. Do I just go to power settings and allow it to go to sleep when not in use? Is this how we get VMs with GPU in low power? Quote Link to comment
testdasi Posted August 14, 2020 Share Posted August 14, 2020 36 minutes ago, jang430 said: @testdasi, you mean Unraid will consume less power with VM started (GPU passed through) than VM turned off? If this is the case, what are the steps to make sure passed through GPU idles properly? I have graphics driver installed properly. Do I just go to power settings and allow it to go to sleep when not in use? Is this how we get VMs with GPU in low power? It's not a "will" - which implies 100% certainty. If your card is passed through to a VM then it would follow any power scheme set by the driver installed in the VM, which generally should do the power management for you automatically. Quote Link to comment
jang430 Posted August 14, 2020 Share Posted August 14, 2020 Thanks for clarifying that. Quote Link to comment
mgutt Posted October 27, 2020 Share Posted October 27, 2020 @T0a Consider using powertop or my script: Reason: https://www.reddit.com/r/homelab/comments/cvs8ue/my_7_watt_homelab/ Thanks to @Miidnight 1 Quote Link to comment
T0a Posted August 28, 2021 Author Share Posted August 28, 2021 (edited) @mgutt Thank you for the hint! I added 'powertop --auto-tune' a while ago to my Go file, but couldn't find the time to measure, analyze and give feedback. Here we go now: Introduction: The system contains all parts listed in 1 with 1.1 as an upgrade. The Samsung 860 EVO M.2 with its controller is marked as passthrough device. For the measurement there is also no device connected to the Inatek KT4006, no other USB devices attached to the host either and no display plugged-in. The iGPU is blacklisted hence it is used for VM passthrough too. I use VM's as my daily driver with an external display connected. I need the Inatek KT4006, because I couldn't get my audio interface Presonos AudioBox USB reliable working otherwise. Bios Settings: The mainboard does not come with a lot of power saving settings in the BIOS. I have disabled audio, enabled Intel Speed Shift, enabled C-States and set C State Limit from AUTO to C10. Boot parameter: kernel /bzimage append video=efifb:off,vesafb:off modprobe.blacklist=i2c_i801,i2c_smbus,snd_hda_intel,snd_hda_codec_hdmi,i915,drm,drm_kms_helper,i2c_algo_bit initrd=/bzroot Measurement: 1. Unraid idle, all disks spin-up, powertop --auto-tune: 26W 2. Unraid idle, only cache spin-up, powertop --auto-tune: 19,6W 3. Unraid idle, only cache spin-up, powertop --auto-tune, Ubuntu VM idle: 19,9W to peaks 21W 4. Unraid idle, only cache spin-up, powertop --auto-tune, Ubuntu VM shutdown: 18W My initial measurement was 15W without the Inatek KT4006, Samsung 860 EVO M.2, using a i3-8100 and no powertop optimizations. In consideration that I've added two new devices an increase of 3W (with powertop optimization included) seems reasonable. However, I wonder why the hardware is not capable of reaching lower C-states. Maybe the USB add-on card prevents lower states. Edited August 28, 2021 by T0a Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.