My silent low-power server build [Update: 01/19/21]


T0a

Recommended Posts

Hello Unraid community,

 

back in October 2019 I was up to build/buy a NAS for data storage that could also host services for my network. I found UnRaid and this great community via a Linus Tech Tips video just before I was going to purchase a pre-build Synology NAS. Glad I didn't! This forum was really helpful for avoiding pitfalls regarding hardware selection and configuration such as backup, plugins and docker containers. My goal was to build a quiet low-power NAS as it runs in my living room and electricity costs are quite high where I am living.

 

Today, I want to present you my final low-power server build. It runs since Nov'19 like a charm and consumes 15W in idle. When both disks are spin-up the server consumes 21W. Feel free to ask questions!

 

1. Hardware

 

I purchased most of the parts used from Ebay. In total, I invested roughly 500 EUR including new WD Red drives for obvious reasons.

 

1.1 Upgraded configuration for VMs (~ +400 EUR)

  • Replaced the Intel Core i3-8100 with a Intel XEON-2146G
  • Added additional 16GB Samsung M391A2K43BB1 ECC DIM RAM
  • Added Inatek KT4006 2 x USB-3.0 PCI-e card for VM passthrough
  • Added Samsung 860 EVO M.2 500 GB for VM disk passthrough
  • Power consumption: 18,3W idle with all disks spin-down (powertop optimization)

 

1.2 IOMMU Groups

IOMMU group 0:	[8086:3e1f] 00:00.0 Host bridge: Intel Corporation 8th Gen Core 4-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 08)
IOMMU group 1:	[8086:3e91] 00:02.0 VGA compatible controller: Intel Corporation 8th Gen Core Processor Gaussian Mixture Model
IOMMU group 2:	[8086:a379] 00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
IOMMU group 3:	[8086:a36d] 00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
	[8086:a36f] 00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
IOMMU group 4:	[8086:a360] 00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
IOMMU group 5:	[8086:a352] 00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
IOMMU group 6:	[8086:a309] 00:1f.0 ISA bridge: Intel Corporation Cannon Point-LP LPC Controller (rev 10)
	[8086:a348] 00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
	[8086:a323] 00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
	[8086:a324] 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
	[8086:15bb] 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10)

 

1.3 Improvements and lesson learned

  • Next time, I would probably buy two 8 GB ECC DIMs instead of a single 16 GB DIM. That way I could utilize the dual-channel mode. However, more memory DIMs consume also more power! When you plan using VMs a lot go with two 16 GB modules.
  • I like the low-power profile of the MB, but it offers only a single Ethernet port. Next time, I would probably buy the Gigabyte C246M-WU4. Can't tell if that one would be as efficient as the Fujitsu though.
  • I can also not pass through the onboard sound device as it is in the same IOMMU group as the single Ethernet controller. Note, ACS override patch does not split up the groups. Here, the Gigabyte C246M-WU4 would have been the better choice due to its additional Ethernet controller in a separate IOMMU group.

 

2. Software & Configuration

 

2.1 Docker

  • Photoprism
  • Cyberchef
  • vm_custom_icons
  • Borgmatic
    • I do my off-site backups via Borg to a Hetzner storage box
  • Guacamole for quick access to several VMs utilizing libvirtwol for WOL
  • Zerotier for remote administration
    • Because my phone provider only offers IPv4 and my DSL provider uses DS-Lite tunnels with a public IPv6 address, I cannot simply utilize a VPN connection.
  • Paperless-ng container for my paperless office
  • Redis for paperless-ng
  • Custom WebDAV Container based on NGINX I build to send PDF documents to my paperless-ng instance
    • I do not like the full-blown solutions with WebDAV integrations such as Nextcloud or Seafile. These applications usually do write a lot to the (cache) drive.

 

2.2 Plugins and Apps

  • Dynamix SSD trim

  • Dynamix Cache Directories (Causes higher CPU load)
  • CA Appdata Backup / Restore (v2)
  • CA Fix Common Problems
  • User Scripts
    • Schedules my incremental backups to an external drives using this script (thank you mgutt!)
  • Dynamix System Stats
  • Rclone for offsite backups to Google Drive (I switched to borgmatic and Hetzner storage box)
  • Community Applications
  • Unassigned Devices
  • CA Cleanup Appdata

  • CA Config Editor

  • ControlR (Waiting for an official Unraid API)

  • Nerd Tools

  • VFIO-PCI CFG (No longer required after upgrade to 6.9)

  • Libvirtwol

 

2.3 VMs

  • Ubuntu as my daily driver with IGD passthrough (srvlx001)
  • Windows 10 Pro with IGD passthrough for playing casual games (srvms001)
  • Kali Linux for cyber security applications and CTFs (srvlx002)
  • Tip: See this thread on how I worked out the IGD passthrough (also take a look at this)

 

2.4 Workflows

 

  • SBM share phone for backing up my mobile phone media files. I use the iOS app PhotoSync for this.
  • With Scanner Pro for iOS, I send my scanned PDF documents to my webDAV docker container that functions as input for paperless-ng

 

3. What's next

 

  • Wait for native ZFS support?

 

Update:

- 05/14/20 Added IOMMU groups, information about IGD passthrough

- 11/16/20 Added Workflows section, updated docker usage and added new VMs, added upgraded hardware

- 01/19/21 Updated docker container, fixed spelling issues, updated backup strategy

Edited by T0a
updated 01/19/21
  • Like 1
Link to comment
10 minutes ago, T0a said:

3. What's next

 

3.1 Buy a GPU for HD gaming in a Windows VM

 

In future, I would like to have a Windows VM for playing games such as WoW. The main challenge will be preserving the low power consumption in idle. Haven't done a lot of research yet.

 

i3 8100 is a pretty weak CPU (4 cores no HT). I think you'd want to upgrade that before trying to run a VM for gaming. It does not even have turbo boost. Looks like your mainboard can take an i7, see if you can find a CPU that have at least 4 cores with HT (8 logical cpus) as you'll want that extra horsepower for a gaming VM with everything else going on in the background.

Link to comment
4 minutes ago, testdasi said:

Unraid and lower power can go together fine. Gaming + low power is like trying to do an F1 race with a Fiat 500.

 

That's what I meant - Gaming+unraid+low power = Unlikely. My UnRaid box runs around 30w on idle so I know about lower power 🙂

Link to comment
  • 1 month later...
On 2/25/2020 at 10:20 PM, Chess said:

 

i3 8100 is a pretty weak CPU (4 cores no HT). I think you'd want to upgrade that before trying to run a VM for gaming. It does not even have turbo boost. Looks like your mainboard can take an i7, see if you can find a CPU that have at least 4 cores with HT (8 logical cpus) as

you'll want that extra horsepower for a gaming VM with everything else going on in the background.

 

I was hoping to be able to play games like WoW or Diablo 3 with minimum graphics configuration with the system. Considering the VM overhead you might be right.

 

After your input @witalit, I plan to upgrade my rick with a core i7 9700. Do you think the idle power consumption (i.e. the state in which the system is most of the time) is comparable to my current i3 8100?

 

I would then also buy a decent graphics card, which I only physically add to the system, when I want to play a game. That way, I could maintain the low-power profile of the server while being able to play games when I'm up to.

 

However, this still depends on the idle consumption of the i7 9700, because I cannot remove that one this easy :P What do you think guys @witalit @testdasi?

 

On 3/3/2020 at 1:38 PM, witalit said:

 

That's what I meant - Gaming+unraid+low power = Unlikely. My UnRaid box runs around 30w on idle so I know about lower power 🙂

 

True and I'm fine with higher consumption when having a gaming session. But when the server is in idle, I would like the GPU to consume as less power as possible. However from what I recall from the forum, some GPUs have problems with entering the low power mode with unRaid. Therefore, I would be fine with physically removing the GPU when not needed.

Edited by T0a
updated
Link to comment

You are overthinking it with regards to CPU idle power consumption. The diff in idle consumption between an 8100 and 9700 is so small that if the electricity cost diff is a concern to you then you probably should be more concerned about necessities such as food and water instead.

 

The reason dedicated graphics card used by Unraid doesn't idle properly is because Unraid doesn't contain proper AMD/Nvidia drivers.

If you pass through the card to a VM then the card would run with the right drivers (in the VM) and thus would idle properly. (that is assuming there's always a VM using the card at all times, which should be the case - there's no point shutting down the VM while Unraid is running)

 

Your CPU choices have iGPU so you should configure your BIOS to boot Unraid with the iGPU, so back to my first point about CPU idle power consumption being a non-concern.

Link to comment

Thanks @testdasifor your input. I got my hands on a used Xeon-2146G for about 200 Euro on Ebay - waiting for it to arrive. For all that are interested in a comparison:

 

Intel Xeon E-2146G Benchmarks (Idle is around 32W and maximum power consumption hits just under 109W)

Intel Core i3-9100F Benchmarks (Idle Power (Performance Mode): 31W, STH 70% Load: 75W, STH 100% Load: 92W)

 

Note that I ruled out the Core i7 option, because it does not support ECC RAM.

Link to comment
  • 2 months later...
On 5/10/2020 at 7:19 PM, testdasi said:

You are overthinking it with regards to CPU idle power consumption. The diff in idle consumption between an 8100 and 9700 is so small that if the electricity cost diff is a concern to you then you probably should be more concerned about necessities such as food and water instead.

 

The reason dedicated graphics card used by Unraid doesn't idle properly is because Unraid doesn't contain proper AMD/Nvidia drivers.

If you pass through the card to a VM then the card would run with the right drivers (in the VM) and thus would idle properly. (that is assuming there's always a VM using the card at all times, which should be the case - there's no point shutting down the VM while Unraid is running)

 

Your CPU choices have iGPU so you should configure your BIOS to boot Unraid with the iGPU, so back to my first point about CPU idle power consumption being a non-concern.

@testdasi, you mean Unraid will consume less power with VM started (GPU passed through) than VM turned off?  

 

If this is the case, what are the steps to make sure passed through GPU idles properly?  I have graphics driver installed properly.  Do I just go to power settings and allow it to go to sleep when not in use?  Is this how we get VMs with GPU in low power?

Link to comment
36 minutes ago, jang430 said:

@testdasi, you mean Unraid will consume less power with VM started (GPU passed through) than VM turned off?  

 

If this is the case, what are the steps to make sure passed through GPU idles properly?  I have graphics driver installed properly.  Do I just go to power settings and allow it to go to sleep when not in use?  Is this how we get VMs with GPU in low power?

It's not a "will" - which implies 100% certainty.

 

If your card is passed through to a VM then it would follow any power scheme set by the driver installed in the VM, which generally should do the power management for you automatically.

 

Link to comment
  • 2 months later...
  • T0a changed the title to My silent low-power server build [Update: 01/19/21]
Posted (edited)

@mgutt

Thank you for the hint! I added 'powertop --auto-tune' a while ago to my Go file, but couldn't find the time to measure, analyze and give feedback.  Here we go now:

 

Introduction: The system contains all parts listed in 1 with 1.1 as an upgrade. The Samsung 860 EVO M.2 with its controller is marked as passthrough device. For the measurement there is also no device connected to the Inatek KT4006, no other USB devices attached to the host either and no display plugged-in. The iGPU is blacklisted hence it is used for VM passthrough too. I use VM's as my daily driver with an external display connected. I need the Inatek KT4006, because I couldn't get my audio interface Presonos AudioBox USB reliable working otherwise. 

 

Bios Settings: The mainboard does not come with a lot of power saving settings in the BIOS. I have disabled audio, enabled Intel Speed Shift, enabled C-States and set C State Limit from AUTO to C10.

 

Boot parameter:

kernel /bzimage
append video=efifb:off,vesafb:off modprobe.blacklist=i2c_i801,i2c_smbus,snd_hda_intel,snd_hda_codec_hdmi,i915,drm,drm_kms_helper,i2c_algo_bit initrd=/bzroot

 

Measurement:

1. Unraid idle, all disks spin-up, powertop --auto-tune: 26W

2. Unraid idle, only cache spin-up, powertop --auto-tune: 19,6W

 

speed_shift_c10_limit_audio_off_unraid_idle.thumb.png.24a96aa371aedce0ffcd746b51441690.png

 

3. Unraid idle, only cache spin-up, powertop --auto-tune, Ubuntu VM idle: 19,9W to peaks 21W

 

speed_shift_c10_limit_audio_off_unraid_ubuntu_idle.thumb.png.5ec3eec71b8f52943f22148b37196fac.png

 

4. Unraid idle, only cache spin-up, powertop --auto-tune, Ubuntu VM shutdown: 18W

 

speed_shift_c10_limit_audio_off_unraid_ubuntu_shutdown_idle.thumb.png.0d2a80fb0eec3f13f6890ddb4755ff9b.png

 

My initial measurement was 15W without the Inatek KT4006, Samsung 860 EVO M.2, using a i3-8100 and no powertop optimizations. In consideration that I've added two new devices an increase of 3W (with powertop optimization included) seems reasonable. However, I wonder why the hardware is not capable of reaching lower C-states. Maybe the USB add-on card prevents lower states.

 

 

Edited by T0a
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.