Build(log): UnNASty

[SiNtEnEl]

Hi guys!

I'm SiNtEnEl from the Netherlands and I had a huge wish. Building my own storage and server solution to release me from the unpleasure that i had with certain NAS solutions.

Translated in my build and server name: (UnNAS) No more "NAS" solutions (UnNASty) Pleasure! and later on (Un) for Unraid!

This buildlog reflects on my decisions for this build and hopefully useful for others with similar issues or wishes, took me quite a while (two months) to research and test all options. I'll going try to update this post as much as I can and feel free to ask anything about my build. Any idea's or improvements? Feel free to post them as well.
 

The requirements:

• Pricing & Cost, should be lower than out of the box NAS solutions.
• Software updates, there should be long term maintenance and frequent.
• Stability, Proven solution that can keep my data save enough, as not all data is backupped.
• Scalability, should scale with my needs and not limit me in usage.
• Upgradability, components can be upgraded easily without buying a new solution.
• Limited Locations, it should be able to operate in the living room, where I got certain options that are wife approved.
• Usage, should handle a couple of dockers and VM's. Including a Plex instance.
• Plex, hardware transcoding x265 and x264 content at minimum 1080p at remux bitrates (x264).
• Data replication, should be able to replicate data offsite to my "stock" NAS. (cloud, not my thing)
• Power Usage: Keep it under 40 watt power usage under load!

Did two months of testing a lot of solutions like Unraid, Freenas, Snapraid, NAS4Free and OpenMediaVault on my test bench.

Some solutions did not meet my requirements and some did not fit practices I try to use on daily basis in life and work:

• KISS: "Keep it simple stupid" Systems work best if they are kept simple rather than made complicated.
• YAGNI:  "You aren't gonna need it" No bloat or functionality until deemed necessary by me. (Modular, plugin based.)

The only system that fit my requirements and practices was Unraid, and it was time to order the new hardware.

 

Hardware Components Used:

CPU: Intel Pentium G4600 – 2 cores 4 threads – Intel 630 iGPU

Heatsink:  Stock Cooler – Since the low TDP from the CPU.
Motherboard: SuperMicro X11SAE-M (M-ATX)
RAM: Kingston VR ECC UDIMM DDR4-2400 8GB Micron
Case: Chenbro RM42300 4U (445 x 430 x 176 (mm), 17.5” x 16.9” x 6.9”)
Drive Cage(s): LIAN LI EX-332B 4-Slot 120mm fan, Chenbro Build-In 4-slot 3.5 drive bay. (Ordered Supermicro CSE-M35T-1B)
Power Supply: Seasonic G-Series 360Watt (Gold)
Parity Drive:  Seagate 10TB Enterprise EXOS10 (ST10000NM0016)
Array Disk1:  Seagate 10TB Enterprise EXOS10 (ST10000NM0016)

Array Disk2:  Seagate 10TB Enterprise EXOS10 (ST10000NM0016) RMA!

Hot Swap Disk:  Western Digital 5TB RED (WD50EFRX) (Large offline data transfer)

Cache Drive: Crucial Mx500 500gb (Not going to RAID!, backup works fine for me.)

 

Usage:

• Datavault - Storing my private data, backups, photo's and movies.
• VM Instance - Running instance of a linux distro that i use for work, study and hobby projects.
• Media Server - Running Plex docker for 4 instances for now 1080p content. (x264 and x265)
• Various Docker - Running: Unifi, PlexPy, Syncthing, Filebot, OpenVPN and more.

 

Power Usage:
Idle load: 22.1 watt
Full disk load: 33.1 watt

Transcoding 4K content: 35.4 watt

 

Hardware Selection, the options that needed most consideration:

CPU: Low power usage, hardware transcode plex, ECC Support, Cost effective are combined quite limiting in available solutions to buy. ECC support was the biggest pain on the Intel side, as it basically does not exist on consumer grade chips anymore. (stupid in my opinion) Going the AMD route in combination with Unraid gave me doubts, and not being able to use hardware transcoding for x264 and x265 (wish). Plus going for a processor without hardware transcode or iGpu would result in higher power usage as it would need more processing power. Final opions:

• Intel Pentium G4600 (5218 points CPU benchmark) @ 70 euro
• Intel Pentium G4620 (Should be 2% faster than the G4600) @ 95 euro
• Intel Xeon E3-1225 v6 (8163 points CPU benchmark) @ 285 euro
• Intel Xeon E3-1245 v6 (10445 points CPU benchmark) @ 305 euro

Since I had my doubts if I would need the Xeon performance and their TDP and higher power usage where factors to consider the Pentium. But the option to be able to change the Pentium for a Xeon later on was also a big factor. The choice for the Pentium G4600 was made. Cheap and didn't want to pay 25 euro for the 2% extra performance of a 4620.

 

Motherboard: My first wish was a M-ITX form factor, and be able to house a Pentium 4600 and 1245 v6 processor out of the box and minimum 6 SATA ports. ECC Support so that means a C236 Motherboard with ample SATA ports. Based on the wishes the only mainboard that was suitable is the ASRock Rack C236 WSI, but there was no availability of the board in the Netherlands. Reading on various forums there also were a lot of posts with issues with the Rack board. So, I desided to look at other larger form factors. M-ATX offered a lot more options, ATX would result in larger cases and was not an option was my thought. My choice went for a SuperMicro X11SAE-M board, as it ticked all my boxes and have had quite well experience with their hardware.

 

CASE part 1: Oh, I wanted a U-NAS so bad. But the costs of importing it would shoot a HUGE hole in my project budget as it should be cheaper as an out of the box solution. Scaling up the motherboard also resulted in not being able to use certain cases, and the wife approved factor was not something I took lightly in making my selection. My previous NAS solution was placed in my TV furniture, and also gave me limitations of dimensions of the new case. The location where the wife preferred the case. So, my focus was on the TV furniture with the dimensions of 49cm wide x 19,5cm high x 48cm deep. The result was disappointing as I was not able to find a case that was M-ATX and would fit in there and not have ventilation issues. Looking for a minimum 8-bay internal solution or something that has enough 5¼  bays to fit drive bays for a good price was not that hard since I did not have too bad size limitations in the stairs closet. Cheapest option is internal storage, so my choice went for the Fractal Define R5 as it offered hot slot drive bays and plenty of them.
 

CASE part 2: The fractal Define R5 turned out to be a huge disappointment. The drive bay is such a disaster when multiple running 7200 RPM disks, vibrations where so bad in this case and hot swap trays that the disks G-Sense Sensors were going NUTS during the preclear fase. Even worse was that the drive performance was cut in half by the vibrations. Contacting Fractal and the Store where I bought the case, only resulted in more disappointment and no working solution could be provided by them. So, decided to buy a new case again, and was lucky to find the Chenbro RM43200 4U 17,5” Chassis that after some modification to chassis and the TV furniture it fits in nicely. No more G-Sense sensor issues as well. Added two Noctua NF-A8 PWM 80mm fans on the back to keep the inside of the case cool. Front intake is by a 120mm Chenbro stock fan attached to a Lian Li EX-332B drive cage.

 

Current Issues:

• Buildin Chebro RM43200 4 slot 3.5 drivebay, cannot mount the Seagate hard disks when there is a FAN placed. Unless u want to mount them only with 2 screws. This rack is used for SSD.
• Stock 120mm fan of the Chenbro mounted on the Lian LI drive bay is quite loud!! This should be fixed when I replace it for another hotswap drive bay with noctua fans.
• Chenbro power BLUE led is like a FLASHLIGHT! Needs to be replaced for another led.
• One hard disk going in to RMA, due to SMART issues. Maybe related to the disk vibrations. Waiting on replacement or Warranty void reply. For now, I moved all data on the other disk and partial on the hot swap drive.
• Playing High bitrate x265 4k content in Transcode in memory and running File integrity Plugin with Blake2 just resulted in a system lockup. No Memory available issue!
  Switching back to SSD solves the issue. But opting to buy more memory soon.

 

Plans:

• Adding two Supermicro CSE-M35T-1B drive bays, ordered one to see if it fits my wishes and cage. Have some concerns with my power cables and the cage dept.
• Adding one extra Kingston VR ECC UDIMM DDR4-2400 8GB Micron, memory pressure is effecting in memory transcodes on higher bitrates (killing it).
• Find a Intel Xeon E3-1245 v6 engineering sample in China or else where for cheap.

Performance:
Having a fiber connection is nice, especially when it's a 1000mbit up and down. But when u route all your traffic trough a openVPN this can be quite taxing on a Pentium G4600. Hitting 50-60MB's on network transfer pulls about 40% CPU usage in the OpenVPN process, and thus quite demanding. Haven't had any issues pulling down my plex transcodes.

Testing the Plex transcoder performance using x265 samples supplied by http://www.libde265.org/downloads-videos/, on a 2 core Pentium G4600 with Hardware Transcoding!

• 3840px × 2160px "bbb-3840x2160-cfg02.mkv" Audio bitrate: 238.58 kbps Video bitrate: 5741.7 kbps Result: avg 44% CPU usage: No playback issues @ 1080p 10Mbps!
• 4096px × 1720px "tos-4096x1720-tiles" Audio bitrate: 546.76 kbps Video bitrate: 2693.8 kbps Result: avg 47% CPU usage: No playback issues @ 1080p 10Mbps

Can handle any content supplied by the libde265.org on a single hardware transcoded stream. So no issues casting to a Chromecast for example.

Used http://www.h264info.com/clips.html for samples of x264:

• 2048x × 858px "Gravity – 2K (2048×858) Trailer" Audio bitrate: 150 kbps Video bitrate: 21 Mbps Result: avg 27% CPU usage: No playback issues @ 1080p 10Mbps!

Edited January 29, 2018 by SiNtEnEl
Updates!

[pwm]

On 1/29/2018 at 6:47 PM, SiNtEnEl said:

as not all data is backupped

Think twice about this part. This should only apply to data you are ok with losing.

 

Notice that a RAID gives redundancy from a disk failure - but this is basically intended to give better availability. I.e. your data aren't offline for many hours or days while you restore backups.

But let one bad program perform a nasty write to your RAID, and there will be no ability for the RAID system to undo that write. Accidental file overwrites means lost files. A virus that attacks all shares that has R/W access means big havoc.

[SiNtEnEl]

Thanks @pwm, I learned it the hard way, in 2008 when kroll-ontrack had to repair 2 dead disks and restore my 5 disk raid array. (sadly)

i have roughly 9,5TB in backup data, that is synced with versioning and checked on integrity and also make monthly offline backups off.

Current cost in EU Netherlands, don't justify spending extra 950 euro on a new 30TB backup disks to backup all data sadly.

If the prices drop, i will upgrade the backup NAS or replace it for a Unraid server.

So some of the data, can be replaced or willing i'm willing to loose.

 

Good that users warn each other about the risks of raid / parity, and not being a proper backup solution.

 

 

Edited January 30, 2018 by SiNtEnEl

[pwm]

If you have 30 TB of data, then I guess quite a lot is unchanging data that you could just make archival copies of to cheapest possible USB-3 disks that you then store at a friend/family etc. And then dedicate the backup server for versioned backups of documents and other file data that may sometimes be changed. And possibly let the backup server have a share for pending data - new static files that might later be moved to a new USB-3 drive when you have bulked up enough files.

[SiNtEnEl]

Currently cheapest "dumpster" drives are €0,026 per gigabyte, still would hit me for 850 euro in disks. (need to by a extra disk due to overlap)

Cheapest NAS capable disks that i would consider are €0,029 per gigabyte, so i rather save a bit more and do a larger NAS solution.

Older disks will be kept for offline backup, what i'm doing at the moment, but that is always less then the size of my current storage.