Khazad-dum, the full flash server, USB, SATA, NVMe, Optane

[gyto6]

0

Quote

"Moria. You fear to go into those mines. The Dwarves dug too greedily and too deep. You know what they awoke in the darkness of Khazad-dûm... shadow and flame."

 

Deep in my cave, is stored a silent server, only its fans wheels can be heard like a whisper. This server is a SuperMicro 1018GR-T.

 

This 1U server is connected from the deepest places of my home to the brightest places of the internet by Mikrotik RB4011-GS+ directly connected to my French ISP (Orange) GPON.

up.

 

Once the server is opened, most of these secrets shows Its components are brighter than the shade around us.

 

First of all, the heart of the beast : a Micron e230 USB DOM. This tiny card of 4GB/Go is equipped with ECC components that shall assure that Unraid doesn’t suffer from corrupted file due to magnetic interferences.. How can it be reached in such as dark place where no light seems to descends ?

 

A second vital element then shows up : an SSD-DM06 from SuperMicro, this special SATA port is able to give it the power to run. This cards is dedicated to store the Unraid’s logs.

 

Not so far can be seen 8 sticks of RAM from Micron. It appears that the server is equipped with 128GB/Go ECC RAM from Micron. Enough to run one Chrome tab with a bright sun if this last one would come down here.

 

The machine is equipped with 6 SATA SSD from Kingston. With a capacity of 11.52TB, this server could store a quarter of Star Citizen at max SATAIII speed 600MB/Mo’s. With its Power Loss Protection (PLP), the data cached in its own RAM can safely be written to the NAND in case of power outage.

 

Those 6 drives are connected to a AOC-S3108L-H8iR from SuperMicro. As the SATA SSD saturates the DMI which needs to vehiculate ethernet packets too, the motherboard needs it to not suffocate.

 

Finally, a green card can catch our eye. This RTX Quadro 4000 is more than enough to run Minecraft Java.

 

This less blinky card show up. The dark plate was hiding a silver one on the otherside. It seems to hide something in it…

 

The card dissected, some jewels can catch our attention. An Optane P4801x ! This memory’s so fast, that no RAM are required to access the data in its cells quickly. PLP is still present as some capacitors appears on its body. It let the controller the time to run the last received instructions and store it in the 100GB/Go storage.

 

Some more conventional flash cells are also on the Hyper M.2 Asus card. 2 Micron 7400 Pro with PLP are there as long as the 110mm Optane stick’s length. A little 80mm stick remains, a Crucial P5 with 2TB/To of storage, as much as the Micron sticks.

Finally, a V7 1500VA UPS is calmly sitting in the floor of this rack to assure the best to run endlessly, even in case of power outage for 40 minutes 

Edited March 23, 2023 by gyto6

[gyto6]

Reserved - Hardware Choices and how to skip bottleneck

Edited March 23, 2023 by gyto6

[gyto6]

Reserved - ZFS and Usage

Edited March 23, 2023 by gyto6

[doesntaffect]

well, that's a nice rig

Any special apps you run that, where do you see most benefit from all the Flash drives?

[BVD]

Not OP, but my use-case for seemingly 'crazy' amounts of NAND is CockroachDB - I used it to basically create a blobstore (sort of a distributed filesystem across multiple servers) so the services could survive a complete server failure/crash. As you can imagine, such a thing is highly latency sensitive, NAND (but most especially optane) is perfect for this stuff.

[Pri]

Great photos! looking forward to seeing more.

[gyto6]

Hi everyone,

 

As I added a picture of my new UPS, you'll notify that the picture is, for now, worst as I had to take it with my mobile phone. I'd take a better shot with my camera soon, I hope.

 

Whatever, I'd be glad to anwser some questions, but they'd be mostly answered within the 2nd and 3rd post. I simply want to fully explain each point, in order for neophytes to understand what a server might implies, in hardware as software, and detailed with pleasant visio files.

But It will take time, and I'm going through a really bad time in my life... So I'm no more located where my server is sitting, and I'm doing what I need to heal myself mentally...
I'd be glad to come soon to fill and finish this humble but passionate description about my server.

[Holle88]

unraid-porn (speaking in phrases of food-porn),

i downgraded my server today from a 3700x and 32GB RAM with GTX1070 to my mini-ITX J3455 for energy saving purpose.

(not used the streaming machine at all and my 2 websites and 10 docker containers can be easy driven be 4 cores with not SMT)

Idle power goes down from 130W to 50W  (drives not spinning)

[gyto6]

22 hours ago, Holle88 said:

unraid-porn (speaking in phrases of food-porn),

i downgraded my server today from a 3700x and 32GB RAM with GTX1070 to my mini-ITX J3455 for energy saving purpose.

(not used the streaming machine at all and my 2 websites and 10 docker containers can be easy driven be 4 cores with not SMT)

Idle power goes down from 130W to 50W  (drives not spinning)

🤣

 

That’s nice indeed! With a lot of works, I was able to drop the power consumption to 70W. It’s really tricky but interesting !

 

However, I forgot that I had to finish this post so thank for the reminder ;)

[BVD]

Hope things are going better for you these days @gyto6 ♥️

 

Looking forward to hearing more about the use cases and implementation!