First time caller. Long time listener.


Grimlok

Recommended Posts

Hey Team!

Been lurking around about 2 years with the plan that when my current server (6+ years old) starts to give up the ghost. I’ll take the plunge into unraid.

 

Quick into. I like fast cars. Music with guitars. And a cold beer on a hot day. I’ve been voiding the warranty since I could hold a screwdriver. I break fix for a living. And love to tinker around with pretty much anything.

 

Saw a chassis that was exactly what I wanted. 8x3.5 internal. 3x5.25 external. 1x3.5 external. Rack mounts. ATX mobo and psu mounting. Let’s me chuck 12x3.5s and 2x2.5 SSDs in.

 

That’s a 6xSata mobo with an 8 way sas addon. Room for some pci expansion to throw unassigned drives in a couple PCI racks, and 4xDIMMs to eventually throw a stupid amount of RAM for a file server in.

 

A reasonably powerful cpu as I intend on running at least a couple VMs (though not at the same time.) Plex. MySQL. NFS. SMB. Kodi. I’d like to run a tvheadend server with multiple tuners. But I haven’t done a lot of research into the pvr aspect. Headless. Deals with itself after a power failure. And will talk to a ups.

 

I work For a small mom and pop shop. So I’ve managed to take advantage of my employee discount on a lot of this build.

 

Drive wise. I’ll be looking at WD Red 8 tb drives. Starting with a 24tb array. And .5tb cache across 2x256mb NVMe drives. (Or SSDs. I haven’t finished weighing up all the pros and cons of each yet) Excuse the upside down box shot. I’m left handed and always end up with upside down photos. And I’m faaaaar to lazy to flip them around)

 

I think at this stage an i5 quad or hex core around 3ghz is plenty for what I want to do at this point in time.

 

Initially I was after a full ATX. But I was willing to loose a couple of PCIe slots for NVMe. As long as it was genuine NVMe and not going to disable a sata port.

 

This showed up at a stupid price for the features around the same time as I found the case. (Ok. I found the case and started spec’ing it out and couldn’t resist. Don’t tell the wife)

 

Asus prime H370M-plus. Ticks more of the boxes than anything else I could find.

 

the board it’s replacing is an old gigabyte p45 chipset with a 771 to 775 hack allowing me to run a 3.6ghz quad core Xeon 771 socket on a 775 board with 16 gb of ram. So if running the vms isn’t an option. I can always repurpose the old server rig as my daily. 

 

So. Opinions on NVMe vs SSD. Is the shorter lifespan of the NVMe the only real downside? Besides not really being that much faster? If at all? From what I understand running them as un assigned drives next to a SSD cache will wear both the SSD and NVMe anyway. So better off wearing just one set of drives out. Is there any advantage to running SSDs at all if I’m hell bent on using the NVMe slots?

FBC2446A-673E-4A4D-AE32-36F7145FD79D.jpeg

3AD8282C-3F85-48F6-A560-E900AA64BE7F.jpeg

Link to comment
2 hours ago, Grimlok said:

Drive wise. I’ll be looking at WD Red 8 tb drives. Starting with a 24tb array. And .5tb cache across 2x256mb NVMe drives. (Or SSDs. I haven’t finished weighing up all the pros and cons of each yet) Excuse the upside down box shot. I’m left handed and always end up with upside down photos. And I’m faaaaar to lazy to flip them around)

 

So. Opinions on NVMe vs SSD. Is the shorter lifespan of the NVMe the only real downside? Besides not really being that much faster? If at all? From what I understand running them as un assigned drives next to a SSD cache will wear both the SSD and NVMe anyway. So better off wearing just one set of drives out. Is there any advantage to running SSDs at all if I’m hell bent on using the NVMe slots?

Hey welcome!

Kudos to cold beer on a hot day. I like to put my beer in the freezer for 10 mins just make sure it's extra cold. Brits drink beer way too warm!

 

About your HDD. Don't get all of them at the same time or if you have to, try to get them from different sellers. That's to avoid possibility of buying all your drives from the same bad batch.

Also remember, for Unraid, you do NOT need to get NAS-rated HDD i.e. you don't have to get WD Red if there are cheaper options.

(I personally also recommend to avoid SMR drives e.g. Seagate Archive if you are going to store a lot of small files)

 

Now about SSD, there are some misconceptions here.

(note: "SSD" includes both SATA and NVMe so what you refer to as "SSD" is actually SATA SSD. All my points below, "SSD" means BOTH types unless I specify otherwise).

  • NVMe SSD lifespan is only shorter than SATA SSD under a perfect storm sorta scenario.
    • NVMe runs hotter because it's faster but it only runs hot under heavy IO. You really have to constantly bombard it with IO for the heat build up to be significant.
    • Such heat is easily dissipated just by having air flow over the SSD so for it to affect the lifespan, you need to have no air flow, which is harder to achieve than you think.
    • Of course, it only applies to M.2 form factor. It never affects the other form factors e.g. PCIe (air flow is pretty much guaranteed + usually come with integrated heat sink) or 2.5" (enterprise level stuff always have heavy heat sink and cooling).
  • The lifespan / endurance of SSD is heavily misunderstood. The warranty / endurance rating by manufacturers is extremely supremely conservative. An SSD rated for 100 TBW (TB written) absolutely does NOT mean it will die after 100TB has been written! SSD tends to fail gracefully as dead cells are replaced by reserved cells. Even when reserved cells have been depleted, the rest of the cells will continue to function and you will just lose capacity as more cells die.
    • Just to illustrate how hard it is to kill an SSD, 2 of my SSDs (1 M.2 and 1 SATA) are way over the TBW rating and they just keep on running like Rambo. The M.2 one, in particular, is right under my GPU so it gets full blast of heat throughout every gaming session - IDLE temp reaches 60+ Celsius!
    • SSD wears out by having data written to it (and not by reading data from it). So if you write some data to the SSD and just leave it there, you are not wearing it out.
      • Age is a factor but it is so minor that it can be ignored. Many components will die before a SSD cell will die from age alone.
  • NVMe SSD is faster than SATA SSD, always. The key is to have the right condition for such speed to be valuable to you.
    • When mounted in the cache pool, for max performance, you need to access the NVMe through /mnt/cache (instead of /mnt/user). That bypasses the overhead by shfs (which is insignificant with SATA but majorly significant with NVMe).
      • When mounted as UD, you will by pass shfs by default to you will get full speed.
    • NVMe allows parallelism (i.e. multi-tasking). In practice, this means NVMe won't grind to a halt under heavy IO. This happens too many times to me with SATA SSD to count.
  • Whether there is a point to have both SATA and NVMe SSD or not entirely depends on your use case. Particularly for Unraid, NVMe SSD is valuable because it saves you from needing more SATA ports that can be used to expand your array.
    • If you are going to get NVMe SSD, avoid QLC NVMe (e.g. Intel 660p) and DRAM-less models. Both are pointless as NVMe devices.
    • If you are planning to pass through a NVMe SSD to a VM using the PCIe method, make sure you google the controller of the NVMe before buying because some (e.g. those running SM2263 controller) requires special workaround / limitation and some (e.g. Intel 660p) just refuses to be passed through.
  • Regardless if SATA or NVMe, you should still engage in good practice with SSD to maximise its endurance.
    • Run TRIM frequently (there's a plugin for that).
    • Minimise the amount of static data stored on a SSD, especially one that is used for heavy-write activities.
      • The more free space available on an SSD, the less write amplification will take place and thus the longer it will last. Writing 100 TB to a 90%-free and a 90%-full SSD's cause majorly different amount of write amplification.
    • Never pre-clear an SSD.
Link to comment

Shudder. Warm beer. I too am guilty of chucking a few in the freezer to get them nice and cold if I’m in the workshop tinkering with something.

 

I can’t see heat being a major problem. I won’t be running any gfx card. Both the NVMe slots are on top of the mobo. And I can easily leave the top off the chassis and run 6 120mm fans pulling air accross the top of the whole chassis. As well as a couple 120s in front of drives. Only a couple of 80s out the back. But If I add some sinks to the NVMe sticks and the ram. It should cool ok. Noise isn’t a problem. As my rack is in a seperate workshop/garage from the house.

 

What you say about static data makes total sense. But I’m in a worst use case. MySQL and plex databases are mostly static. VMs are mostly static. They will benefit most from Solid state speed. Yet I might as well run my pool cache on the same drives. And PVR will use them for recording. Good to know about the NVMe controller. My options are pretty wide open here. I haven’t looked into it but We stock several brands at work.

 

Yup. Standard practice when building arrays is to spread it across suppliers so as to ensure variety of batch. Personally not a fan of Segate drives. But that is entirely a personal opinion. I currently have 2x4tb wd greens. 2x2tb WD greens. 4x2tb hitachi hgsts and a 1tb WD caviar. Just looking to get rid of the drives <2tb for now and build the array.

 

Im pretty OCD about the layout inside a case. I’ve got 8x3.5 spaces set up as 2x4 bays on the left side of the case I’d like to keep free for an 8x sas card. Run either a 4 into 3 bay in the 5.25 bays with a 2x2.5 in the external 3.5 bay. Or a 5 into 3 with a single 3.5 in the External. Using all 6 onboard Sata ports.

 

Which is the direction I’m going. As it allows me to take advantage of the M.2 sockets and free up more Sata ports, and get the total array to around 100tb with 8tb drives. IE enough room to not have to really do anything other than throw more drives in it for at least a couple years.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.