High performance Unraid storage


Recommended Posts

Looking for a bit of a gut check here.

 

I'm needing a high performance storage solution - this unraid box will be doing nothing but minio or maybe SMB shares for backups in a corp setting.

 

Inbound, I have 3 servers (all SSD) on 10g networks each.  

 

Currently, total nightly diff data backup is about 1tb, full weekends are 7tb-ish.  My weekend backup windows are currently 35+ hours on a gigabit network.

 

Plan is 4 port 10gbe nic (2 ports in use) in UnRaid, running 10, 10TB SAS He10 spinners on a 6 or 12gb backplane.

 

My question is real speed.  Cache side and saturation.

 

Am I wasting my time and money on dual 8tb nvme's or SAS SSD's?  

 

I'm trying to get my backup windows down to a much more reasonable timeframe and would be fine with loading up the cache for speed purposes to then offload onto the spinning disks.

 

What are the real world thoughts on actual speeds - both for 6 and 12g backplanes for the spinners, as well as possible 6 or 12g for the SAS SSD's, or all out nvme riser?

 

Thanks for any input....

 

 

 

 

Link to comment

SAS spinners won't make much if any difference, maybe if you use 15krpm devices, SSDs or NVMe devices yes, I have an SSD pool thant can sustain writes of about 800MB/s, same for single device NVMe cache, but note that if your data is mostly small files it will likely be slower, but still faster than spinners. 

Link to comment

Problem with either SSD or NVMe is once you fill up the onboard write cache (and/or hit the temperature throttle) your write speeds will drop significantly, defeating the purpose of what you're trying to accomplish by using them.

 

This has been a hot topic of debate lately but there's enough articles out there talking about this issue to completely ignore it.  I don't know if there are better types of drives made for a corporate type of use vs a home consumer, but your average consumer SSD/NVMe drive is only going to have less than a couple hundred GB of fast write cache before it starts to slow down.  Google it.

 

My only suggestion is to reconsider whether you actually need those backups :P

Link to comment
10 minutes ago, Energen said:

Problem with either SSD or NVMe is once you fill up the onboard write cache (and/or hit the temperature throttle) your write speeds will drop significantly, defeating the purpose of what you're trying to accomplish by using them.

OP mentioned 8TB NVMe devices, that should be enough for the weekend backup, or a multi-SSD cache pool, I have a 6TB usable SSD pool that I use for day to day data, so it's always fast, array is just for cold storage.

Link to comment

Benchmark results on the Sabrent Rocket Q 8TB NVMe, the only 8TB that popped up as the first result.

 

Sabrent’s Rocket Q features a massive dynamic SLC write cache that spans a quarter of the SSD’s available capacity. The 8TB Rocket Q wrote a little over 2.1TB of data at 2.9 GBps before degrading to an average speed of 276 MBps after the write cache filled.

 

So on a drive like this, I guess it would depend on how fast the write cache is cleared out and written to memory. 

 

I'm not an SSD expert, but if you're writing 8TB to an 8TB drive, and the write cache fills up at 2TB before slowing down to 276MBps, that sounds like it's going to be 6TB at slow speeds as I guess it only clears out the cache at around 276MBps? Same speed out, same speed in.

 

 

Link to comment

On the topic of file sizes.. if I run MinIO, I can control the file chunk size up to 5gb I think.  Same if I use SMB - I expect to be writing 1 very large file (lets say my bigger SQL server is 4tb compressed).  

 

I'm game for stacking the backup window one after another if it seems that is the preferred method to let the SSD bake a bit.

 

My SQL server is 4TB compressed.  Currently in one file for weekly, and then smaller daily chunks.

 

Maybe run that 4tb backup job, let it rest for a bit, and then use the unraid mover service to get the file off the SSD, then run the other backup jobs?

 

Seems that HHHL AIC would be my preferred installation method.  PBlaze looks bad ass, but non existent inventory it seems.

Edited by TheSnotRocket
Link to comment

I'm aware of the cache limitations of the consumer nvme's - the sabrent Q was my first thought as well.

 

So, maybe ideally, I'm looking at a pair of these:

 

http://www.acmemicro.com/Product/16580/Intel-NVMe-7-6TB-Solid-State-Drive-SSDPE2KE076T801-DC-P4610-Series-U-2-15mm-3200-MB-s-Read-3D2-TLC-NAND?c_id=622

 

With... an adapter like this?  

 

https://www.amazon.com/StarTech-com-U-2-PCIe-Adapter-PEX4SFF8639/dp/B073WGN61Y?th=1

 

(edit) oh look, they even have a picture of a similar drive on that adapter - pex4sff8639.e.jpg

 

I can't find any info on the cache levels or sustained write on the Intel drives...

Edited by TheSnotRocket
Link to comment
44 minutes ago, jonathanm said:

Keep in mind that because of the way Unraid works, each file can only occupy a single drive. So you are limited in the size of file that you can write by the free space on any single drive.

Thanks for the reminder on that - I guess I'm good with that.  10, 10TB drives, 8 useable gives me 16 backups of my SQL server (4tb) and I only keep 5 full weeks.  My other backups are smaller large files (couple hundred gig each all the way down to 50gb).  I can always add more drives into the array if I need.

Link to comment
  • 3 weeks later...

Just an update on this.

 

I ended up with this config:

 

SuperMicro 12x Bay 2U with SAS3 Backplane
Processors: 2x Xeon E5-2620 v3 2.4GHz 6-Core CPU's
Memory: 96GB (12x 8GB) DDR4 Registered Memory
Storage Controller: LSI 9311-8i (IT MODE)
PCIe Network Card: Intel X520-DA2 Dual Port 10GbE SFP+ PCIe Adaptor
Power Supply: 2x 1000W 80+ Platinum Power Supplies

Added in the above Intel SSDPE2KE076T801 SSD, running with that StarTech adapter.

 

I'm able to write my full backup set to the SSD at full speed (sustained 930+MB/sec) and then blow the data out to the SAS drives at a whopping 135ish MB/sec sustained transfers.

 

Super happy camper.

 

Primary box is a 22 Disk + 2 Parity (All SAS) + 2tb SSD drive, 215TB usable server for home use, this second one is 5+1 (3 disk + 2 parity, 30TB usable all SAS currently) + 1 cache drive for business use.  I'll probably throw in 5 more, 4 TB SATA's to this box as I pull them out of the QNAP it's replacing.

 

Thank you folks.  This makes for my second unRaid config.  Both have been mostly super easy :)

 

 

 

Edited by TheSnotRocket
  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.