SSD Cache Question

[sphbecker]

I am setting up my first unRaid server and planned to buy a decent spec SSD as a cache. However, after seeing the setting to choose a time of day to write the cache to primary story, that leaves me assuming this is a simple file-level cache, which would be far more limited. Block or byte-level cache would typically write back as soon as possible (while also keeping a copy in cache).

 

Is the cache nothing but a separate filesystem that holds new/updated files until they are written back? If so, then I assume it offers no benefit to VM disk images and doesn't do much to help read performance (unless the file you are reading happens to be in cache from a write that same day).

 

I hoped the SSD cache would function as storage auto-tearing (more frequently used bytes--not files--are on the SSD for rapid access). If it doesn't work that way, if it is basically just a faster staging area for write operations, then I will probably just drop an old 256 GB SSD in and keep my expectations low, instead of buying a high-speed 1 TB SSD.

Edited June 27, 2022 by sphbecker

[trurl]

https://wiki.unraid.net/Manual/Overview#Cache

 

https://wiki.unraid.net/Manual/Storage_Management#Pool_.28cache.29_Operations

 

There is no requirement for files to get moved from cache to the array. For better performance, you can have some shares configured to stay on a faster pool (such as cache). This is typically done with appdata, domains, system shares so docker/VM performance won't be impacted by parity, and so array disks can spin down since these files are always open.

 

https://wiki.unraid.net/Manual/Shares#Default_Shares

[sphbecker]

Yes, all those links indirectly confirm my theory about how unRaid uses the cache. Kind of a shame, that is about the least useful way to design a cache. I hoped it worked more like how PrimoCache for Windows works, or how a real storage array uses smart-tiering. I understand the limitation, XFS doesn't have native support for an SSD cache, so unRaid would have to design their own driver-level software, which would be a huge undertaking. Probably better for them to put that effort into switching to ZFS, which does have native support for cache drives.

 

I didn't plan to install 2 SSDs and keeping VMs on non-redundant storage doesn't work for me, but the good news is that my VMs don't need fast I/O so keeping them on a slower HDD based array is fine.

 

Glad I asked before wasting $100 on an SSD, thanks!

Edited June 27, 2022 by sphbecker

[trurl]

44 minutes ago, sphbecker said:

file-level cache

Each data disk in the parity array is an independent filesystem. This is how Unraid allows different sized disks in the parity array. Each file exists completely on one single disk. Cache has to work in a similar manner.

 

You might also want to take a look at how parity updates work.

 

https://wiki.unraid.net/Manual/Storage_Management#Performance

 

18 minutes ago, sphbecker said:

wasting $100 on an SSD

 

Many of us use fast pools of SSDs outside the parity array for performance. These can have redundancy.

 

https://wiki.unraid.net/Manual/Storage_Management#Pool_.28cache.29_Operations

 

Many other things in wiki that you might take a look at since there are some things about the way Unraid works that may be different than other systems you are familiar with.

 

 

 

[JonathanM]

5 minutes ago, sphbecker said:

the good news is that my VMs don't need fast I/O.

vdisk files living on the parity array are workable, but keep in mind random writes are going to be way slower than any typical disk. Orders of magnitude slower. I would never actively use a VM with vdisk files stored on the parity array, I only use them for tasks that aren't time sensitive, background processing stuff only.

 

9 minutes ago, sphbecker said:

confirm my theory about how unRaid uses the cache.

cache is a holdover term from when it was first introduced, back when 40pin IDE interfaces were the norm. At that point it was preferable to write new data at the normal speed of a single disk, then overnight that data could be moved at the parity write restricted speed to the storage array.

 

It never was a real read / write cache.

 

Nowadays the more accurate way to describe things is a single parity protected array, with multiple pools of single or BTRFS RAID array drives for higher speed usage. The parity protected array is still intended as a low speed mass storage destination with flexible disk size and single disk recovery even when you exceed the parity recovery threshold. If you have dual parity, you can recover from 2 disk failures, more failures than that and you can still read all the good remaining disks, instead of losing it all like traditional RAID arrays when you exceed the fault tolerance.

 

16 minutes ago, sphbecker said:

a real storage array uses smart-tiering.

Unraid isn't positioned as an enterprise SAN, it's a hobby OS that can be used in a business setting with careful management. With tweaking and high end parts, it's capable of quite a bit, and as a media station, it's formidable.

 

Based on the tone of your posts I'm afraid your expectations are much greater than Unraid is capable of delivering.

[sphbecker]

Thank you both for your feedback. Yes, I understand unRaid isn't Enterprise class, but being a nerd I enjoy going for the most powerful configuration I can...even if I don't need it :-).

My VMs and Dockers (so far) include an OPNsense Firewall, Home Assistant and UniFi controller. None of those use a lot of I/O, so other than slightly slower boot times, I am really not worried about array performance.

 

I'd love to see unRaid move to ZFS...I feel like that would be a huge win for them, allowing them to get a lot of these more advanced features for basically free.

[trurl]

11 minutes ago, trurl said:

Many of us use fast pools of SSDs outside the parity array for performance

Another reason is so array disks can spin down. Another benefit of the independent filesystems is so disks not in use can spin down. Dockers and VMs will keep some files open and so keep disks spunup.

 

4 minutes ago, sphbecker said:

ZFS

That is planned for future. There is already a plugin for ZFS

 

[JonathanM]

3 minutes ago, sphbecker said:

I'd love to see unRaid move to ZFS

Already available as a third party modification.

 

3 minutes ago, sphbecker said:

other than slightly slower boot times

Not sure what you are getting at. Unraid unpacks the OS into RAM, and boots from there. No impact from disk formats.

 

Or are you talking about boot times of the VM's? In which case yes, a 10 second boot from a decent NVME drive vs. 3 minutes on the parity array is probably about normal.

[trurl]

6 minutes ago, sphbecker said:

enjoy going for the most powerful configuration I can

Then you definitely want multiple pools of fast SSDs with redundancy.

[sphbecker]

14 minutes ago, JonathanM said:

Not sure what you are getting at. Unraid unpacks the OS into RAM, and boots from there. No impact from disk formats.

I was talking about VM boot times, not unRaid.

 

13 minutes ago, trurl said:

Then you definitely want multiple pools of fast SSDs with redundancy.

Fair, I guess I meant "most powerful configuration I can have with a reasonable amount of effort and money." I don't want to pay for 2 SSDs for this purpose right now. I will monitor the disk spin-down comment, that might be reason enough to pay for SSDs.

[JonathanM]

1 minute ago, sphbecker said:

I don't want to pay for 2 SSDs for this purpose right now.

No need, you can run just 1. RAID is never a substitute for backups anyway, I personally have single volume pools with regular backups in the VM set to write to the parity array.

 

59 minutes ago, sphbecker said:

keeping VMs on non-redundant storage doesn't work for me

In my experience, it's better to keep good backups than worry about redundant storage for the base OS.

 

I too run my firewall (pfSense vs OPNsense) as a VM on Unraid, and regular XML backups allow both quick rebuilds if needed as well as temporary transitioning to a hardware firewall when needed for longer maintenance tasks on the main array.

 

I use homeseer instead of home assistant, but it's probably a similar situation, where standing up a new OS and restoring from a backup is quick and easy.

 

It's not worth it for me to dedicate a second SSD for redundancy. I'd rather keep good backups that allow moving to completely new installs if needed rather than wasting hardware on a single failure mode recovery solution. Disk redundancy only helps with drive failure, it can't deal with the myriad of other risks that backups cover.

 

The parity array is another risk assessment, and having disk failure recovery that only "wastes" one or two drives to cover the entire array of 11 (for one of my servers) drives is worth it, and I don't have backups for all the media that is recoverable by re-ripping blu-rays or whatever if the worst case happens.

 

Regardless of disk redundancy, you must have backups elsewhere of anything important. A power supply failure is all it could take to wipe out all your drives, redundant or not.

[sphbecker]

26 minutes ago, JonathanM said:

No need, you can run just 1. RAID is never a substitute for backups anyway...

 

All valid points. RAID offers redundancy for better uptime, while backups are for disaster recovery. The main value of RAID is so that systems remain running after a failure so that you don't have to rush to restore from a backup (even if that process is fast and easy). This is actually what I do for a living, so I understand the best practices, I am just trying to figure out the best way to meet them with unRAID.

 

You are correct that OPNsense and UniFi both have super simple backup/restore operations. Home Assistant does have a simple backup/restore, however with the way its plugins work there is some manual setup needed. Basically, you log into the fresh vanilla install, add any plugins you were using (which may be more complicated than it sounds if using unsupported "HACS" plugins) and then you can restore your backup.

 

I prefer a solution that can remain running without me having to do anything, but I guess I am already accepting the unRAID server as a single point of failure, so adding the SSD to the list isn't the end of the world. I don't even have a redundant power supply, so I really can't get on my high-horse about fault tolerance 🙂 

 

Another project I am looking at is finding an old/cheap embedded x86 device with at least 3 NICs to use as a HA failover for OPNsense. It doesn't need to be powerful enough to keep up with full gigabit traffic, but would automatic failover and keep the network running if the OPNsense VM went down for any reason, even a total unRAID failure. Super overkill for a home network, but would be a fun project. The only thing I am unsure of is exactly how CARP's virtual address works, if it maintains a consistent virtual MAC address across both devices, or if it only layer-3 IP address that stays the same and the layer-2 MAC address changes. I am not sure my ISP would play nice with that WAN MAC address changing, that typically requires a modem reboot.

[JonathanM]

10 minutes ago, sphbecker said:

Another project I am looking at is finding an old/cheap embedded x86 device with at least 3 NICs to use as a HA failover for OPNsense. It doesn't need to be powerful enough to keep up with full gigabit traffic, but would automatic failover and keep the network running if the OPNsense VM went down for any reason, even a total unRAID failure. Super overkill for a home network, but would be a fun project. The only thing I am unsure of is exactly how CARP's virtual address works, if it maintains a consistent virtual MAC address across both devices, or if it only layer-3 IP address that stays the same and the layer-2 MAC address changes. I am not sure my ISP would play nice with that WAN MAC address changing, that typically requires a modem reboot.

Good luck, let us know how that goes. I tried to do something very similar a while back and gave up. I keep an old low power no fan system semi up to date with pfSense so I can power it up when needed. I've never had an issue with the modem needing to reboot, but I've also never had both ports up simultaneously. It doesn't seem to mind if one MAC goes offline and a second MAC comes up assigned with the same IP. Maybe it automatically releases when it detects no active ports.

 

I don't have HA, I have to be onsite to power up the second machine while ensuring the first is fully down, but it's close enough for me. I so rarely have to touch the machine, I think it's been over a year since I've had the other pfSense running. Probably time to make sure it still works. 🤣

[Michael_P]

In HA, the modem should only see the virtual interface

[sphbecker]

On 6/27/2022 at 11:54 AM, Michael_P said:

In HA, the modem should only see the virtual interface

Actually you don't really want to use a virtual interface at all for pfSense or OPNsense on unRAID. The native KVM VirtIO NIC driver does not work on FreeBSD guests, so you can only use the Intel or VMWare emulated drivers, and those have a massive performance penalty. That is not theoretical; I tried it and was only able to get about 300mbps with VMWare and 200mbps with Intel.

 

I had an old Intel 4x gigabit server card from another project, I put that in and used SR-IOV to pass 2 of its ports to the VM as PCIe devices so that OPNsense can use them natively. One port is assigned as WAN and plugs into the modem, the other is LAN and plugs into my switch. Another port on that same card is assigned to the unRAID host and plugs into the switch. That might seem silly to have two connections from the same physical system plugged into the same switch, but any other configuration would result in a big performance drop (also, I have gigabit internet and only a gigabit switch, so this config allows one user to potentially max out the internet connection while another maxes out data transfer to unRAID and they will not have to share the same gigabit connection).

 

With proper interaction between your hypervisor and virtual firewall using a virtual interface is fine. I did this exact same setup on HyperV and got full bandwidth on virtual NICs. I am sure a Linux based firewall on unRAID would also work, but not a FreeBSD based firewall.

Edited July 12, 2022 by sphbecker

[Michael_P]

14 hours ago, sphbecker said:

on unRAID

 

I didn't say anything about using unraid to host it