Hi All,

 

TLDR:  Media Center (windows) / Gaming (windows) / large NAS (unRaid) / DNS Host + Plex + Torrent related Dockers (unRaid) - Looking for the optimal drive configuration when considering Cache SSD (or HDD) vs SSD outside of the array.

 

I've done a bit of searching on this subject but haven't found a solid recommendation. 

 

My configuration options seem to be:

1. Single PCIE SSD cache (512GB)
2. Multiple drives set up as cache - not sure if it would be possible to separate file transfer (write to main array) cache from VM only cache
3. 1X 256-512GB Cache (HDD or SSD) & 1x 256-512GB PCIE SSD outside of the array  (Is there any benefit to doing this?)

 

The use cases are:

• Docker images that run regularly
• Windows VM
• Applications which use the windows VM

 

I don't want to spend more than I need to, and if possible I would like to reduce complexity by limiting the number of drives that I have (In favor of Cache only).  However, I want to ensure that the windows VM is not unnecessarily spinning up drives, and is not bottlenecked by other processes which may be happening on cache - I will be using the windows VM regularly for gaming and media center.

 

Best,

Rob

 

 

Edited January 26, 20188 yr by cablecutter

SSDs can serve many needs. The decision rests with what uses you have planned. Below is an indepth coverage of the topic that I hope to use to direct others that ask this rather common question:

 

1 - Cache was originally conceived as a landing zone for files, e.g., from a workstation. When this feature was added, it was not uncommon for writes to be as slow as 8-10 MB/sec. These transfers were very lengthy, and people were annoyed they had to wait seemingly forever for a digital movie to transfer, for example. (Overnight unRAID moved these files to the array - which was very slow - but since you were sleeping it doesn't matter). But things have changed. unRAID writes are now often much faster - 40-60 MB/sec+. And there is a new write feature called "reconstruct writes" that virtually eliminates the write penalty at the expense of spinning up the entire array. When using cache as a landing zone, files added to cache are NOT protected by parity. So adding redundancy (i.e., a RAID1 cache), files were protected when they hit the cache. But like I said, I suggest turning OFF caching on all shares, and using reconstruct write mode when you have heavy writes to do that you want to finish quickly. That will more than saturate a gigabit LAN, and works quite well.

 

2 - Work area for downloads. Some of our users download media from various sources. Those downloads often require checking / correcting errors (e.g., PAR processing) and decompressing (e.g., RAR). The cache drive can act as the location to improve performance. Using cache would make these operations quite fast. Problem is this is very high I/O - and SSDs lifespans are tied to the number of writes. So the question is - how much is it inconveniencing you to have these operations take their own sweet time? Only you can decide, but many users opt to not use cache for this. The alternative to use an array disk is pretty painful. It often involves writes multiple write streams which slug down array writes even further. If this is a high volume it can lag very far behind downloads. An option is to use an unassigned device (UD), which is a disk running outside the array. Unrestrained by parity, the I/O is much faster. In many cases a UD spinner can keep up with downloads, but it depends on the DL speed. If it is too slow, a second SSD - maybe an older one you're not using for anything else, and if you wear it out it is not a big issue. Or you can use the cache drive, understanding the pros and cons this is perfectly fine. I don't think this requires to be backed up, so no need for RAID1 IMO.

 

3 - VM images. VM images are a primary use case for the cache drive IMO. It is basically your "C" drive inside a Windows VM. Just as in a bare metal computer, booting and running from an SSD makes a huge difference. Adding a second SSD and creating a RAID0 array can make the access even faster. Depending our your system, this may or may not be meaningful. NVMe drives are also faster and running them in a RAID configuration raises performance further. I recommend storing VM images on an SSD/NVMe cache drive. Writes to the VM image are typically not excessive. If you are doing high I/O operations inside the VM, you can always use array or UD devices for that I/O, and not have to write all to your C: drive. (I found a way to move the "TEMP" directory off my C: drive and hence off cache to eliminate these often high volume writes. Trickier than I thought, but it works. Ask if there is interest and I can explain how to do this.) RAID1 can be useful I suppose for backup (or RAID10 for both speed and redundancy). I personally don't. I just backup the VM image from time to time to the array. And if it ever crashes, I can go back to that. Virtually all of my data lives on the array or UDs so only newly installed apps (few and far between) and maybe OS/antivirus updates are lacking, and they take care of themselves).

 

4 - Appdata. This is another primary use case for cache. An extremely popular unRAID feature is dockers. Dockers typically store their configuration data to an "appdata" share. Having this data on a fast drive makes the Dockers faster, and most users opt to do this. The space consumed is typically small EXCEPT for media apps like Plex (which I will cover separately below). I recommend putting this on the cache drive, although you can put on a UD (spinner or SSD) or even on an array disk with little impact for most Dockers. Different Docker config data can be put in different places, but most just use the cache appdata share. RAID0 for this makes little difference IMO. RAID1 would provide redundancy and help overcome a failure. But typically appdata can be reconstructed with only a minor inconvenience, esp. if you have a backup of the config files for the Dockers. I don't consider RAID1 to be required here personally.

 

5 - Big appdata (e.g., Plex). Plex stores its metadata library in its config directory. This can get quite large. And be made up of a huge volume of tiny files. It is quite annoying actually. (Try to copy it, or even get a size of it, sometime). If there is a use can to move a docker's appdata to a different disk, Plex and similar apps are the prime candidates. An SSD does substantially speed up media browsing, so it definitely deserves to be on an SSD IMO. So most people just keep it in their appdata share on the cache drive and grumble a little. This is an area I would like redundancy. I tweak my metadata (posters, tags, sort titles, etc.) to my preferences - and losing that and reconstructing would be a PITA. (But whether it is worth 2x the cost for 2 SSDs, I'm not sure. I'd rather just be able to back it up easily.) I'm actively exploring creating a loopback device on the cache disk for this which would make backing it up quite easy. Don't think anyone has done this yet. I am actually moving my array to a new server and redoing all of my Dockers and settings, and will start populating to a loopback device. I will be doing the metadata rebuild this one time and endure the PITA. It will allow me to restructure which I've wanted to do for a while.

 

6 - Working disks from a VM. While the VM disk itself is on the cache, while working inside the VM you can write to any array or UD on the server. You can even map drive letters to different drives / shares on the server. (Remember everything on the server is local to the VM, and despite looking like they are network drives, they are very near native speed). I have a UD spinner dedicated to my VM for day to day stuff that gets a drive letter. And have another letter for an array location for personal files that I want stored on the array. I also have one to my second SSD, which I use to store VMware disk images that I use inside the KVM VM (yes that does work with one setting change). This is a big comparably slow SSD but gives very good performance to my VMware VMs that I've had for years.

 

Hope this helps. Encourage others to post their thoughts on optimal use of SSDs in unRAID. Most of this is based on my experience, and I'm sure others will disagree or have other suggestions.

 

(#ssdindex - Using SSDs in your array)

 

SSDs can serve many needs. The decision rests with what uses you have planned. Below is an indepth coverage of the topic that I hope to use to direct others that ask this rather common question:
 
1 - Cache was originally conceived as a landing zone for files, e.g., from a workstation. When this feature was added, it was not uncommon for writes to be as slow as 8-10 MB/sec. These transfers were very lengthy, and people were annoyed they had to wait seemingly forever for a digital movie to transfer, for example. (Overnight unRAID moved these files to the array - which was very slow - but since you were sleeping it doesn't matter). But things have changed. unRAID writes are now often much faster - 40-60 MB/sec+. And there is a new write feature called "reconstruct writes" that virtually eliminates the write penalty at the expense of spinning up the entire array. When using cache as a landing zone, files added to cache are NOT protected by parity. So adding redundancy (i.e., a RAID1 cache), files were protected when they hit the cache. But like I said, I suggest turning OFF caching on all shares, and using reconstruct write mode when you have heavy writes to do that you want to finish quickly. That will more than saturate a gigabit LAN, and works quite well.
 
2 - Work area for downloads. Some of our users download media from various sources. Those downloads often require checking / correcting errors (e.g., PAR processing) and decompressing (e.g., RAR). The cache drive can act as the location to improve performance. Using cache would make these operations quite fast. Problem is this is very high I/O - and SSDs lifespans are tied to the number of writes. So the question is - how much is it inconveniencing you to have these operations take their own sweet time? Only you can decide, but many users opt to not use cache for this. The alternative to use an array disk is pretty painful. It often involves writes multiple write streams which slug down array writes even further. If this is a high volume it can lag very far behind downloads. An option is to use an unassigned device (UD), which is a disk running outside the array. Unrestrained by parity, the I/O is much faster. In many cases a UD spinner can keep up with downloads, but it depends on the DL speed. If it is too slow, a second SSD - maybe an older one you're not using for anything else, and if you wear it out it is not a big issue. Or you can use the cache drive, understanding the pros and cons this is perfectly fine. I don't think this requires to be backed up, so no need for RAID1 IMO.
 
3 - VM images. VM images are a primary use case for the cache drive IMO. It is basically your "C" drive inside a Windows VM. Just as in a bare metal computer, booting and running from an SSD makes a huge difference. Adding a second SSD and creating a RAID0 array can make the access even faster. Depending our your system, this may or may not be meaningful. NVMe drives are also faster and running them in a RAID configuration raises performance further. I recommend storing VM images on an SSD/NVMe cache drive. Writes to the VM image are typically not excessive. If you are doing high I/O operations inside the VM, you can always use array or UD devices for that I/O, and not have to write all to your C: drive. (I found a way to move the "TEMP" directory off my C: drive and hence off cache to eliminate these often high volume writes. Trickier than I thought, but it works. Ask if there is interest and I can explain how to do this.) RAID1 can be useful I suppose for backup (or RAID10 for both speed and redundancy). I personally don't. I just backup the VM image from time to time to the array. And if it ever crashes, I can go back to that. Virtually all of my data lives on the array or UDs so only newly installed apps (few and far between) and maybe OS/antivirus updates are lacking, and they take care of themselves).
 
4 - Appdata. This is another primary use case for cache. An extremely popular unRAID feature is dockers. Dockers typically store their configuration data to an "appdata" share. Having this data on a fast drive makes the Dockers faster, and most users opt to do this. The space consumed is typically small EXCEPT for media apps like Plex (which I will cover separately below). I recommend putting this on the cache drive, although you can put on a UD (spinner or SSD) or even on an array disk with little impact for most Dockers. Different Docker config data can be put in different places, but most just use the cache appdata share. RAID0 for this makes little difference IMO. RAID1 would provide redundancy and help overcome a failure. But typically appdata can be reconstructed with only a minor inconvenience, esp. if you have a backup of the config files for the Dockers. I don't consider RAID1 to be required here personally.
 
5 - Big appdata (e.g., Plex). Plex stores its metadata library in its config directory. This can get quite large. And be made up of a huge volume of tiny files. It is quite annoying actually. (Try to copy it, or even get a size of it, sometime). If there is a use can to move a docker's appdata to a different disk, Plex and similar apps are the prime candidates. An SSD does substantially speed up media browsing, so it definitely deserves to be on an SSD IMO. So most people just keep it in their appdata share on the cache drive and grumble a little. This is an area I would like redundancy. I tweak my metadata (posters, tags, sort titles, etc.) to my preferences - and losing that and reconstructing would be a PITA. (But whether it is worth 2x the cost for 2 SSDs, I'm not sure. I'd rather just be able to back it up easily.) I'm actively exploring creating a loopback device on the cache disk for this which would make backing it up quite easy. Don't think anyone has done this yet. I am actually moving my array to a new server and redoing all of my Dockers and settings, and will start populating to a loopback device. I will be doing the metadata rebuild this one time and endure the PITA. It will allow me to restructure which I've wanted to do for a while.
 
6 - Working disks from a VM. While the VM disk itself is on the cache, while working inside the VM you can write to any array or UD on the server. You can even map drive letters to different drives / shares on the server. (Remember everything on the server is local to the VM, and despite looking like they are network drives, they are very near native speed). I have a UD spinner dedicated to my VM for day to day stuff that gets a drive letter. And have another letter for an array location for personal files that I want stored on the array. I also have one to my second SSD, which I use to store VMware disk images that I use inside the KVM VM (yes that does work with one setting change). This is a big comparably slow SSD but gives very good performance to my VMware VMs that I've had for years.
 
Hope this helps. Encourage others to post their thoughts on optimal use of SSDs in unRAID. Most of this is based on my experience, and I'm sure others will disagree or have other suggestions.
 
(#ssdindex - Using SSDs in your array)
 

Hi, really interesting summary you wrote. Could you elaborate further how to move off TEMP from your c:?

8 minutes ago, luca2 said:

Hi, really interesting summary you wrote. Could you elaborate further how to move off TEMP from your c:?

 

Yes - if you experiment a little, you will learn that assigning the TEMP directory to a network drive doesn't work. The network drive, even if given a drive letter, is not accessible in administrator mode. And since the TEMP directory is used in administrator mode, you'll notice that some things just won't work. 

 

So to get around that, you need to assign the TEMP directory to a native disk (at least what Windows thinks is a native disk). And you only have one of those - and that is the C: drive. What you need to do is to create a second disk inside your VM. You do that with the VMmanager GUI. You want to put that second disk on a UD. You can size as you like - I think mine is 20G. That disk will then be assigned a drive letter at boot (mine in the D: drive). Youll have to partition and format that disk. Then make a "TEMP" directory and map the TEMP environment variable to the D:\TEMP. Windows is happy and all temp files are written to a UD.

 

I usually point the "TEMP" and "TMP" environment variables to that directory: TMP may be a holdover from an earlier age, but I still map it just in case. No harm.

 

Directions for Windows 10. (Windows 7/8 are similar or the same):

Go to Control Panel -> System -> Advanced System Settings -> Advanced tab (should already there) -> Environment Variables. Use the System variables section (lower part of the panel). Do not make the changes in the User variables for {userid} section. After changing, click OK. I think you have to reboot. 

 

Good luck!

 
Yes - if you experiment a little, you will learn that assigning the TEMP directory to a network drive doesn't work. The network drive, even if given a drive letter, is not accessible in administrator mode. And since the TEMP directory is used in administrator mode, you'll notice that some things just won't work. 
 
So to get around that, you need to assign the TEMP directory to a native disk (at least what Windows thinks is a native disk). And you only have one of those - and that is the C: drive. What you need to do is to create a second disk inside your VM. You do that with the VMmanager GUI. You want to put that second disk on a UD. You can size as you like - I think mine is 20G. That disk will then be assigned a drive letter at boot (mine in the D: drive). Youll have to partition and format that disk. Then make a "TEMP" directory and map the TEMP environment variable to the D:\TEMP. Windows is happy and all temp files are written to a UD.
 
I usually point the "TEMP" and "TMP" environment variables to that directory: TMP may be a holdover from an earlier age, but I still map it just in case. No harm.
 
Directions for Windows 10. (Windows 7/8 are similar or the same):
Go to Control Panel -> System -> Advanced System Settings -> Advanced tab (should already there) -> Environment Variables. Use the System variables section (lower part of the panel). Do not make the changes in the User variables for {userid} section. After changing, click OK. I think you have to reboot. 
 
Good luck!

Thx.