Please help explain cache disk

[maxse]

Hey guys,

 

So I've been running unraid 5 for about 5 years and love it! I am in the process of finally upgrading to unraid 6. I'm also going to use an older HP n40l for backup (but that's another post).

 

My question is about the new pooled/mirrored cache drives. I haven't messed with this in a while as unraid just works so beautifully but this is my understanding of cache drives. 

 

The cache drive is where all the apps (dockers) get installed to. Such as say plex and any other apps. Also in terms of file movement, files copied to the unraid get deposited to the cache drive first, then get moved by the mover script in the middle of the night, and this is to increase performance so that the array and parity doesn't have to spin up every time a file gets copied... I get that...

------

 

1) My concern is with the apps and configuration. The configs for plex, and any other apps are store on the USB drive. But if I were to get rid of the cache drive (I don't care too much about the write performance), where would the apps then reside? Are they just on the array and also get copied to the parity drive? This is especially important for the backup unraid because I can only have a max of 6 drives, and don't want to waste a drive space for cache...

 

2) Now for the main server, I'll have a cache drive so that plex could transcode to it without having to spin the array every time. My biggest issue is what happens when a cache drive fails (if I only have 1 and not mirrored)? Would I then loose all of the apps/dockers, and would have to reconfigure the apps and re-install them again? How would such a thing be handled on a server (such as my backup server) when there is NO cache drive?

 

I guess I want to protect the unraid so that if the cache drive does go down, I won't have to re-install apps and configs (if my understanding of this is all correct from above), so that would mean I need a mirror for my cache drives correct? That way if one cache drive goes down, everything continues to run as it should, and I just swap out the bad cache drive.

 

Sorry about the long post but I want to make sure I plan correctly going forward. Thanks so much!

[trurl]

Each user share has a setting for whether and how it uses cache. There is no requirement to cache writes to any user share. Some of us (including me) don't. You can just use cache for the apps. There is a plugin (CA Backup) that will make a backup of your apps if you don't want to mirror cache.

1 hour ago, maxse said:

Are they just on the array and also get copied to the parity drive?

Maybe you worded this wrong but just in case. Parity doesn't contain any of your data and nothing is ever copied to it. Parity just allows the data for a missing disk to be calculated from parity PLUS ALL the other disks. Any write to an array disk also updates parity which is the reason for the slower writes, but the parity disk is not a copy of anything.

[maxse]

Perhaps I did word it wrong. Yes I meant if the apps aren’t on a cache drive I wouldn’t necessarily want them to be included on the parity calculation...

 

I’m still a bit confused though... So it’s not  necessary to have a cache drive in order to install apps? If not, would it be some kind of a setting that I can create a share and just force whatever is on the share to be kept on a specific drive and not be moved? How about something like Sabnzbd. Whatever goes into the data folder... I wouldn’t want every time there’s a new file, or unpacking, etc... for two drives to spin up (the drive it’s loaded on and also the parity?) How would something like that work?thank you so much for explaining this! I got a bunch of drives and will upgrade and slowly format to XFS so want to make sure I do this right from the start. Thank you!

[trurl]

It's not necessary to have cache for apps, but that is the simplest place for them. The point I was making is you can put your apps on cache and make them stay there, and not even use cache for your other data.

 

If you put your apps on the array then parity will always be involved.

 

 

[dukiethecorgi]

Why not install and use the backup appdata plugin?

[trurl]

4 minutes ago, dukiethecorgi said:

Why not install and use the backup appdata plugin?

Yes I mentioned that.

1 hour ago, trurl said:

There is a plugin (CA Backup) that will make a backup of your apps if you don't want to mirror cache.

 

[maxse]

Yep I wasnt familiar with that plugin. Will research it now. 

Yes, I would only use cache drive for apps like I do now, no data except temporary stuff from sabnzbd, unrarring, then final file goes on the array. It would be awesome if I could avoid using a cache drive all together. How would that be accomplished?

 

I have plex on another machine so transcoding doesnt happen on the server

 

Especially on the HP Microserver where 6 drives is max and it will only be used as a backup, although the backup will need some kind of app possibly. Wouldnt want to waste a space just for cache...

[trurl]

28 minutes ago, maxse said:

It would be awesome if I could avoid using a cache drive all together. How would that be accomplished?

You could use an Unassigned Device, but people usually do that in addition to a cache drive. Why do you want to avoid having a cache drive?

[trurl]

31 minutes ago, maxse said:

I have plex on another machine so transcoding doesnt happen on the server

If your media is on Unraid it really makes more sense to have Plex docker there as well and let it do the transcoding. If you are only using plex on the machine it is installed on then it probably doesn't matter but why get the media across the network to plex then have it go back out on the network to a plex client?

 

 

[trurl]

For the backup server cache might not be necessary. Lots of ways to do backup including maybe some docker. For just that purpose it would probably be OK to let the app live on the array.

[maxse]

Got it, thanks guys! I guess I will ask for some help setting up a backup server without a cache drive in a remote location. Good to know that could be done so i don't waste valuable physical space on a cash drive.

 

I'll leave the cache on the  main server then as I do now. So I wouldn't need to have mirrored cache drives just for protection of the apps and their configs in case cache drive goes down if I get the app you guys spoke about earlier in this thread correct?

 

Interesting question about plex server. I have it running on a Mac mini now connected to the network with the file server as the source. If it has to transcode, it grabs media from file server and transcodes it locally on the mac mini. Would consolidate really improve anything? The other issue is that I am using an old AMD Sempron processor for unraid. It works fine with sabznbd, and one other app. Consolidating would require me to spend about $560 on mobo, cpu and RAM to future proof 4k transcoding. As it stands right now, the AMD Sempron isn't able to transcode not even 1 1080p stream, and I have about 13 drives worth of data now. I need to be able to transcode about 3 streams to iphones/apple TVs remotely, and the mac mini from 2012 has an i5 with some hardware transcoding capability. 

 

Any thoughts? Would eventually it still be best to have unraid and plex be on the same machine? As opposed to having a seperate plex server machine?

[trurl]

Since you mention your old hardware I wonder if you have enough horsepower for Unraid 6.

 

How much RAM do you have?

[WashingtonMatt]

Giving unRAID/Plex it's own GPU, combined with transcoding to RAM is the cats meow. Completely offloads the work from the CPU. unRAID sleeps through the hardware transcoding. So you might be able to breathe a little life into your existing hardware, or if buying new, that's the direction you should plan to go. 

Edited December 7, 2018 by WashingtonMatt

[maxse]

I actually looked in to GPU transcoding. But from what I understand unbraid does not support it as it doesn’t have Nvidia drivers? Unless something has changed?

im not familiar with VMs and not looking to complicate the set up if that’s still the only way to get drivers to work?

 

i believe I have 8 gigs of RAM which is the max for the motherboard. 

[Hoopster]

11 hours ago, maxse said:

Got it, thanks guys! I guess I will ask for some help setting up a backup server without a cache drive in a remote location. Good to know that could be done so i don't waste valuable physical space on a cash drive.

Your desired scenario is very similar to the one I currently have in place.  Look at the two systems in my signature.

 

My main server has a single cache drive (exclusively for dockers apps and temporary downloads) and an unassigned device (SSD) for VMs and for appdata backups done through CA Backup.  My backup server does not have a cache drive.

 

I run Plex as a docker on my main server.  It has a CPU with QSV support so it does hardware transcoding.  The Backup server (also a CPU with QSV support), having no cache drive, still has Plex installed as a docker.  It and OpenVPN-AS are the only dockers on the backup server and they live on the array.  I have Plex there since, if I have to use the backup server, that is what most of my users care about.

 

I run an rsync backup via script once a week which copies new content from the main server to the backup server.  Of course, it could be done at any interval, but, weekly is often enough for me.

 

 

[Hoopster]

6 minutes ago, maxse said:

I actually looked in to GPU transcoding. But from what I understand unbraid does not support it as it doesn’t have Nvidia drivers? Unless something has changed?

 

Hardware transcoding in Linux/unRAID requires an Intel CPU with iGPU with Quick Sync Video support.  Both my main and backup servers have this.

 

A discrete nvidia GPU cannot be used with unRAID due to lack of nvidia drivers as you stated.  Currently (although some claim otherwise), the only way to use a discrete GPU for hardware transcoding on unRAID is to pass it through to a Windows VM running Plex. 

[maxse]

Thanks. Wouldn’t that make sense to have a cache mirror pool then since you’re effectively using 2 drives that can’t be part of the array anyway this way if one cache goes down the app data is still safe and seamless, no?

 

I’ll still need to look into the system that I’ll use for backups, like incremental or how that even works. Big reason is to protect against crypto viruses also. If Rsync adds new data wouldn’t it ask load a crypto type of virus on the new machine? (But that’s a question for another topic I get it). 

 

So  for my main server. Still have 1 cache and use CA backup app which effectively copy the apps and it’s settings from the cache drive on to the array?

 

In the event of cache drive failure what happens?

 

i install new cache drive lod CA backup onto it then use that app to retrieve whatever CA backup saved to the array to in effect restore the new cache drive exactly to how it was? Do I have this right? Is it easy to do? Or should I just go ahead and suck it up and mirror cache drive to make this process fool-proof and easier?

my main thing is simplicity and to make it easy. 

 

Processor is AMD Sempron 2400. Figured if an HP N40l can run unraid 6 this should as well? I’m going to be upgrading to Unraid 6 on the backup machine today. New drives are still being tested to be out into the main server. 

[maxse]

Right that’s what I thought about hardware transcoding. My machine can’t handle it even 1080pnstreams with the AMD Sempron. Not looking into VMs at all. So it would cost me about $560 for new mobo, ram, cpu that would allow hardware transcoding and future proof for 4K transcoding. Not looking to spend that right now.  That’s why I’m using the Mac mini and the unraid server just for sabnzbd and storage. 

Edited December 7, 2018 by maxse

[Hoopster]

10 minutes ago, maxse said:

Thanks. Wouldn’t that make sense to have a cache mirror pool then since you’re effectively using 2 drives that can’t be part of the array anyway this way if one cache goes down the app data is still safe and seamless, no?

Yes, that would certainly be a viable option; however, since I run a couple of VMs, I have the 2nd SSD mounted as an unassigned devices for hosting the VMs and also use it for backing up the cache disk appdata.

 

I eventually plan to put in another SSD for a cache pool, but, right now, with only 8 SATA ports on the MB (I am trying to avoid an HBA for the time being), I am making due with the current configuration.  Next upgrade, I won't have those limitations.

[maxse]

Got it, thank you soooo much. You guys are really the best, love this place.

 

Am I correct with my understanding of the cache pool set up in mirror mode? That it would be seamless if one of the cache drive goes down and will just be an easy method as opposed to cache backup app? 

 

Also, I'm thinking about what you all are saying. My mac mini has an i5 2.5 GHz i5-3210M processor (2012). Passmark score is 3800 I believe, but I THINK it can do hw transcoding too. I looked at Socket AM3 processors that should still work with my motherboard. I could get a AMD Phenom II X6 1045t which has a passmark score of 4800, but def not hw transcoding. I am happy with the transcoding needs of my mac mini

 

Do you guys think it will benefit me to upgrade my unraid to the new processor and consolidate plex all into one machine? Will plex perform better or at least the same as my experience running it off the mac mini?

 

Lastly, depending on what you guys say, if plex is put on the unraid, I read the it's not optimal to have plex on an SSD because it burns through the life span of the SSD by creating a lot of small files, also the constant writing to the ssd for transcoding etc... Any thoughts? Again im not concerned with the speed of writing files TO the unraid... I definitely would like to be able to have the plex cover art load fast on the plex app though like an apple TV, right now I get some delay sometimes for the art to load even on the mac mini. Not sure if it's a hard drive issue or processor issue or something else, would an SSD speed this up? Or would a standard 3.5" drive be enough.

 

 

I know lots of questions, but I want to make sure I plan this correctly as I upgrade to V6 and convert file systems. I want many years trouble free like I've been having now. Thanks so much in advance!

[trurl]

2 hours ago, maxse said:

Am I correct with my understanding of the cache pool set up in mirror mode? That it would be seamless if one of the cache drive goes down and will just be an easy method as opposed to cache backup app? 

I have both a mirror and a backup. The plugin makes it easy to have backup. And of course, a mirror is not a backup. If something is deleted or overwritten on the mirror, it is gone, but it may still be on the backup.

[Hoopster]

3 hours ago, maxse said:

Lastly, depending on what you guys say, if plex is put on the unraid, I read the it's not optimal to have plex on an SSD because it burns through the life span of the SSD by creating a lot of small files, also the constant writing to the ssd for transcoding etc... Any thoughts? Again im not concerned with the speed of writing files TO the unraid... I definitely would like to be able to have the plex cover art load fast on the plex app though like an apple TV, right now I get some delay sometimes for the art to load even on the mac mini. Not sure if it's a hard drive issue or processor issue or something else, would an SSD speed this up? Or would a standard 3.5" drive be enough.

 

I have Plex in a docker on the cache drive (appdata).  Yes, it writes quite a bit of data to the database and lots of small files; however, my cache SSD (a Samsung 830 which is six years old) is still going strong and is in good health.  I have Plex configured to transcode in RAM, so that does not use the SSD.

 

I have four households, in addition to my own, which access my Plex server.  I never know when they might get on, so having it all consolidated on the unRAID server in a docker is the only way I would consider doing things in my use case.  Fortunately, with an Intel Xeon CPU/iGPU (10,400 passmark) supporting Quick Sync Video and hw transcoding, my server hums along just fine even if all are on simultaneously.

 

In addition, I have HDHomeRun TV tuners on my network which are integrated with Plex and they provide live OTA TV and DVR services which uses unRAID for storage.  One of the households often has poor OTA TV reception in their area but, they can watch Live OTA TV when necessary via Plex since they have good Internet speed.

 

HDHR now also offers a "cable" streaming service which may be of interest to me.  If I go with that, then those channels can also be recorded via Plex to unRAID storage.

 

Having Plex, OTA TV, DVR, storage, etc. all on unRAID makes remote access and troubleshooting a lot easier..

 

I am not trying to tell you that what you're doing won't work, as it clearly does; however, consider how your use case may expand in the future and at least plan a path to get where you need to be should your needs outgrow your current configuration.

 

 

Edited December 7, 2018 by Hoopster

[maxse]

Thank you! 

So I did a clean install today unraid 6 today on the backup server and formated to XFS encrypted. I decided to keep a cache drive on it since I had a 2.5" 320gig hard drive already connected inside. I was able to cram 6 drives this way in the HP Proliant N40l. So I'll have 1 parity, 1, cache, and space for 4 drives for data. So for now that's set (except need to actually figure out back up workflow).

 

Main server... I'm trying to be cost effective here especially since everything is working the way it is, just need to upgrade to V6 and got some drives on sale... So I take it upgrading to the AMD Phenom II X6 4035t, and getting rid of the mac mini, would not give me much of a performance boost over the current i5 in the mac mini correct? Especially since the AMD can't do HW transcode, but the i5 in my mac mini can? (I also read that only the 8th gen intel processors can HW transcode, so Im a bit confused by this).  Also, is it easy to set up these things now with dockers? Back in the day, it took me a few days just setting everything up and getting plugins to work, but I really was starting from scratch and didn't know what I was doing. Setting apps up on the mac mini like plex was easier.

----------

 

Hoopster, I keep my mac mini on at all times. And have shares mounted, it has been very stable. I even thought about just loading, sabnzb, plex, sonar, on and just running all those things on the mac mini to avoid having to learn how to use dockers even if it's just for sab and sonar... Is there a real disadvantage boxes like this? Otherwise I would need to spend about $450 on a new mobo, ram and intel processor just to get rid of the mac mini and run plex on unraid with hw transcoding capability...

 

I only have an interest in iptv and wish there was some kind of way to record it onto the unraid array, but plex doesn't support iptv well and as far as I know there is no simple way to do it with a DVR-like interface.

 

Thoughts?

[Hoopster]

I keep my mac mini on at all times. And have shares mounted, it has been very stable. I even thought about just loading, sabnzb, plex, sonar, on and just running all those things on the mac mini to avoid having to learn how to use dockers even if it's just for sab and sonar... Is there a real disadvantage boxes like this? Otherwise I would need to spend about $450 on a new mobo, ram and intel processor just to get rid of the mac mini and run plex on unraid with hw transcoding capability...

 

If it works for you and you have not experienced problems, then there is no need to spend extra money now just for the sake of a nice-to-have setup that is more than you currently need.  I simply laid out my use case and reasoning for wanting to have it all consolidated with my unRAID server the center of everything.  Perhaps you will have such a need in the future, but, if what you have works for you now, there is no need to replace it just because you can.

 

It is better to start with what you know and move out from there, adding/learning a piece at a time than it is to change everything and throw it all together without the proper knowledge.  That only leads to being baffled when things don't work as expected.

 

[maxse]

You’re right. Thanks so much! Thing is I also recently got an upgrade in speed 100/100 so streaming became more realistic as well... also I’m having issues with sabnzbd unable to unrarnsome files and I can’t update since the plugin hasn’t been supported for about 5 years lol. So need to get on unraid 6 ASAP 

 

Am I correct that hw transcoding can only be done on the 8th gen intel processors? That can’t be right but I know I read something to that affect. Is it that X265 transcoding can only be done on the 8th gen?

 

And here’s the big question: What’s a cost effective CPU/mobo/ram combo? I don’t mind getting used server parts but it’s gotta work smooth with unraid obviously. Years back you guys helped me put

together an awesome system that I’m currently using! 

I guess the only thing I would bring over is my PSU and my M1015 that I flashed to IT. Realistically, what would I need to spend? Assuming transcoding 5-7 streams from 1080p. I set a limit for remote streams to 

10mbit. 

 

Now you got me all excited to have everything nice and sexy on one box. The ultimate machine 

 

I just checked and I think I’ll be able to sell my Mac mini for around $200.