Planning Plex layout

[paqman]

I'm in the process of planning the migration of my Plex server/file server from an Ubuntu machine to unRAID. (utilizing the same hardware, just rebuilding).  I do plan on migrating my plex library as it is, so I can migrate the metadata and stuff with it, but if I find that a different layout will be optimal, I am willing to rebuild it.   Currently I just have it layed out as:

/plex/media/Movies

/plex/media/TV Shows

/plex/config...

etc.

 

I like this layout, and it works well for me.  But my question isn't really about how to layout plex in general, it more has to do with how it uses the different drives in my array.  Currently my Plex library is only about 5TB.  It grows slowly, but it does grow.  I have probably twice that much data in home videos, pictures, and VHS transfers.  I host a lot of my family's home videos for sharing with other family members.  I will have a 16TB parity drive, with a 14TB, 6TB, and a couple 3TB drives to start out.  I have a 256GB SSD that I am planning to use as a cache drive, but may spring for a 1TB nvme drive for that if I'm feeling spry.

 

Do I just put the whole library in /mnt/user share somewhere and let unRAID decide how to balance out the data?  Or do I limit my Plex library to specific drives for performance or power saving reasons?  I do plan on putting the metadata on my cache drive. (does your cache drive get protection from the parity drive like the regular drives?)  

 

Anyway, that's my main question, is it best to let unRAID decide where to put my data?  Or should I control it?  Thanks, any suggestions for this would be great.  I'm trying to spend time planning this out before jumping right in and building the thing.

[apandey]

1 hour ago, paqman said:

Anyway, that's my main question, is it best to let unRAID decide where to put my data?  Or should I control it?

It is for you to decide. The factors are any organizational preferences you may have, and any disk spindown preferences you may have (like if you have music files spread across all disks, playing them in a Playlist will keep all disks spinning) 

 

I personally have individual movies and TV show same seasons stay on same disk, which gives me a balanced array for now (my drives are uniformly sized 8TBs). But I know others who have a different preference 

 

I have an extra level of nesting for movies to use same split-level setting for both movies and shows

[paqman]

13 minutes ago, apandey said:

It is for you to decide. The factors are any organizational preferences you may have, and any disk spindown preferences you may have (like if you have music files spread across all disks, playing them in a Playlist will keep all disks spinning) 

 

I personally have individual movies and TV show same seasons stay on same disk, which gives me a balanced array for now (my drives are uniformly sized 8TBs). But I know others who have a different preference 

 

I have an extra level of nesting for movies to use same split-level setting for both movies and shows

Thanks, yeah I guess I really just don't know how it affects performance if they were to all get spread out on the array. If one episode of a show were on one disk, but the next one were on another, would there be noticeable lag trying to load up the next show or something?

 

I like the idea of containing them all together, but I also like the idea of not having to manage space, and letting it just put them wherever. If I were to limit it to one drive, what do I do if I fill up that drive? Do I have to re-organize or something? 

[apandey]

6 hours ago, paqman said:

would there be noticeable lag trying to load up the next show or something?

If the other disk is spun down, there will be a slight lag. But that's it. I don't worry about this for a use case like streaming a show. Single files are self contained on a single disk, so it's not like reads are randomly jumping disks in real time

 

As for managing across drives, you should look at user shares documentation here: https://wiki.unraid.net/Manual/Shares#User_Shares

 

Specifically, allocation method, free space strategy and split levels and how they work together to manage storage across drives

[paqman]

8 hours ago, apandey said:

If the other disk is spun down, there will be a slight lag. But that's it. I don't worry about this for a use case like streaming a show. Single files are self contained on a single disk, so it's not like reads are randomly jumping disks in real time

 

As for managing across drives, you should look at user shares documentation here: https://wiki.unraid.net/Manual/Shares#User_Shares

 

Specifically, allocation method, free space strategy and split levels and how they work together to manage storage across drives

Thanks, that was a good doc.  Not entirely sure if I want to use split levels for my plex library or not, but I do like the explanations of the allocation methods.  Kind of like High Water mark for my mixed bag of disks.

[Frank1940]

2 hours ago, paqman said:

Not entirely sure if I want to use split levels for my plex library or not

I would avoid split levels (unless you are saving DVD's in a folder and file structure rather than in ISO or MKV format).  Split level can cause problems if you are not careful setting things up and, even then, you can end up with things split over two disks (very occasionally) because (1) the amount of storage on a disk is a fixed quality and (2) the logic the setting employs is not that smart.  Leave it turned off and if you encountered a problem, the Dynamix File Manager provides a idea tool for the user to intelligently 'fix' the problem.   (I would estimate that 99.99% of the files would never need to be 'fixed'...)  

 

You have seen the WIKI on the User Share settings, make absolutely certain that you set the Minimum Free Space setting on each share to a value larger than the largest file you are going to be saving on that share.  If you set that and leave the Split level setting off, you will never be back asking why Unraid is refusing to store a 3GB file on a share that has 15 TB of free space on it!

Edited February 8, 2023 by Frank1940

[paqman]

19 hours ago, Frank1940 said:

I would avoid split levels (unless you are saving DVD's in a folder and file structure rather than in ISO or MKV format).  Split level can cause problems if you are not careful setting things up and, even then, you can end up with things split over two disks (very occasionally) because (1) the amount of storage on a disk is a fixed quality and (2) the logic the setting employs is not that smart.  Leave it turned off and if you encountered a problem, the Dynamix File Manager provides a idea tool for the user to intelligently 'fix' the problem.   (I would estimate that 99.99% of the files would never need to be 'fixed'...)  

 

You have seen the WIKI on the User Share settings, make absolutely certain that you set the Minimum Free Space setting on each share to a value larger than the largest file you are going to be saving on that share.  If you set that and leave the Split level setting off, you will never be back asking why Unraid is refusing to store a 3GB file on a share that has 15 TB of free space on it!

Thanks, yes, I will make sure to set my minimum free space appropriately.  

 

I do understand the problems that split level can create if I don't set it up right.  Just trying to decide if it's worth it for me on at least TV shows.  We tend to go from one episdoe to another, and while it wouldn't be a big deal if there were a few seconds of wait time inbetween episodes, I am trying to keep the wife acceptance factor fairly good lol. Thanks for the warning though.  I was leaning towards using split levels for movies and TV shows, but I will think more about it and decide if the possible problems are really worth it.  

 

For what it's worth, I don't do a ton of downloading, and my library doesn't grow super fast.  I will have a 16TB parity drive, with data drives, 14TB, 6TB, 3TB, 2TB, and 2TB.  I'm thinking high water would be ideal for this setup, as it will start utilizing my 14TB drive mostly and won't touch the others for a while.  So maybe with the size of my TV show library (pretty small), maybe split levels won't be needed anyway, as they'll mostly go into that 14TB drive up front.  Good thoughts.

[paqman]

Wondering how my setup is going to look with high water.  I have 4 or 5 disks, but they are vastly different in sizes:

16TB parity

14TB data

6TB data

3TB data

2TB data

2TB data

 

I really only have about 6-7TB of data at the moment.  Don't ask my why I'm starting with a 16TB parity drive, it was on sale, and at the time, I hadn't planned on unRAID, just grabbed it because I was running out of space for local backups.  (mostly because I was keeping two full backup copies, so I didn't realize how little data I actually had).  Anyway, I figured the 16TB drive would be good just cause it would cover me in the future for just about any size drive I wanted to put in my array.  Anyway, I'm realizing now that the 14TB drive is the only drive that will be used for quite some time.  The others will all go unused, using highwater or most free, for a long time.  Now I'm starting to wonder if I just restrict everything to the 14TB drive, and utilize the smaller ones as a local backup for now, as I have an AWS account with everything backing up to there.  That is really just my offsite disaster recovery backups though, it's all glacier archive.  I was thinking after setting up unRAID that I would forgo actual local backups and just rely on my AWS offsite backups for a complete array disaster.  But maybe I've got enough space I can do some local backups in my array anyway.

 

Sorry, I'm kind of thinking out loud right now, trying to plan my array out.  But comments are welcome!

[Frank1940]

Personally, I would only use the 16TB for parity and the 14TB as my only data drive until I had that first data drive about 75% filled.  Then add another data drive.  The reasoning behind this is that each additional device in a complex system adds another point of failure.  The fewer potential points for failure, the more reliable the resulting system will be.  From the limited data, I have seen on hard drives, large capacity drive have the same (or, perhaps, even lower) annual failure rates than small capacity hard drives. 

Edited February 10, 2023 by Frank1940

[paqman]

18 minutes ago, Frank1940 said:

Personally, I would only use the 16TB for parity and the 14TB as my only data drive until I had that first data drive about 75% filled.  Then add another data drive.  The reasoning behind this is that each additional device in a complex system adds another point of failure.  The fewer potential points for failure, the more reliable the resulting system will be.  From the limited data, I have seen on hard drives, large capacity drive have the same (or, perhaps, even lower) annual failure rates than small capacity hard drives. 

I do like that line of thinking. And honestly I can't think of a reason not to do that. I do think that additional drives in an unRaid parity group don't necessarily endanger the other drives by adding an additional point of failure, but I do see your point. Also my other drives could last longer if I let them sit powered off? Or do they go bad lol. 

[Frank1940]

9 hours ago, paqman said:

I do think that additional drives in an unRaid parity group don't necessarily endanger the other drives by adding an additional point of failure, but I do see your point.

That is true but who needs the hassle of dealing with a failure of an unused component?

 

9 hours ago, paqman said:

Also my other drives could last longer if I let them sit powered off? Or do they go bad lol. 

Interesting question.  I have no answer.   It might be a bit of a paradox situation.  I put a hard disk on shelve and store it for seven years.  After the end of that period, I install it and it does not work.  The question becomes-- was it bad when I put it on the shelve or did it go bad while it was setting there?  There is no way to know for sure...   (One of the reasons why there are not stories about such situations is that a seven year hard disk is probably so small that no one would even consider installing it!)  

 

My guess would be that 80+ percent of all hard drives used in Unraid server are not replaced because they went bad but because they were replaced to add storage capacity.

 

 

[paqman]

Ha you're probably right. After thinking about it, I like your idea. Just run with the 14TB and a parity drive for now. However, I was thinking of using the 6TB for local backups. Is there any downside to have my local backups in my array? If the shares I put on my 14tb are restricted to that drive, then I crest a backup share that is restricted to the 6TB, I could do my local backups, with versioning and everything to that, and bonus, it's parity protected. Only drawback to that is it's not an external drive that I can quickly grab in the event of a house fire lol. But all my important stuff is backed up to AWS glacier archive for that case. 

 

Then if/when get close to filling that 14TB drive, I can add more space to the array and modify my shares to start filling other drives. Thanks for hitting this back and forth with me, I appreciate your thoughts on it. 

[Frank1940]

I  don't see a problem with that.  Local online backup can have ransomware vulnerability.  I make my shares 'Private' with read-only permission for the user(s).  There a way to have write access for new data.  Read here for that scheme:

 

      https://forums.unraid.net/topic/58374-secure-writing-strategy-for-unraid-server-using-write-once-read-many-mode

 

I have off--site storage using 2.5" USB hard drives (currently using 1TB drives cost about $50US) that are in my safety deposit box.  These contain only the stuff that is not replaceable from any other source. 

[paqman]

Thanks I'll take a look at that. 

[paqman]

42 minutes ago, Frank1940 said:

  Local online backup can have ransomware vulnerability.  

If I'm not making my shares accessible outside my network is this still a concern? Or are you talking shares even accessible to other computers on my network? If so, isn't my regular data at a danger to that as well? I guess I'm not real experienced with protecting against ransomware. 

Edited February 10, 2023 by paqman

[Frank1940]

Ransomware runs on Client machines (usually), not servers.  It looks for network connections (probably first-- because in a business environment, those are the most valuable assets) to work on.   Doing encryption on local drives is only a bit of frosting as the client machine may be only used by the receptionist.   Invasion of a home network (such as yours) only has a potential of a few hundred dollars compared to tens of thousands if they hit your (say) local hospital.

 

Having a very recent backup of your data locked up on a local server could have you up and running in a few hours (after making sure your rid of the ransomware) after you discover the problem.  

 

I have found that most video players (KODI, VLC) are just fine with read-only media shares when playing back content so I have those read-only also and use cache drive to upload new material as required.  By the way, you can copy files from a read only Unraid drive back to your PC if required.  You can also open them in a read-only mode by the office suites if all you want to do is view or print them.  In helping people on this forum, I often need to look at the contents of my Unraid boot drive,  I have it shared in read-only mode as I never have reason to write on it under normal conditions. (And it only takes less than a minute to change a share from read-only to read-write if it is required.)

 

I do most file operations on the array these days using the Dynamix File Manager which is relativity new and getting more useful with the frequent updates.