Plex server to host 40-50 streams/transcodes (1080p)

[sanfark]

Rebuilding my Unraid server

 

I already purchased parts for 5 days ago (still have 25 days to return). Big ask to help with this but maybe it's pretty simple....

 

Goal:

Plex server that can host a minimum mix of 40-50 streams/transcodes or more

Fully optimized

Doesn't do much else but store some pictures and ftp etc

 

Parts:

Define 7 case

Motherboard: ASRock Z690 Pro RS LGA 1700

CPU: Intel Core i3-12100 - GPU w QuickSync should handle dozens of transcodes

RAM: G.SKILL Ripjaws V Series 16GB (2 x 8GB) 288-Pin PC RAM DDR4 3200

Disks: 5x but mostly HGST Ultrastar 8TB (One Parity drive) & 1x WD Blue (will be upgrading/adding more)

PSU: Seasonic FOCUS GX-750

10gpbs NIC

10gpbs switch

1.5gpbs fibre internet connection

 

 

How system will function:

Implement Sonarr to download TV shows (I will figure this out later)

I currently curate movie selections (only great stuff) and download to main windows PC and simply transfer them over to Unraid

Average movie size 5GB - 30GB depending on awesomeness

No 4K, no HDR, no DoVi.

Users will be a mix of Shield owners and 'who knows what they use' owners.

 

I would like to ask the community what features/functionality I really should be implementing to ensure success as I am not a total newb, but a simply layman +1 intelligence.

 

Concerns/thoughts I have no answers to:

HDD: Can mechanical HDD handle so many people accessing the drive if 40 people just so happen to access data that's on a single drive?

HDD: Can we mitigate this somehow?

CACHE: Do I need cache NMVe? What for? How much? Can it help store files people access as in if someone is watching a series, can Plex auto load a season into cache ahead of time?

RAM: Is there a way to utilize RAM for anything? Return 16GB and upgrade to 32GB?

File Optimization: Store metadata on Plex app data somewhere else? Does this optimize?

Logging/Testing Once Built: Will be running a trial period and slowly add groups of 20 ppl. Unknown number of streams vs transcodes. How to log/track this for review? What else to watch for?

 

If you have any insight on the above parts/concerns or have built a full streaming server; your insight would be greatly appreciated. If anyone really knows their shit (hi jorge lol) and willing to do a short google-meet phone call I can send some paypal your way to show my gratitude as well.

 

I am a no bullshit guy and I really want to get this right from the beginning and not have to keep buying new parts or keep reconfiguring. Basically, how would you do it?

 

Edited July 7 by sannitig

[Frank1940]

9 hours ago, sanfark said:

CPU: Intel Core i3-12100 - GPU w QuickSync should handle dozens of transcodes

Are you sure about this statement?   I would do a bit of research to find out how much of the CPU resources are necessary for each stream that you are planning to transcode.  (The resources necessary will also depend on what the original format is and what output format is required!  I know my computer requires about half the time of the movie playing time to transcode a 480I-60HZ DVD file to 480P-30HZ file.)   You also need to know how many streams your transcoder can do in a single core.  In some case, transcoding can be speed up by using a GPU but you have to have the right software and hardware to do this. 

 

You may be looking at a Storage pool rather than a standard Array to get the speed required.  If I can calculate correctly, you need a min of approximately 220MB/s of data from your drives to handle 50 BluRay streams at a time!.

[Michael_P]

20 hours ago, sanfark said:

HDD: Can mechanical HDD handle so many people accessing the drive if 40 people just so happen to access data that's on a single drive?

This will probably be your issue.

 

11 hours ago, Frank1940 said:

Are you sure about this statement?

1080p transcodes down to something else, yeah it could probably do it - BMB did 18 smooth 4K transcodes on his 13th gen. And if it's not a 4K HDR file, it's more likely that most if not all will be direct streams not needing transcode at all.

 

Audio will still be CPU bound, but if the clients are all using relatively decent hardware (like the Shield), they'll be direct streams, too.

 

I'd fill it up with RAM, tho

[Frank1940]

Personally, I would force any transcoding required to be done by the client on the end of the stream.  Of course, that could move the bottleneck onto the Network...

 

Of course, one could  make sure that the streams were transcoded into the most efficient format for max data compression before they are stored.  That type of transcoding is not time constricted.

 

 

[ConnerVT]

2 hours ago, Frank1940 said:

Of course, one could  make sure that the streams were transcoded into the most efficient format for max data compression before they are stored.  That type of transcoding is not time constricted.

^ This.  Nearly all "commercial" streaming services do this, keeping files transcoded in the most likely demanded formats that are requested.

 

For your use case, it will all come down to addressing the bottlenecks.  40 simultaneous streams - You will have a lot of them.  Drive access speeds, network speed, transcoding, even bandwidth of moving data through the CPU/MB from drive through to NIC.

 

This is before thinking of worst possible cases.  Let's use this as an example - The latest episode of a popular show (think "GoT") just hit your server, and 30 users all sit down that Friday night to watch.  Obviously, they don't all hit Play at the same time.  So one drive is serving the same file, but reading different parts of it...

[JonathanM]

12 hours ago, ConnerVT said:

So one drive is serving the same file, but reading different parts of it...

Hopefully you have enough RAM to cache the entire video.

[ConnerVT]

8 hours ago, JonathanM said:

Hopefully you have enough RAM to cache the entire video.

 

That would be great, if the only use case is always 30 people watching the same video file.

 

Now flip it the other way - 30 people watching 30 different movies.  Ain't easy caching it all to RAM.

 

As they say, the Devil is in the details.  😫

[sanfark]

On 7/8/2024 at 6:29 AM, Michael_P said:

This will probably be your issue.

 

1080p transcodes down to something else, yeah it could probably do it - BMB did 18 smooth 4K transcodes on his 13th gen. And if it's not a 4K HDR file, it's more likely that most if not all will be direct streams not needing transcode at all.

 

Audio will still be CPU bound, but if the clients are all using relatively decent hardware (like the Shield), they'll be direct streams, too.

 

I'd fill it up with RAM, tho

Why the increase in RAM. You're suggesting 32GB?

[sanfark]

On 7/8/2024 at 9:43 AM, ConnerVT said:

^ This.  Nearly all "commercial" streaming services do this, keeping files transcoded in the most likely demanded formats that are requested.

 

For your use case, it will all come down to addressing the bottlenecks.  40 simultaneous streams - You will have a lot of them.  Drive access speeds, network speed, transcoding, even bandwidth of moving data through the CPU/MB from drive through to NIC.

 

This is before thinking of worst possible cases.  Let's use this as an example - The latest episode of a popular show (think "GoT") just hit your server, and 30 users all sit down that Friday night to watch.  Obviously, they don't all hit Play at the same time.  So one drive is serving the same file, but reading different parts of it...

Great scenario. That is exactly the scenario I imagined.

 

1. If I pre transcode the files, then I will have two copies in my library, which is annoying and ugly and space consuming. I want to make sure I have the best possible quality for myself and those who do not require the transcode, which would mean keeping the original larger file

2. Is there a cache option, where, if the system notices something is being played often, it can dump the files to an NMVe drive and simply serve it from there? Essentially removing the mechanical HDD from the equation? Or what about caching to RAM? Is there a setting where this could be done? I could just buy more RAM...

Edited July 17 by sanfark
RAM

[sanfark]

On 7/9/2024 at 6:23 AM, ConnerVT said:

 

That would be great, if the only use case is always 30 people watching the same video file.

 

Now flip it the other way - 30 people watching 30 different movies.  Ain't easy caching it all to RAM.

 

As they say, the Devil is in the details.  😫

Is this a global option to cache to RAM?

 

Would be great to cache a video to RAM, I agree, but not if it caches everything to RAM, that would suck. Specifically your 30 people watching 30 movies scenario.

[Frank1940]

1 minute ago, sanfark said:

Is this a global option to cache to RAM?

 

The Kernel does it for open files across the entire file system.  However, it is also dynamic and flushes older content as new content is added.  More RAM helps by simply having more free RAM available for file caching.  (That is why many commercial servers have a ton of RAM to minimize disk thrashing and provide faster response to requests.) 

 

You do realize that you are well of outside of the normal usage of Unraid for virtually all of us.  Most of us are only streaming to a single household of less than, say, ten people.  In fact, what you are proposing doing may also be outside of the copyright law-- at least, in the USA. 

[sanfark]

I am under the impression that we can share our plex servers with friends (hence the add friends function). I guess if it gets too big one would be outside copyright law?

 

I know some people have 5, 10, 20, 30 friends they share with.

[Maxrad]

One Plex account/ID with Plex Pass allows you to grant remote streaming access to 100 friends (friend with a Plex account/ID) – no more than 100.

[ConnerVT]

45 minutes ago, sanfark said:

I know some people have 5, 10, 20, 30 friends they share with.

 

I know that many people whom I might give a ride in my car.  But they wouldn't all fit at once.

 

If you want to stream to 30 different users/clients simultaneously, you will be hard pressed to do so with consumer hardware.

 

It is all about the bandwidth.  On every subsystem.  Drive access speeds.  Throughput of your NIC.  Internet upload speed.  Throughput of the CPU. 

 

So put it in RAM, that's fast.  But the data still needs to get there, and be managed.  More overhead.  More throughput.

[Frank1940]

3 hours ago, sanfark said:

I am under the impression that we can share our plex servers with friends (hence the add friends function). I guess if it gets too big one would be outside copyright law?

It depends on what you are sharing and whether you have the Rights to be able to share that material.  (Ask the people who ended up being sued for distributing copyright music back about 2000.  I can remember that ASCAP was going after big bucks from individuals hosting the sharing service for failing to pay a license fee for each song.)  I am not a attorney but I can remember that there were many discussions about if you were even allowed to make a copy of actual DVD's that you owned for your own use within your home.  And there was the DMCA law of 1998.  You can read about it here:

 

   https://en.wikipedia.org/wiki/Digital_Millennium_Copyright_Act

 

I do know that the website of a Chinese software that developed software for removing copy protection from DVD's and BluRay's was blocked by US ISP's for a year or two.  They are now back online as bypassing the US restriction was a very simple process!  

 

You can probably ignore the law if you stay under the radar but realize that the more you publicize your activities the more likely you are caught.   Defending yourself will require lots of money as the attorneys who are qualified to handle these cases are expensive. 

Edited Thursday at 02:56 AM by Frank1940

[Michael_P]

6 hours ago, Frank1940 said:

You can probably ignore the law if you stay under the radar but realize that the more you publicize your activities the more likely you are caught. 

 

And whatever you do, don't charge for access - that's criminal copyright infringement with big boy jail time.