[Support] Josh5 - Unmanic - Library Optimiser


Recommended Posts

On 12/4/2019 at 5:49 AM, propman07 said:

ice-

 

I'll give that a try....thanks for the suggestion.

ice-

 

I went ahead and disabled audio conversion, and tried the same file. It appears that did the trick. I've got the audio coming out of the correct channels. Thanks for the suggestion, and another thanks to Josh for a great plugin. Merry Christmas.

Link to comment
2 minutes ago, Cpt. Chaz said:

Has anybody tried setting /tmp directory to /dev/shm? I wouldn't advise if you're low on ram, but so far i'm not experiencing any performance issues and it's been great for keeping my file system clean.

I haven't tried it because I'm afraid it would consume all of my ram, i have alot, but if you dont configure it correctly out of the box it will eat all of your cores imagine if you let it free on all of your ram.

Link to comment
3 hours ago, Cpt. Chaz said:

Has anybody tried setting /tmp directory to /dev/shm? I wouldn't advise if you're low on ram, but so far i'm not experiencing any performance issues and it's been great for keeping my file system clean.

How does this work for history? I noticed there are files in the temp folder for every conversion that's been done. Wouldn't these be cleared out of temp on reboot? Does it matter?

 

Edit: I think I'm confused. So /dev/shm is ram?  Are you changing this in the docker itself? Or mapping it to there? I know on Plex mapping transcode to ram maps /transcode to /tmp... I thought tmp was ram in unraid?

Edited by letrain
Link to comment
5 hours ago, Cpt. Chaz said:

Has anybody tried setting /tmp directory to /dev/shm? I wouldn't advise if you're low on ram, but so far i'm not experiencing any performance issues and it's been great for keeping my file system clean.

I map /tmp to /tmp, works great. HOWEVER...

I only have 2 workers. Each worker keeps /tmp occupied with the conversion file, so if you are converting 25GB movie files and have 5 workers running, you are probably going to have a bad time unless you are swimming in excess RAM.

 

YMMV, etc.

Link to comment
6 hours ago, letrain said:

How does this work for history? I noticed there are files in the temp folder for every conversion that's been done. Wouldn't these be cleared out of temp on reboot? Does it matter?

 

Edit: I think I'm confused. So /dev/shm is ram?  Are you changing this in the docker itself? Or mapping it to there? I know on Plex mapping transcode to ram maps /transcode to /tmp... I thought tmp was ram in unraid?

/dev/shm is allocated to only use 50% of ram, whereas /tmp has no limit. Wasn’t sure what to expect for ram consumption trying this out, so erred on the side of caution. 
 

the history part had not occurred to me tho. I pretty much leave the container running 24/7, except for updates, etc. I’m not overly concerned with long history, but the next time I restart the docker I’ll report back.

Link to comment
3 hours ago, jonathanm said:

I map /tmp to /tmp, works great. HOWEVER...

I only have 2 workers. Each worker keeps /tmp occupied with the conversion file, so if you are converting 25GB movie files and have 5 workers running, you are probably going to have a bad time unless you are swimming in excess RAM.

 

YMMV, etc.

Good to know! I’m utilizing 3 workers for tv episodes at the moment, no where near 25gb file sizes so I should be in good shape there. Once I get to the movies I may keep a closer eye on it. 

Link to comment
9 hours ago, ijuarez said:

I haven't tried it because I'm afraid it would consume all of my ram, i have alot, but if you dont configure it correctly out of the box it will eat all of your cores imagine if you let it free on all of your ram.

I un-prioritized core usage in the dockers extra parameters, is that what you’re talking about?

 

also, /dev/shm is also allocated to just 50% of your total ram, a handy safeguard. But I’ve also got 192gb of ram in the arsenal, so shouldn’t be a problem for me either way. But for folks with less, I’d definitely use that over /tmp 

Link to comment
/dev/shm is allocated to only use 50% of ram, whereas /tmp has no limit. Wasn’t sure what to expect for ram consumption trying this out, so erred on the side of caution. 
 
the history part had not occurred to me tho. I pretty much leave the container running 24/7, except for updates, etc. I’m not overly concerned with long history, but the next time I restart the docker I’ll report back.
I would think it would use the same amount of ram that it would for disk space, i.e. 8gb transcode would need 8gb of ram. It fully converts then moves the file to overwrite the original. I tried it out with tmp (I have 96gb of ram so I'm not worried at all), And so far so good. And my history was there. It must be kept somewhere else (logs?). The file folders it uses to store the transcoded file before moving it must just be for while it's converting, but they never get deleted by program so I assumed it was history of some sort.

I am glad you brought this up. I've used /tmp for plex transcode and this is a great option. I have an SSD in my cache pool that complains about errors if I let unmanic run for over a day ( crc errors). But is happy any other time. So this means I can leave unmanic plugging along and not have to worry about over working that drive. Also it means mover and parity check won't be slowed down when unmanic is running.

Thank again.

Sent from my Pixel 2 XL using Tapatalk

Link to comment
I un-prioritized core usage in the dockers extra parameters, is that what you’re talking about?
 
also, /dev/shm is also allocated to just 50% of your total ram, a handy safeguard. But I’ve also got 192gb of ram in the arsenal, so shouldn’t be a problem for me either way. But for folks with less, I’d definitely use that over /tmp 
What do you mean "un- prioritized" ?

Sent from my Pixel 2 XL using Tapatalk

Link to comment
5 hours ago, letrain said:

I would think it would use the same amount of ram that it would for disk space, i.e. 8gb transcode would need 8gb of ram. It fully converts then moves the file to overwrite the original. I tried it out with tmp (I have 96gb of ram so I'm not worried at all), And so far so good. And my history was there. It must be kept somewhere else (logs?). The file folders it uses to store the transcoded file before moving it must just be for while it's converting, but they never get deleted by program so I assumed it was history of some sort.

I am glad you brought this up. I've used /tmp for plex transcode and this is a great option. I have an SSD in my cache pool that complains about errors if I let unmanic run for over a day ( crc errors). But is happy any other time. So this means I can leave unmanic plugging along and not have to worry about over working that drive. Also it means mover and parity check won't be slowed down when unmanic is running.

Thank again.

Sent from my Pixel 2 XL using Tapatalk
 

Awesome! Glad it worked, I got to thinking about it at first, not knowing if it would even work, but also not knowing why it wouldn’t. Seems good so far for me too. It always bugged me the empty folders left behind...

Link to comment
6 minutes ago, TexasDave said:

@Cpt. Chaz you say:

 

 

Can you tell me where this is happening? Want to check (and clean) on my system as needed. Thanks!

Sure yeah - in the container settings under “encoding cache directory”. The docker mapping is /tmp/unmanic, the host mapping is the file path right above it. Different people map it to different places, that’s what piqued my curiosity about mapping it to ram so instead of disk leaving the empty folders. Ram mapping will clear it out. 

  • Like 1
Link to comment

Another issue that I noticed....if I add a single movie to a folder that I have unmanic pointed to watch, it will start both workers (I have unmanic set to use two workers) on the same movie....with one being a few percent ahead of the second one. I was wondering if anyone else noticed this behavior.

 

Thanks.

Link to comment
31 minutes ago, Zer0Nin3r said:

In my quick tests in the past, folders inside /tmp are cleared after a reboot. *shrugs*

yeah, that's what i'm going for. otherwise, it leaves countless empty folders behind that have to be cleared out. It also appears that these folders aren't integral to the log/history, so they serve no purpose that i can see.

Link to comment
18 minutes ago, Zer0Nin3r said:

The folders are cleared out automatically after a reboot. No maintenance is necessary from my understanding. Unless, you like strict order Vs. chaos. 🤣

In my experience that is only true if you have them on a RAM location (which is where /tmp is located).    If you have them on physical media they do not seem to get cleared out.

Link to comment
33 minutes ago, itimpi said:

In my experience that is only true if you have them on a RAM location (which is where /tmp is located).    If you have them on physical media they do not seem to get cleared out.

Roger that. Then I believe my /tmp folder is in RAM then as the folders are cleared out after a reboot; good to know.

 

That being said, I haven't had any issues with RAM overflow while encoding with Unmanic (3 workers).

Link to comment

Wanted to share what i think is another awesome way of utilizing Unmanic. I don't mean to hijack the thread here, but this certainly seems like a good place to share with fellow unmanic users. 

 

It requires two different unraid servers, so if you only have one, I don't see this being applicable for you. But essentially, i've "found" a way to use two instances of unmanic on one library. i'll just give some bullet points, if anyone is actually interested in details feel free to ask any questions.

 

For my setup, i have an unraid server at "home", and one at a separate location at my "office", so they do not share a LAN. Unraid 6.8 has introduced support for wireguard VPN and that's what makes this all possible. I won't go into the details of setting that up, but for anyone that missed that intro look here for setup info. "home" server is where all my media server content is housed, and where my primary unmanic instance has been running the past few months. "office" server is a way over-powered ryzen 2700 file server.

 

Reference: "home" = 192.168.1.121

                 "office" = 192.168.0.121

  1. Setup "home" wireguard VPN tunnel and add "office" as a Lan to Lan peer
  2. with an active wireguard connection, use the unassigned devices plugin to add remote smb path on "office" server pointing to path "192.168.1.121/media" (this is the plex media library location on "home") and mount the path.
  3. install unmanic docker on "office". now the library paths will look something like this:
    1. /library/movies -> /mnt/disks/192.168.1.121_media/Plex/Movies/
    2. /library/tv shows -> *intentionally left blank, see explanation below*
    3. change access mode for both paths to "RW/Slave"
  4. As i mentioned earlier in the thread here, i've changed my cache directory to use ram instead of disk so now it's either /tmp or /dev/shm/ if you go this route
  5. apply the settings and let the container install. access gui, and make sure tv path is left empty

 

The reason i've not mapped the tv path is because i don't want the containers fighting over the same files. So doing this, i've got "home" converting TV shows, and "office" remotely converting Movies. your own configuration may vary here, but it's important not to point both containers at the same media folders simultaneously. I see this causing problems.

 

So far, i'm 24 hours into this experiment, "home" wireguard is showing 59.9gb up and and 45.9gb down, with 22 successful movie conversions. i've not had a single file failure yet, and i've just cut my overall library conversion time (more or less) in half. I've got my dual xeon server at home and my ryzen 2700 8-core both crunching away as we speak. Of course i have cpu pinning and prioritization in place on both servers. hopefully somebody can find this useful. YMMV. good luck!

 

 

 

Link to comment

@Josh.5, could you please give a brief overview of what operations you do on /tmp/unmanic inside the container? I was experimenting with allowing my second server to help with the load, and I ran into some permission issues, it seemed to indicate it couldn't move the file from /tmp/unmanic to the final destination. So, I mapped /tmp/unmanic to a spot on the same mount point, and it appears to have recursively deleted my media files out of the folders in that mount, leaving the folders empty.

 

I assumed with a description of "encoding cache directory" that I could safely map that anywhere I wanted. That appears not to be the case.

Link to comment
On 12/30/2019 at 4:22 PM, jonathanm said:

@Josh.5, could you please give a brief overview of what operations you do on /tmp/unmanic inside the container? I was experimenting with allowing my second server to help with the load, and I ran into some permission issues, it seemed to indicate it couldn't move the file from /tmp/unmanic to the final destination. So, I mapped /tmp/unmanic to a spot on the same mount point, and it appears to have recursively deleted my media files out of the folders in that mount, leaving the folders empty.

 

I assumed with a description of "encoding cache directory" that I could safely map that anywhere I wanted. That appears not to be the case.

@jonathanm i think josh.5 may be clocked out for a few months, per this post here

Link to comment
  • 2 weeks later...

Problem: radarr keeps replacing transcoded films.

 

From radarr's history, it looks like radarr is noticing when unmanic deletes the old version of the film but does not rescan the folder to see that the h265 version is in there so thinks it still needs downloading. It also does not see the quality (shows as unknown if manual rescan is done) so presumably any download is an upgrade in quality compared to the unmanic transcode.

 

I think one of these things is behind radarr redownloading the old version of the film, which of course renders unmanic useless. Has anyone else noticed this or figured out a way to work around it?

 

I haven't noticed this happening with sonarr yet but unmanic has mostly been transcoding films so far.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.