[Support] Josh5 - Unmanic - Library Optimiser

[Josh.5 98]

26 minutes ago, Transient said:

I can, although I'm not really comfortable sharing it here.

 

That said, checking unmanic.log, I found my workers all report this:

 

2020-10-22T11:28:30:ERROR:Unmanic.Worker-0 - [FORMATTED] - Exception in processing job with Worker-0: - expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/unmanic/libs/foreman.py", line 224, in run
    self.process_task_queue_item()
  File "/usr/local/lib/python3.6/dist-packages/unmanic/libs/foreman.py", line 203, in process_task_queue_item
    self.current_task.set_success(self.process_item())
  File "/usr/local/lib/python3.6/dist-packages/unmanic/libs/foreman.py", line 159, in process_item
    common.ensure_dir(self.current_task.task.cache_path)
  File "/usr/local/lib/python3.6/dist-packages/unmanic/libs/common.py", line 63, in ensure_dir
    directory = os.path.dirname(file_path)
  File "/usr/lib/python3.6/posixpath.py", line 156, in dirname
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType

 

Are there any fancy characters in the name or path of your files?

[Transient 4]

11 minutes ago, Josh.5 said:

Are there any fancy characters in the name or path of your files?

Just letters, numbers, spaces and round brackets ( ). Nothing unusual.

[Josh.5 98]

1 hour ago, Transient said:

Just letters, numbers, spaces and round brackets ( ). Nothing unusual.

May be a bug in the python dist package. Without the full path I cannot know for sure.

I did a full update of the Docker Ubuntu base image the other day, perhaps that is why this is now broken

[Spritzup 6]

Minor complaint, but how do I stop it from auto creating the Media directory in my user share?  Everytime I delete it, it comes back.

 

Thanks!

~Spritz

[Josh.5 98]

12 hours ago, Spritzup said:

Minor complaint, but how do I stop it from auto creating the Media directory in my user share?  Everytime I delete it, it comes back.

 

Thanks!

~Spritz

Do you by any chance have it as a map on your Unmanic docker container config? I'm not sure what "Media" directory you are talking about. Unmanic does not do anything with a "Media" directory in it's defaults.

[eagle470 6]

3 hours ago, Josh.5 said:

Do you by any chance have it as a map on your Unmanic docker container config? I'm not sure what "Media" directory you are talking about. Unmanic does not do anything with a "Media" directory in it's defaults.

Actually it does, your default template it's 'clean' of the mount points you use to store your test data on your systems. The /library location in the container goes to /mnt/user/Media/TV and /mnt/user/Media/Movies locations, which I can only assume is how you have your test boxes mapped.

[Transient 4]

17 hours ago, Josh.5 said:

May be a bug in the python dist package. Without the full path I cannot know for sure.

I did a full update of the Docker Ubuntu base image the other day, perhaps that is why this is now broken

Check your private messages.  

[Josh.5 98]

1 hour ago, eagle470 said:

Actually it does, your default template it's 'clean' of the mount points you use to store your test data on your systems. The /library location in the container goes to /mnt/user/Media/TV and /mnt/user/Media/Movies locations, which I can only assume is how you have your test boxes mapped.

O right. Well there we go then. Spritzup needs to modify the Docker template.

[CoZ 13]

@Josh.5 you've been pushing a ton of updates lately!  Almost daily I get an update container notification on my unRaid server for your docker. 

[Josh.5 98]

17 minutes ago, CoZ said:

@Josh.5 you've been pushing a ton of updates lately!  Almost daily I get an update container notification on my unRaid server for your docker. 

Indeed. It's hacktober. I took a few days off work so I could spend some time kick-starting development again. The issue list on GitHub was starting to backup a bit.

[TexasDave 6]

On 10/24/2020 at 2:57 AM, Josh.5 said:

Indeed. It's hacktober. I took a few days off work so I could spend some time kick-starting development again. The issue list on GitHub was starting to backup a bit.

 

It is much appreciated - Thank you!!

[Cpt. Chaz 20]

@Josh.5 i'm familiar with the basics of ffmpeg, but wondering if you can clarify use on the advanced options in this container for me a little bit. the command example shown uses input and output file paths, so does that mean that these additional options only work on a per file basis? can more than one at a time be entered in? if it's not on a per file basis, how does one override the unmanic defaults for all files? thx

[Josh.5 98]

1 hour ago, Cpt. Chaz said:

@Josh.5 i'm familiar with the basics of ffmpeg, but wondering if you can clarify use on the advanced options in this container for me a little bit. the command example shown uses input and output file paths, so does that mean that these additional options only work on a per file basis? can more than one at a time be entered in? if it's not on a per file basis, how does one override the unmanic defaults for all files? thx

This overwrites the command options for all files.

 

You can enter as many as you like separated either by a new line or space.

 

And adding an option to this field that already exists will replace the unmanic default for all files.

 

Just note that the example is just an example. The command does differ in some ways for each of your files based on its streams and container.

[Spritzup 6]

On 10/23/2020 at 11:48 AM, Josh.5 said:

O right. Well there we go then. Spritzup needs to modify the Docker template.

I'm not sure what you mean by modify the template, but I will say that every time I update the container, those mappings come back... even after I've deleted them.  It's irritating, because then those shares are created, and it appears that it's only this container that does it.

 

~Spritz

[eagle470 6]

What happened to the /config data? I have no idea how, but on my system it's entirely empty. Did you accidentally move the config data to the /library folder? There used to be a settings file and an unmanic.db file, both of which are entirely gone on my system.

[Cpt. Chaz 20]

2 hours ago, eagle470 said:

What happened to the /config data? I have no idea how, but on my system it's entirely empty. Did you accidentally move the config data to the /library folder? There used to be a settings file and an unmanic.db file, both of which are entirely gone on my system.

it's in the same place, just in a "hidden" folder, ie mnt/user/appdata/unmanic/.unmanic

 

in terminal, navigate to the unmanic appdata folder and run ls -a and you'll see it. if using a file browser in windows / mac, enable hidden files and folders respectively

[Cpt. Chaz 20]

On 10/27/2020 at 1:06 PM, Josh.5 said:

This overwrites the command options for all files.

 

You can enter as many as you like separated either by a new line or space.

 

And adding an option to this field that already exists will replace the unmanic default for all files.

 

Just note that the example is just an example. The command does differ in some ways for each of your files based on its streams and container.

cool, thanks!

[eagle470 6]

Well, I'm no longer picking up anything within either my movies or TV directory and even if I do, rarely, transcoding fails 100% of the time. I'm not sure what happened, but this thing has totally stopped working for me....

[Transient 4]

21 minutes ago, eagle470 said:

Well, I'm no longer picking up anything within either my movies or TV directory and even if I do, rarely, transcoding fails 100% of the time. I'm not sure what happened, but this thing has totally stopped working for me....

You might be having the same problem as me. I traced it down to the cache path (where it does its transcoding) sometimes evaluating as None, but I couldn't see how the situation arises. I sent some details over to Josh.5 but I don't think he's had a chance to review it yet.

 

I would suggest you also share your unmanic.log with him. If someone else is reporting the same issue, I imagine it would help identify what's going wrong.

[eagle470 6]

On 10/30/2020 at 11:25 PM, Transient said:

You might be having the same problem as me. I traced it down to the cache path (where it does its transcoding) sometimes evaluating as None, but I couldn't see how the situation arises. I sent some details over to Josh.5 but I don't think he's had a chance to review it yet.

A long time ago I had issues with transcoding in /tmp as I didn't have enough RAM for larger files  and I don't run ECC RAM in my server so Plex would sometimes crap out on me in the middle of transcoding a movie. 

 

I have a Datacenter grade Intel x8 PCIe 3.0 NVMe card I use as my cache, which is where I have my transcode directory, it's at /mnt/cache/transcode. My app data location is under a user share at /mnt/user/appdata.

 

 

 

unmanic.log

Edited November 2, 2020 by eagle470

[Transient 4]

26 minutes ago, eagle470 said:

A long time ago I had issues with transcoding in /tmp as I didn't have enough RAM for larger files  and I don't run ECC RAM in my server so Plex would sometimes crap out on me in the middle of transcoding a movie. 

 

I have 64GB of ECC RAM, so it shouldn't be that, but I'll give that a try anyway. Who knows, the part that creates the temp file might be failing which is why it ends up set to None.

 

I had a quick look through your log and it looks like you have something else going on. It doesn't seem to have any errors or even pick up any files to transcode at all.

[Cpt. Chaz 20]

just a heads up for anyone transcoding to ram, an 8gb movie file will use 8gb of ram. 3 workers transcoding 8gb movie files at once will use 24gb of ram. an insufficient amount of ram will cause transcode failures, so if you're in short supply, try dropping the number of workers, or it may be the best case is to transcode to a cache disk or unassigned SSD instead.

 

Also, if you do wanna try transcoding to ram and you don't want unmanic to use it all, you can set the path to /dev/shm which allocates only 50% of the ram. you can also decrease unmanics CPU priority by adding --cpu-shares=2 to the extra parameters field in the container.

[eagle470 6]

18 hours ago, Cpt. Chaz said:

 it may be the best case is to transcode to a cache disk or unassigned SSD instead.

This is what I ended up doing on 4 SSD's in a RAID 10 as my cache originally. It was super fast. 

 

Currently I have a 1.6TB NVMe SSD I'm using that fills that role, though I don't think it's as fast as those 4 SATA SSD's....

Edited November 2, 2020 by eagle470

[eagle470 6]

OK, so I enabled debugging, deleted the database and started over. I've attached the logs after a number of failed conversions.

 

Edit: Attached a second log with some successes and many failures.

unmanic.log

unmanic.log

Edited November 2, 2020 by eagle470

[Nosirus 0]

Some mkv's keep the file size of the stream, anyone know how to remove it without removing the language from the track?