[SUPPORT] reven - FileFlows

[reven]

7 hours ago, Xenu said:

Any ideas why the internal node is ignoring the schedule I've set for it?

I assume I make the hours I want the node to function purple, and the rest is when it sleeps? If so, as I say, its ignoring the schedule I set.

Created a ticket for this on github
https://github.com/revenz/FileFlows/issues/24

I'll fix it in the next release (probably this weekend).

[Xenu]

On 3/30/2022 at 1:21 AM, reven said:

Created a ticket for this on github
https://github.com/revenz/FileFlows/issues/24

I'll fix it in the next release (probably this weekend).

Thank you! Is there any plan to add an option to strip audio files appart from the language specified?

[reven]

4 hours ago, Xenu said:

Thank you! Is there any plan to add an option to strip audio files appart from the language specified?

I'm not even sure what you mean by this?   strip a music file/audio file???? Ive only ever seen these with single tracks.

[Eraxar]

Hiho, i have a Question, is there a way to ignore a folder that contains a specific word?

maybe its possible to add an option in the libary setting for that like the file filter?

[reven]

1 minute ago, Eraxar said:

Hiho, i have a Question, is there a way to ignore a folder that contains a specific word?

maybe its possible to add an option in the libary setting for that like the file filter?

You can use a library "Filter" to match items.  So in this example (which is "SABnzbd Videos" its ignoring files that contain _UNPACk or _FAILED or sample.   This regex is tested against the full path of files it finds, so you can get quite complex with this if you want. 

[Eraxar]

Awesome, have no clue i can put it there, iam not so familiar with regex. thank you verry much

[Bolagnaise]

How would i go about forcing plex to rescan the files if i have setup a basic x265/ac3 encode? Is there a plex plugin on the horizon? Could i just delete the original and force plex to rescan the folder?

[Faceman]

Loving this app, I now have multiple flows handling various things on my system like magic.

 

However, I'm getting out of memory errors.

 

Previously I limited the container to 4gb but it didn't like that, so I bumped it to 6gb. That seemed fine since it tended to hang around 2-3gb in normal use. but occasionally it will start to shoot up and will quickly crash.

If I remove the limit entirely it will crash my whole docker system eventually.

 

I'm suspecting this is a bug rather than the app actually needing dozens of gigs of ram.

 

I also noticed it was generating huge logs, a gigs worth of them in the appdata folder.

[reven]

Memory usage:  this could be ffmpeg and depends on what you are processing.  e.g. the HDR to SDR can consume a lot of memory.

 

Logs:  This will be fixed soon in an upcoming release

[Faceman]

no HDR to SDR being done, 90% of what file flows is doing through the day is adding an audio track based on simple rules. sometimes Plex forces a video transcode when only the audio needs transcoding, so I add compatible tracks in FileFlows to minimize that.

 

there is a small amount of video transcoding being done overnight, but its just basic h264 > h265 on files around 2-3gb is size. this does seem to be what kills it though. I'm going to read through the logs and see if it's certain files doing it or something else..

[reven]

HDR to SDR was an example.

FileFlows currently only supports a sqlite database and due to limitation of that database, it caches all objects also in memory to improve performance.   This obvisouly uses more memory, and the bigger the library the more memory it will use.

 

You mentioned gigs of log files, which suggests you have a large library of files processed.  This could be the the reason for the memory usage.

 

I am planning on adding support for an external database very shortly (my next big thing to add), which if configured (it would be opt in, sqlite by default) would mean nothing needs to be stored in memory and just loaded when needed.  

 

But I am aware of the memory usage problem and will be focusing on reducing this as much as possible.

[Faceman]

That seems like it could do it, I have tens of thousands of files, and it is keeping track of every one of them ( i guess to avoid re-processing files through the same flow every scan?)

 

I cant see how that would grow to several gigs of ram in just a couple of days though?

maybe a couple of hundred megs? how much data is cached for each library entry?

[reven]

its storing the in memory object and all its properties.   C# stores string internally as unicode, utf-16, so that increases memory.   if you got tens of thousands of files, thats all the dates, filename, records of what flow parts were run against each file etc.  

 

So that definetly has an effect on memory usage.   To what amount, I cannot tell until I can easily switch to an external database and compare the differences.

[Faceman]

In that case I think I'll keep running it just on a limited library with --memory='8g' for now since it isn't hurting anything if it does crash the container. at least with that memory limit command it doesn't crash the whole server.

 

 

[juanro]

Loving it so far, thanks for all the work!

I tried adding my PC as a node, my server currently doesn't have a GPU so it's taking a while to go through my whole queue. Adding it to the docker was pretty straightforward, it all seems to be set up properly but when it tries to process a file on that external node it gives me a "Library file exists but is not accessible from node" error, and it keeps on moving through my whole queue like that. I can access/open the files from my PC as well as all the paths where it's picking up/replacing the file once it's processed.

 

Any idea what might be causing that error? Do I have to setup some permissions for that node?

[reven]

https://docs.fileflows.com/guides/external-processing-node

 

The mapping section.

[juanro]

2 hours ago, reven said:

https://docs.fileflows.com/guides/external-processing-node

 

The mapping section.

 

That seems to have fixed that error, getting an access denied one now. Can't figure out what I'm missing now.

 

2022-08-16 19:13:01.825 [INFO] -> ======================================================================
2022-08-16 19:13:01.827 [INFO] -> Executing Node 9: MoveFile [FileFlows.BasicNodes.File.MoveFile]
2022-08-16 19:13:01.828 [INFO] -> ======================================================================
2022-08-16 19:13:01.834 [INFO] -> About to move file to: \media\file.mkv
2022-08-16 19:13:01.836 [INFO] -> Mapped destination path: \media\file.mkv
2022-08-16 19:13:01.838 [INFO] -> Moving file: "C:\localtempfolder\Runner-413460f7-8c13-4f83-9b80-fa12802aad0c\624ff387-2a4c-45bc-9205-d2112572c703.mkv" to "\media\file.mkv"
2022-08-16 19:13:01.840 [INFO] -> File moved successfully
2022-08-16 19:13:01.841 [INFO] -> Initing new moved file
2022-08-16 19:13:01.900 [INFO] -> Deleting original file: \\ip\pathtofile\file.mp4
2022-08-16 19:13:02.556 [WARN] -> Failed to delete original file: Access to the path '\\ip\pathtofile\file.mp4' is denied.
2022-08-16 19:13:02.557 [INFO] -> Node execution time: 00:00:00.7276076
2022-08-16 19:13:02.558 [INFO] -> ======================================================================

 

I think I've set up the mapping correctly, it added the ffmpeg.exe mapping by itself. Added the 3 folders I'm picking up files from and another folder where it moves the files after they're processed. Same path as they've been working so far on the server side, copy pasted from the file explorer as they show up on Windows for the external node side.

Edited August 16, 2022 by juanro
Adding info

[reven]

that suggests FileFlows doesnt have write permissions to that path.  check if you can open that path in file explorer and manually delete it.

[juanro]

46 minutes ago, reven said:

that suggests FileFlows doesnt have write permissions to that path.  check if you can open that path in file explorer and manually delete it.

Can confirm I can open the paths on file explorer. Could copy, rename and delete files without a problem in all of the folders.  Tried running the node as an admin and it gave me the old "Library file exists but is not accessible from node" error. Ran it again without admin and it seems to work until it gets to the MoveFile stage.

[juanro]

Seems to be working now. For whatever reason I'm still getting the 'Failed to delete original file: Access to the path 'U:\path\file.mp4' is denied.' on the Move File stage but now I can see/play the new .mkv in the output directory.

 

Tried a couple different things (wiping Windows credentials, making a new Unraid user, reinstalling, installing the DotNet sdk required for the server installation, etc), think the one that finally did the trick was mounting the shares as a lettered drive, had them as just a network drive before (from \\LocalP\path to U:path\). Not sure if it was a combination of the other things I tried or if I just needed to map the drives.

 

Files are processing a lot faster with the GPU, really looking forward to getting one for the server. Thanks for all your help!

[juanro]

7 hours ago, juanro said:

Seems to be working now. For whatever reason I'm still getting the 'Failed to delete original file: Access to the path 'U:\path\file.mp4' is denied.' on the Move File stage but now I can see/play the new .mkv in the output directory.

 

Tried a couple different things (wiping Windows credentials, making a new Unraid user, reinstalling, installing the DotNet sdk required for the server installation, etc), think the one that finally did the trick was mounting the shares as a lettered drive, had them as just a network drive before (from \\LocalP\path to U:path\). Not sure if it was a combination of the other things I tried or if I just needed to map the drives.

 

Files are processing a lot faster with the GPU, really looking forward to getting one for the server. Thanks for all your help!

Left it running overnight. Seems the external node is having problems deleting files. For the library that grabs and replaces the file in the same path it's just erroring out. For the ones where it grabs a file and copies it over to a new folder it's doing 90% of the work, but also not deleting the original file, that's where it kept giving me the 'Failed to delete original file' error.

[juanro]

3 hours ago, juanro said:

Left it running overnight. Seems the external node is having problems deleting files. For the library that grabs and replaces the file in the same path it's just erroring out. For the ones where it grabs a file and copies it over to a new folder it's doing 90% of the work, but also not deleting the original file, that's where it kept giving me the 'Failed to delete original file' error.

Making one last post for anyone that runs into similar issues. Ran Docker Safe New Perms (under Tools tab, should be installed with Fix Common Problems), looks like some apps took ownership of some files so FileFlows couldn't delete them after, this cleared that problem. Been running through different libraries with the external node and it's worked as expected.

[BlazzPlay]

hello, is working on adding different languages?

[reven]

3 hours ago, BlazzPlay said:

hello, is working on adding different languages?

do you mean to the UI?  if so, sorry I only know english, so can't translate it myself.

 

[deadnote]

Hi

I would like to remove tdarr because of the docker image size and the warning in fix common problems, so i tried fileflows.

Everything seems to be OK, but I noticed the fileflows's logo doesn't display in the dashboard view in unraid. With tdarr, the icon displays after "active apps".

Do I miss something in the configuration ?
Thanks