1 hour ago, wgstarks said:

Thanks. The same idea occurred to me. Might just go ahead and move the db to my nvme cache and see what happens.

I've been beavering away on filtering for bitmagnet and now have a working filter to reduce the crud and thus reduce the size of the database, I'm happy to share if anybody is interested.

31 minutes ago, binhex said:

I've been beavering away on filtering for bitmagnet and now have a working filter to reduce the crud and thus reduce the size of the database, I'm happy to share if anybody is interested.

Please do.

36 minutes ago, wgstarks said:

Please do.

Take a look here, the main meat of the filtering happens on line 45, you probably will want to tweak this to your prefererence:-  https://github.com/binhex/documentation/blob/master/docker/configs/bitmagnet/classifier.yml

Edit - if you want to see what it would delete without actually deleting anything then uncomment these lines:- https://github.com/binhex/documentation/blob/126eecd01f39402f440b149c22374c2600a6a92b/docker/configs/bitmagnet/classifier.yml#L47-L48 and comment https://github.com/binhex/documentation/blob/126eecd01f39402f440b149c22374c2600a6a92b/docker/configs/bitmagnet/classifier.yml#L46

1 hour ago, binhex said:

Edit - if you want to see what it would delete without actually deleting anything

How do you view the results?

45 minutes ago, wgstarks said:

How do you view the results?

'Torrent Tag' on the left hand side, expand it and you should see your named tag there with any matches.

I went ahead and used the front le manager plugin to move the app appdata directory to my /mnt/user/cache (nvme drive). Also noticed I had misnamed my classifier.yml file to config.yml so corrected that also with file manager.

 

Now after starting the docker I’m back to waiting for postgeSQL to be ready. Thought I would give it some time but it’s been more than 30 minutes.

 

supervisord 8.log

42 minutes ago, wgstarks said:

Also noticed I had misnamed my classifier.yml file to config.yml so corrected that also with file manager.

yeah don't do that, config.yml and classifier.yml are similar but not the same, if you try to simply rename you will get errors in bitmagnet and it will not start, as shown in your log:-

 

2024-09-30 10:32:34,743 DEBG 'start' stdout output:
[31mERROR[0m	worker/worker.go:190	error starting worker	{"key": "queue_server", "error": "1 error(s) decoding:\n\n* '' has invalid keys: classifier"}
github.com/bitmagnet-io/bitmagnet/internal/boilerplate/worker.(*registry).Start
	/tmp/bitmagnet/internal/boilerplate/worker/worker.go:190
github.com/bitmagnet-io/bitmagnet/internal/boilerplate/app/cmd/worker.New.func1
	/tmp/bitmagnet/internal/boilerplate/app/cmd/worker/command.go:42
github.com/urfave/cli/v2.(*Command).Run
	/home/nobody/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:276
github.com/urfave/cli/v2.(*Command).Run
	/home/nobody/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:269
github.com/urfave/cli/v2.(*Command).Run
	/home/nobody/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:269
github.com/urfave/cli/v2.(*App).RunContext
	/home/nobody/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:333
github.com/bitmagnet-io/bitmagnet/internal/boilerplate/cli.New.func4.1
	/tmp/bitmagnet/internal/boilerplate/cli/cli.go:53

2024-09-30 10:32:34,743 DEBG 'start' stdout output:
[34mINFO[0m	worker/worker.go:218	stopped worker	{"key": "http_server"}

2024-09-30 10:32:34,745 DEBG 'start' stderr output:
panic: 1 error(s) decoding:

* '' has invalid keys: classifier

goroutine 30 [running]:
github.com/bitmagnet-io/bitmagnet/internal/boilerplate/cli.New.func4.1()
	/tmp/bitmagnet/internal/boilerplate/cli/cli.go:54 +0x47
created by github.com/bitmagnet-io/bitmagnet/internal/boilerplate/cli.New.func4 in goroutine 29
	/tmp/bitmagnet/internal/boilerplate/cli/cli.go:52 +0xb8

2024-09-30 10:32:34,754 DEBG 'start' stderr output:
2024-09-30 10:32:34.754 EDT [107] LOG:  received smart shutdown request

Also ensure you pull down the latest image, this includes an example classifier.yml and symlinks to the correct location inside the container.

16 minutes ago, binhex said:

yeah don't do that, config.yml and classifier.yml are similar but not the same, if you try to simply rename you will get errors in bitmagnet and it will not start.

also ensure you pull down the latest image, this includes an example classifier.yml.

Changed the file name back and I can get to the webUI now but not sure it’s actually working. Getting errors about duplicate values.

 

supervisord.log

 

Maybe I should just nuke the db and let it start over?

 

 

19 minutes ago, wgstarks said:

Changed the file name back and I can get to the webUI now but not sure it’s actually working. Getting errors about duplicate values.

 

supervisord.log 7.67 MB · 0 downloads

 

Maybe I should just nuke the db and let it start over?

 

 

hmm those errors do look database related, you could leave it, it may sort itself out, if not then the easiest course of action would be nuke the db (as in delete /config/postgres).

 

if you are going to go for the nuke option you may want to ensure the filtering is as you want as this is the time to sort it if you want a nice clean database.

4 hours ago, binhex said:

Take a look here, the main meat of the filtering happens on line 45, you probably will want to tweak this to your prefererence:-

If I’m reading this correctly it’s filtering any file with foreign characters, right?

 

Cant really see what will be removed with a brand new db.

2 hours ago, wgstarks said:

it’s filtering any file with foreign characters, right?

well its slightly more targeted than that, from the comment in the file:-
 

# remove any magnet in content type movies or tv series that matches the extensions video_movie and is 200 MB or larger and contains a torrent filename with 2 or more consecutive non ascii characters

In my experience of running this (using tagging to view the match) i have not seen any false positive matches, all chinese/japanese/koren/etc movie files get tagged for movies and tv series.

 

But of course this is what I want, what you may want to filter is completely down to you, and it maybe that you don't care about the wasted db space and simply want to grab as many magnets as possible, some people have a pokemon attitude to magnets 'gotta catch em all!' 🙂

I have been trying to get filtering working correctly and giving me the results i want for quite some time now ( 6+ hours) but now its there i'm happy, feel free to ask away if you want to achieve a specific filter type, i may not know the answer though straight away, CEL is still very new to me.

3 minutes ago, binhex said:

well its slightly more targeted than that, from the comment in the file:-
 

# remove any magnet in content type movies or tv series that matches the extensions video_movie and is 200 MB or larger and contains a torrent filename with 2 or more consecutive non ascii characters

In my experience of running this (using tagging to view the match) i have not seen any false positive matches, all chinese/japanese/koren/etc movie files get tagged for movies and tv series.

 

But of course this is what I want, what you may want to filter is completely down to you, and it maybe that you don't care about the wasted db space and simply want to grab as many magnets as possible, some people have a pokemon attitude to magnets 'gotta catch em all!' 🙂

I have been trying to get filtering working correctly and giving me the results i want for quite some time now ( 6+ hours) by now its there i'm happy, feel free to ask away if you want to achieve a specific filter type, i may not know the answer though straight away, CEL is still very new to me.

Actually this is just what I have been wanting plus a filter for movies and tv that stops the ones with no English language.

 

I installed the classifier.yml file using file manager to upload it. Noticed that permissions differ (not executable). Do I need to adjust this? If so how? File manager will set read/write but not executable.

 

51 minutes ago, wgstarks said:

 

Permissions look fine, classifier yml is a configuration file so it does not need to be directly executed

Also, noticed that my config.yml gets disabled when I start the docker after adding classifier.yml. Should this be happening? You can see it in the screenshot above.

Edited September 30, 20241 yr by wgstarks

15 minutes ago, wgstarks said:

Also, noticed that my config.yml gets disabled when I start the docker after adding classifier.yml. Should this be happening? You can see it in the screenshot above.

Yes I forced the rename as you cannot have both config and classifier

3 minutes ago, binhex said:

Yes I forced the rename as you cannot have both config and classifier

Got it.

 

I missed the "banned" and "delete_content_types" sections. I need to modify them to match config.yml.

4 hours ago, binhex said:

feel free to ask away if you want to achieve a specific filter type

Is there a possibility for a workflow for allowed keywords (languages)?

 

I see on bitmagnet.io that bitmagnet can filter for language, resolution, etc but no examples specifically for these. I thought about using banned keywords but many torrents have multiple languages so would result in false matches.

 

I'm trying to eliminate all torrents not marked as english.

9 hours ago, wgstarks said:

Is there a possibility for a workflow for allowed keywords (languages)?

 

I see on bitmagnet.io that bitmagnet can filter for language, resolution, etc but no examples specifically for these. I thought about using banned keywords but many torrents have multiple languages so would result in false matches.

 

I'm trying to eliminate all torrents not marked as english.

from the developer:-
 

Quote

There is an open issue for proper language detection BTW. Doing it with regexes might be tricky and/or involve putting lots of character ranges in...

So right now, nope there is no way to pick up english as flagged by bitmagnet, best you could do is a bit of clever regex, but as mentioned its going to be tricky.

I haven't looked too deeply into the yml, so if you could fill out the details needed to actually match the language section (or would it just be title?) that would be appreciated. Some regex that might/should hopefully work

\b[Ee][Nn]([Gg]|([Gg][Ll][Ii][Ss][Hh])?)\b

matches (on a word boundary) En, Eng, or English, case-insensitive. However, if it's a title that would need some tweaking to prevent false positive/negatives.

Edited October 1, 20241 yr by FunkeCoder23

For the record the DB data compresses really well (about 3/4th savings) so if you're on btrfs/zfs enable compression on that appdata folder.

My folder is 22GB, and compressing it with zstd on btrfs freed 20GB between the compression and defragging (5.6GB actual use now).

Edited October 2, 20241 yr by Kilrah

10 hours ago, FunkeCoder23 said:

I haven't looked too deeply into the yml, so if you could fill out the details needed to actually match the language section (or would it just be title?) that would be appreciated. Some regex that might/should hopefully work

\b[Ee][Nn]([Gg]|([Gg][Ll][Ii][Ss][Hh])?)\b

matches (on a word boundary) En, Eng, or English, case-insensitive. However, if it's a title that would need some tweaking to prevent false positive/negatives.

The trouble with that is that you would be deleting a lot of english movies/tv that do not contain 'En, Eng, or English, case-insensitive' which is the vast majority of releases, if you want to do this your best bet is to negatively match on stuff you want to delete for example match on 'fr, FRE, French' for example if you do not want movies/tv shows with french audio, you could be a little clever and put in a ^((?!en).)*$ or similar as part of the regex to ensure it does not pick up movies that contain (in this example) french AND english audio.

 

HOWEVER disk space is cheap at the end of the day and it's not really the job of the index server to filter, i have put in filtering for non ascii titles as i cannot even tell what the movie/tv sow is, but really this is a job for the scripts/apps searching the index site, i.e. sonarr/radarr etc, but that is just my personal opinion.

My quick stab at this, so the following regex will match on fr, fre, french, ge, ger, germany (feel free to extend languages) but will NOT match if en, eng, or english are (also) present.
 

^(?!.*\b(en|eng|english)\b).*?\b(fr|fre|french|ge|ger|germany)\b.*$

 

Quote

However, if it's a title that would need some tweaking to prevent false positive/negatives.

To be clear this could either match on torrent name or filename within the torrent.

Recording how far i got with language filtering, so the below code doesn't work as expected, it looks like for whatever reason the regex set of separators is not being used, leading to bad partial matching on languages in the movie title, but I'm recording it here in case anybody is interested and wants to try and complete it:-
 

    - if_else:
        condition: "result.contentType in [contentType.movie]"
        if_action:
          find_match:
            - if_else:
                condition:
                  or:
                    # TODO regex currently seems to incrrectly do partial matches need to fix
                    - "torrent.baseName.matches(r'^\\(?!.*[\\.\\-_\\s]\\(eng|english\\)[\\.\\-_\\s]\\).*[\\.\\-_\\s]\\(fre|french|ger|germany|ita|italy\\)[\\.\\-_\\s].*$')"
                    - "torrent.files.filter(f, f.extension in extensions.video_movie && f.size > 200*mb).map(f, f.basePath).join(' ').matches(r'^\\(?!.*[\\.\\-_\\s]\\(eng|english\\)[\\.\\-_\\s]\\).*[\\.\\-_\\s]\\(fre|french|ger|germany|ita|italy\\)[\\.\\-_\\s].*$')"
                if_action:
                  add_tag: foreign-movie

Of note CEL requires a lot of escaping, thus the ugliness in an already rather lengthy regex!.

Recently i changed my cache SSD and had to move appdata folder/share.

After moving it i couldn't access the WebUI and i was getting the missing Postgres db error.

I deleted (nuke you said ) the /config/postgres dir and everything is fine now.

31 minutes ago, francrouge said:

i'm trying to configure binhex-iventoy but i cant get the webgui working and im getting a lot of errors in the logs

 

anyone has the same issues ?

 

thx

I don’t think this has anything to do with bitmagnet. If you go to the docker tab and click the icon for the iventoy docker you can select “support” and go to the support thread for that docker.