[Support] dyonr - magnetico-environment

[Dyon]

Welcome to my second Docker Container that I've ever created. magnetico-environment. A Docker to run the magnetico project.

 

Overview:

This Docker runs a magneticow and magneticod instance.

In short: magneticod crawls the BitTorrent DHT and adds torrent information that it finds to its database. magneticow is the simple http interface that you can use to search the database.

 

Base: Debian 10

Automated Build: No

Application: https://github.com/boramalper/magnetico

Docker Hub: https://hub.docker.com/r/dyonr/magnetico-environment/

GitHub: https://github.com/DyonR/docker-magnetico-environment

Edited July 27, 2020 by Nord
Fixed hyperlinks

[frakman1]

@Nord Looks promising. I installed it and logged into the WebUI but anything I search for returns nothing. I tried searching for "ubuntu" like the screenshot on the github page shows and it returned nothing too. Am I missing something?

Edited July 20, 2020 by frakman1

[Dyon]

6 hours ago, frakman1 said:

@Nord Looks promising. I installed it and logged into the WebUI but anything I search for returns nothing. I tried searching for "ubuntu" like the screenshot on the github page shows and it returned nothing too. Am I missing something?

Magneticod indexes the DHT network. Before it is able to find torrents, it needs to crawl around on the DHT network and find torrents.

It is kinda like hosting your own torrent website, but it does look for torrents by itself and not depend on submissions of .torrent.

Note that I did not develop magneticod or magneticow, thus I recommend looking at the project page over here: https://github.com/boramalper/magnetico

There is also a community database dump, in which multiple people that run magnetico all merged their database together: https://github.com/boramalper/magnetico/issues/218

The latest dump is from the GitHub user kescherCode and can be found here: https://www.kescher.at/magneticod-db. This database contains about 13 million torrents and is about 40GB decompressed.

 

I really think this project in interesting, thus I decided to make a Docker container for it, since it's more flexable than the Dockers provided by magnetico itself

[frakman1]

I let it run overnight but it still doesn't find anything. I looked at the Github page. 

Are you able to get it to work? If so, how?

How does the docker container get this giant database of torrents? Do I need to provide some run parameter or is it baked in?  Do I have to place a file in the appdata fole "/mnt/user/appdata/magnetico" ? I checked and there are already three databases there "database.sqlite3  database.sqlite3-shm  database.sqlite3-wal"

 

I keep getting this:

[Dyon]

2 hours ago, frakman1 said:

I let it run overnight but it still doesn't find anything. I looked at the Github page. 

Are you able to get it to work? If so, how?

How does the docker container get this giant database of torrents? Do I need to provide some run parameter or is it baked in?  Do I have to place a file in the appdata fole "/mnt/user/appdata/magnetico" ? I checked and there are already three databases there "database.sqlite3  database.sqlite3-shm  database.sqlite3-wal"

 

I keep getting this:

<image removed>

 

That is quite odd. Out of the box without any configuration it should work fine.

It could be possible that your network/router blocks the traffic, since it's a quite heavy application on the network. I personally route the container through a VPN (I do it via my Firewall, but routing it through any of Binhex's VPN containers should work also, for reference check out this Spaceinvader One tutorial about on how to do this: https://www.youtube.com/watch?v=znSu_FuKFW0.

 

Could you set the environment variable 'MAGNETICOD_VERBOSE' to 'true', open the console of the container and execute the following command:

'tail -f ./Logs/magneticod.txt'

 

This is what a normal working environment should look like:

 

If yours doens't look like this, most likely not 'Fetching' anything, I recommend submitting an issue to the magnetico GitHub.

Note, there are multipe reports about magneticod not discovering torrents, but these are all quite old:

https://github.com/boramalper/magnetico/issues/85

https://github.com/boramalper/magnetico/issues/132

 

Edit: another thing that you could try is lowering the MAGNETICOD_NEIGHBORS to 100 for example, default is 1000.

Edit 2: About the databases, you can take the database of someone else and use that, but you can also let magneticod generate it's own database, this is what gets generated in /mnt/user/appdata/magnetico, you don't need to place anything there.

Edited July 20, 2020 by Nord

[frakman1]

Thanks for the tip.

I don't have any restrictive router firewalls. My UnRAID server can talk to anything on the outside.

I set that variable to 'true' and re-ran it. This is what my output log file looks like:

 

[Dyon]

3 minutes ago, frakman1 said:

Thanks for the tip.

I don't have any restrictive router firewalls. My UnRAID server can talk to anything on the outside.

I set that variable to 'true' and re-ran it. This is what my output log file looks like:

 

Could you do the same but with MAGNETICOD_NEIGHBORS set to 100?

[frakman1]

Ahh yes. Sorry I missed that. Thank you for the reminder.

That seems to have fixed it. Now I see entries in the log and results when I search.

Thank you!

 

[Dyon]

Just now, frakman1 said:

Ahh yes. Sorry I missed that. Thank you for the reminder.

That seems to have fixed it. Now I see entries in the log and results when I search.

Thank you!

 

Awesome! You can also play around with the neighbors value to see at what point it stops works. I believe that the neighbors value basically means with how many nodes magneticod will connect, so the more the better. Also, don't forget to disable verbose. Since the log file can build up quite quickly.

[sniffton]

Is there a way to use this with privoxy?

[Dyon]

4 hours ago, sniffton said:

Is there a way to use this with privoxy?

magneticod does not support proxies. Hence the Docker does also not support this.

I do recommend routing the Docker through another Docker that suppots a VPN. Spaceinvader One has an amazing video about how to do this right here: 

 

[sniffton]

Thank you!!

[Dyon]

9 minutes ago, storagehound said:

Hey, Nord.  I'm giving this a go.  I am a Jackett user and would love to compare the experience.   I am using the VPN ask suggested.  I can see a ~### number on the bottom left of the magnetico page that is slowly incrementing as I refresh so I assume it is working find and building it's database.  I'm upp to ~662 at the moment.

 

Question:  You suggested that frakman1 lower his MAGNETICOD_NEIGHBORS to 100 to troubleshoot his issues.   I am curious how that works.  Does that mean you can only have a maximum of 100 or does it mean that the data is collected in increments of 100 as opposed to hitting the pipes 1000 at a time.   Apologies if it's a silly question.

 

Thank you

That's a good question, however, how Magneticod exactly works is beyond my knowledge. 

 

From my own testing I noticed that a higher neighbor count increases the amount of torrents found per hour, and lower numbers resulted in lower results, so it seems logicial that it has to do with to how many nodes it connect. However, when running `netstat` on my machine to check the amount of connections I didn't see a significant change in open connections.

magnetico has, in my opinion, no real good documentation about this. The GitHub Issues page of magnetico often gets replies from people with more knowledge about it. If you wish to know how it works you can ask there 😄

[storagehound]

1 minute ago, Nord said:

That's a good question, however, how Magneticod exactly works is beyond my knowledge. 

 

From my own testing I noticed that a higher neighbor count increases the amount of torrents found per hour, and lower numbers resulted in lower results, so it seems logicial that it has to do with to how many nodes it connect. However, when running `netstat` on my machine to check the amount of connections I didn't see a significant change in open connections.

magnetico has, in my opinion, no real good documentation about this. The GitHub Issues page of magnetico often gets replies from people with more knowledge about it. If you wish to know how it works you can ask there 😄

I agree with you about their site.  I did try reading that first before I asked.  I had deleted my question ( I began to think it was a bad one) but I am glad you had already been responding to it.  Interesting info about non significant changes in connections.

[RustyMachine]

I noticed some of you were recommending to use a VPN is this just because your guys ISP is blocking the traffic? or is it recommended because of the possibilities of DMCA takedown notices. I am not downloading any of torrents on my network though so should I be fine, right? As magnetico is just a peer discovery for torrents correct? Also would port forwarding magneticod allow me to discover more torrents? As I would be more connectible?

[Dyon]

6 hours ago, RustyMachine said:

I noticed some of you were recommending to use a VPN is this just because your guys ISP is blocking the traffic? or is it recommended because of the possibilities of DMCA takedown notices. I am not downloading any of torrents on my network though so should I be fine, right? As magnetico is just a peer discovery for torrents correct? Also would port forwarding magneticod allow me to discover more torrents? As I would be more connectible?

You indeed don't have too. magnetico(d) indexes a lot of illegal torrents, which, depending on your region, could cause potential trouble. You are indeed not downloading any torrents. It is indeed a peer discovery. Also, have a look at this warning from the creator of magnetico:

magnetico/README.md at master · boramalper/magnetico · GitHub

Another reason why I do like using a VPN is because if you don't, some weird torrents may show up as that you downloaded it on; https://iknowwhatyoudownload.com/

 

I haven't personally forwarded my ports, and set a static port in magnetico, so I can't tell you if that works better. 

[mihu]

Not sure since when, but it keeps crashing for me right after start. I have completely removed it and the template. This happens on a fresh install.

 

LOG:

2021-02-12 20:54:08.312940 [INFO] Changing the ownership of the database
ownership of '/root/.local/share/magneticod' retained as 99:users
----------- magneticow -----------
2021-02-12 20:54:08.347295 [INFO] Refreshing the credentials file
Adding password for user magnetico
2021-02-12 20:54:08.689373 [INFO] Setting magneticow address to 0.0.0.0
2021-02-12 20:54:08.719119 [INFO] Enabling verbosity for magneticow
----------- magneticod -----------
2021-02-12 20:54:08.749972 [INFO] Setting magneticod indexing address to 0.0.0.0
2021-02-12 20:54:08.780145 [INFO] Setting magneticod indexing port to 0
2021-02-12 20:54:08.811124 [INFO] Setting magneticod indexing interval to 1
2021-02-12 20:54:08.842619 [INFO] Setting magneticod maximum number of neighbors to 1000
2021-02-12 20:54:08.873005 [INFO] Setting magneticod maximum number of leeches to 50
2021-02-12 20:54:08.905086 [INFO] Enabling verbosity for magneticod
----------------------------------
2021-02-12 20:54:08.947026 [INFO] Starting magneticow...
2021-02-12 20:54:08.997816 [INFO] Log file /opt/magnetico/Logs/magneticow.txt doesn't exist. Creating it...
2021-02-12 20:54:09.031738 [INFO] Logfile created. Logging to /opt/magnetico/Logs/magneticow.txt

magnetico.txt contents:
/opt/magnetico/magneticow: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /opt/magnetico/magneticow)

 

[Dyon]

6 minutes ago, mihu said:

Not sure since when, but it keeps crashing for me right after start. I have completely removed it and the template. This happens on a fresh install.

 

LOG:

2021-02-12 20:54:08.312940 [INFO] Changing the ownership of the database
ownership of '/root/.local/share/magneticod' retained as 99:users
----------- magneticow -----------
2021-02-12 20:54:08.347295 [INFO] Refreshing the credentials file
Adding password for user magnetico
2021-02-12 20:54:08.689373 [INFO] Setting magneticow address to 0.0.0.0
2021-02-12 20:54:08.719119 [INFO] Enabling verbosity for magneticow
----------- magneticod -----------
2021-02-12 20:54:08.749972 [INFO] Setting magneticod indexing address to 0.0.0.0
2021-02-12 20:54:08.780145 [INFO] Setting magneticod indexing port to 0
2021-02-12 20:54:08.811124 [INFO] Setting magneticod indexing interval to 1
2021-02-12 20:54:08.842619 [INFO] Setting magneticod maximum number of neighbors to 1000
2021-02-12 20:54:08.873005 [INFO] Setting magneticod maximum number of leeches to 50
2021-02-12 20:54:08.905086 [INFO] Enabling verbosity for magneticod
----------------------------------
2021-02-12 20:54:08.947026 [INFO] Starting magneticow...
2021-02-12 20:54:08.997816 [INFO] Log file /opt/magnetico/Logs/magneticow.txt doesn't exist. Creating it...
2021-02-12 20:54:09.031738 [INFO] Logfile created. Logging to /opt/magnetico/Logs/magneticow.txt

magnetico.txt contents:
/opt/magnetico/magneticow: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /opt/magnetico/magneticow)

 

 

That's odd. I haven't changed anything about the container for a long time and still works fine for me.  

I assume you just run Unraid, right?

 

Edit: You are right, I am getting the same error after a clean install. I will look into this asap.

Edited February 12, 2021 by Dyon

[Dyon]

@mihu I have updated to container and it seems to work fine now. Please update the container in Unraid 😁

What I've changed; switched the Docker from being based on debian:10-slim to debian:bullseye-slim (Bullseye is Debian 11, but still actually in beta) and changed a minor thing in the run script.

Since Debian Bullseye is beta, this will hopefully be a temporary solution for now.

I was unable to find out why it stopped working in Debian 10 😔

[mihu]

@Dyon Thanks everything works fine now.

[brainbudt]

is it possible to use it with sonarr and radarr? or do you recommend it ?

[SLNetworks]

What gives with the container? I start it, it stops.. cycle repeats.

 

Edit: Username must be lowercase apparently.

Edited April 7, 2023 by SLNetworks
Solution

[Dyon]

21 minutes ago, SLNetworks said:

What gives with the container? I start it, it stops.. cycle repeats.

 

Edit: Username must be lowercase apparently.

 

Oh, if username must be lowercase, I can make a check for it

[SLNetworks]

That would be a good move. There's nothing specifying so in the template either.

[Dyon]

2 hours ago, somebody said:

Maybe switch for https://github.com/bitmagnet-io/bitmagnet

 

https://news.ycombinator.com/item?id=37777347

Looks interesting, whenever I have time I will look into it