[Support] Linuxserver.io - diskover


Recommended Posts

So Diskover you should be able to get from CA, Elastic search requires a set version, follow the instructions on the project page. I also mentioned how I installed it on Page 1. If Diskover isn't working, throwing errors etc. Uninstall it, completely, then reinstall it. Use the default settings. 

 

As for Gource, is a Graphical Source for GIT, meant to show the development of a project over time. However, you can load data into it via file or via the STDIN pipe. I've not tried this with Diskover yet. Whilst it was fun to see how our website had developed over the years I was using it to watch users arrive on, and crawl the website.

 

Currently, Diskover won't work for me, I THINK it's seeing the wrong version of REDIS, though a compatible version is installed.

Link to comment
On 12/22/2018 at 4:05 PM, OFark said:

Right, so I've painstakingly created a container template for ElasticSearch v5:

 

Add this to your Docker Template repositories:

https://github.com/OFark/docker-templates/

 

Then you'll see ElasticSearch5 in the list.

 

This is my first container, it's taken me an hour or two, because I had no idea what I was doing. I may have done something wrong. How can I/we get it added to CA?

ElasticSeearch now starts. 

but diskover get no data. 

Error massage in webui: 
 No data in Elasticsearch index! After a crawl starts it can take up to 30 sec (refresh time) for index to be updated... Reload.

Link to comment
2 hours ago, Wuast94 said:

ElasticSeearch now starts. 

but diskover get no data. 

Error massage in webui: 
 No data in Elasticsearch index! After a crawl starts it can take up to 30 sec (refresh time) for index to be updated... Reload.

Check the logs, you're probably getting an error about redis version. Since rc25 of Diskover rc24 and rc25 have been broken of the linuxserver version. See my post on the GitHub issue. What you'll need to do is regress diskover to v1.5.0-rc24-ls8 (replace the repository with "linuxserver/diskover:v1.5.0-rc24-ls8" in the unRAID container config) until a fix is released. Which would normally be Friday, but what with the holidays this may take another week.

Edited by OFark
Link to comment
24 minutes ago, OFark said:

Check the logs, you're probably getting an error about redis version. Since rc25 of Diskover rc24 and rc25 have been broken of the linuxserver version. See my post on the GitHub issue. What you'll need to do is regress diskover to v1.5.0-rc24-ls8 (replace the repository with "linuxserver/diskover:v1.5.0-rc24-ls8" in the unRAID container config) until a fix is released. Which would normally be Friday, but what with the holidays this may take another week.

ok works fine now :) thanks for your help :)

did anyone find out how i can grouce kive data to my windows machine ? there is only one command i found for that but i don´t know if i can ssh in my docker container... becouse i must run this command in my windows machine to pipe it out i think

ssh user@diskovervm '( python diskover.py --gourcert -i diskover-indexname )' | sh diskover-gource.sh -r

 

Link to comment
On 12/25/2018 at 1:12 AM, Wuast94 said:

ok works fine now :) thanks for your help :)

did anyone find out how i can grouce kive data to my windows machine ? there is only one command i found for that but i don´t know if i can ssh in my docker container... becouse i must run this command in my windows machine to pipe it out i think


ssh user@diskovervm '( python diskover.py --gourcert -i diskover-indexname )' | sh diskover-gource.sh -r

 

@Wuast94 https://github.com/shirosaidev/diskover/wiki/Gource-visualization-support has some information at the bottom about getting gource data out of diskover onto another machine. You will need ssh access to the diskover docker container. Windows needs to be running a bash shell with gource installed and diskover github files to run that diskover-gource.sh script on the windows machine. The | (pipe) is redirecting the ssh output of the diskover.py python command (you may need to set this and python to absolute paths) running on the diskover docker container to your windows machine.

Link to comment
4 hours ago, shirosai said:

@Wuast94 https://github.com/shirosaidev/diskover/wiki/Gource-visualization-support has some information at the bottom about getting gource data out of diskover onto another machine. You will need ssh access to the diskover docker container. Windows needs to be running a bash shell with gource installed and diskover github files to run that diskover-gource.sh script on the windows machine. The | (pipe) is redirecting the ssh output of the diskover.py python command (you may need to set this and python to absolute paths) running on the diskover docker container to your windows machine.

I can test it. But if I understand it right I must run this command on the windows machine not in the docker container. And to connect from windows to the container direktly I need "user@server" and I didn't find a way for this for docker container. Or am I wrong?

Link to comment
11 hours ago, shirosai said:

It looks like there is a new built v1.5.0-rc25-ls10 image on docker hub. Please try it out and reply back if working okay. Thank you.

So if I force it to that version tag the REDIS version issue has gone but now I have a new problem:

starting crawler with following options: --finddupes -i diskover-
Traceback (most recent call last):
  File "/usr/lib/python3.6/configparser.py", line 1138, in _unify_values
    sectiondict = self._sections[section]
KeyError: 'ownersgroups'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./diskover.py", line 1851, in <module>
    config, configfile = load_config()
  File "./diskover.py", line 172, in load_config
    configsettings['ownersgroups_uidgidonly'] = config.get('ownersgroups', 'uidgidonly').lower()
  File "/usr/lib/python3.6/configparser.py", line 781, in get
    d = self._unify_values(section, vars)
  File "/usr/lib/python3.6/configparser.py", line 1141, in _unify_values
    raise NoSectionError(section)
configparser.NoSectionError: No section: 'ownersgroups'

 

Link to comment
4 hours ago, OFark said:

So if I force it to that version tag the REDIS version issue has gone but now I have a new problem:


starting crawler with following options: --finddupes -i diskover-
Traceback (most recent call last):
  File "/usr/lib/python3.6/configparser.py", line 1138, in _unify_values
    sectiondict = self._sections[section]
KeyError: 'ownersgroups'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./diskover.py", line 1851, in <module>
    config, configfile = load_config()
  File "./diskover.py", line 172, in load_config
    configsettings['ownersgroups_uidgidonly'] = config.get('ownersgroups', 'uidgidonly').lower()
  File "/usr/lib/python3.6/configparser.py", line 781, in get
    d = self._unify_values(section, vars)
  File "/usr/lib/python3.6/configparser.py", line 1141, in _unify_values
    raise NoSectionError(section)
configparser.NoSectionError: No section: 'ownersgroups'

 

copy over any missing sections from diskover.cfg.sample to diskover.cfg

Link to comment
11 hours ago, Wuast94 said:

I can test it. But if I understand it right I must run this command on the windows machine not in the docker container. And to connect from windows to the container direktly I need "user@server" and I didn't find a way for this for docker container. Or am I wrong?

Try what I wrote at the bottom and let me know if it works, you can also just install python2 or python3 in your windows host with gource and install the elasticsearch 5 python module using pip, then you don't need to ssh or run anything in docker container and just pull the es index data right from the windows host that has diskover github files on it.

https://github.com/shirosaidev/diskover/wiki/Gource-visualization-support

Edited by shirosai
Link to comment
On 12/30/2018 at 8:53 PM, OFark said:

Has anyone got --finddupes working? If I put it in either DISKOVER_OPTS or WORKER_OPTS then Diskover fails to create any workers and throws errors in the dispatcher.log. I'm not even sure if either of those is the right place, but I don't know where else to try.

Have you tried to bash into the container and running it from there? DISKOVER_OPTS would be the right place to to put it at container startup but you also need to specify the index name you want to run finddupes on using -i <index>. I think there is an issue with DISKOVER_OPTS not getting set in the diskover container in Unraid, can anyone verify this? Try adding -w 20 to WORKER_OPTS and see if 20 bots start up.

Link to comment

Ahh well, if I put in --finddupes into the DISKOVER_OPTS the log shows an error that says something about not finding the index, so that makes sense.

it appears it always adds the -i diskover- option anyway

 

Several options from the config appear to be missing in this version, but not on GitHub (redis_retry_on_timeout for example) in diskover_connection.py

 

If I set that and the next option (session keepalive) to True then it starts up but bots are starting up but none are surviving. There are never any workers showing. I've posted in Github under my previous issue.

 

Going back to v1.5.0-rc24-ls8 and everything is working again.

Edited by OFark
Link to comment
On 1/3/2019 at 5:35 PM, OFark said:

Ahh well, if I put in --finddupes into the DISKOVER_OPTS the log shows an error that says something about not finding the index, so that makes sense.

it appears it always adds the -i diskover- option anyway

 

Several options from the config appear to be missing in this version, but not on GitHub (redis_retry_on_timeout for example) in diskover_connection.py

 

If I set that and the next option (session keepalive) to True then it starts up but bots are starting up but none are surviving. There are never any workers showing. I've posted in Github under my previous issue.

 

Going back to v1.5.0-rc24-ls8 and everything is working again.

Try v1.5.0-rc28-ls14 when released later today or tomorrow and let me know if all working good. I pushed a pr into the lsio container to update the default cfg with the container and @hackerman accepted it so that will be in the new container builds now.

Edited by shirosai
Link to comment
  • 3 weeks later...
6 hours ago, oliver said:

This is a very poorly put together container.  Diskover in general just seems like a poorly mashed together jumble of dependancies.  

 

Why not make a single docker with everything embedded in it?  

 

Why not make it yourself if you don't like it? 

  • Like 2
Link to comment
9 hours ago, oliver said:

This is a very poorly put together container.  Diskover in general just seems like a poorly mashed together jumble of dependancies.  

 

Why not make a single docker with everything embedded in it?  

Because that is not the intention of how docker containers are run.  They're meant to be run as independent micro-services.  We experimented with containers with multiple services in it before and it was a complete fuster-cluck.

 

But as you're an expert critic of what makes a well put together docker container fork it and make your own.

Link to comment

I was having no luck at all getting this to work, so I:

  • installed docker-compose and git through the Nerd Pack
  • SSHed into the UnRAID box
  • Started tmux (optional but recommended)
  • cd to my repos share (where I check out git crap)
  • git clone https://github.com/shirosaidev/diskover-web.git

  • cd diskover-web

  • Edited the docker-compose.yml to change the port from 8080 to 9090 (I already have an 8080)

  • docker-compose build

  • docker-compose up -d

Now, I can go to :9090 and see the web interface, which is MUCH further than I got before. Now I just have to figure out how to make it look nice, but honestly I'm just going to put it in Organizr anyway.

 

I would highly recommend anyone else wanting to use this do the same or similar to the above.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.