OFark Posted December 20, 2018 Share Posted December 20, 2018 So Diskover you should be able to get from CA, Elastic search requires a set version, follow the instructions on the project page. I also mentioned how I installed it on Page 1. If Diskover isn't working, throwing errors etc. Uninstall it, completely, then reinstall it. Use the default settings. As for Gource, is a Graphical Source for GIT, meant to show the development of a project over time. However, you can load data into it via file or via the STDIN pipe. I've not tried this with Diskover yet. Whilst it was fun to see how our website had developed over the years I was using it to watch users arrive on, and crawl the website. Currently, Diskover won't work for me, I THINK it's seeing the wrong version of REDIS, though a compatible version is installed. Quote Link to comment
OFark Posted December 22, 2018 Share Posted December 22, 2018 Right, so I've painstakingly created a container template for ElasticSearch v5: Add this to your Docker Template repositories: https://github.com/OFark/docker-templates/ Then you'll see ElasticSearch5 in the list. This is my first container, it's taken me an hour or two, because I had no idea what I was doing. I may have done something wrong. How can I/we get it added to CA? Quote Link to comment
Wuast94 Posted December 24, 2018 Share Posted December 24, 2018 On 12/22/2018 at 4:05 PM, OFark said: Right, so I've painstakingly created a container template for ElasticSearch v5: Add this to your Docker Template repositories: https://github.com/OFark/docker-templates/ Then you'll see ElasticSearch5 in the list. This is my first container, it's taken me an hour or two, because I had no idea what I was doing. I may have done something wrong. How can I/we get it added to CA? ElasticSeearch now starts. but diskover get no data. Error massage in webui: No data in Elasticsearch index! After a crawl starts it can take up to 30 sec (refresh time) for index to be updated... Reload. Quote Link to comment
OFark Posted December 24, 2018 Share Posted December 24, 2018 (edited) 2 hours ago, Wuast94 said: ElasticSeearch now starts. but diskover get no data. Error massage in webui: No data in Elasticsearch index! After a crawl starts it can take up to 30 sec (refresh time) for index to be updated... Reload. Check the logs, you're probably getting an error about redis version. Since rc25 of Diskover rc24 and rc25 have been broken of the linuxserver version. See my post on the GitHub issue. What you'll need to do is regress diskover to v1.5.0-rc24-ls8 (replace the repository with "linuxserver/diskover:v1.5.0-rc24-ls8" in the unRAID container config) until a fix is released. Which would normally be Friday, but what with the holidays this may take another week. Edited December 24, 2018 by OFark Quote Link to comment
Wuast94 Posted December 24, 2018 Share Posted December 24, 2018 24 minutes ago, OFark said: Check the logs, you're probably getting an error about redis version. Since rc25 of Diskover rc24 and rc25 have been broken of the linuxserver version. See my post on the GitHub issue. What you'll need to do is regress diskover to v1.5.0-rc24-ls8 (replace the repository with "linuxserver/diskover:v1.5.0-rc24-ls8" in the unRAID container config) until a fix is released. Which would normally be Friday, but what with the holidays this may take another week. ok works fine now thanks for your help did anyone find out how i can grouce kive data to my windows machine ? there is only one command i found for that but i don´t know if i can ssh in my docker container... becouse i must run this command in my windows machine to pipe it out i think ssh user@diskovervm '( python diskover.py --gourcert -i diskover-indexname )' | sh diskover-gource.sh -r Quote Link to comment
shirosai Posted December 26, 2018 Share Posted December 26, 2018 It looks like there is a new built v1.5.0-rc25-ls10 image on docker hub. Please try it out and reply back if working okay. Thank you. Quote Link to comment
shirosai Posted December 26, 2018 Share Posted December 26, 2018 On 12/25/2018 at 1:12 AM, Wuast94 said: ok works fine now thanks for your help did anyone find out how i can grouce kive data to my windows machine ? there is only one command i found for that but i don´t know if i can ssh in my docker container... becouse i must run this command in my windows machine to pipe it out i think ssh user@diskovervm '( python diskover.py --gourcert -i diskover-indexname )' | sh diskover-gource.sh -r @Wuast94 https://github.com/shirosaidev/diskover/wiki/Gource-visualization-support has some information at the bottom about getting gource data out of diskover onto another machine. You will need ssh access to the diskover docker container. Windows needs to be running a bash shell with gource installed and diskover github files to run that diskover-gource.sh script on the windows machine. The | (pipe) is redirecting the ssh output of the diskover.py python command (you may need to set this and python to absolute paths) running on the diskover docker container to your windows machine. Quote Link to comment
Wuast94 Posted December 26, 2018 Share Posted December 26, 2018 4 hours ago, shirosai said: @Wuast94 https://github.com/shirosaidev/diskover/wiki/Gource-visualization-support has some information at the bottom about getting gource data out of diskover onto another machine. You will need ssh access to the diskover docker container. Windows needs to be running a bash shell with gource installed and diskover github files to run that diskover-gource.sh script on the windows machine. The | (pipe) is redirecting the ssh output of the diskover.py python command (you may need to set this and python to absolute paths) running on the diskover docker container to your windows machine. I can test it. But if I understand it right I must run this command on the windows machine not in the docker container. And to connect from windows to the container direktly I need "user@server" and I didn't find a way for this for docker container. Or am I wrong? Quote Link to comment
saarg Posted December 26, 2018 Share Posted December 26, 2018 There is no SSH server in the container. Quote Link to comment
Wuast94 Posted December 26, 2018 Share Posted December 26, 2018 And that's the problem were I don't know how to get it to work. What shiroasai says on the diskover github page is the way of diskover runs nativ on hw. But we run it in a container..So it don't work Quote Link to comment
OFark Posted December 26, 2018 Share Posted December 26, 2018 11 hours ago, shirosai said: It looks like there is a new built v1.5.0-rc25-ls10 image on docker hub. Please try it out and reply back if working okay. Thank you. So if I force it to that version tag the REDIS version issue has gone but now I have a new problem: starting crawler with following options: --finddupes -i diskover- Traceback (most recent call last): File "/usr/lib/python3.6/configparser.py", line 1138, in _unify_values sectiondict = self._sections[section] KeyError: 'ownersgroups' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./diskover.py", line 1851, in <module> config, configfile = load_config() File "./diskover.py", line 172, in load_config configsettings['ownersgroups_uidgidonly'] = config.get('ownersgroups', 'uidgidonly').lower() File "/usr/lib/python3.6/configparser.py", line 781, in get d = self._unify_values(section, vars) File "/usr/lib/python3.6/configparser.py", line 1141, in _unify_values raise NoSectionError(section) configparser.NoSectionError: No section: 'ownersgroups' Quote Link to comment
shirosai Posted December 26, 2018 Share Posted December 26, 2018 4 hours ago, OFark said: So if I force it to that version tag the REDIS version issue has gone but now I have a new problem: starting crawler with following options: --finddupes -i diskover- Traceback (most recent call last): File "/usr/lib/python3.6/configparser.py", line 1138, in _unify_values sectiondict = self._sections[section] KeyError: 'ownersgroups' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./diskover.py", line 1851, in <module> config, configfile = load_config() File "./diskover.py", line 172, in load_config configsettings['ownersgroups_uidgidonly'] = config.get('ownersgroups', 'uidgidonly').lower() File "/usr/lib/python3.6/configparser.py", line 781, in get d = self._unify_values(section, vars) File "/usr/lib/python3.6/configparser.py", line 1141, in _unify_values raise NoSectionError(section) configparser.NoSectionError: No section: 'ownersgroups' copy over any missing sections from diskover.cfg.sample to diskover.cfg Quote Link to comment
shirosai Posted December 26, 2018 Share Posted December 26, 2018 (edited) 11 hours ago, Wuast94 said: I can test it. But if I understand it right I must run this command on the windows machine not in the docker container. And to connect from windows to the container direktly I need "user@server" and I didn't find a way for this for docker container. Or am I wrong? Try what I wrote at the bottom and let me know if it works, you can also just install python2 or python3 in your windows host with gource and install the elasticsearch 5 python module using pip, then you don't need to ssh or run anything in docker container and just pull the es index data right from the windows host that has diskover github files on it. https://github.com/shirosaidev/diskover/wiki/Gource-visualization-support Edited December 26, 2018 by shirosai Quote Link to comment
OFark Posted December 28, 2018 Share Posted December 28, 2018 Yes, copying over the missing sections from the sample config fixed this, why aren't they in the container by default? Quote Link to comment
shirosai Posted December 29, 2018 Share Posted December 29, 2018 7 hours ago, OFark said: Yes, copying over the missing sections from the sample config fixed this, why aren't they in the container by default? I've sent a message to @hackerman to add them in. I think the config in the lsio github for diskover is old and it uses that instead of getting new versions from diskover github when building. Quote Link to comment
OFark Posted December 30, 2018 Share Posted December 30, 2018 Has anyone got --finddupes working? If I put it in either DISKOVER_OPTS or WORKER_OPTS then Diskover fails to create any workers and throws errors in the dispatcher.log. I'm not even sure if either of those is the right place, but I don't know where else to try. Quote Link to comment
shirosai Posted January 3, 2019 Share Posted January 3, 2019 On 12/30/2018 at 8:53 PM, OFark said: Has anyone got --finddupes working? If I put it in either DISKOVER_OPTS or WORKER_OPTS then Diskover fails to create any workers and throws errors in the dispatcher.log. I'm not even sure if either of those is the right place, but I don't know where else to try. Have you tried to bash into the container and running it from there? DISKOVER_OPTS would be the right place to to put it at container startup but you also need to specify the index name you want to run finddupes on using -i <index>. I think there is an issue with DISKOVER_OPTS not getting set in the diskover container in Unraid, can anyone verify this? Try adding -w 20 to WORKER_OPTS and see if 20 bots start up. Quote Link to comment
OFark Posted January 3, 2019 Share Posted January 3, 2019 (edited) Ahh well, if I put in --finddupes into the DISKOVER_OPTS the log shows an error that says something about not finding the index, so that makes sense. it appears it always adds the -i diskover- option anyway Several options from the config appear to be missing in this version, but not on GitHub (redis_retry_on_timeout for example) in diskover_connection.py If I set that and the next option (session keepalive) to True then it starts up but bots are starting up but none are surviving. There are never any workers showing. I've posted in Github under my previous issue. Going back to v1.5.0-rc24-ls8 and everything is working again. Edited January 3, 2019 by OFark Quote Link to comment
shirosai Posted January 6, 2019 Share Posted January 6, 2019 (edited) On 1/3/2019 at 5:35 PM, OFark said: Ahh well, if I put in --finddupes into the DISKOVER_OPTS the log shows an error that says something about not finding the index, so that makes sense. it appears it always adds the -i diskover- option anyway Several options from the config appear to be missing in this version, but not on GitHub (redis_retry_on_timeout for example) in diskover_connection.py If I set that and the next option (session keepalive) to True then it starts up but bots are starting up but none are surviving. There are never any workers showing. I've posted in Github under my previous issue. Going back to v1.5.0-rc24-ls8 and everything is working again. Try v1.5.0-rc28-ls14 when released later today or tomorrow and let me know if all working good. I pushed a pr into the lsio container to update the default cfg with the container and @hackerman accepted it so that will be in the new container builds now. Edited January 24, 2019 by shirosai Quote Link to comment
oliver Posted January 26, 2019 Share Posted January 26, 2019 (edited) This is a very poorly put together container. Diskover in general just seems like a poorly mashed together jumble of dependancies. Why not make a single docker with everything embedded in it? Edited January 26, 2019 by oliver Quote Link to comment
saarg Posted January 26, 2019 Share Posted January 26, 2019 6 hours ago, oliver said: This is a very poorly put together container. Diskover in general just seems like a poorly mashed together jumble of dependancies. Why not make a single docker with everything embedded in it? Why not make it yourself if you don't like it? 2 Quote Link to comment
oliver Posted January 26, 2019 Share Posted January 26, 2019 2 hours ago, saarg said: Why not make it yourself if you don't like it? That doesn't answer the question. Quote Link to comment
saarg Posted January 26, 2019 Share Posted January 26, 2019 16 minutes ago, oliver said: That doesn't answer the question. I thought it was a pretty clear answer. Quote Link to comment
CHBMB Posted January 26, 2019 Share Posted January 26, 2019 9 hours ago, oliver said: This is a very poorly put together container. Diskover in general just seems like a poorly mashed together jumble of dependancies. Why not make a single docker with everything embedded in it? Because that is not the intention of how docker containers are run. They're meant to be run as independent micro-services. We experimented with containers with multiple services in it before and it was a complete fuster-cluck. But as you're an expert critic of what makes a well put together docker container fork it and make your own. Quote Link to comment
falcor Posted January 29, 2019 Share Posted January 29, 2019 I was having no luck at all getting this to work, so I: installed docker-compose and git through the Nerd Pack SSHed into the UnRAID box Started tmux (optional but recommended) cd to my repos share (where I check out git crap) git clone https://github.com/shirosaidev/diskover-web.git cd diskover-web Edited the docker-compose.yml to change the port from 8080 to 9090 (I already have an 8080) docker-compose build docker-compose up -d Now, I can go to :9090 and see the web interface, which is MUCH further than I got before. Now I just have to figure out how to make it look nice, but honestly I'm just going to put it in Organizr anyway. I would highly recommend anyone else wanting to use this do the same or similar to the above. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.