April 2, 20197 yr Application Name: elasticsearch Application Site: https://www.elastic.co/ Docker Hub: https://hub.docker.com/_/elasticsearch Template Repo: https://github.com/FoxxMD/unraid-docker-templates Overview Elasticsearch is an open-source, RESTful, distributed search and analytics engine built on Apache Lucene. Since its release in 2010, Elasticsearch has quickly become the most popular search engine, and is commonly used for log analytics, full-text search, security intelligence, business analytics, and operational intelligence use cases. This template defaults to Elasticsearch 6.6.2. This can easily be changed by modifying the Repository field (in Advanced View) on the template to use any available tags from dockerhub. Note: I have not tested any alpine variants. How To Use You must follow these steps if using Elasticsearch 5 or above -- vm.max_map_count must be increased or this container will not work Install CA User Scripts from Community Apps Create a new script named vm.max_map_count with this contents: #!/bin/bash sysctl -w vm.max_map_count=262144 3. Set script schedule to At Startup of Array 4. Run the script now to affect changes General Instructions (all versions) Check the default exposed ports and volume mappings in the template to ensure no conflicts. Happy searching! Acknowledgements Big thanks to @Jclendineng for instructions on how to modify max_map_count for elasticsearch Thanks to @OFark for additional arguments needed to get elasticsearch running Edited April 2, 20197 yr by FoxxMD
April 11, 20197 yr Thank you very much for the docker. Works nicely. I have a request if possible. Would you please install "ingest-attachment" plugin in the docker. Nextcloud needs it for indexing. It can be added by /bin/elasticsearch-plugin install ingest-attachment
April 12, 20197 yr Author I'm sorry but I can't help with that. This is the official docker container by elasticsearch, not mine. Additionally I could not find that executable anywhere in the container.
May 1, 20197 yr The correct command inside the docker container would be /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch ingest-attachment ... is there an easy possibility in unRAID to run this command after the Docker container has started?
May 3, 20197 yr Author On 5/1/2019 at 2:26 PM, lousek said: The correct command inside the docker container would be /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch ingest-attachment ... is there an easy possibility in unRAID to run this command after the Docker container has started? I don't know about automatically running it but you could put a script in CA User Scripts that makes it a one-click operation after the container is running. I haven't test this but it should be close to working: #!/bin/bash # Return name of docker container with elasticsearch in it -- assuming only one container for elasticsearch is present con="$(docker ps --format "{{.Names}}" | grep -i elasticsearch)" # execute command inside container docker exec -i "$con" /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch ingest-attachment Edited May 3, 20197 yr by FoxxMD
May 5, 20197 yr On 4/2/2019 at 1:57 PM, FoxxMD said: Application Name: elasticsearch Application Site: https://www.elastic.co/ Docker Hub: https://hub.docker.com/_/elasticsearch Template Repo: https://github.com/FoxxMD/unraid-docker-templates Overview Elasticsearch is an open-source, RESTful, distributed search and analytics engine built on Apache Lucene. Since its release in 2010, Elasticsearch has quickly become the most popular search engine, and is commonly used for log analytics, full-text search, security intelligence, business analytics, and operational intelligence use cases. This template defaults to Elasticsearch 6.6.2. This can easily be changed by modifying the Repository field (in Advanced View) on the template to use any available tags from dockerhub. Note: I have not tested any alpine variants. How To Use You must follow these steps if using Elasticsearch 5 or above -- vm.max_map_count must be increased or this container will not work Install CA User Scripts from Community Apps Create a new script named vm.max_map_count with this contents: #!/bin/bash sysctl -w vm.max_map_count=262144 3. Set script schedule to At Startup of Array 4. Run the script now to affect changes General Instructions (all versions) Check the default exposed ports and volume mappings in the template to ensure no conflicts. Happy searching! Acknowledgements Big thanks to @Jclendineng for instructions on how to modify max_map_count for elasticsearch Thanks to @OFark for additional arguments needed to get elasticsearch running Hey, thanks for tips and directions. I'm running into an issue that I'm not sure how to proceed since this docker thing's new to me. After following your directions and creating the script with the command in it - all I get is "/tmp/user.scripts/tmpScripts/vm.max_map_count/script: line 2: $'sysctl\357\273\277': command not found". Any hints?
May 8, 20197 yr I have been trying to setup nextcloud fulltextsearch using elasticsearch on unraid but I can not get the search to return any results. Where should I look or what could I be missing in the configuration? root@f59188334fbf:/config/www/nextcloud# sudo -u abc ./occ files:scan userid --all Starting scan for user 1 out of 7 (admin) Starting scan for user 2 out of 7 (admin2) Starting scan for user 3 out of 7 (Ben) Starting scan for user 4 out of 7 (Bryce) Starting scan for user 5 out of 7 (Fritz) Starting scan for user 6 out of 7 (Sam) Starting scan for user 7 out of 7 (Steph) +---------+-------+--------------+ | Folders | Files | Elapsed time | +---------+-------+--------------+ | 30 | 127 | 00:00:02 | +---------+-------+--------------+
May 19, 20197 yr When I go to the webgui for this, all I get is the below - should I be doing something else ? : { "name" : "HQdTNvT", "cluster_name" : "docker-cluster", "cluster_uuid" : "0jYRnGHBTimntwyla4MEIg", "version" : { "number" : "6.6.2", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "3bd3e59", "build_date" : "2019-03-06T15:16:26.864148Z", "build_snapshot" : false, "lucene_version" : "7.6.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }
May 20, 20197 yr Thanks @FoxxMD...... I guess I need to know how to use it then ? I was hoping I would go to a config page, add my areas I want indexes, type of files etc, then have a search page of some type.
May 21, 20197 yr Author @vw-kombi elasticsearch is just a backend service. It does not provide a UI but rather is used by other applications to query for data. When you are visiting the HTTP port you are making a simple API request and ES sends a response in the same format it would send all other data in -- so the text you got back in your first post is basically a sample of how all data returned from ES will look. There are a few web-based frontends for querying and viewing this data: https://www.elastic.co/guide/en/sense/current/introduction.html https://github.com/appbaseio/dejavu But from your description of what you want I have a feeling that's not what you are looking for. I did find this tutorial for a simple full-text search https://github.com/triestpa/Guttenberg-Search but you would have to write a new Dockerfile for it and publish it to DockerHub to make it useable on unraid.
May 21, 20197 yr Hmmmm - OK - thanks for the info. Im not very technical on dockers beyond installing from unraid community apps / docker hub. I will delete this one and keep looking.
August 22, 20196 yr How would I come about getting the rest of the ELK stack on Unraid? I have taken the official docker images of Kibana and Logstash from the docker hub and installed them, however I need to know of filemappings, portmappings, variables and such. I don’t really know anything about docker, so I was hoping that you had made those available on Unraid as well, but unfortunately no - and noone else has done it, it seems. Thanks, /Klaus
August 26, 20196 yr Author @klausagnoletti I'm sorry but I can't help you with your specific issues, I don't run an ELK stack and don't have experience with logstash/kibana. You probably aren't finding what you're looking for because: 1. ELK is an orchestration of services. Unraid lets you run individual dockers (docker-compose is not available) so an ELK stack can't be easily packaged up and deployed as "one docker" for unraid. 2. Unraid's apps are basically just templates/configuration presets for docker images from dockerhub. There is nothing special about docker images run on unraid other than the app creator can provide some convenience to other unraid users. These templates don't provide any extra functionality not already found in the docker image from dockerhub. Having said that you can easily run any docker image from dockerhub, as you have discovered. I suspect your issue is an unfamiliarity with the ELK stack, not docker -- as docker is usually pretty transparent. If you can setup an ELK stack in a regular environment you shouldn't have much trouble doing it with docker containers. Some tips that might help you though with template configuration for docker containers in unraid: variable - an environmental variable provided to the docker container. Elasticsearch requires the variable discovery.type to be provided so you would created variable named discovery.type and put its value to single-node directory/folder mapping -- provides mapping from a local directory (in your array) to a directory of your choosing in the docker container. If the ELK stack requires shared files between the services you could setup a folder /mnt/user/appdata/elk that is mapped to all three dockers. port - a local port (on the host machine) mapped to a port in the docker container. If need to point docker container A to docker container B you would give it the host IP + local (mapped) port of container B. EX elasticsearch exposes port 9200. port mapping could be 9200 (docker) -> 8200 (local). If kibana needed to access elasticsearch you would map a variable elastic_endpoint to 192.168.0.1:8200 (or whatever your host IP is) Edited August 26, 20196 yr by FoxxMD
August 26, 20196 yr 5 hours ago, FoxxMD said: Unraid lets you run individual dockers (docker-compose is not available) Docker compose is available in nerdpack. https://forums.unraid.net/topic/35866-unraid-6-nerdpack-cli-tools-iftop-iotop-screen-kbd-etc/?do=findComment&comment=653719
September 24, 20196 yr On 5/8/2019 at 2:08 PM, pepper said: I have been trying to setup nextcloud fulltextsearch using elasticsearch on unraid but I can not get the search to return any results. Where should I look or what could I be missing in the configuration? root@f59188334fbf:/config/www/nextcloud# sudo -u abc ./occ files:scan userid --all Starting scan for user 1 out of 7 (admin) Starting scan for user 2 out of 7 (admin2) Starting scan for user 3 out of 7 (Ben) Starting scan for user 4 out of 7 (Bryce) Starting scan for user 5 out of 7 (Fritz) Starting scan for user 6 out of 7 (Sam) Starting scan for user 7 out of 7 (Steph) +---------+-------+--------------+ | Folders | Files | Elapsed time | +---------+-------+--------------+ | 30 | 127 | 00:00:02 | +---------+-------+--------------+ Did you manage to get this working? I installed the nextcloud and elasticsearch docker. Installed the nescessary apps: Full text search Full text search - Elasticsearch Platform Full text search - Files Configured nextcloud gui to use elasticsearch for full text search Scheduled the script at docker start: # Return name of docker container with elasticsearch in it -- assuming only one container for elasticsearch is present con="$(docker ps --format "{{.Names}}" | grep -i elasticsearch)" # execute command inside container docker exec -i "$con" /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch ingest-attachment But... I guess this is not enough. I do not get results. Does anyone have the necessary step by step to complete this task?
February 14, 20206 yr On 9/24/2019 at 3:12 AM, jsspanjer said: Did you manage to get this working? I installed the nextcloud and elasticsearch docker. Installed the nescessary apps: Full text search Full text search - Elasticsearch Platform Full text search - Files Configured nextcloud gui to use elasticsearch for full text search Scheduled the script at docker start: # Return name of docker container with elasticsearch in it -- assuming only one container for elasticsearch is present con="$(docker ps --format "{{.Names}}" | grep -i elasticsearch)" # execute command inside container docker exec -i "$con" /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch ingest-attachment But... I guess this is not enough. I do not get results. Does anyone have the necessary step by step to complete this task? It appears to be an issue with this version of elasticsearch. I was finally able to get it working after switching to 7.6.0.
July 8, 20205 yr Hello. I want to share, because I myself was looking for Old. I have NextCloud production, nextcloud repository: production-apache. Version 17. 1. I entered the script from the topic in the terminal sysctl -w vm.max_map_count=262144 2. I installed 4 plug-ins with a magnifier in the NextCloud, on the NextCloud web. 3. I installed the plugin for NextCloud. ingest-attachment Entered in the terminal: docker exec -it elasticsearch bash /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment 4. Then I made the settings in NextCloud, as in the photo. 5. Then started indexing. In terminal: docker exec --user www-data nextcloud php occ fulltextsearch:index ______________ I hope this helps someone. Edited July 9, 20205 yr by muwahhid
July 28, 20205 yr On 7/8/2020 at 11:16 PM, muwahhid said: Hello. I want to share, because I myself was looking for Old. I have NextCloud production, nextcloud repository: production-apache. Version 17. 1. I entered the script from the topic in the terminal sysctl -w vm.max_map_count=262144 2. I installed 4 plug-ins with a magnifier in the NextCloud, on the NextCloud web. 3. I installed the plugin for NextCloud. ingest-attachment Entered in the terminal: docker exec -it elasticsearch bash /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-attachment 4. Then I made the settings in NextCloud, as in the photo. 5. Then started indexing. In terminal: docker exec --user www-data nextcloud php occ fulltextsearch:index ______________ I hope this helps someone. Thank you for your help is worked so far for me exept of the last step when i want to start with the indexing it give me de following error root@Homeserver:~# docker exec --user www-data nextcloud php occ fulltextsearch:index unable to find user www-data: no matching entries in passwd file do you haven an Idea what i have to do to solve this error?
July 28, 20205 yr 2 minutes ago, Wingold said: Thank you for your help is worked so far for me exept of the last step when i want to start with the indexing it give me de following error root@Homeserver:~# docker exec --user www-data nextcloud php occ fulltextsearch:index unable to find user www-data: no matching entries in passwd file do you haven an Idea what i have to do to solve this error? I have permissions to the "appdata/nextcloud" folder of the www-data user. You most likely have another user, so the error.
July 28, 20205 yr The www-data user is the default in Nextcloud, if you have another user, you can return everything this way. open terminal on docker nextcloud, and write: chown -R www-data:www-data /var/www/html/data
September 15, 20205 yr For those that are stuck like me on the ingest-attachment plugin issue: You need to stop the elasticsearch docker and restart it after you have executed the command to install the plugin so it gets loaded into elasticsearch. Here my steps: 1.) get elasticsearch docker (7.9.1 works) do a clean install (delete old elasticsearch in /mnt/user/appdata/) 2.) Download the full text search packages in nextcloud app store (at least 3 packages) 3.) Configure your Nextcloud search platform to "Elasticsearch" and address of Servlet to: http://YOUR_IP:9200/ It needs to be configured to the port of the REST API 4.) Install the plugin for elasticsearch, by either opening a console inside the elasticsearch docker and type /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch ingest-attachment OR through the User scripts Unraid plugin as stated above. 5.) Restart the elasticsearch container 6.) Test everything by opening a new shell in the nextcloud container then navigate to the occ directory (/var/www/html) and type ./occ fulltextsearch:test If everything is ok, then you can continue with the index: ./occ fulltextsearch:index Edited September 15, 20205 yr by rob_robot
October 24, 20205 yr I've been using Elasticsearch 7.9.1. On a docker image reinstall, I'm failing to get this to work again. I've moved to 7.9.3 in case it was a bug fix. I have a heap size issue. The pull down shows completed successfully, see the environment variable: -e 'ES_JAVA_OPTS'='-Xms4g -Xmx4g' which I entered. Later in it shows this as well: -e "ES_JAVA_OPTS"="-Xms512m -Xmx512m" When I do an "echo ES_JAVA_OPTS" inside the container, it shows that only 512m memory is being used. It appears as if the 512m is taking precedent. I've also set the minimum and maximum heap size in the jvm.options file. It doesn't seem to make a difference. Below is the container pull: Quote root@localhost:# /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker run -d --name='elasticsearch2' --net='bridge' -e TZ="Australia/Brisbane" -e HOST_OS="Unraid" -e 'discovery.type'='single-node' -e 'ES_JAVA_OPTS'='-Xms4g -Xmx4g' -p '9200:9200/tcp' -p '9300:9300/tcp' -v '/mnt/user/appdata/elasticsearch/data':'/usr/share/elasticsearch/data':'rw' -e "ES_JAVA_OPTS"="-Xms512m -Xmx512m" --ulimit nofile=262144:262144 'elasticsearch:7.9.3' 82d34c8772e2291a617e86a9c4f21dd227a392c3d02f1522384e7b02332566fa The command finished successfully! I've tried this for most of the day. I've not had this problem before, I urgently need to get this up and running. Can anybody advise?
October 24, 20205 yr Quote {"type": "server", "timestamp": "2020-10-24T16:07:56,878+10:00", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "82d34c8772e2", "message": "JVM home [/usr/share/elasticsearch/jdk]" } {"type": "server", "timestamp": "2020-10-24T16:07:56,879+10:00", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "82d34c8772e2", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms4g, -Xmx4g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-10094349604196076055, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
October 25, 20205 yr Okay, So for those following after me. I'm not very experienced with docker, but I pulled down the container manually and by doing so changed the environment variables, changing the heap size to 2g. Quote docker run -d --name='elasticsearch' --net='bridge' -e TZ="Australia/Brisbane" -e HOST_OS="Unraid" -e 'discovery.type'='single-node' -e 'ES_JAVA_OPTS'='-Xms2g -Xmx2g' -p '9200:9200/tcp' -p '9300:9300/tcp' -v '/mnt/user/appdata/elasticsearch/data':'/usr/share/elasticsearch/data':'rw' -e "ES_JAVA_OPTS"="-Xms2g -Xmx2g" --ulimit nofile=262144:262144 'elasticsearch:7.9.3' I changed the second set of ES_JAVA_OPTS (in red) from Xms512mb -Xmx512mb to 2g instead. However, I've just noticed that there are two ES_JAVA_OPTS entries, the other one I've marked in yellow. This was a copy and paste from the original container install. No matter what I did I couldn't get the heap size to change from 512mb to any other size by setting ES_JAVA_OPTS as a variable in the CA settings for the container or via jvm.options inside the container itself. Anybody shed light on this? I would like to just pull the container down without all this phaff.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.