Hi - I'm interested in using this docker image, but ran into a few issues trying to start it.

 

https://hub.docker.com/r/sebp/elk/

 

Has anyone else successfully implemented this docker? Has anyone tried installing three separate containers of Elasticsearch, Logstash, and Kibana?

 

Looking forward to the responses because having this log server would be awesome to complement Grafana!

 

 

EDIT: Changed the topic from "Request ELK Stack" to "Elastic Stack Setup", since Beats has been added to the stack.

Edited April 2, 20197 yr by surfshack66

Looks like Elasticsearch is failing to start.

 

Quote

ErrorWarningSystemArrayLogin

* Starting periodic command scheduler cron
...done.
* Starting Elasticsearch Server
...done.
waiting for Elasticsearch to be up (1/30)
waiting for Elasticsearch to be up (2/30)
waiting for Elasticsearch to be up (3/30)
waiting for Elasticsearch to be up (4/30)
waiting for Elasticsearch to be up (5/30)
waiting for Elasticsearch to be up (6/30)
waiting for Elasticsearch to be up (7/30)
waiting for Elasticsearch to be up (8/30)
waiting for Elasticsearch to be up (9/30)
waiting for Elasticsearch to be up (10/30)
waiting for Elasticsearch to be up (11/30)
waiting for Elasticsearch to be up (12/30)
waiting for Elasticsearch to be up (13/30)
waiting for Elasticsearch to be up (14/30)
waiting for Elasticsearch to be up (15/30)
waiting for Elasticsearch to be up (16/30)
waiting for Elasticsearch to be up (17/30)
waiting for Elasticsearch to be up (18/30)
waiting for Elasticsearch to be up (19/30)
waiting for Elasticsearch to be up (20/30)
waiting for Elasticsearch to be up (21/30)
waiting for Elasticsearch to be up (22/30)
waiting for Elasticsearch to be up (23/30)
waiting for Elasticsearch to be up (24/30)
waiting for Elasticsearch to be up (25/30)
waiting for Elasticsearch to be up (26/30)
waiting for Elasticsearch to be up (27/30)
waiting for Elasticsearch to be up (28/30)
waiting for Elasticsearch to be up (29/30)
waiting for Elasticsearch to be up (30/30)
Couln't start Elasticsearch. Exiting.
Elasticsearch log follows below.
[2018-09-13T09:37:55,954][INFO ][o.e.n.Node ] [] initializing ...
[2018-09-13T09:37:56,025][INFO ][o.e.e.NodeEnvironment ] [o7wlA7C] using [1] data paths, mounts [[/var/lib/elasticsearch (shfs)]], net usable_space [72gb], net total_space [238.4gb], types [fuse.shfs]
[2018-09-13T09:37:56,025][INFO ][o.e.e.NodeEnvironment ] [o7wlA7C] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-09-13T09:37:56,027][INFO ][o.e.n.Node ] [o7wlA7C] node name derived from node ID [o7wlA7CsSem_dAXjTL3sTA]; set [node.name] to override
[2018-09-13T09:37:56,027][INFO ][o.e.n.Node ] [o7wlA7C] version[6.4.0], pid[94], build[default/tar/595516e/2018-08-17T23:18:47.308994Z], OS[Linux/4.14.49-unRAID/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
[2018-09-13T09:37:56,027][INFO ][o.e.n.Node ] [o7wlA7C] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.2ABInCTu, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.enforce.bootstrap.checks=true, -Des.path.home=/opt/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2018-09-13T09:37:57,742][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [aggs-matrix-stats]
[2018-09-13T09:37:57,742][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [analysis-common]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [ingest-common]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [lang-expression]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [lang-mustache]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [lang-painless]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [mapper-extras]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [parent-join]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [percolator]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [rank-eval]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [reindex]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [repository-url]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [transport-netty4]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [tribe]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [x-pack-core]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [x-pack-deprecation]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [x-pack-graph]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [x-pack-logstash]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [x-pack-ml]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [x-pack-monitoring]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [x-pack-rollup]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [x-pack-security]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [x-pack-sql]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [x-pack-upgrade]
[2018-09-13T09:37:57,743][INFO ][o.e.p.PluginsService ] [o7wlA7C] loaded module [x-pack-watcher]
[2018-09-13T09:37:57,744][INFO ][o.e.p.PluginsService ] [o7wlA7C] no plugins loaded
[2018-09-13T09:38:01,189][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/121] [Main.cc@109] controller (64 bit): Version 6.4.0 (Build cf8246175efff5) Copyright (c) 2018 Elasticsearch BV
[2018-09-13T09:38:01,474][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-09-13T09:38:01,655][INFO ][o.e.d.DiscoveryModule ] [o7wlA7C] using discovery type [zen]
[2018-09-13T09:38:02,229][INFO ][o.e.n.Node ] [o7wlA7C] initialized
[2018-09-13T09:38:02,229][INFO ][o.e.n.Node ] [o7wlA7C] starting ...
[2018-09-13T09:38:02,334][INFO ][o.e.t.TransportService ] [o7wlA7C] publish_address {172.17.0.7:9300}, bound_addresses {0.0.0.0:9300}
[2018-09-13T09:38:02,347][INFO ][o.e.b.BootstrapChecks ] [o7wlA7C] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-09-13T09:38:02,350][ERROR][o.e.b.Bootstrap ] [o7wlA7C] node validation exception
[2] bootstrap checks failed
[1]: max file descriptors [40960] for elasticsearch process is too low, increase to at least [65536]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-09-13T09:38:02,353][INFO ][o.e.n.Node ] [o7wlA7C] stopping ...
[2018-09-13T09:38:02,406][INFO ][o.e.n.Node ] [o7wlA7C] stopped
[2018-09-13T09:38:02,406][INFO ][o.e.n.Node ] [o7wlA7C] closing ...
[2018-09-13T09:38:02,413][INFO ][o.e.n.Node ] [o7wlA7C] closed
[2018-09-13T09:38:02,414][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started

Two reasons for failing that I see are..

 

1. Max file descriptors need to increase

2. Max virtual memory areas need to increase

 

Supposedly, this is a fix for the second issue, but not sure if I want to be messing with unraids max_map_count.

 

Quote

A limit on mmap counts equal to 262,144 or more

!! This is the most frequent reason for Elasticsearch failing to start since Elasticsearch version 5 was released.

On Linux, use sysctl vm.max_map_count on the host to view the current value, and see Elasticsearch's documentation on virtual memory for guidance on how to change this value. Note that the limits must be changed on the host; they cannot be changed from within a container.

 

Anyone from Linuxserver.io interested in making an unraid compatible container for this?  

Found this recent thread referencing the ELK docker.

 

 

Tried the script @Jclendineng mentioned, but received an error

 

 sysctl: cannot stat /proc/sys/vm/max_map_count: No such file or directory

 

Is no one else using an ELK stack?

@sparklyballs Are you interested in this docker? I believe this is something similar to your pf-logstash docker, which I used prior to 2.4

I am running it, it works great. I am having a hard time figuring the best way to import data but the elk stack itself works well.

I am using my own ELK stack. Its a bit old, as I've had it setup and running since the 6.0beta days (https://hub.docker.com/r/roninkenji/docker-elk/) I have the template attached but you won't get much support from me...

I might see if I have some time to refresh my stack and publish it.

 

ELK.xml

https://hub.docker.com/r/sebp/elk/

 

This is the up-to-date ELK stack, you just need to do a couple things.

 

 

Install this in docker : ELK

 

Go through the wiki linked on the docker page, and make sure the variables are correct.

 

Add a variable : MAX_OPEN_FILES set to 65536

 

To get this to stick you need to set the ELK image as privileged (need to toggle advanced)

 

Download community apps script manager

 

Add the script below to run at start of array:

 

sysctl -w vm.max_map_count=262144

 

After this elk stack is fully running, you will still need to set it up with index and all that to parse data.

Thanks @Jclendineng

 

I added the script but received an error

sysctl: cannot stat /proc/sys/vm/max_map_count: No such file or directory

 

I attached a screenshot of the script

Hmm, I just added a new script, named it vm.max_map_count, and used that command, saved.

The error happens when the script runs...which is after rebooting the server and, subsequently, after the array starts...

 

 

This is what mine looks like, Just tested and it runs fine

Not a lot of room for user error here...I'm not sure how to debug this issue.

 

I used MC to see if the file exists and it does... 

@Jclendineng

 

Alright. So I was able to change the max map count by running the command as root.

 

Next issue is this:

[1]: max file descriptors [40960] for elasticsearch process is too low, increase to at least [65536]

 

Did you come across this issue? If so, how did you address it?

Did you end up getting this to work with unraid? 

Also having an issue changing vm.max_map_count. Running as root did not help.

On 3/10/2019 at 7:57 PM, bobokun said:

Did you end up getting this to work with unraid? 

I have elasticsearch and kibana running but not pointing any logs to it yet. Ultimately, I'd like to use Beats, so I have a Metricbeat container but also not fully configured yet.

 

On 3/27/2019 at 1:05 PM, FoxxMD said:

Also having an issue changing vm.max_map_count. Running as root did not help.

I don't think I ever solved this either...

 

This project has taken a back seat for some time but I'd like to get this up and running soon.

@surfshack66 I actually got the max_map_count working by renaming the script to vm.max_map_count. I'm not sure why this works but it does ¯\_(ツ)_/¯

 

EDIT: I created an app for CA for elasticsearch, check it out.

 

Edited April 3, 20197 yr by FoxxMD