Pfsense Log sytem / docker image

[Aric]

Hi I found this log monitoring docker for pfsense. But I am not sure how to add it to my unraid since it is not in the normal container system we use here.

 

the url on github is https://github.com/northshorenetworks/elk

 

the url for the program is here http://elijahpaul.co.uk/monitoring-pfsense-2-1-logs-using-elk-logstash-kibana-elasticsearch/

 

I am sure I am not the only one that uses unraid and pfsense.

 

So any help to install this would be appreciated.

 

[guruleenyc]

I'm interested in this as well.

[Squid]

I just did a quick search on docker hub, and I believe that this is the same thing

 

https://registry.hub.docker.com/u/northshorenetworks/elk/dockerfile/

 

You can just paste that link into the docker search plugin http://lime-technology.com/forum/index.php?topic=38879.0, and you're off to the races (or Bob's your uncle depending upon your preference)

 

You are going to have to manually add the four ports specified in the instructions manually when you add the container because the dockerfile doesn't explicitly expose those ports so that the conversion process can pick them up: 80tcp 5140tcp 5140udp, and 9200 tcp

[sparklyballs]

I just did a quick search on docker hub, and I believe that this is the same thing

 

https://registry.hub.docker.com/u/northshorenetworks/elk/dockerfile/

 

You can just paste that link into the docker search plugin http://lime-technology.com/forum/index.php?topic=38879.0, and you're off to the races (or Bob's your uncle depending upon your preference)

 

You are going to have to manually add the four ports specified in the instructions manually when you add the container because the dockerfile doesn't explicitly expose those ports so that the conversion process can pick them up: 80tcp 5140tcp 5140udp, and 9200 tcp

 

 

Also editing the config file might be a PITA if you add this via a template.

[smdion]

Dang.. why did I down size away from my main unRAID box and PFSense. This is sexy

[sparklyballs]

I'm trying to shoehorn it into something more amenable to unraid.

[sparklyballs]

I have it built to the same stage as the original docker, now just need to do startup scripts, set volumes etc....

[sparklyballs]

It's in my beta repo, but good luck getting any information in or out of it.

 

Unless you have a degree in grok patterns, it's pretty much useless.

 

 

[sparklyballs]

the repo has the default logstash.conf from the container linked by squid.

 

I haven't included the config files from the link in op, because with 2.2.1 pfsense, they just don't work.

 

nothing but a mile of grok parse failures, earlier versions of pfsense you may better luck with.

[sparklyballs]

I have snort and squid installed on my pfsense, don't know if they're messing the log output for this, i'm not disabling them just for some pretty pictures, but my graylog server handles my pfsense log perfectly well.

 

[sparklyballs]

Finally got this PITA working.

 

the version in my beta repo works for pfsense 2.2 ONLY.

 

pull the container as per usual via template and let it run for a couple of minutes or so to generate the neccesary config files locally.

 

then in your local config folder inside logstash-conf you'll find a file called

 

10-syslog.conf

 

with this content

 

filter {
  if [type] == "syslog" {

    #change to pfSense ip address
    if [host] =~ /192\.168\.1\.1/ {
      mutate {
        add_tag => ["PFSense", "Ready"]
      }
    }


    if "Ready" not in [tags] {
      mutate {
        add_tag => [ "syslog" ]
      }
    }
  }
}

filter {
  if [type] == "syslog" {
    mutate {
      remove_tag => "Ready"
    }
  }
}

filter {
  if "syslog" in [tags] {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM  dd HH:mm:ss" ]
      locale => "en"
    }

    if !("_grokparsefailure" in [tags]) {
      mutate {
        replace => [ "@source_host", "%{syslog_hostname}" ]
        replace => [ "@message", "%{syslog_message}" ]
      }
    }

    mutate {
      remove_field => [ "syslog_hostname", "syslog_message", "syslog_timestamp" ]
    }
#    if "_grokparsefailure" in [tags] {
#      drop { }
#    }
  }
}

 

 

you must change this part to match your pfsense ip address (use ip not name as docker doesn't really do name-servers)

 

#change to pfSense ip address
    if [host] =~ /192\.168\.1\.1/ {

 

Then stop/start the container.

 

 

 

 

make sure you keep the format, the dots and various slashes are very important.

 

 

If your pfsense is on 192.168.1.1 then you're a lucky SOB as this will work out of the box with no editing.

 

Will make a pfsense 2.1 conf setup later.

 

[misterbeetz]

Hey sparkly.  I would like to give this one a go however the docker will not start at all (no log text available or config files in local.)  Any ideas?

[sparklyballs]

Hey sparkly.  I would like to give this one a go however the docker will not start at all (no log text available or config files in local.)  Any ideas?

 

i just loaded the docker from scratch, no problems whatsoever.

[StanC]

Hey, sparkly. Thank you for this. This is one awesome docker. I installed it, made the change to my pfSense firewall and it was immediately up and running. 

[misterbeetz]

This thing is really cool.  I just got it working.  I guess the trick here is to realize that pfsense typically sends out those logs to a specific port by default however the docker suggests a different one which is probably smart.

 

I should also mention that graylog is another amazing docker that can be used to capture logs from pfsense in addition to anything else you would want (desktop syslogs, unraid main syslog etc.)  It's in sparkly's beta repo.