View Nginx Proxy Manager Access Logs In A Beautiful Dashboard (with goaccess)


Recommended Posts

29 minutes ago, stFfn said:

 

hey im kinda new to linux. where and how do i have to us the "nifty" script to change the format?

The link to the script shows exactly how it's used. In any case, you don't have to use the script because I already did and provided the output. Those are the three lines you need to copy into the goaccess.conf file.

Link to comment
18 hours ago, frakman1 said:

The link to the script shows exactly how it's used. In any case, you don't have to use the script because I already did and provided the output. Those are the three lines you need to copy into the goaccess.conf file.

Where do i have to put the lines? Or can i put them at the bottem of the goacces.conf file?

Link to comment
On 12/1/2021 at 12:39 PM, stFfn said:

Where do i have to put the lines? Or can i put them at the bottem of the goacces.conf file?

 

I put mine around line 116:

 

image.png.253f0faf14e3412bf2dbcb262116e4cc.png

 

I now notice that I use log_format while the file seems to be using log-format. Not sure if it makes a difference. You may want to try both if it isn't working like you expect.

 

Link to comment
7 hours ago, frakman1 said:

 

I put mine around line 116:

 

image.png.253f0faf14e3412bf2dbcb262116e4cc.png

 

I now notice that I use log_format while the file seems to be using log-format. Not sure if it makes a difference. You may want to try both if it isn't working like you expect.

 

it doesnt work either way.

 

i also have these Problems :(

 

Parsed 1 lines producing the following errors:

Token '-' doesn't match specifier '%m'

Format Errors - Verify your log/date/time format
usermod: no changes
Running container as dockeruser in group users (99:100)
Variables set:
PUID=99
PGID=100

nginx: [alert] could not open error log file: open() "/var/lib/nginx/logs/error.log" failed (13: Permission denied)

usermod: no changes
Running container as dockeruser in group users (99:100)
Variables set:
PUID=99
PGID=100

nginx: [alert] could not open error log file: open() "/var/lib/nginx/logs/error.log" failed (13: Permission denied)


Parsed 1 lines producing the following errors:

Token '-' doesn't match specifier '%m'

Format Errors - Verify your log/date/time format

 

Link to comment
On 3/12/2021 at 11:03 AM, frakman1 said:

image.thumb.png.b669daca59f442bbfbe6d7958738f8aa.png

 

Pre-requisites

 

- You have Nginx Proxy Manager already installed and working. I am using this one from the Community Applications: jlesage/nginx-proxy-manager:

image.png.8283a62f9d2cce9b0b47427c7cc4343a.png

 

- You have installed goaccess from Community Applications but it's not working out-of-the-box. I am using this one from the Community Applications: gregyankovoy/goaccess

image.png.5b25544197fe1fc26ac7565de41e7166.png

 

There are three main steps

 

1- Have your log generator container (Nginx Proxy Manager) output its logs into a folder that goaccess can, well, access

2- Configure goaccess to look for the right log file 

3- Configure goaccess to understand how to parse the log/date/time format

 

Step One: Map Log File Folder

 

I use Nginx Proxy Manager and by default, it puts its access logs in the file /config/log/default.log. This location is non-configurable. Well, actually it's configured in the file /etc/nginx/nginx.conf with the line:


 

access_log /config/log/default.log proxy;

 

... but nginx.conf is not in a mapped folder so I just left it alone. I just ensured that it mapped its /config/log folder to a folder that both containers could access. In my case, I used /mnt/user/dmz/goaccess/log

 

image.png

 

Step Two: Configure Log File

 

The goaccess container looks for its access logs in the file /opt/log/access.log by default. Luckily, this is configurable in the goaccess.conf file that is mapped to the host's /mnt/user/appdata/goaccess/goaccess.conf file. In there, change the line:

 

log-file /opt/log/access.log

 

To:
 

log-file /opt/log/default.log

 

 

Step Three: Configure Log Format

 

The other thing to do is to provide the log/date/time file format that Nginx Proxy Manager uses in a language that goaccess understands. The nginx format is defined in the same nginx.conf file mentioned above as:

 

log_format standard '[$time_local] $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] "$http_user_agent" "$http_referer"';

 

There is a nifty script that does this mapping for you here. The short story is that it has to look like this for goaccess to understand it otherwise you get parsing errors.

 

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"

 

Now, open the file goaccess.conf again and comment out the line:

 

log-format COMBINED

 

and paste the three lines describing the log/date/time format we want.

 

That's it. You should now have a beautiful dashboard of your Nginx Proxy Manager access logs including which subdomains are getting used most (virtual hosts) and which URLs end up going to 404 (possible attacks) and a whole lot more besides!

 

Sample Dashboard:

 

image.thumb.png.ad5b30c7efa67812055b57fefcd5ea8a.png

 

Note that it should update in real time as long as the settings cog on the left has a green dot near it like this:

 

image.png.7fa2a870a876492719b2bdc37ec1ecb4.png

 

That means that the websocket is connected.

 

BONUS

 

If you want to get all geeky and see the results in a terminal window, you can do that too.

Just open the goaccess container's Console window and paste the three lines of log/date/time format into the file ~/.goaccessrc so it looks like this:

 

# cat ~/.goaccessrc 
time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"

 

and run:

 

 goaccess /opt/log/default.log 

 

And you will get the same information in a terminal window:

(Navigate with TAB and SHIFT+TAB button to jump between sections and ENTER to expand selection. q to quit)

 

image.thumb.png.9db544b70d74ecf0da4c55595b380124.png

 

Having problem with step 3 i think,  also what to use default.log or proxy-host-11_access.log files ?

used 

(#log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"
log_format [%d:%t %^] %^ %^ %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] [Sent-to %^] "%u" "%R"
#log-format [%d:%t %^] %^ %s %^ %^ %m %^ %v "%U" [%^ %h] [%^ %b] %^"%u" "%R")

 

 

 

image.thumb.png.f723deb870c00afd5c2bae472c627bb5.png

Link to comment
On 10/19/2021 at 12:12 AM, DieFalse said:

Ok - I found a MUCH easier way.....

 

After making the changes to goaccess.conf to be:

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %^ %^ %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] [Sent-to %^] "%u" "%R"



log-file /opt/log/proxy_logs.log

 

 

Simply add the following line to each proxy host in NGINX Proxy Manager - Official "advanced" 

access_log /data/logs/proxy_logs.log proxy;

 

like so:  (if you already have advanced stuff here, add the line to the VERY top)

image.thumb.png.6333e6106ec7181d1d3c0bec06681fa4.png

 

Now they all log to the same file, and same format,  simply add the line to all proxy_hosts and remember to add it to any new ones.

 

Did you tryed the official one allinurl/goaccess ?

  • Like 1
Link to comment
  • 3 weeks later...
10 hours ago, Masterwishx said:

 

Do you mean the official new version ?

I was refering to gregyankovoy/goaccess.

Not needed though, I just managed to fix it.. Had a different filename for the log file.

 

Also, since there are 2 types of formatting needed (for the main logfile and the additional proxys), I skipped the main logfile and am just using the proxylogs...

 

Did you manage to find a way to monitor both of them?

Link to comment
1 hour ago, SaiSaijo said:

I was refering to gregyankovoy/goaccess.

Not needed though, I just managed to fix it.. Had a different filename for the log file.

 

Also, since there are 2 types of formatting needed (for the main logfile and the additional proxys), I skipped the main logfile and am just using the proxylogs...

 

Did you manage to find a way to monitor both of them?

 

im using one file for all proxys, you can use proxy_log like in my previos post On 10/19/2021 at 12:12 AM, DieFalse said: 

since log_format is the same you can use it for any logs ,im using format for proxys 

Link to comment
On 12/30/2021 at 9:58 PM, Masterwishx said:

 

im using one file for all proxys, you can use proxy_log like in my previos post On 10/19/2021 at 12:12 AM, DieFalse said: 

since log_format is the same you can use it for any logs ,im using format for proxys 

Im also using one file for all proxys, but that means that i cannot use the logging for the main domain, only for the proxys.

Do you know of any wat to also monitor the main domain?

Link to comment
  • 1 month later...

xavierh - You could provide some info on how to manually add it to UNRAID? As I have added a /config mapped to /appdata folder, yet the goaccess.conf and other files still appear to reside within the docker image.

 

Also if there was some way to allow adding in a MaxMind license key and having the GeoIP City and Country latest database download and unzip (to /config/geoip directory in /appdata of unraid), it would be great !

Edited by JustinBH
Link to comment
On 2/25/2022 at 11:00 PM, JustinBH said:

xavierh - You could provide some info on how to manually add it to UNRAID? As I have added a /config mapped to /appdata folder, yet the goaccess.conf and other files still appear to reside within the docker image.

 

Also if there was some way to allow adding in a MaxMind license key and having the GeoIP City and Country latest database download and unzip (to /config/geoip directory in /appdata of unraid), it would be great !

Not sure when you grabbed the latest but the MaxMind database is included now in the image. There is no direct access to the goaccess.conf as this is set currently only to process proxy_host files. But later I'll let the goaccess.conf be loaded from outside the image. 

 

You should be able to manually add it to unraid going to docker, add container, then click advanced and set up the parameters appropriately.

Name: Whatever you want

Repository: xavierh/goaccess-for-nginxproxymanager:develop

WebUI: http://[IP]:[PORT:7880]/

 

Create a path:

Name: doesn't matter
Container Path: /opt/log

Host Path: /mnt/user/***

Access Mode can be read only

 

Create a port:

Name: doesn't matter

Container Port: 7880

Host Port: 7880

 

 

Capture1.JPG

Capture2.JPG

  • Like 2
Link to comment
  • 2 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.