View Nginx Proxy Manager Access Logs In A Beautiful Dashboard (with goaccess)

[frakman1]

29 minutes ago, stFfn said:

 

hey im kinda new to linux. where and how do i have to us the "nifty" script to change the format?

The link to the script shows exactly how it's used. In any case, you don't have to use the script because I already did and provided the output. Those are the three lines you need to copy into the goaccess.conf file.

[stFfn]

18 hours ago, frakman1 said:

The link to the script shows exactly how it's used. In any case, you don't have to use the script because I already did and provided the output. Those are the three lines you need to copy into the goaccess.conf file.

Where do i have to put the lines? Or can i put them at the bottem of the goacces.conf file?

[stFfn]

?

[frakman1]

On 12/1/2021 at 12:39 PM, stFfn said:

Where do i have to put the lines? Or can i put them at the bottem of the goacces.conf file?

 

I put mine around line 116:

 

 

I now notice that I use log_format while the file seems to be using log-format. Not sure if it makes a difference. You may want to try both if it isn't working like you expect.

 

[stFfn]

7 hours ago, frakman1 said:

 

I put mine around line 116:

 

 

I now notice that I use log_format while the file seems to be using log-format. Not sure if it makes a difference. You may want to try both if it isn't working like you expect.

 

it doesnt work either way.

 

i also have these Problems

 

Parsed 1 lines producing the following errors:

Token '-' doesn't match specifier '%m'

Format Errors - Verify your log/date/time format
usermod: no changes
Running container as dockeruser in group users (99:100)
Variables set:
PUID=99
PGID=100

nginx: [alert] could not open error log file: open() "/var/lib/nginx/logs/error.log" failed (13: Permission denied)

usermod: no changes
Running container as dockeruser in group users (99:100)
Variables set:
PUID=99
PGID=100

nginx: [alert] could not open error log file: open() "/var/lib/nginx/logs/error.log" failed (13: Permission denied)


Parsed 1 lines producing the following errors:

Token '-' doesn't match specifier '%m'

Format Errors - Verify your log/date/time format

 

[stFfn]

No ideas?

Edited December 6, 2021 by stFfn

[Masterwishx]

On 3/12/2021 at 11:03 AM, frakman1 said:

 

Pre-requisites

 

- You have Nginx Proxy Manager already installed and working. I am using this one from the Community Applications: jlesage/nginx-proxy-manager:

 

- You have installed goaccess from Community Applications but it's not working out-of-the-box. I am using this one from the Community Applications: gregyankovoy/goaccess

 

There are three main steps

 

1- Have your log generator container (Nginx Proxy Manager) output its logs into a folder that goaccess can, well, access

2- Configure goaccess to look for the right log file 

3- Configure goaccess to understand how to parse the log/date/time format

 

Step One: Map Log File Folder

 

I use Nginx Proxy Manager and by default, it puts its access logs in the file /config/log/default.log. This location is non-configurable. Well, actually it's configured in the file /etc/nginx/nginx.conf with the line:

 

access_log /config/log/default.log proxy;

 

... but nginx.conf is not in a mapped folder so I just left it alone. I just ensured that it mapped its /config/log folder to a folder that both containers could access. In my case, I used /mnt/user/dmz/goaccess/log

 

 

Step Two: Configure Log File

 

The goaccess container looks for its access logs in the file /opt/log/access.log by default. Luckily, this is configurable in the goaccess.conf file that is mapped to the host's /mnt/user/appdata/goaccess/goaccess.conf file. In there, change the line:

 

log-file /opt/log/access.log

 

To:
 

log-file /opt/log/default.log

 

 

Step Three: Configure Log Format

 

The other thing to do is to provide the log/date/time file format that Nginx Proxy Manager uses in a language that goaccess understands. The nginx format is defined in the same nginx.conf file mentioned above as:

 

log_format standard '[$time_local] $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] "$http_user_agent" "$http_referer"';

 

There is a nifty script that does this mapping for you here. The short story is that it has to look like this for goaccess to understand it otherwise you get parsing errors.

 

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"

 

Now, open the file goaccess.conf again and comment out the line:

 

log-format COMBINED

 

and paste the three lines describing the log/date/time format we want.

 

That's it. You should now have a beautiful dashboard of your Nginx Proxy Manager access logs including which subdomains are getting used most (virtual hosts) and which URLs end up going to 404 (possible attacks) and a whole lot more besides!

 

Sample Dashboard:

 

 

Note that it should update in real time as long as the settings cog on the left has a green dot near it like this:

 

 

That means that the websocket is connected.

 

BONUS

 

If you want to get all geeky and see the results in a terminal window, you can do that too.

Just open the goaccess container's Console window and paste the three lines of log/date/time format into the file ~/.goaccessrc so it looks like this:

 

# cat ~/.goaccessrc 
time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"

 

and run:

 

 goaccess /opt/log/default.log 

 

And you will get the same information in a terminal window:

(Navigate with TAB and SHIFT+TAB button to jump between sections and ENTER to expand selection. q to quit)

 

 

Having problem with step 3 i think,  also what to use default.log or proxy-host-11_access.log files ?

used 

(#log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"
log_format [%d:%t %^] %^ %^ %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] [Sent-to %^] "%u" "%R"
#log-format [%d:%t %^] %^ %s %^ %^ %m %^ %v "%U" [%^ %h] [%^ %b] %^"%u" "%R")

 

 

 

[Masterwishx]

[08/Dec/2021:16:19:21 +0200] - 200 200 - GET https mysite.com "/" [Client 172.70.162.210] [Length 529] [Gzip 2.12] [Sent-to 192.168.0.199] "Uptime-Kuma/1.11.0" "-"

this is my line in proxy-host-10_error.log

[Masterwishx]

Finally got it working with proxy_logs.log ,some one tryed the official version from allinurl/goaccess its much newer ?

[Masterwishx]

On 10/19/2021 at 12:12 AM, DieFalse said:

Ok - I found a MUCH easier way.....

 

After making the changes to goaccess.conf to be:

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %^ %^ %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] [Sent-to %^] "%u" "%R"



log-file /opt/log/proxy_logs.log

 

 

Simply add the following line to each proxy host in NGINX Proxy Manager - Official "advanced" 

access_log /data/logs/proxy_logs.log proxy;

 

like so:  (if you already have advanced stuff here, add the line to the VERY top)

 

Now they all log to the same file, and same format,  simply add the line to all proxy_hosts and remember to add it to any new ones.

 

Did you tryed the official one allinurl/goaccess ?

[SaiSaijo]

On 12/9/2021 at 9:18 AM, Masterwishx said:

Finally got it working with proxy_logs.log ,some one tryed the official version from allinurl/goaccess its much newer ?

 

 

Mind helping me out in how you exactly fixed it?

I have the same problem and have no idea what to do...

 

Thanks!

[Masterwishx]

12 hours ago, SaiSaijo said:

Mind helping me out in how you exactly fixed it?

I have the same problem and have no idea what to do...

 

Do you mean the official new version ?

[SaiSaijo]

10 hours ago, Masterwishx said:

 

Do you mean the official new version ?

I was refering to gregyankovoy/goaccess.

Not needed though, I just managed to fix it.. Had a different filename for the log file.

 

Also, since there are 2 types of formatting needed (for the main logfile and the additional proxys), I skipped the main logfile and am just using the proxylogs...

 

Did you manage to find a way to monitor both of them?

[Masterwishx]

1 hour ago, SaiSaijo said:

I was refering to gregyankovoy/goaccess.

Not needed though, I just managed to fix it.. Had a different filename for the log file.

 

Also, since there are 2 types of formatting needed (for the main logfile and the additional proxys), I skipped the main logfile and am just using the proxylogs...

 

Did you manage to find a way to monitor both of them?

 

im using one file for all proxys, you can use proxy_log like in my previos post On 10/19/2021 at 12:12 AM, DieFalse said: 

since log_format is the same you can use it for any logs ,im using format for proxys 

[SaiSaijo]

On 12/30/2021 at 9:58 PM, Masterwishx said:

 

im using one file for all proxys, you can use proxy_log like in my previos post On 10/19/2021 at 12:12 AM, DieFalse said: 

since log_format is the same you can use it for any logs ,im using format for proxys 

Im also using one file for all proxys, but that means that i cannot use the logging for the main domain, only for the proxys.

Do you know of any wat to also monitor the main domain?

[Masterwishx]

On 1/1/2022 at 5:33 PM, SaiSaijo said:

Do you know of any wat to also monitor the main domain?

im not using for now main domain, its in 404 of npm and its also monitored by (access_log /data/logs/proxy_logs.log proxy;).

[xavierh]

Recently created this for myself https://hub.docker.com/r/xavierh/goaccess-for-nginxproxymanager . Its not an unraid community app but I hope it is one day when its farther along. Currently only reading proxy logs. It works and updates itself. Running GoAccess 1.5.5.

 

Not sure if anyone wants to give it a try.

[JustinBH]

xavierh - You could provide some info on how to manually add it to UNRAID? As I have added a /config mapped to /appdata folder, yet the goaccess.conf and other files still appear to reside within the docker image.

 

Also if there was some way to allow adding in a MaxMind license key and having the GeoIP City and Country latest database download and unzip (to /config/geoip directory in /appdata of unraid), it would be great !

Edited February 26, 2022 by JustinBH

[xavierh]

On 2/25/2022 at 11:00 PM, JustinBH said:

xavierh - You could provide some info on how to manually add it to UNRAID? As I have added a /config mapped to /appdata folder, yet the goaccess.conf and other files still appear to reside within the docker image.

 

Also if there was some way to allow adding in a MaxMind license key and having the GeoIP City and Country latest database download and unzip (to /config/geoip directory in /appdata of unraid), it would be great !

Not sure when you grabbed the latest but the MaxMind database is included now in the image. There is no direct access to the goaccess.conf as this is set currently only to process proxy_host files. But later I'll let the goaccess.conf be loaded from outside the image. 

 

You should be able to manually add it to unraid going to docker, add container, then click advanced and set up the parameters appropriately.

Name: Whatever you want

Repository: xavierh/goaccess-for-nginxproxymanager:develop

WebUI: http://[IP]:[PORT:7880]/

 

Create a path:

Name: doesn't matter
Container Path: /opt/log

Host Path: /mnt/user/***

Access Mode can be read only

 

Create a port:

Name: doesn't matter

Container Port: 7880

Host Port: 7880

 

 

[xavierh]

If at some point enough people use it I might try to create a community app for it. I'm still debugging a few things and would want to use an alpine image rather than a ubuntu image.

[Archonw]

Thanks for that. It works and i just could detect some network issues inside my netowork with this.