View Nginx Proxy Manager Access Logs In A Beautiful Dashboard (with goaccess)


Recommended Posts

image.thumb.png.b669daca59f442bbfbe6d7958738f8aa.png

 

Pre-requisites

 

- You have Nginx Proxy Manager already installed and working. I am using this one from the Community Applications: jlesage/nginx-proxy-manager:

image.png.8283a62f9d2cce9b0b47427c7cc4343a.png

 

- You have installed goaccess from Community Applications but it's not working out-of-the-box. I am using this one from the Community Applications: gregyankovoy/goaccess

image.png.5b25544197fe1fc26ac7565de41e7166.png

 

There are three main steps

 

1- Have your log generator container (Nginx Proxy Manager) output its logs into a folder that goaccess can, well, access

2- Configure goaccess to look for the right log file 

3- Configure goaccess to understand how to parse the log/date/time format

 

Step One: Map Log File Folder

 

I use Nginx Proxy Manager and by default, it puts its access logs in the file /config/log/default.log. This location is non-configurable. Well, actually it's configured in the file /etc/nginx/nginx.conf with the line:


 

access_log /config/log/default.log proxy;

 

... but nginx.conf is not in a mapped folder so I just left it alone. I just ensured that it mapped its /config/log folder to a folder that both containers could access. In my case, I used /mnt/user/dmz/goaccess/log

 

image.png

 

Step Two: Configure Log File

 

The goaccess container looks for its access logs in the file /opt/log/access.log by default. Luckily, this is configurable in the goaccess.conf file that is mapped to the host's /mnt/user/appdata/goaccess/goaccess.conf file. In there, change the line:

 

log-file /opt/log/access.log

 

To:
 

log-file /opt/log/default.log

 

 

Step Three: Configure Log Format

 

The other thing to do is to provide the log/date/time file format that Nginx Proxy Manager uses in a language that goaccess understands. The nginx format is defined in the same nginx.conf file mentioned above as:

 

log_format standard '[$time_local] $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] "$http_user_agent" "$http_referer"';

 

There is a nifty script that does this mapping for you here. The short story is that it has to look like this for goaccess to understand it otherwise you get parsing errors.

 

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"

 

Now, open the file goaccess.conf again and comment out the line:

 

log-format COMBINED

 

and paste the three lines describing the log/date/time format we want.

 

That's it. You should now have a beautiful dashboard of your Nginx Proxy Manager access logs including which subdomains are getting used most (virtual hosts) and which URLs end up going to 404 (possible attacks) and a whole lot more besides!

 

Sample Dashboard:

 

image.thumb.png.ad5b30c7efa67812055b57fefcd5ea8a.png

 

Note that it should update in real time as long as the settings cog on the left has a green dot near it like this:

 

image.png.7fa2a870a876492719b2bdc37ec1ecb4.png

 

That means that the websocket is connected.

 

BONUS

 

If you want to get all geeky and see the results in a terminal window, you can do that too.

Just open the goaccess container's Console window and paste the three lines of log/date/time format into the file ~/.goaccessrc so it looks like this:

 

# cat ~/.goaccessrc 
time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"

 

and run:

 

 goaccess /opt/log/default.log 

 

And you will get the same information in a terminal window:

(Navigate with TAB and SHIFT+TAB button to jump between sections and ENTER to expand selection. q to quit)

 

image.thumb.png.9db544b70d74ecf0da4c55595b380124.png

Edited by frakman1
  • Like 5
Link to comment
  • 2 weeks later...
  • 4 months later...
On 3/31/2021 at 1:29 AM, 062bel313 said:

I couldn't find nginx.conf in my NPM /config directory. Do you know where it is located, I am using Nginx Proxy manager docker. I searched using find everything and it results empty search. 😞


I address this point in Step One of my post.
The idea is not to have to change anything in nginx.conf and map the folders appropriately.

Link to comment
  • 4 weeks later...
2 hours ago, hjaltioj said:

Do you know how to do this with Nginx-Proxy-Manager-Official docker? Cant find the default.log file for that docker?

 

I am using this docker jlesage/nginx-proxy-manager

 

If you're talking about jc21/nginx-proxy-manager:latest

 

Then its /etc/nginx/nginx.conf configuration has this line:

 

access_log /data/logs/fallback_access.log proxy;

 

Which mean you will need to map the /data/logs folder to a location that the goaccess container can also access.

You will also need to change the goaccess configuration to look for that log file instead of the one currently configured

 

Link to comment
36 minutes ago, frakman1 said:

 

I am using this docker jlesage/nginx-proxy-manager

 

If you're talking about jc21/nginx-proxy-manager:latest

 

Then its /etc/nginx/nginx.conf configuration has this line:

 

access_log /data/logs/fallback_access.log proxy;

 

Which mean you will need to map the /data/logs folder to a location that the goaccess container can also access.

You will also need to change the goaccess configuration to look for that log file instead of the one currently configured

 

Hi again.
I use jc21 version, but for me fallback_access.log is empty?

 

these are the logs that get createt.

 

log.PNG

Link to comment

hi again. this is the error i get in NPM

 

error: skipping "/data/logs/fallback_access.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

Link to comment

Sorry, I misunderstood earlier. I didn't know there was a second 'official' app on Community Applications. 

I'll have to try is out and get back to you later.

 

 

UPDATE:


Sorry, it's highly unlikely that I will revisit this on a different version of the app. It seems it was added after I started using the one from Grack's Repository. I updated my post to show which version I am using. You can use that one if you like.

 

Edited by frakman1
  • Like 1
Link to comment
On 8/31/2021 at 9:32 PM, hjaltioj said:

hi again. this is the error i get in NPM

 

error: skipping "/data/logs/fallback_access.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

 

Hi,

 

I got the same error, did you fix it?

Link to comment
  • 1 month later...

I spent time with this and got several things to work, but it seems with the Nginx Proxy Manager - Official container, the log files are different between fallback_access.log and the proxy host logs.  Adding them all will not work and missing info will occur since you can have either all the proxy host logs OR fallback_access since they're formatted differently.

 

As for the permissions, use a folder or share not in NginxProxyManager/GoAccess'es appdata and it will allow it to read correctly.  I made a share called Logs that I use for various logging and mapped to /mnt/user/Logs/NPM/

Link to comment

Here is the log format that NGINX Proxy Manager - Official uses for:

Proxy hosts:

'[$time_local] $upstream_cache_status $upstream_status $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] [Sent-to $server] "$http_user_agent" "$http_referer"'

 

standard:

'[$time_local] $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] "$http_user_agent" "$http_referer"'

 

Someone that knows the variables for goaccess, will need to convert the "proxy" one.

Link to comment
22 minutes ago, fmp4m said:

Here is the log format that NGINX Proxy Manager - Official uses for:

Proxy hosts:

'[$time_local] $upstream_cache_status $upstream_status $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] [Sent-to $server] "$http_user_agent" "$http_referer"'

 

standard:

'[$time_local] $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] "$http_user_agent" "$http_referer"'

 

Someone that knows the variables for goaccess, will need to convert the "proxy" one.

 

 

Nevermind,  I got it:

 

goaccess.conf:

comment out the existing time/date/log formats and add this:

 

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %^ %^ %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] [Sent-to %^] "%u" "%R"
 

Then add under the log file your list of proxy-host-log files like so: (note this is my list and is not the same as your list, find this in your NGINX Proxy Manager - Official appdata logs and add each you want to track.

 

log-file /opt/log/proxy-host-12_access.log
log-file /opt/log/proxy-host-13_access.log
log-file /opt/log/proxy-host-14_access.log
log-file /opt/log/proxy-host-15_access.log
log-file /opt/log/proxy-host-3_access.log
log-file /opt/log/proxy-host-4_access.log
log-file /opt/log/proxy-host-5_access.log
log-file /opt/log/proxy-host-6_access.log
log-file /opt/log/proxy-host-8_access.log
log-file /opt/log/proxy-host-9_access.log

Link to comment

Ok - I found a MUCH easier way.....

 

After making the changes to goaccess.conf to be:

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %^ %^ %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] [Sent-to %^] "%u" "%R"



log-file /opt/log/proxy_logs.log

 

 

Simply add the following line to each proxy host in NGINX Proxy Manager - Official "advanced" 

access_log /data/logs/proxy_logs.log proxy;

 

like so:  (if you already have advanced stuff here, add the line to the VERY top)

image.thumb.png.6333e6106ec7181d1d3c0bec06681fa4.png

 

Now they all log to the same file, and same format,  simply add the line to all proxy_hosts and remember to add it to any new ones.

  • Like 2
Link to comment
  • 2 weeks later...

I am trying to deploy goaccess-docker but this is the error I am seeing:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
4530bb8cfa7ba7172d4118ebe3a814665ee45e4d1bd124ab5d221b3a1f5a624d

 

Docker Container logs show the following:

standard_init_linux.go:228: exec user process caused: exec format error

 

 

 

OS details:
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID_LIKE=debian

 

Can some one help me with this please?

Edited by MS12345
Link to comment

Great utility... thanks!! I have it all working (using SWAG/NGINX - /opt/log directory path is set to my SWAG /mnt/user/appdata/swag/log/nginx/ folder) and pulling into the access.log. I am seeing this error message though:

 

nginx: [alert] could not open error log file: open() "/var/lib/nginx/logs/error.log" failed (13: Permission denied)

 

Couldn't find much from a Google search. Any idea how to fix it? The error.log file is in the same directory as access.log so I assume it can read it.

Link to comment
  • 2 weeks later...
On 10/18/2021 at 2:44 PM, DieFalse said:

goaccess.conf:

comment out the existing time/date/log formats and add this:

 

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %^ %^ %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] [Sent-to %^] "%u" "%R"
 

Then add under the log file your list of proxy-host-log files like so: (note this is my list and is not the same as your list, find this in your NGINX Proxy Manager - Official appdata logs and add each you want to track.

 

log-file /opt/log/proxy-host-12_access.log
log-file /opt/log/proxy-host-13_access.log
log-file /opt/log/proxy-host-14_access.log
log-file /opt/log/proxy-host-15_access.log
log-file /opt/log/proxy-host-3_access.log
log-file /opt/log/proxy-host-4_access.log
log-file /opt/log/proxy-host-5_access.log
log-file /opt/log/proxy-host-6_access.log
log-file /opt/log/proxy-host-8_access.log
log-file /opt/log/proxy-host-9_access.log

 

 

 

OMG OMG OMG!!!

THANK YOU SOOOO MUCH!!!!!!!

 

🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩 

 

 

 

 

Link to comment
On 11/3/2021 at 1:31 AM, urmyboyblue said:

Great utility... thanks!! I have it all working (using SWAG/NGINX - /opt/log directory path is set to my SWAG /mnt/user/appdata/swag/log/nginx/ folder) and pulling into the access.log. I am seeing this error message though:

 

nginx: [alert] could not open error log file: open() "/var/lib/nginx/logs/error.log" failed (13: Permission denied)

 

Couldn't find much from a Google search. Any idea how to fix it? The error.log file is in the same directory as access.log so I assume it can read it.

 

jupp, same here. Is it even essential?

Link to comment
  • 2 weeks later...

Fairly good guide, but needs more content - you've fallen into the same trap that a lot of OSS guides fall into, and made some leaps of faith in wording and knowledge.

To make this easier for people you should probably elaborate more on the below sections - to be clear, I am not asking for help myself, but for a more complete guide for others - 

 

 

...In my case, I used /mnt/user/dmz/goaccess/log

Why?

Is DMZ a folder you've setup?

If so, what does it store?

Just these files, and is needed for this setup, etc.?

 

 

There is a nifty script that does this mapping for you here...

What does the script do?

How and where is it ran?

Is it a one time thing? or every time we want to see the logs?

What is the effect on NPM?

What are the manual steps?

 

Now, open the file goaccess.conf again and comment out the line:

Is there a reason why this isn't done in the earlier editing of the goacces.conf file?

If so, why?

 

Minor adjustments to an otherwise good guide.

 

Thanks.

Edited by boomam
Link to comment
On 3/12/2021 at 10:03 AM, frakman1 said:

image.thumb.png.b669daca59f442bbfbe6d7958738f8aa.png

 

Pre-requisites

 

- You have Nginx Proxy Manager already installed and working. I am using this one from the Community Applications: jlesage/nginx-proxy-manager:

image.png.8283a62f9d2cce9b0b47427c7cc4343a.png

 

- You have installed goaccess from Community Applications but it's not working out-of-the-box. I am using this one from the Community Applications: gregyankovoy/goaccess

image.png.5b25544197fe1fc26ac7565de41e7166.png

 

There are three main steps

 

1- Have your log generator container (Nginx Proxy Manager) output its logs into a folder that goaccess can, well, access

2- Configure goaccess to look for the right log file 

3- Configure goaccess to understand how to parse the log/date/time format

 

Step One: Map Log File Folder

 

I use Nginx Proxy Manager and by default, it puts its access logs in the file /config/log/default.log. This location is non-configurable. Well, actually it's configured in the file /etc/nginx/nginx.conf with the line:


 

access_log /config/log/default.log proxy;

 

... but nginx.conf is not in a mapped folder so I just left it alone. I just ensured that it mapped its /config/log folder to a folder that both containers could access. In my case, I used /mnt/user/dmz/goaccess/log

 

image.png

 

Step Two: Configure Log File

 

The goaccess container looks for its access logs in the file /opt/log/access.log by default. Luckily, this is configurable in the goaccess.conf file that is mapped to the host's /mnt/user/appdata/goaccess/goaccess.conf file. In there, change the line:

 

log-file /opt/log/access.log

 

To:
 

log-file /opt/log/default.log

 

 

Step Three: Configure Log Format

 

The other thing to do is to provide the log/date/time file format that Nginx Proxy Manager uses in a language that goaccess understands. The nginx format is defined in the same nginx.conf file mentioned above as:

 

log_format standard '[$time_local] $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] "$http_user_agent" "$http_referer"';

 

There is a nifty script that does this mapping for you here. The short story is that it has to look like this for goaccess to understand it otherwise you get parsing errors.

 

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"

 

Now, open the file goaccess.conf again and comment out the line:

 

log-format COMBINED

 

and paste the three lines describing the log/date/time format we want.

 

That's it. You should now have a beautiful dashboard of your Nginx Proxy Manager access logs including which subdomains are getting used most (virtual hosts) and which URLs end up going to 404 (possible attacks) and a whole lot more besides!

 

Sample Dashboard:

 

image.thumb.png.ad5b30c7efa67812055b57fefcd5ea8a.png

 

Note that it should update in real time as long as the settings cog on the left has a green dot near it like this:

 

image.png.7fa2a870a876492719b2bdc37ec1ecb4.png

 

That means that the websocket is connected.

 

BONUS

 

If you want to get all geeky and see the results in a terminal window, you can do that too.

Just open the goaccess container's Console window and paste the three lines of log/date/time format into the file ~/.goaccessrc so it looks like this:

 

# cat ~/.goaccessrc 
time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"

 

and run:

 

 goaccess /opt/log/default.log 

 

And you will get the same information in a terminal window:

(Navigate with TAB and SHIFT+TAB button to jump between sections and ENTER to expand selection. q to quit)

 

image.thumb.png.9db544b70d74ecf0da4c55595b380124.png

 

hey im kinda new to linux. where and how do i have to us the "nifty" script to change the format?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.