View Nginx Proxy Manager Access Logs In A Beautiful Dashboard (with goaccess)

[frakman1]

 

Pre-requisites

 

- You have Nginx Proxy Manager already installed and working. I am using this one from the Community Applications: jlesage/nginx-proxy-manager:

 

- You have installed goaccess from Community Applications but it's not working out-of-the-box. I am using this one from the Community Applications: gregyankovoy/goaccess

 

There are three main steps

 

1- Have your log generator container (Nginx Proxy Manager) output its logs into a folder that goaccess can, well, access

2- Configure goaccess to look for the right log file 

3- Configure goaccess to understand how to parse the log/date/time format

 

Step One: Map Log File Folder

 

I use Nginx Proxy Manager and by default, it puts its access logs in the file /config/log/default.log. This location is non-configurable. Well, actually it's configured in the file /etc/nginx/nginx.conf with the line:

 

access_log /config/log/default.log proxy;

 

... but nginx.conf is not in a mapped folder so I just left it alone. I just ensured that it mapped its /config/log folder to a folder that both containers could access. In my case, I used /mnt/user/dmz/goaccess/log

 

 

Step Two: Configure Log File

 

The goaccess container looks for its access logs in the file /opt/log/access.log by default. Luckily, this is configurable in the goaccess.conf file that is mapped to the host's /mnt/user/appdata/goaccess/goaccess.conf file. In there, change the line:

 

log-file /opt/log/access.log

 

To:
 

log-file /opt/log/default.log

 

 

Step Three: Configure Log Format

 

The other thing to do is to provide the log/date/time file format that Nginx Proxy Manager uses in a language that goaccess understands. The nginx format is defined in the same nginx.conf file mentioned above as:

 

log_format standard '[$time_local] $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] "$http_user_agent" "$http_referer"';

 

There is a nifty script that does this mapping for you here. The short story is that it has to look like this for goaccess to understand it otherwise you get parsing errors.

 

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"

 

Now, open the file goaccess.conf again and comment out the line:

 

log-format COMBINED

 

and paste the three lines describing the log/date/time format we want.

 

That's it. You should now have a beautiful dashboard of your Nginx Proxy Manager access logs including which subdomains are getting used most (virtual hosts) and which URLs end up going to 404 (possible attacks) and a whole lot more besides!

 

Sample Dashboard:

 

 

Note that it should update in real time as long as the settings cog on the left has a green dot near it like this:

 

 

That means that the websocket is connected.

 

BONUS

 

If you want to get all geeky and see the results in a terminal window, you can do that too.

Just open the goaccess container's Console window and paste the three lines of log/date/time format into the file ~/.goaccessrc so it looks like this:

 

# cat ~/.goaccessrc 
time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"

 

and run:

 

 goaccess /opt/log/default.log 

 

And you will get the same information in a terminal window:

(Navigate with TAB and SHIFT+TAB button to jump between sections and ENTER to expand selection. q to quit)

 

Edited September 1, 2021 by frakman1

[062bel313]

Nice one, i will give this a try. Thank you for sharing.

[062bel313]

I couldn't find nginx.conf in my NPM /config directory. Do you know where it is located, I am using Nginx Proxy manager docker. I searched using find everything and it results empty search. 😞

[frakman1]

On 3/31/2021 at 1:29 AM, 062bel313 said:

I couldn't find nginx.conf in my NPM /config directory. Do you know where it is located, I am using Nginx Proxy manager docker. I searched using find everything and it results empty search. 😞

I address this point in Step One of my post.
The idea is not to have to change anything in nginx.conf and map the folders appropriately.

[hjaltioj]

Do you know how to do this with Nginx-Proxy-Manager-Official docker? Cant find the default.log file for that docker?

[frakman1]

2 hours ago, hjaltioj said:

Do you know how to do this with Nginx-Proxy-Manager-Official docker? Cant find the default.log file for that docker?

 

I am using this docker jlesage/nginx-proxy-manager

 

If you're talking about jc21/nginx-proxy-manager:latest

 

Then its /etc/nginx/nginx.conf configuration has this line:

 

access_log /data/logs/fallback_access.log proxy;

 

Which mean you will need to map the /data/logs folder to a location that the goaccess container can also access.

You will also need to change the goaccess configuration to look for that log file instead of the one currently configured

 

[hjaltioj]

36 minutes ago, frakman1 said:

 

I am using this docker jlesage/nginx-proxy-manager

 

If you're talking about jc21/nginx-proxy-manager:latest

 

Then its /etc/nginx/nginx.conf configuration has this line:

 

access_log /data/logs/fallback_access.log proxy;

 

Which mean you will need to map the /data/logs folder to a location that the goaccess container can also access.

You will also need to change the goaccess configuration to look for that log file instead of the one currently configured

 

Hi again.
I use jc21 version, but for me fallback_access.log is empty?

 

these are the logs that get createt.

 

[hjaltioj]

hi again. this is the error i get in NPM

 

error: skipping "/data/logs/fallback_access.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

[frakman1]

Sorry, I misunderstood earlier. I didn't know there was a second 'official' app on Community Applications. 

I'll have to try is out and get back to you later.

 

 

UPDATE:

Sorry, it's highly unlikely that I will revisit this on a different version of the app. It seems it was added after I started using the one from Grack's Repository. I updated my post to show which version I am using. You can use that one if you like.

 

Edited September 19, 2021 by frakman1

[Kopernikus]

On 8/31/2021 at 9:32 PM, hjaltioj said:

hi again. this is the error i get in NPM

 

error: skipping "/data/logs/fallback_access.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

 

Hi,

 

I got the same error, did you fix it?

[1471]

Nothing  to add other than I get the same error

[1471]

On 9/8/2021 at 1:05 PM, Kopernikus said:

 

Hi,

 

I got the same error, did you fix it?

 

Hey, did you have any luck getting sorted?

[DieFalse]

I spent time with this and got several things to work, but it seems with the Nginx Proxy Manager - Official container, the log files are different between fallback_access.log and the proxy host logs.  Adding them all will not work and missing info will occur since you can have either all the proxy host logs OR fallback_access since they're formatted differently.

 

As for the permissions, use a folder or share not in NginxProxyManager/GoAccess'es appdata and it will allow it to read correctly.  I made a share called Logs that I use for various logging and mapped to /mnt/user/Logs/NPM/

[MrLinford]

Got mine all setup but not showing Proxy Hosts

[DieFalse]

Here is the log format that NGINX Proxy Manager - Official uses for:

Proxy hosts:

'[$time_local] $upstream_cache_status $upstream_status $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] [Sent-to $server] "$http_user_agent" "$http_referer"'

 

standard:

'[$time_local] $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] "$http_user_agent" "$http_referer"'

 

Someone that knows the variables for goaccess, will need to convert the "proxy" one.

[DieFalse]

22 minutes ago, fmp4m said:

Here is the log format that NGINX Proxy Manager - Official uses for:

Proxy hosts:

'[$time_local] $upstream_cache_status $upstream_status $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] [Sent-to $server] "$http_user_agent" "$http_referer"'

 

standard:

'[$time_local] $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] "$http_user_agent" "$http_referer"'

 

Someone that knows the variables for goaccess, will need to convert the "proxy" one.

 

 

Nevermind,  I got it:

 

goaccess.conf:

comment out the existing time/date/log formats and add this:

 

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %^ %^ %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] [Sent-to %^] "%u" "%R"
 

Then add under the log file your list of proxy-host-log files like so: (note this is my list and is not the same as your list, find this in your NGINX Proxy Manager - Official appdata logs and add each you want to track.

 

log-file /opt/log/proxy-host-12_access.log
log-file /opt/log/proxy-host-13_access.log
log-file /opt/log/proxy-host-14_access.log
log-file /opt/log/proxy-host-15_access.log
log-file /opt/log/proxy-host-3_access.log
log-file /opt/log/proxy-host-4_access.log
log-file /opt/log/proxy-host-5_access.log
log-file /opt/log/proxy-host-6_access.log
log-file /opt/log/proxy-host-8_access.log
log-file /opt/log/proxy-host-9_access.log

[DieFalse]

Ok - I found a MUCH easier way.....

 

After making the changes to goaccess.conf to be:

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %^ %^ %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] [Sent-to %^] "%u" "%R"



log-file /opt/log/proxy_logs.log

 

 

Simply add the following line to each proxy host in NGINX Proxy Manager - Official "advanced" 

access_log /data/logs/proxy_logs.log proxy;

 

like so:  (if you already have advanced stuff here, add the line to the VERY top)

 

Now they all log to the same file, and same format,  simply add the line to all proxy_hosts and remember to add it to any new ones.

[MrLinford]

@fmp4m thank you for sharing this will be useful. 

[MS12345]

I am trying to deploy goaccess-docker but this is the error I am seeing:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
4530bb8cfa7ba7172d4118ebe3a814665ee45e4d1bd124ab5d221b3a1f5a624d

 

Docker Container logs show the following:

standard_init_linux.go:228: exec user process caused: exec format error

 

 

 

OS details:
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID_LIKE=debian

 

Can some one help me with this please?

Edited November 3, 2021 by MS12345

[urmyboyblue]

Great utility... thanks!! I have it all working (using SWAG/NGINX - /opt/log directory path is set to my SWAG /mnt/user/appdata/swag/log/nginx/ folder) and pulling into the access.log. I am seeing this error message though:

 

nginx: [alert] could not open error log file: open() "/var/lib/nginx/logs/error.log" failed (13: Permission denied)

 

Couldn't find much from a Google search. Any idea how to fix it? The error.log file is in the same directory as access.log so I assume it can read it.

[DrLucasMendes]

On 10/18/2021 at 2:44 PM, DieFalse said:

goaccess.conf:

comment out the existing time/date/log formats and add this:

 

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %^ %^ %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] [Sent-to %^] "%u" "%R"
 

Then add under the log file your list of proxy-host-log files like so: (note this is my list and is not the same as your list, find this in your NGINX Proxy Manager - Official appdata logs and add each you want to track.

 

log-file /opt/log/proxy-host-12_access.log
log-file /opt/log/proxy-host-13_access.log
log-file /opt/log/proxy-host-14_access.log
log-file /opt/log/proxy-host-15_access.log
log-file /opt/log/proxy-host-3_access.log
log-file /opt/log/proxy-host-4_access.log
log-file /opt/log/proxy-host-5_access.log
log-file /opt/log/proxy-host-6_access.log
log-file /opt/log/proxy-host-8_access.log
log-file /opt/log/proxy-host-9_access.log

 

 

 

OMG OMG OMG!!!

THANK YOU SOOOO MUCH!!!!!!!

 

🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩  🤩 

 

 

 

 

[3oD]

On 11/3/2021 at 1:31 AM, urmyboyblue said:

Great utility... thanks!! I have it all working (using SWAG/NGINX - /opt/log directory path is set to my SWAG /mnt/user/appdata/swag/log/nginx/ folder) and pulling into the access.log. I am seeing this error message though:

 

nginx: [alert] could not open error log file: open() "/var/lib/nginx/logs/error.log" failed (13: Permission denied)

 

Couldn't find much from a Google search. Any idea how to fix it? The error.log file is in the same directory as access.log so I assume it can read it.

 

jupp, same here. Is it even essential?

[BBQnBEER]

I have two questions.

 

1) I'm also geting 13: Permission denied error when I'm starting the Goaccess Container.  Looks like NPM did write files to that location.

2)Step 3 do I run script from the Goaccess terminal?

[boomam]

Fairly good guide, but needs more content - you've fallen into the same trap that a lot of OSS guides fall into, and made some leaps of faith in wording and knowledge.

To make this easier for people you should probably elaborate more on the below sections - to be clear, I am not asking for help myself, but for a more complete guide for others - 

 

 

...In my case, I used /mnt/user/dmz/goaccess/log

Why?

Is DMZ a folder you've setup?

If so, what does it store?

Just these files, and is needed for this setup, etc.?

 

 

There is a nifty script that does this mapping for you here...

What does the script do?

How and where is it ran?

Is it a one time thing? or every time we want to see the logs?

What is the effect on NPM?

What are the manual steps?

 

Now, open the file goaccess.conf again and comment out the line:

Is there a reason why this isn't done in the earlier editing of the goacces.conf file?

If so, why?

 

Minor adjustments to an otherwise good guide.

 

Thanks.

Edited November 24, 2021 by boomam

[stFfn]

On 3/12/2021 at 10:03 AM, frakman1 said:

 

Pre-requisites

 

- You have Nginx Proxy Manager already installed and working. I am using this one from the Community Applications: jlesage/nginx-proxy-manager:

 

- You have installed goaccess from Community Applications but it's not working out-of-the-box. I am using this one from the Community Applications: gregyankovoy/goaccess

 

There are three main steps

 

1- Have your log generator container (Nginx Proxy Manager) output its logs into a folder that goaccess can, well, access

2- Configure goaccess to look for the right log file 

3- Configure goaccess to understand how to parse the log/date/time format

 

Step One: Map Log File Folder

 

I use Nginx Proxy Manager and by default, it puts its access logs in the file /config/log/default.log. This location is non-configurable. Well, actually it's configured in the file /etc/nginx/nginx.conf with the line:

 

access_log /config/log/default.log proxy;

 

... but nginx.conf is not in a mapped folder so I just left it alone. I just ensured that it mapped its /config/log folder to a folder that both containers could access. In my case, I used /mnt/user/dmz/goaccess/log

 

 

Step Two: Configure Log File

 

The goaccess container looks for its access logs in the file /opt/log/access.log by default. Luckily, this is configurable in the goaccess.conf file that is mapped to the host's /mnt/user/appdata/goaccess/goaccess.conf file. In there, change the line:

 

log-file /opt/log/access.log

 

To:
 

log-file /opt/log/default.log

 

 

Step Three: Configure Log Format

 

The other thing to do is to provide the log/date/time file format that Nginx Proxy Manager uses in a language that goaccess understands. The nginx format is defined in the same nginx.conf file mentioned above as:

 

log_format standard '[$time_local] $status - $request_method $scheme $host "$request_uri" [Client $remote_addr] [Length $body_bytes_sent] [Gzip $gzip_ratio] "$http_user_agent" "$http_referer"';

 

There is a nifty script that does this mapping for you here. The short story is that it has to look like this for goaccess to understand it otherwise you get parsing errors.

 

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"

 

Now, open the file goaccess.conf again and comment out the line:

 

log-format COMBINED

 

and paste the three lines describing the log/date/time format we want.

 

That's it. You should now have a beautiful dashboard of your Nginx Proxy Manager access logs including which subdomains are getting used most (virtual hosts) and which URLs end up going to 404 (possible attacks) and a whole lot more besides!

 

Sample Dashboard:

 

 

Note that it should update in real time as long as the settings cog on the left has a green dot near it like this:

 

 

That means that the websocket is connected.

 

BONUS

 

If you want to get all geeky and see the results in a terminal window, you can do that too.

Just open the goaccess container's Console window and paste the three lines of log/date/time format into the file ~/.goaccessrc so it looks like this:

 

# cat ~/.goaccessrc 
time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"

 

and run:

 

 goaccess /opt/log/default.log 

 

And you will get the same information in a terminal window:

(Navigate with TAB and SHIFT+TAB button to jump between sections and ENTER to expand selection. q to quit)

 

 

hey im kinda new to linux. where and how do i have to us the "nifty" script to change the format?