Sharing my ChatGTP adventure

Hi,

 

I have been playing around with ChatGTP lately and "we" have created something that -might- be usefull for others.

 

I am using PiHole as both my DNS server as well as my DHCP server. Since that functionality is kind of crucial I am running two PiHole dockers, unfortunately it is a bad idea (and not really possible unless you become very strict in your networking) to have two different DHCP servers on your network. So only one of the two PiHole servers has DHCP switched on.

 

Now my problem was that I wanted to know whenever DHCP would not work so at least I could manually switch, and preferably before my whole family falls on top of me because "the internet is not working".

 

So I needed some kind of monitoring that could really check if the DHCP was working on the primary server. The API for PiHole is kind of a mess so that was not a good route, I succeeded though and now have the following monitoring page updated every three minutes:

 

 

The basis of this thing is that I have two different PiHoles running (one in an unraid VM, one on a Pi), I have exposed both of their folders containing logfiles so I can mount them on the system that creates the HTML.

 

On all servers install the NFS networking package:

sudo apt-get update
sudo apt-get install nfs-kernel-server

 

Then, on every server that has a folder you want to share, create a file /etc/exports with the following content:

/var/log/pihole    192.168.2.34(ro,sync)

 

After that export those folders with the following command on each server:

sudo exportfs -a

 

Then make sure that that networking package starts with every reboot:

sudo systemctl start nfs-kernel-server
sudo systemctl enable nfs-kernel-server

 

Then, on the server that needs to create the HTML file do:

sudo mount -t nfs yourservername:/var/log/pihole /some/local/directory

 

That mounts the folders of the servers as readable folders on the server that needs to create the html. Make sure that "/some/local/directory" exists (you can just create some random folder somewhere and use that, nice to do it in /mnt as good linux people do.

 

Although this will make it work the mounts will be gone after a reboot, to make them persistent you need to add them to fstab:

 

sudo nano /etc/fstab

 

And add the following lines into the file that now opens (IP is the IP of the server that has the logfiles). Ofcourse you need to do that twice, once for every server with lofiles:

192.168.2.253:/var/log/pihole /mnt/pihole1 nfs defaults 0 0

 

Now the monitoring server has everything it needs to create the html file.

 

Now create a bash script on the monitoring server:

sudo nano pihole_monitor.sh

 

And copy the following code in:

#!/bin/bash

echo Starting the Pi-Hole High Availability script

# Configuration variables

echo Declaring variables ..

pihole1_ip="192.168.2.253"
pihole2_ip="192.168.2.252"
pihole1_log="/mnt/pihole1/pihole.log"
pihole2_log="/mnt/pihole2/pihole.log"
test_interval_seconds=60
test_fail_threshold=3
pihole1_available="yes"
pihole2_available="yes"
pihole1_unavailability_log="/mnt/download/logs/pihole1_unavailability.log"
pihole2_unavailability_log="/mnt/download/logs/pihole2_unavailability.log"
check_time=""
max_age_minutes=60  # Adjust this to the desired value
pushover_user_key="YOUR PUSHOVER USERKEY"
pushover_token="YOUR PUSHOVER TOKEN"

echo "Pi-Hole 1 IP: $pihole1_ip"
echo "Pi-Hole 1 API: $pihole1_api_key"
echo
echo "Pi-Hole 2 IP: $pihole2_ip"
echo "Pi-Hole 2 API: $pihole2_api_key"
echo
echo
echo "We will test every $test_interval_seconds and fail a server when this has failed $test_fail_treshold times within the executing of this script"
echo

# Function to check the availability of a Pi-hole server
check_pihole_availability() {
  echo Checking availability ...
  local pihole_ip=$1
  local failed_attempts=0

  for i in $(seq 1 $test_fail_threshold); do
    ping -c 1 -W 1 "$pihole_ip" > /dev/null 2>&1
    if [ $? -eq 0 ]; then
      check_time=$(date '+%Y-%m-%d %H:%M:%S') 
      return 0
    fi
    failed_attempts=$((failed_attempts + 1))
    if [ $failed_attempts -lt $test_fail_threshold ]; then
      sleep $test_interval_seconds
    fi
  done
  check_time=$(date '+%Y-%m-%d %H:%M:%S')
  return 1
}

# Check the availability of both Pi-hole servers and update the variables
check_pihole_availability "$pihole1_ip"
if [ $? -eq 0 ]; then
  pihole1_available="yes"
else
  pihole1_available="no"
fi

check_pihole_availability "$pihole2_ip"
if [ $? -eq 0 ]; then
  pihole2_available="yes"
else
  pihole2_available="no"
fi

echo Availability check finished...
echo
echo "Pi-Hole1: $pihole1_available"
echo "Pi-Hole2: $pihole2_available"
echo




get_dhcpack_logs() {
  local pihole_log=$1
  local log_lines=$(grep "DHCPACK" "$pihole_log" | tail -n 5)
  echo "<pre>$log_lines</pre>"
}

get_unavailability_times() {
  local unavailability_log=$1
  if [ ! -f "$unavailability_log" ]; then
    echo "No unavailability times yet" > "$unavailability_log"
  fi
  local log_lines=$(tail -n 5 "$unavailability_log")
  echo "<pre>$log_lines</pre>"
}

check_last_dhcpack() {
  local pihole_log=$1
  local max_age_minutes=$2
  local max_age_seconds=$((max_age_minutes * 60))
  local last_dhcpack_line=$(grep "DHCPACK" "$pihole_log" | tail -n 1)
  local last_dhcpack_time=$(echo $last_dhcpack_line | cut -d' ' -f1,2,3)
  local last_dhcpack_seconds=$(date --date="$last_dhcpack_time" +%s)
  local current_seconds=$(date +%s)
  local age_seconds=$((current_seconds - last_dhcpack_seconds))
  local age_minutes=$((age_seconds / 60))
  local age_hours=$((age_minutes / 60))
  local age_days=$((age_hours / 24))

  if (( age_seconds > max_age_seconds )); then
    echo "<span style='color:red;'>Server has not served a DHCP request since $age_days day(s), $age_hours hour(s), $age_minutes minute(s). Check the server please.</span>"
  else
    echo "<span style='color:green;'>Server seems to be healthy and responding to DHCP Requests</span>"
  fi
}

get_last_log_timestamp() {
  local pihole_log=$1
  local last_log_timestamp=$(grep "DHCPACK" "$pihole_log" | tail -n 1 | cut -d' ' -f1,2,3)
  echo "$last_log_timestamp"
}

get_dhcpack_logs_time() {
  local pihole_log=$1
  local last_log_time=$(grep "DHCPACK" "$pihole_log" | tail -n 5 | cut -d' ' -f1,2,3)
  echo "$last_log_time"
}


# Function to send a Pushover notification
send_pushover_notification() {
  local message=$1
  curl -s \
    --form-string "token=$pushover_token" \
    --form-string "user=$pushover_user_key" \
    --form-string "message=$message" \
    https://api.pushover.net/1/messages.json
}

# Function to check if pihole1 is available and send a notification if it is
check_and_notify_pihole1() {
  if [ "$pihole1_available" = "no" ]; then
    send_pushover_notification "Pi-hole 1 is unavailable."
  fi
}

check_and_notify_pihole1

# Function to generate the HTML page
echo Starting generating HTML...

generate_html() {
  local output_file="/var/www/html/pihole_status.html"
  local current_time=$(date '+%Y-%m-%d %H:%M:%S')

cat > "$output_file" << EOF
<!DOCTYPE html>
<html>
<head>
  <title>Pi-hole High Availability Status</title>
  <style>
    table {
      width: 100%;
      border-collapse: collapse;
    }
    th, td {
      padding: 8px;
      text-align: left;
      border-bottom: 1px solid #ddd;
    }
    .dark-red {
      color: darkred;
      font-size: 1em;
    }
    .dark-blue {
      color:darkblue;
      font-size: 1em;
    }
    .header-title {
      font-size: 2em; /* Adjust this value to change the font size of the header */
    }
  </style>
</head>
<body>
  <h1 class="header-title dark-red">Pi-hole High Availability Status</h1>
  <p class="dark-blue">Primary (Unraid): $(check_last_dhcpack "$pihole1_log" "$max_age_minutes")</p>
  <p class="dark-blue">Secondary (Raspberry): $(check_last_dhcpack "$pihole2_log" "$max_age_minutes")</p>
  <br/>
  <br/>
  <table>
    <tr class="dark-red">
      <th>Server</th>
      <th>Availability</th>
      <th>Last Checked</th>
    </tr>
    <tr class="dark-red">
      <td><a href="http://$pihole1_ip/admin" class="dark-red">Primary (Unraid)</a></td>
      <td>$pihole1_available</td>
      <td>$check_time</td>
    </tr>
    <tr>
      <td colspan="3" class="dark-blue">Last 5 DHCPACK log lines: $(get_dhcpack_logs_time "$pihole1_log")</td>      
    </tr>
    <tr>
      <td colspan="3" class="dark-blue">$(get_dhcpack_logs "$pihole1_log")</td>
    </tr>
    <tr>
      <td colspan="3" class="dark-blue">Last 5 Unavailability Times for Primary (Unraid):</td>
    </tr>
    <tr>
      <td colspan="3" class="dark-blue">$(get_unavailability_times "$pihole1_unavailability_log")</td>
    </tr>
    <tr class="dark-red">
      <td><a href="http://$pihole2_ip/admin" class="dark-red">Secondary (Raspberry)</a></td>
      <td>$pihole2_available</td>
      <td>$check_time</td>
    </tr>
    <tr>
      <td colspan="3" class="dark-blue">Last 5 DHCPACK log lines: $(get_dhcpack_logs_time "$pihole2_log")</td>    
    </tr>
    <tr>
      <td colspan="3" class="dark-blue">$(get_dhcpack_logs "$pihole2_log")</td>
    </tr>
    <tr>
      <td colspan="3" class="dark-blue">Last 5 Unavailability Times for Secondary (Raspberry):</td>
    </tr>
    <tr>
      <td colspan="3" class="dark-blue">$(get_unavailability_times "$pihole2_unavailability_log")</td>
    </tr>
  </table>
</body>
</html>
EOF
}


generate_html

echo HTML created..
echo
echo Script finished.

 

Make sure to change the variables in the beginning to fit your environment. Create a (free) pushover account and you will receive push notifications if the primary server goes offline.

 

The way the system checks if a DHCP server is functioning is looking at the logfiles, and looking at the latest line that shows a DHCP package has been delivered, this should be no longer then 60 minutes ago and you need to allign that with your number of devices and your DHCP lease time. Those are all variables.

 

To give as much troubleshooting info as possible the script also does a ping test to see if the server is reachable at all.

 

The primary and secundary headers in the html are clickable and bring you to the specific PiHole server.

 

You need to schedule the script in CRON (or with userscripts plugin) to make it run as often as you want, I am running it every three minutes.

 

Its probably a quite specific usecase but also a nice example on how to collaborate with ChatGTP. The things makes errors that you need have it correct and this still took some hours in total but I would not have been able to do it without ChatGTP and I learned some stuff in the process.

 

Extra tip: If you are running pihole in a docker then you would need to share the specific "appdata" folder to the monitoring server, -OR- you do it a bit differently and create another script that copies over the logfile out of the docker to a central location every few minutes, that is actually how I first built the script before I realised that just having the script look at the mounted logfiles uses less resources and gives more up-to-date data.

Edited May 19, 20233 yr by Helmonder