[Support] devzwf - Pihole Sync

[ZappyZap]

Overview: Support for Docker image Pihole Sync 

Application Name: Pihole Sync
Application:  https://github.com/ShiromMakkad/docker-pihole-sync

Documentation: https://github.com/ShiromMakkad/docker-pihole-sync/blob/master/README.md

 

Make sure to look at the complete documentation

 

Please post any questions/issues relating to this docker you have in this thread.

[rutherford]

I was looking into getting https://github.com/vmstan/gravity-sync going for my two piHole instances. I would like to use the hardware rasperrypi as the primary, and the docker instance as the secondary - I'm thinking that isn't what this docker does. I guess I could swap 'em around? thanks!

[ZappyZap]

Ho this is the sender so for the primary 
i can creater a receiver template if you need ....
 

[rutherford]

no no, that's OK! As I started to look into a little more: read/write to an SD card (which my raspberri pi has) it seems like too big a thing. I rarely make changes to the pihole anyhow, and I can just manually do it every once and a while.

AND: thanks for your work and containers!

[ZappyZap]

Ok No problem
I will still make a receiver i think just in case

[Nodiaque]

Hello, just to be sure, this is only the "sync" part of pihole and not pihole directly right? Meaning I still have to install a pihole server?

 

Thank you

[ZappyZap]

You are correct 

this is used when you have 2 pihole server and you want to keep them in sync

[Nodiaque]

ok great. I just set it up yesterday. I see everything is getting sync, but is it normal I don't have the same amount of blocked domains? I though this would sync the gravity list when it's updated on the master, but it seems it's waiting on the gravity trigger that I triggered manually after 12 hours or so. There's no error in log from either receiver and sender, and when I look at the sender during a gravity update, it does says it's sending a lots of stuff.

[Nodiaque]

Oh I forgot to add, there's an error in the receiver template, the default mode is sender and since it's a hidden value, you don't see it when configuring the template per default.

[ZappyZap]

2 hours ago, Nodiaque said:

Oh I forgot to add, there's an error in the receiver template, the default mode is sender and since it's a hidden value, you don't see it when configuring the template per default.

Thanks for the report ,  i will take a look and fix 
Edit : This has been fixed

Edited December 14, 2022 by ZappyZap

[geekazoid]

Hi, I'm just wondering how this is supposed to be operable. When I deploy the images they just drop into a reboot loop forever and I can't get a console long enough to set up ssh.

[ZappyZap]

6 minutes ago, geekazoid said:

Hi, I'm just wondering how this is supposed to be operable. When I deploy the images they just drop into a reboot loop forever and I can't get a console long enough to set up ssh.

Can you please provide some log ?
would be a good start

[geekazoid]

I've deleted the verbose detail of my troubleshooting because its not relevant to the community. Instead I'm going to turn my findings into a Quick Start Guide for pihole-sync:

 

Quick Start Guide

Environment

• Primary pihole on the unraid docker host A (unraidA) has it's files typically in /mnt/user/appdata/pihole. There are two important directories there that this app is going to sync:
  • /mnt/user/appdata/pihole/pihole
  • /mnt/user/appdata/pihole/dnsmasq.d
• The pihole-sync sender is going to reside on the same docker host as the master pihole server as it has direct access to the same filesystem. A sensible path for this would be
  • /mnt/user/appdata/pihole-sync-sender (on unraidA)
• Secondary pihole on unraid docker host B (unraidB) should probably have a similar filesystem setup for consistency. Of course the path for the receiver should be similarly
  • /mnt/user/appdata/pihole-sync-receiver (on unraidB)
• NETWORKING:
  • If you use Bridge mode for your containers, the sync will be employing TCP ports 22222. Defaults will work for you.
  • If you use br0 mode for a dedicated IP per container, you can just use the standard ssh port 22 (change 22222 to 22).
    Remember that your sync hosts are going to be on different IPs than your piholes if you use br0 method, so don't confuse that in your setup.
• Operations: you should be able to ssh from unraidA to unraidB. Otherwise you can use the clipboard and web terminal.

 

Install the Receiver

1. On unraidB, use Community Apps to install Pihole-Sync-Receiver. In the Add Container dialog you are going to have the following fields to fill:
   • Name: I recommend renaming the docker image to pihole-sync-receiver (lowercase) for consistency (this will make it less tedious in the CLI later)
   • Network: as discussed above in Networking
   • Remote Host IP: as discussed above in Networking
   • Root directory: /mnt/user/appdata/pihole-sync-receiver/root
   • Etc-ssh: /mnt/user/appdata/pihole-sync-receiver/ssh
   • Pi-Hole Path: /mnt/user/appdata/pihole-sync-receiver/pihole  (we're going to change this later to a symlink)
   • Pi-Hole DNSmasq path: /mnt/user/appdata/pihole-sync-receiver/dnsmasq.d   (we're going to change this later to a symlink)
   • <click Show more settings>
     • Note Type: receiver
     • Remote SSH port: as discussed above in Networking
   • <Apply>
2. Stop the Receiver container for now.
3. Open a Terminal on unraidB

   • cd /mnt/user/appdata/pihole-sync-receiver

   • mkdir root/.ssh

   • later you will be symlinking the pihole and dnsmasq.d directories here to your secondary pihole server's paths one level up. Let's get this working first.

 

Note: We installed the Receiver first because the sender is going to thrash against the receiver trying to connect until we complete the ssh key installation.

 

Install the Sender and copy the ssh key to the Receiver

1. On unraidA, use Community Apps to install Pihole-Sync-Sender. In the Add Container dialog you are going to have the following fields to fill:
   • Name: I recommend renaming the docker image to pihole-sync-sender for consistency (this will make it less tedious in the CLI later)
   • Network: as discussed above in Networking
   • Remote Host IP: as discussed above in Networking
   • Root directory: /mnt/user/appdata/pihole-sync-sender/root
   • Etc-ssh: /mnt/user/appdata/pihole-sync-sender/ssh
   • Pi-Hole Path: /mnt/user/appdata/pihole-sync-sender/pihole  (we're going to change this later to a symlink)
   • Pi-Hole DNSmasq path: /mnt/user/appdata/pihole-sync-sender/dnsmasq.d   (we're going to change this later to a symlink)
   • <click Show more settings>
     • Note Type: sender
     • Remote SSH port: as discussed above in Networking
   • <Apply>
2. On first startup, the container is going to generate a ssh host key. If you used the paths defined above, it will be located in
   • /mnt/user/appdata/pihole-sync-sender/root/.ssh/
3. Open Terminal on unraidA

   • docker logs pihole-sync-sender

     • there is a message early on in the logs that mentions the steps to copy the ssh key to the receiver. I'll re-iterate these instructions below.

   • cd /mnt/user/appdata/pihole-sync-sender/root/.ssh/

   • scp id_ed25519.pub root@unraidB:/mnt/user/appdata/pihole-sync-receiver/root/.ssh/authorized_keys

     • enter the root password for unraidB when prompted to authenticate this secure copy
4. Now start the Receiver on unraidB
5. Open Terminal on unraidB

   • docker exec -it pihole-sync-receiver /bin/bash

   • cd /

   • chown root root

   • chgrp root root

   • exit

6. On unraidA
   • cd /mnt/user/appdata/pihole-sync-sender
   • touch pihole/psynctestfile
   • touch dnsmasq.d/dsynctestfile
7. Now start the Sender on unraidA
   • docker logs pihole-sync-sender
8. On unraidB
   • docker logs pihole-sync-receiver
   • cd /mnt/user/appdata/pihole-sync-receiver
   • ls -a pihole  (do you see your psynctestfile ?)
   • ls -a dynsmasq.d   (do you see your dsynctestfile ?)
9. Once you know its syncing properly, the setup is ready to be connected to your primary and secondary piholes.
   • stop both the sender and receiver containers
   • delete the sync test files at both ends
   • Use symlinks:
     • on unraidA (sender) it would be something like this:
       • ln -s /mnt/user/appdata/pihole/pihole /mnt/user/appdata/pihole-sync-sender/pihole
       • ln -s /mnt/user/appdata/pihole/dnsmasq.d /mnt/user/appdata/pihole-sync-sender/dnsmasq.d
     • on unraidB (receiver) it would be something like:
       • ln -s /mnt/user/appdata/pihole/pihole /mnt/user/appdata/pihole-sync-receiver/pihole
       • ln -s /mnt/user/appdata/pihole/dnsmasq.d /mnt/user/appdata/pihole-sync-receiver/dnsmasq.d
   • Start the Receiver, then the Sender... check the logs!
10. This is where the more specific pihole configuration begins (beyond scope of this quick start guide)

 

Edited January 1, 2023 by geekazoid
Made my post useful for others

[ZappyZap]

I will have look

and get back to you

[geekazoid]

I've fixed the ssh pubkeygen auth issue. 

 

Authentication refused: bad ownership or modes for directory /root

 

cd /
chown root root
chgrp root root

 

I made an issue on github for this, and I submitted an edit to the README. Above I've written a Quick Start Guide which I'll probably build upon later when I get the whole stack dialed. When done I'll bring the final product to github, but I figure being on first page it will be helpful for the next guy here.

Have a Happy New Year!

Edited January 1, 2023 by geekazoid
updated to reflect final solution

[Nodiaque]

Quick question, does it sync the gravity database? I notice that they don't have the same number of domain in the blocklist unless I sync both gravity at the sametime. I do see a lot of log saying it's sending files when I start a sync on the sender, but the receiver doesn't seems to "read" the new values (Although I did see all other setting getting updated).

[ZappyZap]

5 hours ago, geekazoid said:

I've fixed the ssh pubkeygen auth issue. 

 

Authentication refused: bad ownership or modes for directory /root

 

cd /
chown root root
chgrp root root

 

I made an issue on github for this, and I submitted an edit to the README. Above I've written a Quick Start Guide which I'll probably build upon later when I get the whole stack dialed. When done I'll bring the final product to github, but I figure being on first page it will be helpful for the next guy here.

Have a Happy New Year!

Thanks  @geekazoid
Awesome Job , did not get this issue on mine but probably because my receiver is a docker <> Unraid....

 

5 hours ago, Nodiaque said:

Quick question, does it sync the gravity database? I notice that they don't have the same number of domain in the blocklist unless I sync both gravity at the sametime. I do see a lot of log saying it's sending files when I start a sync on the sender, but the receiver doesn't seems to "read" the new values (Although I did see all other setting getting updated).

I notice that as well, but i am pretty sure the  rsync send the db over....
i will investigate

 

Happy New Year

[geekazoid]

I built this again from scratch and it went off without a hitch using the procedure I laid out in my comment above. What a delight!

 

I also figured out what causes the root permissions bug. If you create the path manually before installation, it doesn't happen. So I think I will update my guide to move those steps to the beginning and include the symlink steps as well.

I also found a little nugget people might appreciate:

If you want to set the hostname of your pihole, select Advanced View in the Edit page and append " --hostname=your_hostname" to Extra Parameters.

[ZappyZap]

31 minutes ago, geekazoid said:

Question: what is the purpose of the ssh path since .ssh is in root?

This is the directory in which the SSH server key files and the SSH daemon config will be stored, so it needs to be persistent.
only relevant to receiver..... i might go head and remove from Sender ....

Edited January 1, 2023 by ZappyZap

[ZappyZap]

24 minutes ago, geekazoid said:

I built this again from scratch and it went off without a hitch using the procedure I laid out in my comment above. What a delight!

 

I also figured out what causes the root permissions bug. If you create the path manually before installation, it doesn't happen. So I think I will update my guide to move those steps to the beginning and include the symlink steps as well.

I also found a little nugget people might appreciate:

If you want to set the hostname of your pihole, select Advanced View in the Edit page and append " --hostname=your_hostname" to Extra Parameters.

If you want make a PR in to https://github.com/devzwf/unraid-docker-templates/tree/main/guides
and i will add the guide in the readme first section of the template perhaps ?

let me know

[geekazoid]

Another configuration nugget for those using pihole's dhcp service.

In dnsmasq.d create a file called 99-extra-dns.conf, contents:
 

dhcp-option=6,pihole1_ip,pihole2_ip

So your dhcp clients are aware of the primary and secondary dns servers.

[ZappyZap]

33 minutes ago, geekazoid said:

Another configuration nugget for those using pihole's dhcp service.

In dnsmasq.d create a file called 99-extra-dns.conf, contents:
 

dhcp-option=6,pihole1_ip,pihole2_ip

So your dhcp clients are aware of the primary and secondary dns servers.

this should go in pihole thread ....

[axeman]

Thanks for your work!

 

Would this be able to sync to piholes where one isn't in a docker? 

 

My primary pihole runs as a VM on an ESXi machine. I'd like to sync that to a potential docker container on Unraid (potential because I need to understand how to get them to sync before deploying). Also - does the "disable pihole for x minutes" sync or is it just the lists that are getting synced? 

 

 

[ZappyZap]

11 minutes ago, axeman said:

Thanks for your work!

 

Would this be able to sync to piholes where one isn't in a docker? 

 

My primary pihole runs as a VM on an ESXi machine. I'd like to sync that to a potential docker container on Unraid (potential because I need to understand how to get them to sync before deploying). Also - does the "disable pihole for x minutes" sync or is it just the lists that are getting synced? 

 

 

should doable but with some adjustment.
I dont think the "Disable phole for x minutes" write anything to file , and all is in mem so i dont think this getting synced ... need to confirm

[geekazoid]

1 hour ago, axeman said:

does the "disable pihole for x minutes" sync or is it just the lists that are getting synced? 

PiHole Browser Extension supports multiple PiHoles.

 

For the other stuff:

You can run dockers on ESXi. Then you can deploy pihole in docker and use the shared storage to run pihole-sync. There may be licensing requirements from vmware. My last certification was vSphere 5; I don't know.

 

Otherwise you'd need to set up some kind of shared storage (nfs) that your linux pihole VMs can mount. Note: this would not be mounted to the hypervisor but to the VMs directly. Basically a bunch of work and adding points of failure. Then you could fork the project and adapt the pihole-sync scripts to whatever linux you want to run etc.