ich777 Posted July 3, 2021 Share Posted July 3, 2021 Prometheus unRAID Exporter Plugins This is the support thread for the Prometheus unRAID Exporter plugins. If you got a request for a Prometheus Exporter plugin feel free to ask here in the thread. Following plugins for Prometheus Exporters are available (click on the name to go to the tutorial post): Node Exporter (this post) Data from your unRAID Server nvidia-smi Exporter Data from your Nvidia card(s) passed through to Docker PiHole Exporter Data from your PiHole, whether from Docker or your local network MSI Afterburner Exporter Data from your local PC/VM Wireguard Exporter Data from unRAID Wireguard AdGuard Exporter Data from your AdGuard, whether from Docker or your local network Fritzbox Exporter Data from your Fritzbox Tutorial Grafana, Prometheus & Node Exporter base configuration to get the basic readings from unRAID: Go to the CA App and download the Grafana Docker container: Download the Prometheus Docker container from the CA App (please note that it will be stopped after the installation, that's just fine and will be fixed in the next steps): Download and install the Prometheus Node Exporter plugin from the CA App: Download the 'prometheus.yml' and place it in your '.../appdata/prometheus/etc' directory on your unRAID server: prometheus.yml Open the prometheus.yml and edit the part 'YOURUNRAIDSERVERIP' so that it matches your server IP, save and close the file: Go to your Docker page and start the Prometheus Docker container: After it is started click the Prometheus container again and select WebUI: On the top click on "Status -> Targets": You should see now a list with two entries, the entry "YOURSERVERIP:9100" should display "UP" like in the following screenshot: Go to the Docker page and start the Grafana WebUI: Login with your Username and Password (default: admin | admin): If you left it at default you should change the password in the next screen to something secure or press "Skip": Next we are going to configure the Prometheus datasource in Grafana, click on the "Gear Icon -> Data sources": Click on "Add data source": At Prometheus click on "Select": In the next screen enter the IP and Port from your Prometheus container like this: (You can check the IP:PORT on your Docker page): At the bottom click on "Save & Test": You should see this message: In Grafana click on "+ -> Import": Now we are going to import a preconfigured Dashboard for the Node Exporter from Grafana.com (Source), to do this simply enter the ID from the Dasboard and click "Load": In the next screen be sure to select "Prometheus" as the data source and rename the Dashboard to your liking: Click on "Import": Now you should be greeted with something like this (please keep in mind that the Dashboard can display N/A at some values since there is not enough data available, wait a few minutes and you will see that the values are filled in): (Please also keep in mind that some values will not be filled at all because unRAID by default don't have a SWAP) 2 8 Quote Link to comment
ich777 Posted July 3, 2021 Author Share Posted July 3, 2021 Dashboard Hints Create your very own customized Dashboard from the templates: Open two window side by side from Grafana In the first window open up the Dashboard from which you want to import panels In the second window click on "+ -> Dashboard" and click on "Add an empty panel" and click on the back arrow on the top: Back in the first window click on the little arrow that appears when you hover over the name from the panel with the cursor that you want to move over to your new Dashboard and select "Inspect -> Panel JSON": Copy the whole text from the appearing side panel in your clipboard with CTRL+C: Go to the second window and hover over the name from the empty panel with the cursor and select "Inspect -> Panel JSON": Paste the contents from Step 5 with CTRL+V from the appearing side panel and click "Apply": If you got an empty panel like in this case your source Panel from which you copied the JSON has most likely variables configured that you can also define in your new Dashboard: Click on the little Gear icon on the very top in your first and also in your second window: Click in both windows on "Variables" and add the missing variables from the first to the second window (the missing variables that you have to add are displayed on the bottom): Copy over all fields from the first to the second window so that they match and click "Update": Once you copied over all variables and none missing Variables are displayed anymore click on "Save dashboard" and in the following dialog box on "Save": Now you should see your panel in it's full glory: Note if you added (a) missing variable(s) from a "old" Dashboard to a "new" these variables apply globally to the "new" Dashboard so you don't have to configure it for each Panel. Make your Dashboard non editable: Open the Dashboard and click on the Gear icon: In the General tab go switch from "Editable" to "Read-only": Click on "Save dashboard": In the next dialogue click on "Save": Click on the "Back Arrow" on top to go back to your template: To make Dashboard editable again do the steps from above and in Step 2 switch from "Read-only" to "Editable". 2 Quote Link to comment
ich777 Posted July 3, 2021 Author Share Posted July 3, 2021 Prometheus nvidia-smi Exporter (For this Exporter you need to have the Nvidia-Driver plugin installed and made sure that you Nvidia Card is not bound to VFIO) If you want to watch the Status of a passed through graphics card in a VM or your local computer, install the Prometheus MSI Afterburner Exporter plugin. Download and install the Prometheus nvidia-smi Exporter plugin from the CA App: Open up the prometheus.yml (Step 4 + 5 from the first post), add a line with '- targets: ["YOURSERVERIP:9202"]' (please change "YOURSERVERIP" to your Server IP), save and close the file: Go to the Docker page and restart Prometheus: Open up the Grafana WebUI: In Grafana click on "+ -> Import": In the following windows click on "Upload JSON file" and upload the nvidia-smi.json file: nvidia-smi.json Rename the Dashboard to your liking and click on "Import": Now you should be greeted with something like this (please keep in mind that the Dashboard can display N/A at some values since there is not enough data available, wait a few minutes and you will see that the values are filled in): 1 1 Quote Link to comment
ich777 Posted July 3, 2021 Author Share Posted July 3, 2021 Prometheus PiHole Exporter Note: You can connect to any PiHole on your local network and of course if you run it on unRAID in a Docker container or VM. Download and install the Prometheus PiHole Exporter plugin from the CA App: Go to the plugin settings by clicking on "Settings -> Pi-Hole Exporter (at the bottom of the Settings page)": Enter your IP from PiHole and also your API Token and click on "Confirm & Start": (Please note that if you run your PiHole in a Docker container in a Custom network like br0 you have to enable the option "Enable host access" in your Docker settings, otherwise the plugin can't connect to your PiHole instance) To get your API Token go to your PiHole instance, Login and click on "Settings -> API / Web interface -> Show API Token -> Yes, show API Token": After that you should see in the right top corner that the Exporter is running and details about it: Open up the prometheus.yml (Step 4 + 5 from the first post), add a line with '- targets: ["YOURSERVERIP:9617"]' (please change "YOURSERVERIP" to your Server IP), save and close the file: Go to the Docker page and restart Prometheus: Open up the Grafana WebUI: In Grafana click on "+ -> Import": Now we are going to import a preconfigured Dashboard for the PiHole Exporter from Grafana.com (Source), to do this simply enter the ID (10176) from the Dasboard and click "Load": In the next screen rename the Dashboard to your liking and click on "Import" Now you should be greeted with something like this (please keep in mind that the Dashboard can display N/A at some values since there is not enough data available, wait a few minutes and you will see that the values are filled in): (Now you will notice that this warning: "Panel plugin not found: grafana-piechar-panel" appears on the Dasboard, to fix this follow the next steps) Go to your Docker page and click on Grafana and select "Console": In the next window enter the following 'grafana-cli plugins install grafana-piechart-panel' and press RETURN: After that close the Console window and restart the Grafana Docker container: Now go back to your PiHole Dashboard within Grafana and you should now see that the Dasboard is fully loaded: ATTENTION Please note if you restart your PiHole container the Exporter will stop and you have to manually start it from the plugin configuration page with the "START" button. This is also applies if you have CA Backup installed and the container is beeing backed up. To workaround that you don't have to manually restart it after each CA Backup do the following steps: Go to Settings and click on the bottom on "Backup/Restore Appdata": Confirm the Warning that pops up and scroll all the way down to the bottom and click on "Show Advanced Settings": At Pi-Hole make sure that you click on the switch so that it shows "Don't Stop": Scroll down to the bottom and click "Apply": NO DATA SHOWING UP IN THE PIHOLE DASHBOARD If no data is showing up in the PiHole Dashboard it is most likely the case that you have configured another Datasource like Telegraf in Grafana, to solve this issue go to this post: 2 2 Quote Link to comment
ich777 Posted July 3, 2021 Author Share Posted July 3, 2021 Prometheus MSI Afterburner Exporter With this plugin you can export the status from your local PC or a VM, running for example on unRAID, to Prometheus. Note: This plugin needs MSI Afterburner and MSI Afterburner Remote Server installed to work on the machine you want to watch (you can get them here )! Download and install the Prometheus MSI Afterburner Exporter plugin from the CA App: Go to your local local PC/VM and install MSI Afterburner and also MSI Afterburner Remote Server: (In the case for MSI Afterburner Remote Server I recommend to extract the folder to C:\Program Files (x86)\RemoteServer folder and configure the "MSIAfterburnerRemoteServer.exe" to run on boot as Administrator) Go to the plugin settings by clicking on "Settings -> Afterburner Exporter (at the bottom of the Settings page)": Enter your IP from your local PC/VM and click on "Confirm & Start": To get your IP go to your local PC/VM and doubleclick on the MSI Afterburner Remote Server icon and it will tell you the IP (in this case the second line is relevant since 127.0.0.1 = localhost is only reachable from the local PC/VM and your server actually can't connect to this address): (It is strongly recommended to assign your local PC/VM a static IP in your Router/Firewall) After you started it you should see in the right top corner that the Exporter is running and details about it: Open up the prometheus.yml (Step 4 + 5 from the first post), add a line with '- targets: ["YOURSERVERIP:9891"]' (please change "YOURSERVERIP" to your Server IP), save and close the file: Go to the Docker page and restart Prometheus: Open up the Grafana WebUI: In Grafana click on "+ -> Import": In the following windows click on "Upload JSON file" and upload the MSI Afterburner.json file: MSI Afterburner.json (thanks to @alturismo who created that Dashboard) In the next screen rename the Dashboard to your liking and click on "Import": Now you should be greeted with something like this (please keep in mind that the Dashboard can display N/A at some values since there is not enough data available, wait a few minutes and you will see that the values are filled in): 3 Quote Link to comment
ich777 Posted July 3, 2021 Author Share Posted July 3, 2021 Prometheus Wireguard Exporter With this plugin you can monitor your Wiregard VPN Server that is running on your unRAID Server. Download and install the Prometheus Wireguard Exporter plugin from the CA App: Open up the prometheus.yml (Step 4 + 5 from the first post), add a line with '- targets: ["YOURSERVERIP:9586"]' (please change "YOURSERVERIP" to your Server IP), save and close the file: Go to the Docker page and restart Prometheus: Open up the Grafana WebUI: In Grafana click on "+ -> Import": In the following windows click on "Upload JSON file" and upload the WireGuard.json file: WireGuard.json In the next screen rename the Dashboard to your liking and click on "Import": Now you should be greeted with something like this (please keep in mind that the Dashboard can display N/A at some values since there is not enough data available, wait a few minutes and you will see that the values are filled in, of course a client has to be connected and transmit/receive data): Rename Public Keys to match your devices Click on the down arrow on the table that shows all information about the clients and select "Edit": Copy the Public Key for the device you want to rename: In the right sidebar scroll down to "Override 2" click on "Add override property" and select "Value mappings > Value mappings": Click on "Add value mappings": Click on "Add a new mapping": In the following pop-up choose "Value": Then paste the copied Public Key from Step 2 in the "Value" field and at "Display text" your preferred device name and click on "Update": Then you should see that the Public Key is now changed to the entered "Display text" from Step 6: Don't forget to save your changes to the Dashboard by clicking on "Save" on the top right corner: In the pop-up click again "Save": And finally click on "Apply" on the top right corner: If you have more devices that you want to rename simply repeat this steps for each Public Key. 1 2 Quote Link to comment
ich777 Posted July 3, 2021 Author Share Posted July 3, 2021 Prometheus AdGuard Exporter Note: You can connect to any AdGuard Home on your local network and of course if you run it on unRAID in a Docker container or VM. Download and install the Prometheus AdGuard Exporter plugin from the CA App: Go to the plugin settings by clicking on "Settings -> AdGuard Exporter" (at the bottom of the Settings page) : Enter your IP from AdGuard, Port, admin username & the password and click on "Confirm & Start": (Please note that if you run your AdGuard in a Docker container in a Custom network like br0 you have to enable the option "Enable host access" in your Docker settings, otherwise the plugin can't connect to your AdGuard instance) After that you should see in the right top corner that the Exporter is running and details about it: Open up the prometheus.yml (Step 4 + 5 from the first post), add a line with '- targets: ["YOURSERVERIP:9617"]' (please change "YOURSERVERIP" to your Server IP), save and close the file: Go to the Docker page and restart Prometheus: Open up the Grafana WebUI: In Grafana click on "+ -> Import": Now we are going to import a preconfigured Dashboard for the AdGuard Exporter from Grafana.com (Source), to do this simply enter the ID (13330) from the Dasboard and click "Load": In the next screen rename the Dashboard to your liking, select "Prometheus" as datasource and click on "Import": Now you should be greeted with something like this (please keep in mind that the Dashboard can display N/A at some values, especiall at the gauges, since there is not enough data available, wait a few minutes and you will see that the values are filled in): (Now you will notice that this warning: "Panel plugin not found: grafana-piechar-panel" appears on the Dasboard, to fix this follow the next steps) Go to your Docker page and click on Grafana and select "Console": In the next window enter the following 'grafana-cli plugins install grafana-piechart-panel' and press RETURN: After that close the Console window and restart the Grafana Docker container: Now go back to your AdGuard Dashboard within Grafana and you should now see that the Dasboard is fully loaded: ATTENTION Please note if you restart your AdGuard container the Exporter will stop and you have to manually start it from the plugin configuration page with the "START" button. This also applies if you have CA Backup installed and the container is beeing backed up. To workaround that you don't have to manually restart it after each CA Backup do the following steps: Go to Settings and click on the bottom on "Backup/Restore Appdata": Confirm the Warning that pops up and scroll all the way down to the bottom and click on "Show Advanced Settings": At AdGuard make sure that you click on the switch so that it shows "Don't Stop": Scroll down to the bottom and click "Apply": 2 1 Quote Link to comment
falconexe Posted July 4, 2021 Share Posted July 4, 2021 (edited) Great work man! Thanks for DMing me about your projects. I know what I’m adding to the Ultimate UNRAID Dashboard (UUD) Version 1.7! 😂 @SpencerJ You see this yet? This might just be juicy enough to pull me out of retirement/hiatus and get the UUD 1.7 out. I’ll look into this in August. I’m getting married this month! Edited July 4, 2021 by falconexe 4 Quote Link to comment
SpencerJ Posted July 4, 2021 Share Posted July 4, 2021 This looks great @ich777! Will share this. 6 hours ago, falconexe said: I’ll look into this in August. I’m getting married this month! Congrats @falconexe! I hope you have a great wedding 1 1 Quote Link to comment
SirCadian Posted July 4, 2021 Share Posted July 4, 2021 Great write-up! I've set up Prometheus & Grafana before on a Pi. This was some time ago and, at the time, there wasn't a way to import SMART stats into Prometheus. I'm building an Unraid server soon and had ruled out Prometheus for this reason and was going to use Influxdb instead. Is it still difficult/impossible to get SMART data into Prometheus? Quote Link to comment
ich777 Posted July 4, 2021 Author Share Posted July 4, 2021 54 minutes ago, SirCadian said: Is it still difficult/impossible to get SMART data into Prometheus? I have to look into this if tgere already someone made an exporter or not, can think of various difficults (drives spinning up when trying to read the data,...). I will get back to you and let you know about my findings. From what I've read so far, it should be possible. You can always use InfluxDB with Promtheus in Grafana. EDIT: Already found something will let you know how this is working and if it's even working... Quote Link to comment
ich777 Posted July 4, 2021 Author Share Posted July 4, 2021 3 hours ago, SirCadian said: Is it still difficult/impossible to get SMART data into Prometheus? Is this something you can work with: Quote Link to comment
SirCadian Posted July 4, 2021 Share Posted July 4, 2021 Wow, thanks for the speedy response! I'm mainly looking for the temps and for: 0x05: Reallocated Sectors Count 0xC5: Current Pending Sector Count 0xC6: Uncorrectable Sector Count I currently use CrystalDiskInfo on a headless Win10 machine that I use as a fileserver. CDI checks those SMART metrics and alerts when a drive is starting to go bad. It has been useful to help me gauge when is time to swap a drive out as it nears the end of its useful life. I'm hoping to monitor for drives starting to go bad using prometheus/grafana. It may be that there's a better way to do this in Unraid as it's more storage/NAS oriented. Quote Link to comment
ich777 Posted July 4, 2021 Author Share Posted July 4, 2021 1 hour ago, SirCadian said: 0x05: Reallocated Sectors Count 0xC5: Current Pending Sector Count 0xC6: Uncorrectable Sector Count These values are read too but the dasboard is just a template, here is another one (I'm not really good at creating dashbaords but you can at least customize it or create one and if you want to share it I'm happy to put it here in the thread): 1 Quote Link to comment
SirCadian Posted July 4, 2021 Share Posted July 4, 2021 Looking good. I'll definitely play around with this once I've got my server up and running. All the kit is sitting in a big pile behind me. Just waiting on a case to be delivered before I build it and start playing around with Unraid. I've been looking for a new tech project. This is going to be fun. 1 Quote Link to comment
falconexe Posted July 4, 2021 Share Posted July 4, 2021 (edited) The Ultimate UNRAID Dashboard that I developed has all of this S.M.A.R.T disk info, but uses Telegraf. It’s nice to have both options. The UUD displays this in a “pretty” format. Let me know if you need a link to it. ich777 can also point you to it as he is aware of my work. Edit: Here is the link: Edited July 4, 2021 by falconexe 1 1 Quote Link to comment
ich777 Posted July 4, 2021 Author Share Posted July 4, 2021 20 minutes ago, falconexe said: ich777 can also point you to it as he is aware of my work. Yes I am... I want to keep everything in Prometheus if possible... Trying to get the script for smart work reliably and then integrate it into the plugin system. One thing I can tell for sure that this plugin for smartctl needs the node exporter installed and won't work without it but the integration to the node exporter will be handled by the smartctl plugin. I know your Dashboards look really nice and I also want it... But I simply haven't got the time to make Dasboards, Plugins, Dockers,... My goal is simply to stick to Prometheus wherever I can (currently I only have Prometheus and Grafana installed). I know you can't do all with Prometheus but at least I try... Have to look into this when I got more time, eventually when I retire from my real job in about 30 years or so... 1 Quote Link to comment
Airwolf99 Posted July 4, 2021 Share Posted July 4, 2021 (edited) Got this installed and going today, works great so far! Is there any way to add the connected UPS data to this so it can be seen in Grafana? Thanks! Edited July 4, 2021 by Airwolf99 Quote Link to comment
ich777 Posted July 4, 2021 Author Share Posted July 4, 2021 43 minutes ago, Airwolf99 said: Got this installed and going today, works great so far! Is there any way to add the connected UPS data to this so it can be seen in Grafana? Thanks! Will look into this which kind of UPS do you have and how do you connect it to unRAID? Have you installed NUT too? Quote Link to comment
Airwolf99 Posted July 4, 2021 Share Posted July 4, 2021 HI, I currently have a APC brand UPS setup, but also use Cyberpower. It is plugged in via USB and uses the default UPS settings under the settings tab (screenshots attached). Being able to see the Line Voltage, Time left, etc.. would be great. Let me know if you need any other info, Thanks for the quick reply! Quote Link to comment
shadowbert Posted July 5, 2021 Share Posted July 5, 2021 I've installed the pihole exporter - but the web port seems to be hardcoded to 80. Can this be exposed as a setting? This doesn't work for people who run pihole in a docker in bridge mode who use port 80 for other things (such as unraid itself)... Quote Link to comment
unw1red Posted July 5, 2021 Share Posted July 5, 2021 I am running two instances of Pi-Hole on two external servers. Can you monitor both? The one I have setup is not populating any data. Would the port be different for external servers? Quote Link to comment
boosting1bar Posted July 5, 2021 Share Posted July 5, 2021 (edited) So I was going to try switch over from UUD to this, deleted all the old stuff including the Grafana appdata folders etc. When I configure the .yaml then check Prometheus I have the localhost entry as up but the IP:9100 is down (404) and I can't figure out how I've broken it. Point me in the right direction? Edited July 5, 2021 by boosting1bar Quote Link to comment
ich777 Posted July 5, 2021 Author Share Posted July 5, 2021 6 hours ago, Airwolf99 said: APC brand UPS setup I have to look into this, can take a while since I don't own a UPS. 5 hours ago, shadowbert said: hardcoded to 80. No it isn't, you can change it by editing the file /boot/config/plugins/prometheus_pihole_exporter/settings.cfg on your USB boot device and edit the port there. EDIT: I will update the configruation page from the plugin so that the port is displayed/changable from there in one of the next updates. 4 hours ago, unw1red said: I am running two instances of Pi-Hole on two external servers. Are you running the two instances with keepalived? 4 hours ago, unw1red said: Can you monitor both? Theoretically it's possible but you have to put in a line in your go file so that you start a second instance of the exporter on another port. 4 hours ago, unw1red said: The one I have setup is not populating any data. Does it say on the plugin page that it's running and in the Prometheus WebGUI that it's up? 3 hours ago, boosting1bar said: When I configure the .yaml then check Prometheus I have the localhost entry as up but the IP:9100 is down (404) Do you run your containers in a custom network on br0? If yes you need to enable Allow Host Access in the Docker settings. 1 Quote Link to comment
Airwolf99 Posted July 6, 2021 Share Posted July 6, 2021 On 7/5/2021 at 12:52 AM, ich777 said: I have to look into this, can take a while since I don't own a UPS. Okay thanks! If there's any logs I can give you or something that would help just let me know. Thanks! Quote Link to comment
Recommended Posts
Posted by SpencerJ,
1 reaction
Go to this post
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.