[Plug-In] SNMP

[Dro]

On 2/28/2022 at 9:24 AM, BigJuanKer said:

Sorry for the delay, I have not been getting notifications of replies to this topic and check in today "just in case"!

 

On the only system I can reboot easily at this time I have now successfully installed the SNMP plugin.  I needed to remove the plugin from the "failed plugins tab", goto /boot/config/plugins and manually delete the snmp directory and all it's contents, reboot and then install the plugin through the app store as normal.  This completed without issue and even gave a good output during the log.  I have not included it as it will not tell you anything.

 

However, on Thor, I have not been able to reboot, but removed the failed plugin, deleted the /boot/config/plugins/snmp directory and then retried installation.  I received the following log:

 

plugin: installing: https://raw.githubusercontent.com/kubedzero/unraid-snmp/main/snmp.plg
plugin: downloading https://raw.githubusercontent.com/kubedzero/unraid-snmp/main/snmp.plg
plugin: downloading: https://raw.githubusercontent.com/kubedzero/unraid-snmp/main/snmp.plg ... done
plugin: downloading: https://raw.githubusercontent.com/kubedzero/unraid-snmp/main/packages/perl-5.32.0-x86_64-1.txz ... done

+==============================================================================
| Skipping package perl-5.32.0-x86_64-1 (already installed)
+==============================================================================

plugin: downloading: https://raw.githubusercontent.com/kubedzero/unraid-snmp/main/packages/libnl-1.1.4-x86_64-3.txz ... done

+==============================================================================
| Skipping package libnl-1.1.4-x86_64-3 (already installed)
+==============================================================================

plugin: downloading: https://raw.githubusercontent.com/kubedzero/unraid-snmp/main/packages/net-snmp-5.9-x86_64-1.txz ... done

+==============================================================================
| Skipping package net-snmp-5.9-x86_64-1 (already installed)
+==============================================================================

plugin: downloading: https://raw.githubusercontent.com/kubedzero/unraid-snmp/main/packages/unraid-snmp-2021.05.21-x86_64-1.txz ... done

+==============================================================================
| Skipping package unraid-snmp-2021.05.21-x86_64-1 (already installed)
+==============================================================================


+==============================================================================
| Testing SNMP by listing mounts, /boot should be present
+==============================================================================

snmpwalk -v 2c localhost -c public hrFSMountPoint
snmpwalk failureCouldn't find /boot mount point. SNMP output:
Timeout: No Response from localhost
plugin: run failed: /bin/bash retval: 1

Updating Support Links



Finished Installing. If the DONE button did not appear, then you will need to click the red X in the top right corner

 

 

I will get a chance to reboot Thor in the next few hours and will report back once I have tested it again.

 

I have now been able to reboot Thor and it also worked.  Deleting the /boot/config/plugins/snmp folder seems to be the key.  Once it downloads EVERYTHING again it seems to work.

 

I have a third server to try but see no reason why I should not have success there also. 

 

Thanks for the help

 

Same issue here i reboot and poof, gone.. I gather from this, delete snmp folder, reinstall, issue is fixed?

 

[BigJuanKer]

On 3/30/2022 at 3:26 AM, Dro said:

 

I gather from this, delete snmp folder, reinstall, issue is fixed?

 

 

Which worked perfectly until the next reboot

 

I'm currently waiting for 6.10 to get out of RC and into stable and then i'll try again.

[Dro]

On 4/2/2022 at 11:13 AM, BigJuanKer said:

 

Which worked perfectly until the next reboot

 

I'm currently waiting for 6.10 to get out of RC and into stable and then i'll try again.

Yep, same. I just reinstall it everytime I need to reboot. Oh well. I use snmp for zabbix. So if I forget it at least reminds me of no snmp data. 

[kubed_zero]

On 3/29/2022 at 10:26 PM, Dro said:

 

Same issue here i reboot and poof, gone.. I gather from this, delete snmp folder, reinstall, issue is fixed?

 

 

On 4/15/2022 at 8:57 PM, Dro said:

Yep, same. I just reinstall it everytime I need to reboot. Oh well. I use snmp for zabbix. So if I forget it at least reminds me of no snmp data. 

 

On 4/2/2022 at 12:13 PM, BigJuanKer said:

 

Which worked perfectly until the next reboot

 

I'm currently waiting for 6.10 to get out of RC and into stable and then i'll try again.

If anyone's able to provide reproduction steps from a clean install of the latest 6.10 RC, let me know and I will investigate. Otherwise, it has been very difficult to evaluate what's going on. I had no issues on a clean install and wasn't able to reproduce, and SimonF also mentioned not being able to reproduce as of 6.10-RC3.

[Squid]

The only place it's going to fail to install would be in this section:

echo ""
echo "+=============================================================================="
echo "| Testing SNMP by listing mounts, /boot should be present"
echo "+=============================================================================="
echo ""

printf "snmpwalk -v 2c localhost -c public hrFSMountPoint\n"
results=$(snmpwalk -v 2c localhost -c public hrFSMountPoint 2>&1) || printf "snmpwalk failure"
if [[ "$results" =~ "/boot" ]]
then
  echo "SNMP appears to be working. Output:"
  echo "$results"
else
  echo "Couldn't find /boot mount point. SNMP output:"
  echo "$results"
  exit 1
fi

and that is because of the exit 1

 

My surmise is that for those users where it fails is because the network is not up and initialized yet, so localhost doesn't actually exist.

 

A plugin cannot make the assumption that the network actually exists or is initialized at plugin installation time, because it really depends upon the hardware, networking gear etc.  

 

For those users who have problems with this a reinstall, the simple way to determine if it's because of networking being down is to simply delete all the .txz's from /boot/config/plugins/snmp but ensure the folder and the .plg still exists in /config/plugins.

 

On a reboot if I'm right it'll fail to download the .txz's

[kubed_zero]

4 minutes ago, Squid said:

A plugin cannot make the assumption that the network actually exists or is initialized at plugin installation time, because it really depends upon the hardware, networking gear etc.  

 

Right, that's why the dependencies are cached to USB on first install, so the SNMP plugin still installs even if the network isn't up on future boots (for example if someone runs pfSense as a VM on their Unraid box).

 

I was under the assumption that localhost/127.0.0.1 would be guaranteed to exist by the point in the boot process that Unraid does plugin initialization, and that regardless of network hardware, localhost would always exist (since I'd expect the WebGUI to also depend on it). Is there documentation that could confirm or deny this?

 

I'm also open to suggestions for how to do a validation that SNMP is functioning correctly without relying on the network, but I don't have any ideas at the moment. 

[Squid]

Like I said, deleting the txz's is the test for why your snmpwalk command fails for those users with the issues.  Hard for me to try and replicate since my network is up and running at plugin install.

 

Maybe 127.0.0.1 since it doesn't require /etc/hosts to be populated yet (if that is what the issue actually is)

[SimonF]

5 hours ago, kubed_zero said:

 

Right, that's why the dependencies are cached to USB on first install, so the SNMP plugin still installs even if the network isn't up on future boots (for example if someone runs pfSense as a VM on their Unraid box).

 

I was under the assumption that localhost/127.0.0.1 would be guaranteed to exist by the point in the boot process that Unraid does plugin initialization, and that regardless of network hardware, localhost would always exist (since I'd expect the WebGUI to also depend on it). Is there documentation that could confirm or deny this?

 

I'm also open to suggestions for how to do a validation that SNMP is functioning correctly without relying on the network, but I don't have any ideas at the moment. 

Happy to look again once back from hols next week. But it would be good for some feeback on use cases that are failing.

[Dro]

For me the issue is not that it won’t install. I have zero issues installing it. It’s just that when I reboot the server the SNMP app is no longer present. I know this because zabbix throws an alert about “no snmp data collection” I go to check and sure enough snmp is missing. I reinstall it, everything comes right back up and I’m good to go. If I reboot again, same issue.  Rinse, repeat. I’m getting no errors on anything that I’m aware of.  I’m on the latest RC

 

Im glad to do any kind of testing or log gathering. Let me know what kind of information is needed. 
 

Thanks

[BigJuanKer]

On my systems, upon reboot, I would find the plugin moved to the "plugin error" section, both in the UI and /config folders on the flash drive.

 

A few days ago I rebuilt the flash on my main storage array.  I hadn't thought to test but seeing notifications about this thread prompted me.  This was a complete clean RC4 install with only the license key used from the old install.  All drives etc have been wiped.  Other than removing an NVidia GPU (and associated drivers), nothing much has changed about the hardware config.

 

In the last hour I installed SNMP on that server and rebooted, it was there and functioning after reboot.  I then added back VLANs and rebooted again, it was there and functioning afterwards.

 

I do not yet have any containers or VMs running on this system and will update if anything I install causes the issue to appear.

 

I have yet to test this on a server that I haven't rebuilt.  I will soon but my other systems are in production and can not be rebooted at the moment.

 

I would post diagnostics but A) there's no point from the working server and B) I want to make sure that the other server still has the issue!

 

Thanks

Edited April 27, 2022 by BigJuanKer
spelling / fat fingers

[autumnwalker]

Having the same issue as Dro. On reboot it's as if the SNMP plugin was never installed / never existed.

[Squid]

11 minutes ago, autumnwalker said:

Having the same issue as Dro. On reboot it's as if the SNMP plugin was never installed / never existed.

Delete the .txz from /config/plugins/snmp (or whatever the folder is)

 

Reboot

 

If the .txz does not re-appear in the folder after rebooting then the author now knows why its failing to reinstall on a reboot.

[kubed_zero]

All right folks, I just updated to 6.10 on my existing install (no Docker or any fancy stuff) and SNMP installed after reboot without any issue. I'll have to give a vanilla install + reboot a shot on a spare flash drive, but I suspect that will yield the same no-issue result. I'll report back if I see anything otherwise. In the meantime, please let me know if reproduction steps are available on a fresh install for this SNMP not installing after reboot issue. 

 

Thanks!

 

On 5/3/2022 at 10:54 AM, autumnwalker said:

Having the same issue as Dro. On reboot it's as if the SNMP plugin was never installed / never existed.

 

[Dro]

On 5/3/2022 at 10:06 AM, Squid said:

Delete the .txz from /config/plugins/snmp (or whatever the folder is)

 

Reboot

 

If the .txz does not re-appear in the folder after rebooting then the author now knows why its failing to reinstall on a reboot.

 

I just performed this step. Deleted the .txz, rebooted, the .txz did NOT comeback.. This is on 6.10 rc3 now.. Latest update...

Only way the .txz is coming back is if I go into apps, search for SNMP, Install.. Then it shows up again and my SNMP works..

[Dro]

Still broken, do we have any update as to why this is occuring? Thanks

[mattie112]

Just to confirm: I just updated my 6.9 to 6.10 a few days ago. No problems at all. Even tried rebooting a couple of times -> works fine.

[kubed_zero]

6 hours ago, Dro said:

Still broken, do we have any update as to why this is occuring? Thanks

@Dro I've been unable to reproduce the issue. I suspect it has something to do with your network configuration messing up the self-test at plugin install time, but I'm not positive. 

Perhaps it would be helpful to fully delete the files associated with SNMP on the boot drive?

- the /boot/config/plugins/snmp folder, which contains an snmp.conf if defined, as well as the .txz files

- the /boot/config/plugins/snmp.plg file

[irishjd]

Unable to install SNMP package. The first time I installed it, it worked fine until I rebooted. Now it shows in the pluggins error tab. I tried to uninstall it and reinstall it, but now it skips all the package downloads saying they are already installed and then it it ends with this:

+==============================================================================
| Testing SNMP by listing mounts, /boot should be present
+==============================================================================

snmpwalk -v 2c localhost -c public hrFSMountPoint
snmpwalk failureCouldn't find /boot mount point. SNMP output:
Timeout: No Response from localhost
plugin: run failed: /bin/bash retval: 1

Updating Support Links

Finished Installing. If the DONE button did not appear, then you will need to click the red X in the top right corner
 

Since UnRAID doesn't think its installed, I can't uninstall it. Is there a way to manually clean all of itup and try again?

 

[kubed_zero]

 

22 minutes ago, irishjd said:

Is there a way to manually clean all of itup and try again?

On 7/6/2022 at 8:09 AM, kubed_zero said:

fully delete the files associated with SNMP on the boot drive?

- the /boot/config/plugins/snmp folder, which contains an snmp.conf if defined, as well as the .txz files

- the /boot/config/plugins/snmp.plg file

Yes! I think this is all you need.

[irishjd]

Thanks, I will give that a shot and see what happens.

[irishjd]

Rats, even after that, I have the same issue. The installer runs through the download and setup, then fails at the snmpwalk, and then UnRAID thinks it isn't installed even though the  files are installed in the /boot/config/plugins directory.

[irishjd]

Pulling image: searxng/searxng:latest

IMAGE ID [692714839]: Pulling from searxng/searxng.
IMAGE ID [213ec9aee27d]: Already exists.
IMAGE ID [9678b253aa07]: Already exists.
IMAGE ID [4f4fb700ef54]: Already exists.
IMAGE ID [c3a321569da3]: Already exists.
IMAGE ID [5bb31db6deee]: Already exists.
IMAGE ID [ef86e400f659]: Pulling fs layer. Downloading 100% of 3 MB. Verifying Checksum. Download complete. Extracting. Pull complete.
IMAGE ID [406100ec900e]: Pulling fs layer. Downloading 100% of 437 KB. Verifying Checksum. Download complete. Extracting. Pull complete.
Status: Downloaded newer image for searxng/searxng:latest

TOTAL DATA PULLED: 3 MB

 

Stopping container: SearXNG

Successfully stopped container 'SearXNG'

 

Removing container: SearXNG

Successfully removed container 'SearXNG'

 

Command:[email protected]:# /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker run -d --name='SearXNG' --net='bridge' -e TZ="America/Chicago" -e HOST_OS="Unraid" -e HOST_HOSTNAME="Unraid-1" -e HOST_CONTAINERNAME="SearXNG" -e 'INSTANCE_NAME'='' -l net.unraid.docker.managed=dockerman -l net.unraid.docker.webui='http://[IP]:[PORT:8080]/' -l net.unraid.docker.icon='https://raw.githubusercontent.com/xthursdayx/docker-templates/master/xthursdayx/images/searxng-icon.png' -p '8080:8080/tcp' -v '/mnt/user/appdata/searxng/':'/etc/searxng':'rw' 'searxng/searxng'

787dd7db9f4423bf6bfc8ac413bb1edbdc10170fd57be8c78180a29416bafcf9

The command finished successfully!

 

Removing orphan image: 73d6407308a2

Successfully removed orphan image '73d6407308a2'

[kubed_zero]

7 minutes ago, irishjd said:

Rats, even after that, I have the same issue. The installer runs through the download and setup, then fails at the snmpwalk, and then UnRAID thinks it isn't installed even though the  files are installed in the /boot/config/plugins directory.

 

My guess, in conjunction with your Docker logs, is that somehow the localhost address isn't configured/accessible. There's some discussion earlier on in the thread, though I don't think we ever collectively got reproduction steps to isolate whether or not this was the case. 

The thinking with this SNMP install script is that localhost should be available since Unraid would just be pinging itself, and then that /boot exists on all Unraid installations and should be good to baseline. If either one of those fails to be true, then the installation script fails. That said, SNMP might have actually been installed, and perhaps these tests aren't passing for some reason. I suspect that Docker networking might be fiddling with localhost, but don't use Docker myself so I've been unable to reproduce.

I'd say you have two options here:
- Try to figure out if localhost is working, or if SNMP is working at all. Mess with Docker networking, install SNMP manually, and just generally deep-dive on your system

- Remove the validation lines from your local copy of the PLG file, which on next install would just skip over the validation and treat it as a successful install. You could strip it all out, or just remove the "exit 1" line https://github.com/kubedzero/unraid-snmp/blob/main/snmp.plg#L195

[irishjd]

Thanks for the comments. I'm going to think about this over a beer or two (Sam Adams Octoberfest is in the stores!), and then try again tomorrow.

[cjhammel]

It looks like the install script only checks with public as the rocommunity string.  I found that if public is not the rocommunity string snmpwalk fails and the plugin fails the check.  After 2 reboots the SNMP plugin uninstalls.  If you have the rocommunity string as public then it will survive the reboot.