[Plug-In] SNMP


Recommended Posts

Thanks for that info @MVLP. I realized I could optimize the way I'm grabbing CPU MHz so I applied that change and stopped using cat, hopefully it won't happen anymore.

 

@cjhammel I also added the change to read the Community string from the snmpd.conf file, so changing the Community should no longer fail the SNMP plugin installation after boot.

 

These changes are a part of 2023.02.19 😄 

  • Like 1
Link to comment
  • 1 month later...
13 minutes ago, irishjd said:

Has anyone seen the issue where the /var/log/snmpd.log fills up the root volume? I can always delete it and restart the service. It would be nice if some logic was added to the plugging where this was done automatically when the volume capacity is less than 20%.

What is it filled with? The file has always been empty whenever I checked or did development on it. 

Link to comment

Its already back up to 25gb.

 

Here are some of the entries:

Received SNMP packet(s) from UDP: [172.17.0.2]:60467->[192.168.100.69]:161
Connection from UDP: [172.17.0.2]:60467->[192.168.100.69]:161
Connection from UDP: [172.17.0.2]:60467->[192.168.100.69]:161
Connection from UDP: [172.17.0.2]:60467->[192.168.100.69]:161
Connection from UDP: [172.17.0.2]:60467->[192.168.100.69]:161
Connection from UDP: [172.17.0.2]:60467->[192.168.100.69]:161

  • Confused 1
Link to comment
On 3/25/2023 at 11:54 PM, irishjd said:

Its already back up to 25gb.

 

Here are some of the entries:

Received SNMP packet(s) from UDP: [172.17.0.2]:60467->[192.168.100.69]:161
Connection from UDP: [172.17.0.2]:60467->[192.168.100.69]:161
Connection from UDP: [172.17.0.2]:60467->[192.168.100.69]:161
Connection from UDP: [172.17.0.2]:60467->[192.168.100.69]:161
Connection from UDP: [172.17.0.2]:60467->[192.168.100.69]:161
Connection from UDP: [172.17.0.2]:60467->[192.168.100.69]:161

I'm seeing the same kind of messages and a frequently large snmp log (100Mb plus) filling up /var/log. Did anyone discover a way to turn off verbose logging?

Link to comment
1 hour ago, stuartcarrison said:

Did anyone discover a way to turn off verbose logging

So more people are having this issue, odd. What's in your logs, the same thing as @irishjd

We were chatting in PMs, but I guess this applicable to more people. I've never had luck with log output.

 

- What version of Unraid are you using?

- Did you apply any customizations to the SNMP config?

- Have you rebooted recently?

- Has uninstalling and reinstalling the SNMP plugin made any difference?


I suspect I know the line that is causing this. Inside the unraid-snmp TXZ file there is the do-inst.sh script that configures and modifies everything. Specifically, there is a line that sets up the logging https://github.com/kubedzero/unraid-snmp/blob/main/source/install/doinst.sh#L58

 

Right now it outputs levels 1=a=alert, 2=c=crit, 3=e=err, 4=w=warn, 5=n=notice into /var/log/snmpd.log. I'd suspect that changing this line to output something else (0-4, 1-4, 1-3, etc) would probably cut down on your log output. That said I've never gotten ANYTHING to show up in /var/log/snmpd.log on my own installations so I've never been able to test. I need reproduction steps to get a better idea of what's happening. 

Link to comment
12 hours ago, kubed_zero said:

- What version of Unraid are you using?

6.11.5

 

- Did you apply any customizations to the SNMP config?

I did not, can attach if desired?

 

- Have you rebooted recently?

Yes, when my log hit 100% on Friday and is now back at 70% already

 

- Has uninstalling and reinstalling the SNMP plugin made any difference?

I haven't attempted

 

 

I noted the config log level recommendation is a later post and I will try that before a remove/reinstall

Link to comment

I was also experiencing this issue and did a little digging and found the following that might be helpful.

 

Some background on my system, this is a long standing install from 2017 and I've only installed stable releases. I tried uninstalling and reinstalling the script as recommended but that did not change anything, the log file would keep growing with the "Connection from UDP:" messages.

 

I look a look at the do-inst.sh script and tried tracing its steps and during install it prompts "Editing SNMP startup options in rc.snmpd to be [-LF 0-5 /var/log/snmpd.log -A -p /var/run/snmpd -a]", but when I take a look at /etc/rc.d/rc.snmpd the variable doesn't appear to be updated and is missing the logging syntax.

#!/bin/sh
#
# rc.snmpd      This shell script takes care of starting and stopping
#               the net-snmp SNMP daemon

[ -r /etc/default/snmpd ] && . /etc/default/snmpd
SNMPD_OPTIONS=${SNMPD_OPTIONS:-"-A -p /var/run/snmpd -a"}

 

I tried updating the rc.snmpd file with the correct options but it kept insisting on using "-A -p /var/run/snmpd -a". I then found that the snmp options are also defined under "/etc/default/snmpd" and if the variable is already set, it will use that and not what is listed under rc.snmpd.

 

cat /etc/default/snmpd
# Options to use when running snmpd:
SNMPD_OPTIONS="-A -p /var/run/snmpd -a"

ps -ef | grep snmp
root      8961     1  0 09:57 ?        00:00:00 /usr/sbin/snmpd -A -p /var/run/snmpd -a -c /etc/snmp/snmpd.conf
root     21364  9181  0 10:13 pts/3    00:00:00 grep snmp

 

So, I went in and edited /etc/default/snmpd and added the missing syntax and voila, problem is fixed for me and the snmpd.log file is no longer growing.

 

cat /etc/default/snmpd
# Options to use when running snmpd:
SNMPD_OPTIONS="-LF 0-5 /var/log/snmpd.log -A -p /var/run/snmpd -a"

ps -ef | grep snmp
root      5131  9181  0 10:32 pts/3    00:00:00 grep snmp
root     24264     1  0 10:17 ?        00:00:00 /usr/sbin/snmpd -LF 0-5 /var/log/snmpd.log -A -p /var/run/snmpd -a -c /etc/snmp/snmpd.conf

 

Hope this helps

  • Like 1
Link to comment
On 4/16/2023 at 12:33 PM, Uncore said:

I look a look at the do-inst.sh script and tried tracing its steps

You didn't need to do this, I explicitly noted this in an earlier message: 

 

 

On 4/2/2023 at 3:29 PM, kubed_zero said:

Specifically, there is a line that sets up the logging https://github.com/kubedzero/unraid-snmp/blob/main/source/install/doinst.sh#L58

 

 

anyway...

 

On 4/16/2023 at 12:33 PM, Uncore said:

when I take a look at /etc/rc.d/rc.snmpd the variable doesn't appear to be updated and is missing the logging syntax

Odd. I'm running 6.9.2 and have no issues:

 

root@Unraid:~# cat /etc/rc.d/rc.snmpd 
#!/bin/sh
#
# rc.snmpd      This shell script takes care of starting and stopping
#               the net-snmp SNMP daemon

OPTIONS="-LF 0-5 /var/log/snmpd.log -A -p /var/run/snmpd -a"

 

root@Unraid:~# ps -ef | grep snmp
root      3744     1  0 Apr15 ?        00:01:35 /usr/sbin/snmpd -LF 0-5 /var/log/snmpd.log -A -p /var/run/snmpd -a -c /etc/snmp/snmpd.conf
root     19665 19643  0 17:33 pts/0    00:00:00 grep snmp

 

 

 

 

 

53 minutes ago, irishjd said:

every time I reboot, the config reverts.

Yes, this is expected. A clean install of SNMP would be:
 

- Uninstall SNMP plugin, and ensure it fully completes and removes all the SNMP-related files in /boot/config/plugins
- Reboot Unraid
- Reinstall SNMP after reboot

 

On 4/16/2023 at 12:33 PM, Uncore said:

the snmpd.log file is no longer growing.


That's great news. That means the issue people are seeing seems to be from SNMP's logging modification not installing properly. Again, odd that I can't reproduce this and haven't seen this myself. What version of Unraid? 

Link to comment

Ah, I haven't tried rebooting since my modifications so good to know it'll likely revert back and reintroduce the issue.

 

The version I'm running is the latest stable 6.11.5. I'm on travel so unable to try your uninstall steps but I'll try that and see if it helps. Any other logs or data points you'd like me to capture?

  • Like 1
Link to comment
On 4/16/2023 at 12:33 PM, Uncore said:

when I take a look at /etc/rc.d/rc.snmpd

 

9 hours ago, Uncore said:

version I'm running is the latest stable 6.11.5


OK, I've figured out the issue and will have a fix out soon. The newest version of the SNMP plugin (2023.02.19) declares a dependency on net-snmp-5.9.3-x86_64-1.txz. Meanwhile the old version (2021.05.21) uses net-snmp-5.9-x86_64-1.txz. 

The problem is that the net-snmp package differs in its /etc/rc.d/rc.snmpd file from 5.9 to 5.9.3, so the do-inst.sh file in my own unraid-snmp plugin fails to modify the file to adjust the logging. 

 

Old /etc/rc.d/rc.snmpd snippet:

OPTIONS="-A -p /var/run/snmpd -a"

start() {
	if [ -x /usr/sbin/snmpd -a -f /etc/snmp/snmpd.conf ]; then
		echo -n "Starting snmpd: "
		/usr/sbin/snmpd $OPTIONS -c /etc/snmp/snmpd.conf
		echo " /usr/sbin/snmpd $OPTIONS -c /etc/snmp/snmpd.conf"
	fi
}

 

New /etc/rc.d/rc.snmpd snippet:

[ -r /etc/default/snmpd ] && . /etc/default/snmpd
SNMPD_OPTIONS=${SNMPD_OPTIONS:-"-A -p /var/run/snmpd -a"}

start() {
	if [ -x /usr/sbin/snmpd -a -f /etc/snmp/snmpd.conf ]; then
		echo -n "Starting snmpd: "
		/usr/sbin/snmpd $SNMPD_OPTIONS -c /etc/snmp/snmpd.conf
		echo " /usr/sbin/snmpd $SNMPD_OPTIONS -c /etc/snmp/snmpd.conf"
	fi
}



My modification code, compatible with 5.9 but not 5.9.3:

    new_flags="-LF 0-5 /var/log/snmpd.log "
    options=$(grep "OPTIONS=" /etc/rc.d/rc.snmpd | cut -d'"' -f 2)
    if [[ $options != *"-L"* ]]; then
        options=$new_flags$options
        echo "Editing SNMP startup options in rc.snmpd to be [$options]"
        sed --in-place=.bak --expression "s|^OPTIONS=.*|OPTIONS=\"$options\"|" /etc/rc.d/rc.snmpd
    else
        echo "SNMP logging flag already present in rc.snmpd, skipping modification"
    fi


I need to change this code to something that can handle the new syntax. Some thoughts I'll be exploring:

  • I saw /etc/default/snmpd included in the new SNMP package but not the old, and perhaps that could be used
  • I need to check the syntax, but I might be able to just set an environment variable SNMPD_OPTIONS as the install script now seems to merge whatever it's set to with the values declared in /etc/rc.d/rc.snmpd
  • I could update my script to just do the text modification inside the /etc/rc.d/rc.snmpd file, the same as it was before but with updated find and replace logic

 

Hopefully I'll have a plugin update out in the next week. Thanks @Uncore for the details!

Link to comment

Ah, I found the true root cause of everyone's issue. https://linux.die.net/man/8/snmpd notes the different flags. Looking more closely at the output from ps, I see the default settings for snmpd are to call with the -a flag which "Logs the source addresses of incoming requests" hence the huge log files.

 

root@Test:~# ps -ef | grep snmp
root      2070     1  0 18:54 ?        00:00:00 /usr/sbin/snmpd -A -p /var/run/snmpd -a -c /etc/snmp/snmpd.conf
root      2933  2715  0 18:55 pts/0    00:00:00 grep snmp


So the root cause is twofold: I didn't have the doinst.sh script built in a way that was compatible with SNMP 5.9.3, and this allowed the default flag to log all addresses and fill up people's disks. 

This is fixed in the version I just released, 2023.04.20 (blaze it). https://github.com/kubedzero/unraid-snmp/releases/tag/2023.04.20 Now the ps output should look the same as what I have below:

 

root@Test:~# ps -ef | grep snmp
root      4749     1  0 19:20 ?        00:00:00 /usr/sbin/snmpd -A -p /var/run/snmpd -LF 0-5 /var/log/snmpd.log -c /etc/snmp/snmpd.conf
root      5247  5167  0 19:21 pts/0    00:00:00 grep snmp


One can also verify the correct behavior by looking at the update log output in the plugin update popup window.

This is wrong, because the desired flags don't match the actual flags:

Editing SNMP startup options in rc.snmpd to be [-LF 0-5 /var/log/snmpd.log -A -p /var/run/snmpd -a]
Restart SNMP daemon now that we've adjusted how rc.snmpd starts it
Starting snmpd: /usr/sbin/snmpd -A -p /var/run/snmpd -a -c /etc/snmp/snmpd.conf


This is the correct output:

Writing extra SNMPD_OPTIONS into /etc/default/snmpd to configure logging
Start SNMP daemon back up now that snmpd.conf and /etc/default/snmpd modifications are done
Starting snmpd: /usr/sbin/snmpd -A -p /var/run/snmpd -LF 0-5 /var/log/snmpd.log -c /etc/snmp/snmpd.conf


Thanks all and let me know if you have issues!

  • Like 1
Link to comment
  • 1 month later...

What, if my output is like this?:

 

root@Raider:~# ps -ef | grep snmp
root      4301  4204  0 08:18 pts/70   00:00:00 grep snmp
root      7046     1  0 Jun02 ?        00:05:27 /usr/sbin/snmpd -A -p /var/run/snmpd -a -c /etc/snmp/snmpd.conf

 

According the plugin info i have already 2023.04.20 installed

 

Thank you,

faxxe

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.