Monitoring Spamd with MRTG

Concept

Like many users of spamassassin, we found that we had to offload spam processing to a separate machine (using spamc/spamd) for performance reasons. Being a stats freak, I wanted to use our existing MRTG system to follow spamd stats. I had already discovered the mailstats program for obtaining information about sendmail, so I decided to come up with a similar approach to spamassassin.

There are several steps you'll need to take, but it's fairly simple. Instructions assume that you're running MRTG and spamd on separate machines, since that's how I'm running it.
 

Define custom port

On both the spamd server and on your MRTG you'll need to add the following (or similar) to /etc/services:

 spam-stats      7300/tcp                        # spam-stats

spamd machine

Install spam-stats.p_ on the spamd machine. This program will parse through your log file, and develop several totals. You will need to create a "/var/local/spam-stats/" directory, or change the location of $statfile and $offsetfile in order to allow spam-stats.l to store its state information between runs.

UPDATE: (7/30/03) I finally got around to writing a perl version. The old bash script is still available (spam-stats) if anyone cares. The old version is a simple bash script using wc. This version requires logtail, and will also need to create some state files (logtail's offset, and a dbm file for the stats). DBM is overkill for the ammount of data, but it was cleaner than parsing and updating a text file.

Then you'll need to get access to spam-stats through the network. Add the following line to /etc/inetd.conf on the spamd machine:

spam-stats      stream  tcp     nowait  root    /usr/local/sbin/spam-stats.pl counts

...and restart inetd
 

For xinet users, Paul Fries paul@cwie.net sent me a copy of his xinetd.d file. A copy can be found at xinetd.d. Thanks for the suggestion, Paul. It took me almost a year to post it, but the info is finally up here.
 

MRTG machine: parser program

Install spamstats.p_ in /usr/local/mrtg/spamstats.pl (or wherever you prefer) on your MRTG station. This program will connect to the spamd machine and print the appropriate results, in the format that MRTG expects to see.

UPDATE: (3/8/09) reamed the .pl files to ".p_". This directory used to offer up .pl files as plain text, but it looks like my ISP wants to execute all .pl files now.

UPDATE: (7/30/03) spamstats.pl has been modified to include uptime and server name.
 

MRTG machine: MRTG settings

Here are some suggested MRTG config file settings. You can also add the gauge option if you'd like to see the rate of spam processing. For some reason, I'm currently most interested in seeing totals. I also have things set to see relative percentage of spam/total processed, nonspam/total processed, and processed/total submitted messages.

#---------------------------------------------------------------
#
# show number of messages that were spam, total number of
#   messages _processed_ by spamd
#
Target[spam_spam]: `/usr/local/mrtg/spamstats.pl host.name.here spam`
MaxBytes[spam_spam]: 100000
Options[spam_spam]: growright,nopercent,perminute
Title[spam_spam]: Spam Statistics
PageTop[spam_spam]: SA-identified spam, SA-processed email </H1>
XSize[spam_spam]: 500
YSize[spam_spam]: 200
WithPeak[spam_spam]: dwmy
YLegend[spam_spam]: No. of messages
ShortLegend[spam_spam]: messages
LegendI[spam_spam]: &nbsp;spam:
LegendO[spam_spam]: &nbsp;total:

#---------------------------------------------------------------
#
# show number of messages that were not spam and total number of
#   messages _processed_ by spamd
#
Target[spam_clean]: `/usr/local/mrtg/spamstats.pl host.name.here nonspam`
MaxBytes[spam_clean]: 100000
Options[spam_clean]: growright,nopercent,perminute
Title[spam_clean]: Spam Statistics
PageTop[spam_clean]: SA-identified clean, SA-processed email</H1>
XSize[spam_clean]: 500
YSize[spam_clean]: 200
WithPeak[spam_clean]: dwmy
YLegend[spam_clean]: No. of messages
ShortLegend[spam_clean]: messages
LegendI[spam_clean]: &nbsp;nonspam:
LegendO[spam_clean]: &nbsp;total:
 

#---------------------------------------------------------------
#
# show number of messages skipped (large messages) total # _submitted_
Target[spam_skipped]: `/usr/local/mrtg/spamstats.pl host.name.here skipped`
MaxBytes[spam_skipped]: 100000
Options[spam_skipped]: growright,nopercent,perminute
Title[spam_skipped]: Spam Statistics
PageTop[spam_skipped]: Messages skipped due to large size, total messages submitted</H1>
XSize[spam_skipped]: 500
YSize[spam_skipped]: 200
WithPeak[spam_skipped]: dwmy
YLegend[spam_skipped]: No. of messages
ShortLegend[spam_skipped]: messages
LegendI[spam_skipped]: &nbsp;skipped:
LegendO[spam_skipped]: &nbsp;total:
 

#---------------------------------------------------------------
#
# show percentage of spam messages
Target[spam_percent_spam]: `/usr/local/mrtg/spamstats.pl host.name.here spam`
MaxBytes[spam_percent_spam]: 100000
Options[spam_percent_spam]: growright,gauge,nopercent,dorelpercent,noi,noo
Title[spam_percent_spam]: Spam Statistics
PageTop[spam_percent_spam]: Percent of messages tagged as spam</H1
XSize[spam_percent_spam]: 500
YSize[spam_percent_spam]: 200
WithPeak[spam_percent_spam]: dwmy
YLegend[spam_percent_spam]: percent of messages
ShortLegend[spam_percent_spam]: messages
LegendI[spam_percent_spam]: &nbsp;skipped:
LegendO[spam_percent_spam]: &nbsp;total:
 

#---------------------------------------------------------------
#
# show percentage of non-spam messages
Target[spam_percent_nonspam]: `/usr/local/mrtg/spamstats.pl host.name.here nonspam`
MaxBytes[spam_percent_nonspam]: 100000
Options[spam_percent_nonspam]: growright,gauge,nopercent,dorelpercent,noi,noo
Title[spam_percent_nonspam]: Spam Statistics
PageTop[spam_percent_nonspam]: Percent of messages tagged as non-spam</H1
XSize[spam_percent_nonspam]: 500
YSize[spam_percent_nonspam]: 200
WithPeak[spam_percent_nonspam]: dwmy
YLegend[spam_percent_nonspam]: percent of messages
ShortLegend[spam_percent_nonspam]: messages
LegendI[spam_percent_nonspam]: &nbsp;skipped:
LegendO[spam_percent_nonspam]: &nbsp;total:
 

Author

Rich Puhek rpuhek@2z.net>

Thanks to:

Tobias Oetiker <oetiker@ee.ethz.ch>
and   Dave Rand <dlr@bungi.com>

...for MRTG itself

Petter Reinholdtsen <pere@td.org.uit.no>

...for the mailstats program that inspired this little idea.
 
last modified: 9/12/03 -- Added note to create /var/local/spam-stats directory.
08/15/2003 -- Added xinetd config info from Paul Fries.
07/30/2003 -- Updated spamstats.pl to include uptime and server name. New spam-stats.pl returns servername and uptime, also uses logtail and a DBM so we don't look at the whole log file all the time.

someday I'll clean up the page format...