About stats – part 2
Last week I wrote about collecting stats. If you collect long enough, you can create some pretty graphs from them.
But you can do more: for instance, you could use these as heartbeats.
Let suppose, one machine stopped sending those collecting email messages. Several problems could be causing this, like the internet connection for that server went down. Or you internet connection went down. Or the email server went down. Anyway, you would like to be notified of such an event, because you would want to look into that.
For that purpose I wrote a cronjob on my central host (the one with the MySQL server), to check when the other servers sent something the last time.
I created a cronjob, that would fire a shell script every five minutes. That shell script would look like this:
#!/bin/sh
HOSTS='validmst04 validmst03 isis muth ra ramses loghost'
CURRENT=`/bin/date +%s`
TENMINUTES=`/bin/expr ${CURRENT} - 600`
for a in ${HOSTS}; do
LASTENTRY=`/usr/local/bin/mysql -BNe \
'select distinct unix_timestamp(date)\
from stats\
where server = "'"${a}"'"\
order by date desc limit 1'`
if [ ${TENMINUTES} -gt ${LASTENTRY} ]; then
if [ -f /tmp/${a}.heartbeat ]; then
touch /tmp/${a}.heartbeat
else
touch /tmp/${a}.heartbeat
echo "No stats received from ${a} since `date -r ${LASTENTRY}`" | mail -s "ALERT: ${a}" peter
fi
else
if [ -f /tmp/${a}.heartbeat ]; then
rm /tmp/${a}.heartbeat
echo "${a} is sending stats again" | mail -s "ALERT END: ${a}" peter
fi
fi
done
The variable HOSTS is filled with all hosts I would like to keep an eye on, and the threshold for a server not sending email I defined as 10 minutes (the variable TENMINUTES substracts 600 seconds from the CURRENT time, which is the number of seconds since January 1, 1970 – date +%s).
In the SQL query, I convert the date of the last entry to seconds (unix_timestamp) and compare this with the value I calculated in TENMINUTES. If the TENMINUTES variable is greater than the last entry in the database for that server, then I send mysql an alert and create a <server>.heartbeat file in /tmp. This file acts as a semaphore: if I would not create this file and test for it, I would receive such an alert every five minutes, as long as my host receives no updates from the collecting host.
If I receive data again, I simply delete the semaphore, and everybody is happy again.
One other problem I’ve encountered was sending the information about the email statistics on my email relays.
For instance, I wanted to know who sends email to whom and when. No problem there. But I also wanted to know which emails were rejected and why.
For rejecting email I use two different methods: the first are DNS based blacklists, which are fairly effective in blocking spam, the other one is the virus scanner (ClamAV) together with MimeDefang. In addition to the normal ClamAV virus definitions, I use the signature files from SaneSecurity in otder to filter spam. In order to make some great stats I needed the sending address, the recipient address, the sending host (IP address), the date/time this message had been delivered, and of course the reason why it had been blocked (or put into quarantaine), thus the ‘virus name’.
These statistics are collected from the maillogs on a daily bases, with this script:
#!/usr/bin/awk -f
#
BEGIN {
FS = ","
}
{
split($1, ve, " ");
if (ve[6] == "MDLOG" && ($3 == "virus" || $3 == "modify" || $3 == "bad_filename" || $3 == "really_bad_filename")) {
printf("%s %s %s;%s;%s;%s;%s;%s;%s;%s\n",ve[2],ve[1],ve[3],ve[4],$3,$4,$2,$5,$6,$7);
}
}
END {}
MimeDefang has a very distinct way of logging, therefore searching the logs for viruses has been made very easy. This script is run on the logging of the yesterdays’ maillog and redirected into /tmp/vrschk-$TODAY (where $TODAY is substituted by the current date).
Because I wanted to mail the results every day to myself, I had to embed them into an email. Here comes the problem.
Some of the signatures look into the headers searching for domain names, and based on the result the email is quarantained. But since I wanted the sending address in my results list as well, the statistics mail would get quarantained as well.
The solution was simple: just encrypt the message, and decrypt it again on my server.
Sending:
/usr/sbin/sendmail -t <<EOF
From: some-email-address
To: another-email-address
X-Process: SECRETKEY-2
Subject: Validmst04 Viruses
`cat /tmp/vrschk-$TODAY | crypt aaa | uuencode testfile`
EOF
In this script I attach the contents of the result of the previous script into an email message, just after I encrypted it with the crypt command (and key ‘aaa’, but that doesn’t really matter, as long as it matches the key the attachment is decrypted with).
The reason for the encryption isn’t security (therefore I use crypt to encrypt it), but to prevent the ClamAV from recognizing the email as spam.
In this email I didn’t use a ‘KEY=SECRETKEY’ construction in the body, but advanced to a better solution and put the identifier in the head of the email. The effect is the same, procmail (who’s processing this email) can make decisions based on keys in the header or in the body, whatever you like. But if it’s in the header it makes processing of the body easier.
This is the promail recipe:
:0 bw:*
^X-Process: SECRETKEY-2
| formail -I "" | uudecode -o /dev/stdout| crypt aaa| grep -v "^$" >>/home/peter/development/programm/php/virus
I search for the X-Process header, and if it matches, I process the body.
‘formail -I “”‘ removes all header, ‘uudecode -o /dev/stdout’ undoes the ‘uuencode’ in the sending script, ‘crypt aaa’ decrypts the contents, and ‘grep’ removes all empty lines and puts the results into a file called /home/peter/development/programm/php/virus. I could have make the contents disappear automatically in a table in my MySQL database, but I chose to do this manually by a php script. In the beginning I always want to have complete control, therefore I do things manually, and for some reason I never changed this to automatic processing.
Looks like a new project 
.
This entry was posted on Monday, June 22nd, 2009 at 8:00 am and is filed under freebsd. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
Leave a Reply