On December 5th 2007 I could not believe my eyes when I had a look at the log files - someone was trying to send a severe amount of spam to my server. Someone? Yes, "someone" targeted his botnet against a hosted domain and this botnet performed a dictionary attack. So what did I do? What can you do?
First of all let's see what happens in detail - a bad guy (tm) who controls a botnet says he wants to spam a certain domain - now the bots start sending mails to that domain using random (dictionary based) recipients that don't exist on the server, i.e.
- Firmi@domain.com
- Smalleyjducz@domain.com
- Mikk-Smollen@domain.com
- stephaniereka@domain.com
- Kimplin@domain.com
- jensenybl@domain.com
- Kajumova@domain.com
- ...
By running the following command against the mail-log you can see that most of the sender IP's are unique:
cat mail.log | grep NOQUEUE | awk -F[ '{print $3}' | awk -F] '{print $1}' | sort | uniq -c | sort -n
You will also see that most IP's are just contacting the server once.
So the first thing to do is blocking the "top 50" (or top 100) of those IP's using iptables. This helps to filter IP's that are coming back on a regular basis (be sure you don't filter any of your users!) and works, but compared to ~1000 unique IP's that are ever-changing it does not really work out. This manually IP filtering shows up in the graph around 14:45 and 18:00.
You are lucky if you are running a greylisting service, that way ~90% of all mails are rejected the first time and the spam bots never come back to try a second time. This is also visible in the graph - usually there is about 1 mail per minute filtered, during the attack this number went up to 22/minute. What I did was increasing the greylisting time-out time to 3000 seconds (50 minutes) to block the bots even longer. But still some of them tried very often and did not get blocked by manually created iptables rules - they came through to the mail server.
Luckily catch-all was disabled on this domain, so the other 10% that came through have been rejected with a "user not found" message. I would estimate that not a single mail has been delivered to any inbox... so, as no mail hit Spamassassin and ClamAV (which are expensive processes) the load of the machine was still fine and it had reserves left even at the peak of the attack around 18:00.
What else to do? Actually, there is nothing you can do if you have such an attack except waiting until it's over. Next steps would have been disabling the targeted domain in the mail server (which makes the denial of service attack successful) or shutting the service down - which also has an impact on other hosted domains. Both options are only the last resort and should be taken if the mails start coming through to Spamassassin and ClamAV and the server load increases dramatically. But it's better shutting one service down than to loose the whole machine.
So, sit down, have a drink and wait.
The conclusion is that you can't really do anything against such attacks as they are coming from a lot of different hosts - but the spammers will stop by themselves after a while and it's going to be over soon. But you better be prepared - be sure you use a greylisting filter and catch-all is disabled. Also closely monitor the system load and be prepared to shut the service down (inform your users before you do that!).
Three additional comments: the spike at 21:15 was a user sending his evening-bunch of "funny fun mails" to a lot of recipients (he should use Youtube for those videos... *grml*). No user reported an issue with the server, it was responding as usual during the attack. Traffic did not increase rapidly during the attack.
