Finding an e-mail spammer on your server

Long times ago, spam has been a nuisance that has plagued many internet users. Most people however, think that spam e-mail is being sent out by an actual person behind an actual computer, while in reality, these spammers (ab)use hacked e-mail/server accounts, or security holes within websites. Most server operators don’t even know that spam is being sent from their own server, until they receive complaints that bonafide e-mail from users on their server is not reaching the recipients, or they find out that their server IP address is suddenly located on blacklists. At that point, the hunt for the spammer is on, but how do you actually locate the spammer?
I work at a web hosting company, and see similar questions on a daily basis. A lot of people renting a VPS or dedicated server have no (or very little) actual server management knowledge, and rely mostly on the web hosting control panels (such as DirectAdmin), and the support desk of their server provider. This article offers several ways to find a spammer on your server, and does assume some basic linux knowledge such as how to use the command line (cli/ssh).

Symptoms

• The server may slow down, and use more resources,
• E-mail is not reaching its intended recipients,
• Messages appearing in the DirectAdmin message log about a user having sent X amount of e-mail,
• The Server IP address is located on one or more blacklists,
• A user complains he is receiving a lot of bounce e-mail for e-mail he did not send.

Spam methods

Spammers have a variety of spam delivery methods available:

• SMTP spam,
• Hacked/abused web hosting account,
• Perl script/server

With SMTP spam, the spammer has obtained the credentials of the e-mail account he is abusing. This is done through either brute-forcing (usually not successful if the password is difficult enough), or by malware that is installed on the actual computer(s) of the owner of that e-mail address. This type of malware either comes in the form of a key logger that logs your keystrokes and sends it to a central location, or malware that scans the computer for password files. A lot of people find the options that some (ftp, e-mail) software offer to automatically store your login credentials very convenient, but this is a major security risk that this kind of malware capitalizes on. These files are found, decrypted, and again sent to a central location where spam bots can abuse it. Once they have the login details, they can send out spam.
The hacked/abused web hosting account is also very common. Bots automatically scan websites for known security holes. This is particularly easy with websites running standard software such as WordPress, Joomla, and their plugins. Webmasters tend to not update their software, even though security risks are being discovered and fixed, or they are unaware of the risks. When a hole is identified by a spam bot, it is quickly abused by uploading a file containing malicious code through that hole, and executing it. A similar thing is done when malware locates the ftp credentials on a computer, uploads a file using ftp, and executes it through a web browser. With that latter method, the file is usually removed immediately after execution, making the hunt a bit more difficult.
The perl script/server method will not be covered in this article at this point, but may be at a later date.

Identifying spam

Identifying/locating a spammer is often difficult. Especially if you have a lot of users/domains on your server. There is very little you can do using the DirectAdmin panel, but it is still useful. The first is to open the “Message System”, which is located at the top/right of the standard DirectAdmin template. If you find recent messages like:

Warning: 600 emails have just been sent by

Then that might be a spammer, or someone who has sent out a newsletter.
The other thing you can do within DirectAdmin is to open the “Mail Queue Administration”. If the Mail Queue Administration will not open due to timing out, you can be fairly certain that your server is being abused to send out spam. This means there are so many messages waiting to be sent out, that this page cannot be opened anymore. If however it does open, and you see a large number of pages you can click on, this also means there are a lot of messages waiting in the queue. If you see a lot of the same sender- or recipient addresses in that list, it may be wise to investigate a few of those e-mails by clicking on the mail ID to the right.
The best way to do this however, is by searching the mailserver (exim) logs, which are located in /var/log/exim. The most recent log inside that directory is called “mainlog“. It is possible to search this log using DirectAdmin’s “Log Viewer” which is located on the admin main page, and then selecting “Exim Mainlog“, however these logs can be “very” large, and therefore very slow to dig through using a web interface such as DirectAdmin. It may be more convenient to open the command line interface (ssh), which is the method I will expand on below.
The first thing I do when I login through ssh, suspecting spam, is to see how many e-mail is actually located in the mailqueue waiting to be handled (sent/received). To see this, type:

[root@server]# exim -bpc
1069713

This outputs a summary of the total amount of e-mail that is sitting in the mailqueue, and is totalling over one million in this example. The server in this example is most definitely sending out spam.
Next, move to:

[root@server]# cd /var/log/exim

There, type:

[root@server]# cat mainlog |more

Just “more mainlog” is also possible. If the dates in front of the log lines are old (e.g. much more than a day), then try this:

[root@server]# tail -n 10000 mainlog |more

The 10000 number is the last number of lines to start displaying. You can tweak this around by modifying the numbers. I sometimes have to change it to 150000 or more before I see some sensible data, because when the spamrun has been active for a while, the bounces start to appear very fast, which clutter the log. You can press the space bar to scroll downward through the log. You may find something like:

2013-12-28 23:48:27 1Vx2g2-0001EU-Cq ** source@example.com F=<> R=lookuphost T=remote_smtp: SMTP error from remote mail server after  RCPT TO:: host aspmx.l.google.com [172.28.15.27]: 550-5.1.1 The email account that you tried to reach does not exist. Please try\n550-5.1.1 double-checking the recipient's email address for typos or\n550-5.1.1 unnecessary spaces. Learn more  at\n550 5.1.1 http://support.google.com/mail/bin/answer.py?answer=6596 l2si45202725een.62 - gsmtp
2013-12-28 23:48:27 1Vx2g2-0001EU-Cq Frozen (delivery error message)

Which is a bounce e-mail. You will almost always find bounces in the log, also from bona fide users, but if you see a whole lot for the same destination address on your server (source@example.com in this example), then you can write down that address to scan on that specific address later.
Something else you may find in the log:

2013-12-29 00:00:47 1Vqqlt-0002LF-0d ** target@targetaddress.com F= R=lookuphost T=remote_smtp: SMTP error from remote mail server after initial connection: host smtp.secureserver.net [172.21.10.17]: 554-m1pismtp01-029.prod.mesa1.secureserver.net \n554 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure  is in error, please contact the intended recipient via alternate means.

Lines like this show that you are being blacklisted, and may come in many different forms depending on the provider and type of blacklist. Bona fide mail may generate lines like this if your server IP address is on blacklists.
The following are the type of lines you may be looking for:

Example A

2013-12-27 11:34:51 1VwUkl-0006D4-Co <= dauser@example.com U=apache P=local S=3436 T="R0L3x AND v14rga" from for target@spamtargetaddress.com 2013-12-27 11:34:52 1VwUkl-0006D4-Co => target@spamtargetaddress.com F= R=lookuphost T=remote_smtp S=3576 H=172.11.80.170 [172.11.80.170] C="250 2.0.0 Ok: queued as E19AA22B0173"
2013-12-27 11:34:52 1VwUkl-0006D4-Co Completed

or:

Example B

2013-12-27 20:36:10 1VwdCb-0001ze-5c <= user@example.com H=([172.25.100.95]) [172.25.100.95] P=esmtpa A=login:user@example.com S=100156 id=XREO1FTV-KAX1-NLMI-0XTF-4BPJKR7EII6F@spamtarget.com T="FREE MONEY EVERYDAY - £1 MILLION IN UNDER 7 YEARS" from for target@spamtargetaddress.com

Example A shows “U=apache“, which means that the e-mail was sent using the web server. If the website (or server) is running suphp or mod_ruid2, then the website is not executed as apache, but the username. In that case you may see “U=dauser“, which in this example is the DirectAdmin username for this specific user. “T=” is the e-mail subject, which is usually very obviously spam. However, there are cases where it may look legitimate such as “Scheduled Home Delivery Problem”, but if you find a large number of the same subject types sent by the same user, combined with a lot of bounces, it is likely spam.
Example B shows “P=esmtpa“, which means it was sent using SMTP, which is the exim mailserver. The sender IP is shown as “H=([172.25.100.95]) [172.25.100.95]” and the e-mail account that was used to send it is shown at “A=login:user@example.com“. This is again a very obvious spam subject which you can see in “T=“, and you may find a large number of those in the log, from the same user. However, the part in “from ” may in some cases actually show a completely unrelated and foreign address. The reason for that is that it is very easy to fake a source e-mail address, so the spammed users will not easily know that it came from your user’s  domain name, unless they look at the e-mail headers.
Once you have identified (or suspect) a possible spammer, you could re-scan the log using just the e-mail address, or username (I use tail here to scan the last part of the log, but you can substitute “tail -n 10000” with “cat” to start from the beginning):

[root@server]# tail -n 10000 mainlog |grep user@example.com |more

or

[root@server]# tail -n 10000 mainlog |grep user |more

or to get all the details regarding a specific mail ID:

[root@server]# cat mainlog |grep 1VwdCb-0001ze-5c

or all lines with a specific subject (grep’s -i flag for case-insensitive search):

[root@server]# tail -n 10000 mainlog |grep -i "FREE MONEY EVERYDAY" |more

To show a list of e-mails sitting in the queue you can also do:

[root@server]# exim -bp |more

The |more part pauses the list, which is a must if there are a large number of e-mails waiting. You will probably find many occurrences of the spamming e-mail address in that list. To show some more detailed information, you can query a single e-mail with:

[root@server]# exim -Mvh 1Vx7MK-0007RR-Re

Where the last part is the e-mail ID. This may output something like:

198P Received: from apache by your.servername.com with local (Exim 4.72)
(envelope-from )
id 1Vx7MK-0007RR-Re
for target@targetaddress.com; Sun, 29 Dec 2013 04:48:12 +0100
038  Date: Sun, 29 Dec 2013 04:48:12 +0100
057I Message-Id:
030T To: target@targetaddress.com
051  Subject: Exper tP harma cy
059  X-PHP-Script: www.example.com/components/com_module/x.php for 172.27.17.218
028F From: dauser@example.com

The “from apache” part shows it was sent using the webserver, and the “X-PHP-Script” part shows which script was used to send it. This is especially useful to find the location of the possible spamscript on your server.
If the e-mail was sent using SMTP, the solution is very easy. You can then simply reset the compromised e-mail address’ password and inform the user that his/her credentials were found, and that the user needs to scan his personal computer for malware. If the spam was sent using the webserver, then a temporary solution is to suspend the user’s web hosting account, which will prevent it from adding any new spam e-mails to the mailqueue. You can then try to find the script that is used to send out the spam.

Finding a spamscript

Locating the script that was used to send out the spam can be difficult, especially if the compromised web hosting account has a lot of files. The “X-PHP-Script” shown in the previous example is not always there, and in that case you will have to hunt for it manually. Also, there may be more than one (spam)script, especially if the user is running very outdated software, and has been doing so for quite a while. Such site may be hacked over and over again by many different bots.
One thing you can do is scan for base64 encoded code, which is still very popular. Such code is immediately executed using php’s eval() function. To search for that, go to the user’s public_html (or private_html) directory:

[root@server]# cd /home/username/domains/example.com/public_html

Where username is the username of the DirectAdmin user, and example.com is his/her domain name. Then type:

[root@server]# find . -name '*.php' | while read FILE; do if grep 'eval(base64_decode' "$FILE"; then echo "$FILE" >> maybeinfected; fi ; done

This code snippet searches through all .php files for the “eval(base64_decode” code, and if found, stores the location of that file inside a file with the name “maybeinfected”.  To not store it in a file, but output it to the console instead:

[root@server]# find . -name '*.php' | while read FILE; do if grep 'eval(base64_decode' "$FILE"; then echo "$FILE"; fi ; done

You can substitute “eval(base64_decode” for any other string, such as “eval(gzinflate(base64_decode“, which may also be used.
A more simplified way to search for the same string:

[root@server]# grep -r "eval(base64_decode" . |more

This will basically do the same thing, but also presents you with the whole line of code for every file it finds. The “-r” flag means it is a recursive search which traverses all directories and files in the path you specified. You can also add a “-i” flag for case-insensitive searches.
If you know when the spamming started, you can also check for files that had a status change within the last X days:

[root@server]# find . -type f -user apache -ctime -3 |more

Where “-user apache” reads as “users owned by apache”, which can be removed if you want to search for all owners, and the value after “-ctime” is the number of days ago to start from, which is in this case 3 days. This will produce a file listing that will almost certainly also show innocent files. However, you will probably see a few files that have strange filenames (x.php, 424.php, sys2674123.php, etc), which you may want to investigate further.
You can also substitute the dot (.) in the above examples with a full path name, including a wildcard (*) to search through more directories such as:

/home/*/domains/*/public_html

Where the first * is a wildcard for all users (you can add a username there to narrow the search), and the second wildcard is for all domains.
Note though that removing the infected scripts may only be a temporary solution, because if the user is running outdated software, chances are high to 100% that this will happen again.

Emptying the e-mail queue

You will probably want to empty the mailqueue, so that the thousands (or in some cases millions) of messages that are waiting to be sent or delivered, are no longer being handled. This can be done in several ways:

[root@server]# exim -bp | awk '{ print $3 }' | xargs exim -Mrm

This will empty the complete queue including any bona fide e-mails. Though this may not be what you want, it is the fastest way to empty the queue, especially if there are over a million e-mails inside it. Even though it is the fastest way, it may still take a very long time to complete.
To only delete the spam e-mails:

[root@server]# exiqgrep -i -f dauser@example.com |xargs exim -Mrm

You can also substitute dauser@example.com with just example.com. Another way to remove only the spam e-mails:

[root@server]# exim -bp | awk '// {print $3}' | xargs exim -Mrm

There are more ways to do this than just the above examples. After the operation has completed, you can check how many e-mails are left in the queue:

[root@server]# exim -bpc

If you still see many messages, then chances are that those messages are “frozen”. To remove those:

[root@server]# exipick -zi | xargs exim -Mrm

Sometimes, even then you are left with a large number of e-mails sitting in the mailqueue. Trying to empty the queue may result in an error like:

[root@server]# exim -bp | awk '{ print $3 }' | xargs exim -Mrm
exim: malformed message id  after -Mrm option

This can be fixed with by first obtaining the e-mail ID the removal is stopping at:

[root@server]# exim -bp | exiqgrep -i
Line mismatch: 50h       1VwxAB-0005m5-7R 

And then removing that e-mail from the queue:

[root@server]# exim -Mrm 1VwxAB-0005m5-7R

You may have to do this several times if there are more malformed e-mails in the queue. After that, you can retry emptying the queue.
When all is done, you should not see many messages waiting in the queue anymore.

Conclusion

There are many ways to solve these spam issues, and this is also not a definite guide. One of the things you may also do in case the webserver is used to send spam, is to check the FTP logs to see if the scripts were stored that way, but information about how to do that is not (yet) part of this article.  If, after reading this article, you have any improvements, then let me know so this article can be updated.