If your server is sending spam, most likely you've already ended up on an RBL blacklist.
The first thing you need to do is to find out who is spamming and stop them.
There are several ways they could be sending spam:
- they've figured out an email password and are sending remotely
- they are sending from a script on the server (insecure script)
- they've created a cronjob to send emails in the background
1) Make sure that you have a send limit enabled
echo 200 > /etc/virtual/limit
This
will put a daily cap on the number of emails a DA user can send. This
is the sum of all their email account's' sends totaled together. http://help.directadmin.com/item.php?id=81
This also requires that you've got the latest version of the /etc/exim.pl
2) Check /etc/virtual/usage to see which User has a
higher-than-average bytes log. If any User stands out, then they're
likely it.
3) If you have the latest exim.pl, and DA 1.41.1 or newer, you should be able to log into DA as that User, and go to:
User Level -> E-Mail Accounts -> E-Mail Usage
which will give you a better breakdown of who is sending what, and to where.
4) Go to:
Admin Level -> Mail Queue Admin
If spam is being sent, there are usually going to be messages stuck in the Queue.
Click one of these messages (the ID) and check the contents (to ensure
it's spam), and the headers of the messages to try and get info about
who sent it (IP), where it's going, and if there are any
X-PHP-Originating-Script headers, telling you wish script did the
sending.
5) Check to see if there are any repeated smpt-auth values:
cd /var/log/exim
grep 'A=login:' mainlog* | less
although, step 3 above should tell you what login was used, if one was used.
7) Although not related to the prevention of sending spam, if you
use DKIM with your email sends, it will both lower your spam score on
remote boxes as it will confirm that your emails are indeed from your
server, and not spoofed from elsewhere. http://www.directadmin.com/features.php?id=1189
8) Newer setups should have php-mail.log files, to log all php sends using the mail() function. Check the log:
Long times ago, spam has been a nuisance that has plagued many
internet users. Most people however, think that spam e-mail is being
sent out by an actual person behind an actual computer, while in
reality, these spammers (ab)use hacked e-mail/server accounts, or
security holes within websites. Most server operators don’t even know
that spam is being sent from their own server, until they receive
complaints that bonafide e-mail from users on their server is not
reaching the recipients, or they find out that their server IP address
is suddenly located on blacklists. At that point, the hunt for the
spammer is on, but how do you actually locate the spammer?
I work at a web hosting company, and see similar questions on a daily
basis. A lot of people renting a VPS or dedicated server have no (or
very little) actual server management knowledge, and rely mostly on the
web hosting control panels (such as DirectAdmin), and the support desk
of their server provider. This article offers several ways to find a
spammer on your server, and does assume some basic linux knowledge such
as how to use the command line (cli/ssh).
Symptoms
The server may slow down, and use more resources,
E-mail is not reaching its intended recipients,
Messages appearing in the DirectAdmin message log about a user having sent X amount of e-mail,
The Server IP address is located on one or more blacklists,
A user complains he is receiving a lot of bounce e-mail for e-mail he did not send.
Spam methods
Spammers have a variety of spam delivery methods available:
SMTP spam,
Hacked/abused web hosting account,
Perl script/server
With SMTP spam, the spammer has obtained the credentials of the
e-mail account he is abusing. This is done through either brute-forcing
(usually not successful if the password is difficult enough), or by
malware that is installed on the actual computer(s) of the owner of that
e-mail address. This type of malware either comes in the form of a key
logger that logs your keystrokes and sends it to a central location, or
malware that scans the computer for password files. A lot of people find
the options that some (ftp, e-mail) software offer to automatically
store your login credentials very convenient, but this is a major
security risk that this kind of malware capitalizes on. These files are
found, decrypted, and again sent to a central location where spam bots
can abuse it. Once they have the login details, they can send out spam.
The hacked/abused web hosting account is also very common. Bots
automatically scan websites for known security holes. This is
particularly easy with websites running standard software such as
WordPress, Joomla, and their plugins. Webmasters tend to not update
their software, even though security risks are being discovered and
fixed, or they are unaware of the risks. When a hole is identified by a
spam bot, it is quickly abused by uploading a file containing malicious
code through that hole, and executing it. A similar thing is done when
malware locates the ftp credentials on a computer, uploads a file using
ftp, and executes it through a web browser. With that latter method, the
file is usually removed immediately after execution, making the hunt a
bit more difficult.
The perl script/server method will not be covered in this article at this point, but may be at a later date.
Identifying spam
Identifying/locating a spammer is often difficult. Especially if you
have a lot of users/domains on your server. There is very little you can
do using the DirectAdmin panel, but it is still useful. The first is to
open the “Message System”, which is located at the top/right of the
standard DirectAdmin template. If you find recent messages like:
Warning: 600 emails have just been sent by
Then that might be a spammer, or someone who has sent out a newsletter.
The other thing you can do within DirectAdmin is to open the “Mail
Queue Administration”. If the Mail Queue Administration will not open
due to timing out, you can be fairly certain that your server is being
abused to send out spam. This means there are so many messages waiting
to be sent out, that this page cannot be opened anymore. If however it
does open, and you see a large number of pages you can click on, this
also means there are a lot of messages waiting in the queue. If you see a
lot of the same sender- or recipient addresses in that list, it may be
wise to investigate a few of those e-mails by clicking on the mail ID to
the right.
The best way to do this however, is by searching the mailserver
(exim) logs, which are located in /var/log/exim. The most recent log
inside that directory is called “mainlog“. It is possible to search this log using DirectAdmin’s “Log Viewer” which is located on the admin main page, and then selecting “Exim Mainlog“,
however these logs can be “very” large, and therefore very slow to dig
through using a web interface such as DirectAdmin. It may be more
convenient to open the command line interface (ssh), which is the method
I will expand on below.
The first thing I do when I login through ssh, suspecting spam, is to
see how many e-mail is actually located in the mailqueue waiting to be
handled (sent/received). To see this, type:
[root@server]# exim -bpc
1069713
This outputs a summary of the total amount of e-mail that is sitting
in the mailqueue, and is totalling over one million in this example. The
server in this example is most definitely sending out spam.
Next, move to:
[root@server]# cd /var/log/exim
There, type:
[root@server]# cat mainlog |more
Just “more mainlog” is also possible. If the dates in front of the log lines are old (e.g. much more than a day), then try this:
[root@server]# tail -n 10000 mainlog |more
The 10000 number is the last number of lines to start displaying. You
can tweak this around by modifying the numbers. I sometimes have to
change it to 150000 or more before I see some sensible data, because
when the spamrun has been active for a while, the bounces start to
appear very fast, which clutter the log. You can press the space bar to
scroll downward through the log. You may find something like:
2013-12-28 23:48:27
1Vx2g2-0001EU-Cq ** source@example.com F=<> R=lookuphost
T=remote_smtp: SMTP error from remote mail server after RCPT
TO:: host aspmx.l.google.com [172.28.15.27]:
550-5.1.1 The email account that you tried to reach does not exist.
Please try\n550-5.1.1 double-checking the recipient's email address for
typos or\n550-5.1.1 unnecessary spaces. Learn more at\n550 5.1.1
http://support.google.com/mail/bin/answer.py?answer=6596
l2si45202725een.62 - gsmtp
2013-12-28 23:48:27 1Vx2g2-0001EU-Cq Frozen (delivery error message)
Which is a bounce e-mail. You will almost always find bounces in the
log, also from bona fide users, but if you see a whole lot for the same
destination address on your server (source@example.com in this example),
then you can write down that address to scan on that specific address
later.
Something else you may find in the log:
2013-12-29 00:00:47
1Vqqlt-0002LF-0d ** target@targetaddress.com
F= R=lookuphost T=remote_smtp: SMTP error from
remote mail server after initial connection: host smtp.secureserver.net
[172.21.10.17]: 554-m1pismtp01-029.prod.mesa1.secureserver.net \n554
Your access to this mail system has been rejected due to the sending
MTA's poor reputation. If you believe that this failure is in error,
please contact the intended recipient via alternate means.
Lines like this show that you are being blacklisted, and may come in
many different forms depending on the provider and type of blacklist.
Bona fide mail may generate lines like this if your server IP address is
on blacklists.
The following are the type of lines you may be looking for:
Example A
2013-12-27 11:34:51
1VwUkl-0006D4-Co <= dauser@example.com U=apache P=local S=3436
T="R0L3x AND v14rga" from for
target@spamtargetaddress.com 2013-12-27 11:34:52 1VwUkl-0006D4-Co
=> target@spamtargetaddress.com F=
R=lookuphost T=remote_smtp S=3576 H=172.11.80.170 [172.11.80.170] C="250
2.0.0 Ok: queued as E19AA22B0173"
2013-12-27 11:34:52 1VwUkl-0006D4-Co Completed
or:
Example B
2013-12-27 20:36:10
1VwdCb-0001ze-5c <= user@example.com H=([172.25.100.95])
[172.25.100.95] P=esmtpa A=login:user@example.com S=100156
id=XREO1FTV-KAX1-NLMI-0XTF-4BPJKR7EII6F@spamtarget.com T="FREE MONEY
EVERYDAY - £1 MILLION IN UNDER 7 YEARS" from
for target@spamtargetaddress.com
Example A shows “U=apache“,
which means that the e-mail was sent using the web server. If the
website (or server) is running suphp or mod_ruid2, then the website is
not executed as apache, but the username. In that case you may see “U=dauser“, which in this example is the DirectAdmin username for this specific user. “T=”
is the e-mail subject, which is usually very obviously spam. However,
there are cases where it may look legitimate such as “Scheduled Home
Delivery Problem”, but if you find a large number of the same subject
types sent by the same user, combined with a lot of bounces, it is
likely spam.
Example B shows “P=esmtpa“, which means it was sent using SMTP, which is the exim mailserver. The sender IP is shown as “H=([172.25.100.95]) [172.25.100.95]” and the e-mail account that was used to send it is shown at “A=login:user@example.com“. This is again a very obvious spam subject which you can see in “T=“, and you may find a large number of those in the log, from the same user. However, the part in “from ”
may in some cases actually show a completely unrelated and foreign
address. The reason for that is that it is very easy to fake a source
e-mail address, so the spammed users will not easily know that it came
from your user’s domain name, unless they look at the e-mail headers.
Once you have identified (or suspect) a possible spammer, you could
re-scan the log using just the e-mail address, or username (I use tail
here to scan the last part of the log, but you can substitute “tail -n 10000” with “cat” to start from the beginning):
To show a list of e-mails sitting in the queue you can also do:
[root@server]# exim -bp |more
The |more part pauses the list, which is a must if there are a large
number of e-mails waiting. You will probably find many occurrences of
the spamming e-mail address in that list. To show some more detailed
information, you can query a single e-mail with:
[root@server]# exim -Mvh 1Vx7MK-0007RR-Re
Where the last part is the e-mail ID. This may output something like:
198P Received: from apache by your.servername.com with local (Exim 4.72)
(envelope-from )
id 1Vx7MK-0007RR-Re
for target@targetaddress.com; Sun, 29 Dec 2013 04:48:12 +0100
038 Date: Sun, 29 Dec 2013 04:48:12 +0100
057I Message-Id:
030T To: target@targetaddress.com
051 Subject: Exper tP harma cy
059 X-PHP-Script: www.example.com/components/com_module/x.php for 172.27.17.218
028F From: dauser@example.com
The “from apache” part shows it was sent using the webserver, and the “X-PHP-Script”
part shows which script was used to send it. This is especially useful
to find the location of the possible spamscript on your server.
If the e-mail was sent using SMTP, the solution is very easy. You can
then simply reset the compromised e-mail address’ password and inform
the user that his/her credentials were found, and that the user needs to
scan his personal computer for malware. If the spam was sent using the
webserver, then a temporary solution is to suspend the user’s web
hosting account, which will prevent it from adding any new spam e-mails
to the mailqueue. You can then try to find the script that is used to
send out the spam.
Finding a spamscript
Locating the script that was used to send out the spam can be
difficult, especially if the compromised web hosting account has a lot
of files. The “X-PHP-Script” shown in the previous example is not always
there, and in that case you will have to hunt for it manually. Also,
there may be more than one (spam)script, especially if the user is
running very outdated software, and has been doing so for quite a while.
Such site may be hacked over and over again by many different bots.
One thing you can do is scan for base64 encoded code, which is still
very popular. Such code is immediately executed using php’s eval()
function. To search for that, go to the user’s public_html (or
private_html) directory:
[root@server]# cd /home/username/domains/example.com/public_html
Where username is the username of the DirectAdmin user, and example.com is his/her domain name. Then type:
[root@server]# find . -name '*.php' | while read FILE; do if grep 'eval(base64_decode' "$FILE"; then echo "$FILE" >> maybeinfected; fi ; done
This code snippet searches through all .php files for the “eval(base64_decode”
code, and if found, stores the location of that file inside a file with
the name “maybeinfected”. To not store it in a file, but output it to
the console instead:
[root@server]# find . -name '*.php' | while read FILE; do if grep 'eval(base64_decode' "$FILE"; then echo "$FILE"; fi ; done
You can substitute “eval(base64_decode” for any other string, such as “eval(gzinflate(base64_decode“, which may also be used.
A more simplified way to search for the same string:
This will basically do the same thing, but also presents you with the
whole line of code for every file it finds. The “-r” flag means it is a
recursive search which traverses all directories and files in the path
you specified. You can also add a “-i” flag for case-insensitive
searches.
If you know when the spamming started, you can also check for files that had a status change within the last X days:
[root@server]# find . -type f -user apache -ctime -3 |more
Where “-user apache” reads as “users owned by apache”, which can be removed if you want to search for all owners, and the value after “-ctime”
is the number of days ago to start from, which is in this case 3 days.
This will produce a file listing that will almost certainly also show
innocent files. However, you will probably see a few files that have
strange filenames (x.php, 424.php, sys2674123.php, etc), which you may
want to investigate further.
You can also substitute the dot (.) in the above examples with a full
path name, including a wildcard (*) to search through more directories
such as:
/home/*/domains/*/public_html
Where the first * is a wildcard for all users (you can add a username
there to narrow the search), and the second wildcard is for all
domains.
Note though that removing the infected scripts may only be a
temporary solution, because if the user is running outdated software,
chances are high to 100% that this will happen again.
Emptying the e-mail queue
You will probably want to empty the mailqueue, so that the thousands
(or in some cases millions) of messages that are waiting to be sent or
delivered, are no longer being handled. This can be done in several
ways:
This will empty the complete queue including any bona fide e-mails.
Though this may not be what you want, it is the fastest way to empty the
queue, especially if there are over a million e-mails inside it. Even
though it is the fastest way, it may still take a very long time to
complete.
To only delete the spam e-mails:
There are more ways to do this than just the above examples. After
the operation has completed, you can check how many e-mails are left in
the queue:
[root@server]# exim -bpc
If you still see many messages, then chances are that those messages are “frozen”. To remove those:
[root@server]# exipick -zi | xargs exim -Mrm
Sometimes, even then you are left with a large number of e-mails
sitting in the mailqueue. Trying to empty the queue may result in an
error like:
[root@server]# exim -bp | awk '{ print $3 }' | xargs exim -Mrm
exim: malformed message id after -Mrm option
This can be fixed with by first obtaining the e-mail ID the removal is stopping at:
[root@server]# exim -bp | exiqgrep -i
Line mismatch: 50h 1VwxAB-0005m5-7R
And then removing that e-mail from the queue:
[root@server]# exim -Mrm 1VwxAB-0005m5-7R
You may have to do this several times if there are more malformed
e-mails in the queue. After that, you can retry emptying the queue.
When all is done, you should not see many messages waiting in the queue anymore.
Conclusion
There are many ways to solve these spam issues, and this is also not a
definite guide. One of the things you may also do in case the webserver
is used to send spam, is to check the FTP logs to see if the scripts
were stored that way, but information about how to do that is not (yet)
part of this article. If, after reading this article, you have any
improvements, then let me know so this article can be updated.
Exim mail queue is one of the most important things to monitor in the
hosting server. Web server with many shared hosting has more
possibility to produce a huge numbers of emails in the Exim mail queue.
And if you read those messages especially the old one, you will realize
that most of them are junk or spam bouncing emails.
I did have the same problem and I had followed many suggestions to
handle this problem including tweak the Exim configuration. Although
Exim is supposed to delete email which more than 7 days old, sometimes
for unknown reasons you still can find so many older mail left in your
Exim mail queue.
Exim will try to resend those emails for certain periods. Too many
mails in queue can take a lot of cpu resources, so you need to maintain
the numbers of messages in your Exim mail queue.
Here is the manual but effective way to clean up the old messages in you Exim mail queue:
Login to your server as a root user.
Run the following command:
exiqgrep -o 86400 -i | xargs exim -Mrm
The above command will delete the messages older than 1 day. To
delete email more than one day you can multiply with the number of days
with 86400.
Depends on how many messages in your Exim mail queue, this command
may take a while to complete. If for some reason you find a locking
error then your server might just in the middle of trying to resend
process of those emails. Take a few moments before you try again.
