Debian Mailserver

Introduction

This is a HowTo to implement a Virtual Mail Server on Debian stable (etch) Distribution.
It is based on the Howto setup a Virtual Mail System with Exim.
So i only will refer on it and describe the specials to implement this concept in a Debian Distribution.

I will describe also some enhancements for using this mailserver with a dynamic IP-adress.
Normally you must use a relay to send emails to big public mailservers like aol.com, yahoo.com or web.de.
But i want also to support a direct delivery of mails to servers who will accept dynamic IP-adresses.
So i created a additional mySQL-table where the domain-names for direct mail delivery can be defined.

Here a overview of the complete system and specially the mail flow:

Changelog

11.12.2007 Initial version
18.12.2007 Added Section Cronjobs and Webmailer Squirrelmail
06.01.2008 Added Fetchmail Installation
12.01.2008 Added alternative Fetchmail Configuration
05.10.2008 Added clamav volatile installation
05.12.2008 Added Autoresponder
11.12.2008 Added configuration files

EXIM

exim runs as User: Debian-exim
/etc/passwd:

Debian-exim:x:100:102::/var/spool/exim4:/bin/false

For exim MTA (v4) daemon with extended features, including exiscan-acl install

apt-get install exim4-daemon-heavy

(Standard installation is exim4-daemon-light)

~# apt-cache search exim | grep scan
 
exim4-daemon-heavy - exim MTA (v4) daemon with extended features, including exiscan-acl
mailscanner - email virus scanner and spam tagger

Attention: Don't install the package sa-exim (Use spamAssassin at SMTP time with the Exim) !

To check the exim-Version type:

~# exim -bV

Exim version 4.63 #1 built 20-Jan-2007 10:42:32
Copyright (c) University of Cambridge 2006
Berkeley DB: Sleepycat Software: Berkeley DB 4.3.29: (September  6, 2005)
Support for: crypteq iconv() IPv6 PAM Perl GnuTLS move_frozen_messages Content_Scanning Old_Demime
Lookups: lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmnz dnsdb dsearch ldap ldapdn ldapm mysql nis nis0 passwd pgsql sqlite
Authenticators: cram_md5 cyrus_sasl plaintext spa
Routers: accept dnslookup ipliteral iplookup manualroute queryprogram redirect
Transports: appendfile/maildir/mailstore/mbx autoreply lmtp pipe smtp
Fixed never_users: 0
Size of off_t: 8
Configuration file is /var/lib/exim4/config.autogenerated

exim4.conf

Globals

Create the file /etc/exim4/exim4.conf by using the example from struction.de
exim4 will then use that file and ignore the file that it generated from the debconf-configuration.

The Database-connection is defines in the form:

hide mysql_servers = YOURDBHOST/YOURDBNAME/YOURDBUSER/YOURDBPASS

The package for the Spamassassin creates the following socket:

spamd_address = /var/run/spamd.sock

The package for clamAV creates the following socket:

av_scanner = clamd:/var/run/clamav/clamd.ctl

When you have a dynamic IP-Adress you must add your relay here:

# 2007-12-05 Karsten:  Added the LAN adresses for relay and define the relay-server
  domainlist relay_to_domains = smtp.relayname.de
  hostlist   relay_from_hosts = 127.0.0.1  : 192.168.0.0/24

Give local LAN-Clients in the subnet 192.168.0.0/24 the possibility to deliver mails.

To test a new configuration for syntactic correctness type:

exim -C /etc/exim4/exim4.conf -bV

Router Konfiguration

For the definition of domains not to relay create the following mySQL-Table:

CREATE TABLE `relay` (
  `id` int(11) NOT NULL auto_increment,
  `domain` varchar(64) NOT NULL COMMENT 'domain for direct delivery',
  `relay` set('YES','NO') NOT NULL default 'NO' COMMENT 'NO = Do not relay emails for this domain',
  `last_modified` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
  PRIMARY KEY  (`id`),
  KEY `domain` (`domain`)
) ENGINE=MyISAM  DEFAULT CHARSET=utf8 COMMENT='relay definitions';

Here are the corresponding changes for the exim4.conf:

begin routers

# 2007-12-05 Karsten
# emails to domains that are marked in the table "relay" are delivered direct
# without relay if the field "relay" is set to "NO"

dnslookup:
  driver = dnslookup
  domains = ${lookup mysql {SELECT GROUP_CONCAT(domain SEPARATOR ':') FROM relay WHERE relay='NO' GROUP BY relay}{$value}}
  transport = remote_smtp
  ignore_target_hosts = 0.0.0.0 : 127.0.0.0/8
  no_more


# 2007-12-05 Karsten
# This is the replacement for the dnslookup before.
# Every external mail that has not to be delivered direct is sent to the SMTP-relay.

relayhost:
  driver = manualroute
  domains = ! +local_domains
  route_list = * smtp.relayname.de
  transport = relay_smtp
  ignore_target_hosts = 0.0.0.0 : 127.0.0.0/8
  no_more

Transports

Add in the sections transports the following:

begin transports

# 2007-12-05 Karsten
# Transport to relay-server

relay_smtp:
  driver = smtp
  hosts_try_auth = smtp.relayname.de
  headers_remove = X-Spam-Report:X-Spam-Bar

Authenticators

Add in the sections authenticators the following:

begin authenticators
...

# 2007-12-05 Karsten
# Authenticating on relay-server

auth_df:
  driver = plaintext
  public_name = PLAIN
  client_send = ^user^password

The client_send defines the user and password for authentication at your mail-account of the relay.

Exim-Documentation

The exim-documentation can be found in /usr/share/doc/exim4-base

zless /usr/share/doc/exim4-base/README.Debian.gz

clamAV

First install the clamAV-packages if needed:

apt-get install clamav clamav-base clamav-daemon clamav-freshclam

Now take the config from struction.de
or edit the example in /etc/clamav/clamd.conf:

LocalSocket /var/run/clamav/clamd.ctl
ArchiveMaxFileSize 32M
StreamMaxLength 32M

Set the rights for the scan of the mails:

chgrp mail /var/spool/exim4
chgrp mail /var/spool/exim4/scan
chmod 750 /var/spool/exim4/scan

And respect this bug:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=250335
Thus clamav can no longer read files from exim's spool folder.
README.Debian also includes a specific explanation of the issue in respect to exim4, with suggestions as to how to fix it.

Fixing the problem involves doing one of the following, assuming you're using the standard exim4 packages (I'd recommend the last option, as would the maintainer given that he does so in README.Debian):

2) Adding `User Debian-exim' to clamav.conf ( -> User clamav Debian-exim )

3) Adding the clamav user to group `Debian-exim', and making sure that clamav.conf contains `User clamav' and `AllowSupplementaryGroups'.

If you don't want to run clamAV or want to deactivate it for test comment the following lines out of exim4.conf:

#  deny message         = This message contains a virus ($malware_name) and is rejected.
#       log_message     = rejected VIRUS ($malware_name) from $sender_address to $recipients
#       condition       = ${if < {$message_size}{20M}}
#       demime          = *
#       malware         = *

Another problem is that clam-av will not be the actual version when you are running debian stable or lenny.
So you should use the package from the debian volatile distribution:

For this you must add to /etc/apt/sources.list

# ClamAV
deb http://ftp.de.debian.org/debian-volatile stable/volatile main
deb http://ftp.de.debian.org/debian-volatile stable-proposed-updates/volatile main

and then you must create a file /etc/apt/preferences with the following content:

Package: *
Pin: release a=stable/volatile
Pin-Priority: 200

Installation of actual clamav now with

apt-get install -t stable/volatile clamav clamav-freshclam

Spamassassin

First install the spamassassin-packages if needed:

apt-get install spamassassin

Edit the config-file in /etc/default/spamassassin:

OPTIONS="--max-children 15 --max-conn-per-child=25 -H -u spamassassin --socketpath=/var/run/spamd.sock --socketowner=spamassassin --socketgroup=spamassassin"

SPAMD_NICELEVEL=5

Autoresponder

I have created an autoresponder for vacation with the following features:

Fully database configurable
Terminated responses with a given start- and end-time
Answers to all mails for a mailaccount and personal answers to senders are possible
Answers to foreign mailaccounts which are fetched over fetchmail are intern possible
Standard protection against mail loops including answering mails only once in 7 days for each sender

You need a additional table for this functionality:

CREATE TABLE `responder` (
  `username` varchar(64) NOT NULL COMMENT 'localpart of email-address',
  `domain` varchar(64) NOT NULL COMMENT 'domain-part of email-address',
  `sender` varchar(64) NOT NULL default '*' COMMENT '* for all or sender adress',
  `activ` enum('NO','YES') NOT NULL default 'NO' COMMENT 'autoresponder is activ',
  `valid_from` datetime NOT NULL default '2000-01-01 00:00:00' COMMENT 'timestamp to response the mails from',
  `valid_to` datetime NOT NULL default '2099-12-31 23:59:59' COMMENT 'timestamp to response the mails up to',
  `subject` varchar(255) NOT NULL COMMENT 'subject of autorespond mail',
  `body` varchar(3000) NOT NULL COMMENT 'body of autorespond mail',
  `last_modified` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
  PRIMARY KEY  (`username`,`domain`,`sender`),
  KEY `activ` (`activ`),
  FULLTEXT KEY `username` (`username`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='autoresponder definitions';

Example entries:

"username";"domain";           "sender";    "activ";"valid_from";        ;"valid_to";           "subject";             "body";          "last_modified"
"foo";     "mydomain.mydyn.de";"*";         "YES";  "2000-01-01 00:00:00";"2008-12-05 23:59:59";"Vacation: Reminder";  "I am not here.";"2008-12-05 16:03:38"
"foo";     "mydomain.mydyn.de";"bar@foo.de";"YES";  "2000-01-01 00:00:00";"2099-12-31 23:59:59";"Vacation: Personally";"Hi and bye.";   "2008-12-05 16:03:19"

If the sender is "*" then mails from every sender is answered for a special mailaccount "foo@mydomain.mydyn.de".
The second entry answers to mails from the sender "bar@foo.de" with a special mailtext.
Generally the autoanswer can be switched on or off with the field "activ", which must contain "YES" or "NO".

The database and logs of responses are saved in a special folder from exim which must be created:

mkdir /var/log/exim4/vacation
chown Debian-exim /var/log/exim4/vacation

The configuration of exim must have two new sections for the router and transport.

Add to the the Router Configuration:
(direct at the begin of the router section)

# 2008-11-22 Karsten
# Check if this eMail has to be autoresponded
# 2008-12-05 Karsten Enhanced Features

mysql_autorespond:
  driver = accept
  no_verify
  no_expn
  senders = "! ^.*-request@.*:\
             ! ^owner-.*@.*:\
             ! ^root@.*:\
             ! ^postmaster@.*:\
             ! ^listmaster@.*:\
             ! ^mailer-daemon@.*"
  condition = ${if eq{} {${lookup mysql{SELECT activ FROM responder WHERE \
              ((username='$local_part' AND domain='$domain') OR (username='$local_part_prefix' AND domain='$local_part_suffix')) \
              AND (sender ='$sender_address' or sender ='$reply_address' or sender ='*') \
              AND activ='YES' AND valid_from <= now() AND valid_to >= now() ORDER BY sender DESC LIMIT 1 }}}{NO}{YES}}
  unseen
  transport = autorespond_mysql

Add to the Transports Configuration:

# 2008-11-22 Karsten
# Check if this eMail has to be autoresponded
# 2008-12-05 Karsten Enhanced Features

autorespond_mysql:
  driver   = autoreply
  log     = /var/log/exim4/vacation/${domain}.${local_part}.log
  once    = /var/log/exim4/vacation/${domain}.${local_part}.once_db
# Message is sent if that number of days has elapsed since a message was last sent to this recipient.
  once_repeat = 7d
#  If original mail should be included uncomment return_message
#  return_message
  to       = ${reply_address}
  from     = ${local_part}@${domain}
  reply_to = "autoresponder-noreply@mydomain.mydyn.de"
  subject  = ${rfc2047:${lookup mysql{SELECT subject FROM responder WHERE \
              ((username='$local_part' AND domain='$domain') OR (username='$local_part_prefix' AND domain='$local_part_suffix')) \
             AND (sender ='$sender_address' or sender ='$reply_address' or sender ='*') \
             AND activ='YES' AND valid_from <= now() AND valid_to >= now() ORDER BY sender DESC LIMIT 1}}}
  text     = ${quote_mysql:${lookup mysql{SELECT body FROM responder WHERE \
              ((username='$local_part' AND domain='$domain') OR (username='$local_part_prefix' AND domain='$local_part_suffix')) \
             AND (sender ='$sender_address' or sender ='$reply_address' or sender ='*') \
             AND activ='YES' AND valid_from <= now() AND valid_to >= now() ORDER BY sender DESC LIMIT 1}}}

There is only one unsolved problem so far:
It is not possible to answer with messages that have multiple lines.
The mailbody must be quoted with "quote_mysql" from the database, so that special characters like "äöü" are not interpreted.
But also inserted linefeeds and returns are converted to "\r\n".

Web-Frontend

I created a little PHP-Webfrontend with phpMyEdit.
You can get this tool at http://www.phpmyedit.org/

First you must create a mySQL-user that has the sufficient rights for full access to the tables:

CREATE USER 'mailadmin'@ 'localhost' IDENTIFIED  BY  '***********';
GRANT  SELECT , INSERT , UPDATE , DELETE  ON  `mail`  .  * TO  'mailadmin'@'localhost';

There is a ready-to-use frontend in the download section:

user.php

The SHA1-function in mySQL does not create the correct results.

So you must generate the SHA1-hash manually for each user with

dovecotpw -s SHA1

relay.php

autoresponder.php

You must alter the access-parameter in each of the files:

// MySQL host name, user name, password, database, and table
$opts['hn'] = 'localhost';
$opts['un'] = 'mailadmin';
$opts['pw'] = 'password';
$opts['db'] = 'mail';

Additionally forms can easily be created with phpMyEditSetup.php.

Maybe a PHP-programmer can create two good forms with user authentification and password-encryption for the user- and autoresponder-table ?

Dovecot

First install the dovecot-packages if needed:

apt-get install dovecot-common dovecot-imapd dovecot-pop3d

If you want to use sieve you have the problem that at this time this is not supported by the debian packages.
Then you must download and compile the actual version of dovecot yourself. You need

v1.0.8 sources
Sieve plugin v1.0.2 for Dovecot v1.0.x's local delivery agent.

You must add the following to dovecot.conf if you want to activate sieve:

protocol lda {
  # If there is no user-specific Sieve-script, global Sieve script is
  # executed if set. (v1.0.1 and older used "global_script_path")
  #sieve_global_path =
  # Support for dynamically loadable plugins. mail_plugins is a space separated
  # list of plugins to load.
  mail_plugins = cmusieve
}

User for dovecot is: dovecot
/etc/passwd:

dovecot:x:114:117:Dovecot mail server,,,:/usr/lib/dovecot:/bin/false

Now look that you have the common user and group mail:
/etc/passwd:

mail:x:8:8:mail:/var/mail:/bin/false

/etc/group:

mail:x:8:dovecot,Debian-exim,karsten

According this you must define in /etc/dovecot/dovecot.conf:

# Valid UID range for users, defaults to 500 and above.
first_valid_uid = 8
last_valid_uid = 8
# Valid GID range for users, defaults to non-root/wheel.
first_valid_gid = 8
last_valid_gid = 8
# UID and GID MUST be set to the correct of user/group 'mail'

# Use this logfile instead of syslog(). /dev/stderr can be used if you want to
# use stderr for logging (ONLY /dev/stderr - otherwise it is closed).
log_path = /var/log/dovecot.log

# For informational messages, use this logfile instead of the default
info_log_path = /var/log/dovecot-info.log

For a correct logrotate you must edit the /etc/logrotate.d/dovecote:

# dovecot SIGUSR1: Re-opens the log files.
/var/log/dovecot*.log {
  missingok
  notifempty
  delaycompress
  sharedscripts
  postrotate
    /bin/kill -USR1 `cat /var/run/dovecot/master.pid 2>/dev/null` 2> /dev/null || true
  endscript
}

fetchmail

First install fetchmail with

apt-get install fetchmail fetchmailconf

(fetchmailconf is optional as tool for configuring fetchmail under X)

Edit in /etc/default/fetchmail

START_DAEMON=yes

Create a file /etc/fetchmailrc with a content described in the next section.
/etc/fetchmailrc should have 600 permissions and be owned by the user fetchmail, so do the following:

touch /etc/fetchmailrc
chmod 600 /etc/fetchmailrc
chown fetchmail /etc/fetchmailrc

Don't forget to add the user fetchmail to the group mail to avoid problems with rights.

If you don't want that fetchmail write his log to the syslog, you must create this directory with the correct rights:

mkdir /var/log/fetchmail
chown fetchmail /var/log/fetchmail
chgrp mail /var/log/fetchmail
chmod 770 /var/log/fetchmail

(In my configuration i use the logfile /var/log/fetchmail/fetchmail.log)

All the options of fetchmail are described in

man fetchmail

Finally you can start fetchmail with

/etc/init.d/fetchmail start

Deliver to Exim

At this time mails where fetched from other (old) Mail-servers and are delivered local to Exim.
Here is an example for a /etc/fetchmailrc

# /etc/fetchmailrc for system-wide daemon mode
# This file must be chmod 0600, owner fetchmail

set daemon 600       # Poll every 10 minutes
set no syslog        # Alternate log
set logfile /var/log/fetchmail/fetchmail.log
set postmaster standarduser   # In case of errors mails are delivered here

set no bouncemail    # avoid loss on 4xx errors
                     # on the other hand, 5xx errors get more dangerous

# Defaults ===============================================================
defaults:
timeout 300
batchlimit 100

##########################################################################
# Accounts to poll
##########################################################################

poll pop.server.net protocol pop3
localdomains my.domainname
user "username" there with password "mypassword"
to "user@domain" here
options fetchall

From http://fetchmail.berlios.de/
Note that fetchmail, until version 6.3.4, did NOT allow full user@domain specifications here, these would never match.
Fetchmail 6.3.5 and newer support user@domain specifications on the left-hand side of a user mapping.

The current fetchmail in the Debian stable is

# fetchmail -V
Dies ist fetchmail Version 6.3.6+NTLM+SDPS+SSL+NLS.

I installed the actual version. When you also must or want to use it, you must download and compile the source yourself.
To do this you also maybe need an additional package:

apt-get install libssl-dev

Then you can compile and install with:

./configure --with-ssl
make
make install

# fetchmail -V
Dies ist fetchmail Version 6.3.8+SSL+NLS.

The original Debian package is installed in /usr/bin/fetchmail (i removed the files), but the standard fetchmail is located in /usr/local/bin/fetchmail.
So you must alter this in /etc/init.d/fetchmail

# Defaults
PATH=/sbin:/bin:/usr/sbin:/usr/local/bin:/usr/bin
DAEMON=/usr/local/bin/fetchmail
USER=fetchmail

Deliver to Dovecot deliver (not working)

When delivering direct to dovecot the virus-scan and spam-functions implemented in exim are not used!
I am fetching mails from an provider who has already done this for the mails.

First we must set the rights that the deliver job can access dovecot.conf, because it runs as user fetchmail.

chgrp mail /etc/dovecot/dovecot.conf
chmod 440 /etc/dovecot/dovecot.conf

-r--r----- 1 dovecot mail 37373 2007-12-18 15:23 dovecot.conf

Normally Dovecot logs everything through its master process, which is running as root.
Deliver however doesn't do this, which means that you need some special configuration for it.
You must add two lines to the /etc/dovecot/dovecot.conf to give alternate logfiles for the deliver job:

protocol lda {
  ...
  log_path = /var/log/fetchmail/dovecot-deliver.log
  info_log_path = /var/log/fetchmail/dovecot-deliver.log
}

auth default {
  ...
  # Authentication socket for /usr/lib/dovecot/deliver
  socket listen {
  # Note that we're setting a master socket. SMTP AUTH for Postfix and Exim uses client sockets.
      master {
          path = /var/run/dovecot/auth-master
          # Auth master socket can be used to look up userdb information for
          # given usernames. This probably isn't very sensitive information
          # for most systems, but still try to restrict the socket access if possible.
          mode = 0600
          user = fetchmail      # User running deliver
          group = mail          # Or alternatively mode 0660 + deliver user in this group
      }
  }
}

But this does not work !!! :-(
deliver does not support prefetch.
/var/log/dovecot.log

dovecot: Jan 06 11:34:39 Error: auth(default): prefetch(user@domain): passdb didn't return userdb entries

http://www.mail-archive.com/dovecot@dovecot.org/msg03210.html
This problem must be solved to get it running ...

/etc/fetchmailrc

# /etc/fetchmailrc for system-wide daemon mode
# This file must be chmod 0600, owner fetchmail

set daemon 600       # Poll every 10 minutes
set no syslog        # Alternate log
set logfile /var/log/fetchmail/fetchmail.log
set postmaster standarduser   # In case of errors mails are delivered here

set no bouncemail    # avoid loss on 4xx errors
                     # on the other hand, 5xx errors get more dangerous

# Defaults ===============================================================
defaults:
timeout 300
batchlimit 100

##########################################################################
# Accounts to poll
##########################################################################

poll pop.server.net protocol pop3
user "username" there with password "mypassword"
mda "/usr/lib/dovecot/deliver -d user@domain"

Delivering mails direct to dovecot:
The Dovecot LDA, called deliver, is a local delivery agent which takes mail from an MTA and delivers it to a user's mailbox, while keeping Dovecot index files up to date.

Cronjobs

Because i had some trouble with the stability of clamAV i put this little script into /etc/cron.hourly

#!/bin/bash
# Cronjob that checks mail-daemons and (re)start them if needed
# 18.12.07 K.Malcher

SOCKET_CLAMAV=/var/run/clamav/clamd.ctl
INIT_CLAMAV=/etc/init.d/clamav-daemon

SOCKET_SPAMAS=/var/run/spamd.sock
INIT_SPAMAS=/etc/init.d/spamassassin

PROC_EXIM=/usr/sbin/exim4
INIT_EXIM=/etc/init.d/exim4

PROC_DOVECOT=/usr/sbin/dovecot
INIT_DOVECOT=/etc/init.d/dovecot


if test -S $SOCKET_CLAMAV; then
        echo "ClamAV O.K."
else
        echo "Starting ClamAV ..."
        $INIT_CLAMAV start
fi

if test -S $SOCKET_SPAMAS; then
        echo "Spamassassin O.K."
else
        echo "Starting Spamassassin ..."
        $INIT_SPAMAS start
fi

if ps -ef | grep $PROC_EXIM | grep -v grep &> /dev/null; then
        echo "Exim O.K."
else
        echo "Starting Exim ..."
        $INIT_EXIM start
fi

if ps -ef | grep $PROC_DOVECOT | grep -v grep &> /dev/null; then
        echo "Dovecot O.K."
else
        echo "Starting Dovecot ..."
        $INIT_EXIM start
fi

Mailclient

Because this mailserver forces the use of TLS-Encryption you must activate it in your external clients.

And because of the virtual addresses think that you must give the complete addresses in the form user@domain

The migration of mails from other POP3 and IMAP accounts can be done with Mozilla-Thunderbird.
There is also a Import-Filter for Outlook-Mails.
Just add all the accounts and then you can simply move the mails with drag and drop.

Webmailer Squirrelmail

To install Squirrelmail use the packages:

apt-get install squirrelmail squirrelmail-locales

For configuration of Squirrelmail use /usr/sbin/squirrelmail-configure

Add Squirrelmail to your Apache-Directory:

ln -s /usr/share/squirrelmail /var/www/squirrelmail

If you want to patch the view of the date to a long date with time in the folder overview, then you must patch

$messages[$msgi]['DATE_STRING'] = getLongDateString($messages[$msgi]['TIME_STAMP']);

in line 709 in the file .../functions/imap_messages.php

Downloads

Configuration Files (zip)

Webfrontend with phpMyEdit (zip)