lf173, System Administration: Building Scalable ISPs with open-source softwares

This article is available in: English Castellano Deutsch Francais Nederlands Turkce

by Atif Ghaffar

About the author:

Atif is a chameleon. He changes his roles, from System Administrator, to programmer, to teacher, to project manager, to whatever is required to get the job done.
Currently he is working as the Internet Development manager at 4unet and is looking for talented people to hire.
More about him can be found at his homepage

Content:

Why LDAP?
Whats ISPMan?
What will this ISP provide?
Virtual domains
Softwares powering our ISP
Directory Design
LDAP Authentification (No system accounts)
Managing Dns via LDAP
Configuring proftpd
Setting up Cyrus
Setting up Postfix
Managing users with ISPMan
Scaling to more machines
providing web mail to users
Resources
Talkback form for this article

Building Scalable ISPs with open-source softwares

Abstract:

In my last article, I introduced you on using LDAP under Linux. I had many requests from people who were curious to know how LDAP combined with some fine open-source software can make a web mail or web hosting system.

In this Article we will setup a scalable ISP service based on LDAP and Linux. It will touch many of the issues and questions asked. We will then use a tool called ISPMan to manage it.

To avoid the unnecessary length with the textual examples, I have embedded them in textarea widgets. If you really want to print this page, then save it locally and change textarea to pre tags.
Perhaps this perl-liner will do the trick.
perl -pi.bak -e 's!textarea.*?>!pre>!g' filename

Building an ISP and then managing it is tricky, especially if you want to make it mostly automated, highly available and scalable. Running an ISP requires a team of system administrators, dedicated to the running of machines, creating accounts, managing web-sites, trouble shooting, running a help desk etc.
Most of the time, the help desk has little or no control on the setup. Most of the help desks I have come in touch with desperately need help.

Why LDAP?

LDAP is an excellent directory. It can manage not only the username/password of users but much more about users and resources. As we go along we will see how LDAP helps to manage a lot of things centrally.

Whats ISPMan?

ISPMan is the software, I wrote, so people from the IT department dont bug me when they need to create a new domain, set up a new webserver or a change a DNS entry. ISPMan is open source software and is available at http://www.ispman.org. I wont try to explain more about ISPMan itself but the concepts behind it. You can then feel free to try it out and enhance it.

What will this ISP provide?

This ISP will provide DNS, mail, webmail, web hosting etc.

The idea is that a customer comes and wants to host domain "exampledomain.com".
This domain is created centrally by just a few clicks and a lot of magic is managed behind the scenes including setting up DNS, setting up the virtual server for mail and a virtual webserver.
The customer gets one ftp username to access his/her web/ftp server. Any number of users can be created within the domain for email access. Users within the domain may or may not get web space.

There is also the issue of providing internet access. This could be a rather straight forward process or it can get really complicated, so we wont be touching it in this article :).

Virtual domains

Every one wants a website and their own domain's email addresses. Email server have been strangely tied to the system accounts for emails. I dont like it.
The problem arises on multiple levels when you want to manage user1@domain1.com and user1@domain2.ch etc.
There is a lot of unneccesary mappings etc to be done. Hopefully future softwares will be written with domain names in mind.
For example, Cyrus, an excellent IMAP server. Cyrus manages mailboxes (not users). It only allows one mailbox to have a name "aghaffar". If I have another customers from domain "linuxrus.com" who wants a username "aghaffar", then I am out of luck. I would have to create a mailbox by another name and map the user to that mailbox. Another trick would be to create a user "aghaffar.linuxrus.com" but Cyrus used "." as the mailbox delimeter, so cant use that one, neither can I use "aghaffar@linuxrus.com".

Also note that not all of us live in US. A lot of examples in some mailing lists suggest user1@domain1 and user1@domain2. They expect all domain names to end in .com. So we have to keep track of "username" "domain" "TLD"(Top Level Domain).

Our design will take all this into account.

So instead of creating a user called "aghaffar", we will create users in this form. "username_domain_tld".
I dont really remember why I used "_" as the delimeter, but "." was not available because of Cyrus, and there were other problems with other delimiters, for example "&" is a bad one both for shells and URLs.

Softwares powering our ISP

This is a list of software that I was able to use nicely together. You may want to use another software if you wish and if it works for you.

openLDAP (the LDAP server)
Postfix (the LDAP aware and very fast SMTP server)
Cyrus imapd (an excellent IMAP/POP3 server)
Cyrus SASL (authentification library required by Cyrus imapd)
Apache (need I say more)
Proftpd (an excellent LDAP aware ftp server)
Perl (Oh come on, you didn't think I can do this without perl)
ISPMan (a software written on this design to manage ISPs)
Bind (the DNS daddy)
pam_ldap (ldap module for authentification)
IMP 2.2 (an excellent PHP based web mail client)

Directory Design

Our directory is based on domains. There are domains, domain users, domain services etc.
For example, a user can only exist in a domain (except some system users such as LDAP admin, Cyrus admin etc)
domain branch has information about users of that domain, the DNS data of the domain, the http data of the domain etc etc
Example

Here we defined a branch for domain "developer.ch", this branch has sub branches for users, dnsdata, and httpdata.
In this example, we defined a uid, gid, homeDirectory etc for the domain, cause we want only the user "domain.tld" to access via ftp.
For example, if the customer who owns developer.ch wants to upload files to her directory, then she will login as user "developer.ch" and the appropriate password to login to the ftpserver which will hopefully chroot to homeDirectory.. etc .. etc.. more about that later.

For other users, we dont define uid, gid, etc cause we dont want them to login via ftp.
You can find a complete LDIF example file here (this file may be outdated cause its from a production machine, and I have added features etc in the new design) or use this link if you want to show it to your boss (again a bit outdated)

LDAP Authentification (No system accounts)

Another great thing is that we do not have to create any system accounts in /etc/passwd, /etc/shadow etc or manage NIS etc.
All accounts are manged in LDAP and authentification is done directly from LDAP.
For this trick we take a lot of help from PAM (Pluggable Authentification Module) pam_ldap. PAM allows you to define which module should take care of the authentification, authorization etc.

For example, my /etc/pam.d/imap, /etc/pam.d/pop and /etc/pam.d/proftpd files say

#%PAM-1.0
auth       sufficient   /lib/security/pam_ldap.so
account    sufficient   /lib/security/pam_ldap.so

Now all imap/pop3/ftp authentifications are handed to the ldap server. So user aghaffar_developer_ch gets authentificated and gets his mails even though he does not exists on the system's passwd or nis records.

Managing Dns via LDAP

DNS does not at the moment have an LDAP back end, and perhaps its not a good idea to have an LDAP backend for DNS except for Dynamica DNS which is used in conjunction with DHCP. Any way, we use LDAP to store information about DNS and then extract it to create DNS zone files. This allows us to manage everything centrally and from a secure machine, while not making any changes to DNS.

The DNS entries in the LDAP look like this

dn: ou=dnsdata, domain=4unet.net, o=ispman domain: 4unet.net ou: dnsdata objectclass: top objectclass: domainrelatedobject objectclass: posixAccount uid: 4unet.net uidNumber: 2000 gidNumber: 1000 homeDirectory: /home/4unet.net userPassword: {crypt}XXffGGHH loginShell: /bin/true
The dnsdata branch definition. It also defined a posixAccount(a user) with the same name as the domain name. This user is sort of webmaster, who can log in via ftp and upload files to the designated areas etc.

dn: cn=soarecords, ou=dnsdata, domain=4unet.net, o=ispman cn: soarecords primary: ns1.4unet.net ou: dnsdata retry: 1800 rootmail: dnsmaster.4unet.net domain: 4unet.net minimum: 432000 objectclass: top objectclass: domainRelatedObject expire: 1209600 refresh: 21600
This defines the SOA records for the DNS for 4unet.net.
A script is responsible to extract the values and format them into a DNS SOA record to be included in the zone file

dn: cn=nsrecords, ou=dnsdata, domain=4unet.net, o=ispman domain: 4unet.net cn: nsrecords ou: dnsdata objectclass: top objectclass: domainRelatedObject record: @,ns1.4unet.net record: @,ns2.4unet.net
And these are the NS records.
A script will grab all the record attributes, and split them by character "," and get the target and nameserver to be added to the zonefile

dn: cn=mxrecords, ou=dnsdata, domain=4unet.net, o=ispman domain: 4unet.net cn: mxrecords ou: dnsdata objectclass: top objectclass: domainRelatedObject record: @,10, mx1.4unet.net record: @,100, mx2.4unet.net
Same as above, but MX records this time.
These records also contains the priority field which is accordingly adjusted in the MX records of the zonefile

dn: cn=arecords, ou=dnsdata, domain=4unet.net, o=ispman objectclass: top objectclass: domainRelatedObject domain: 4unet.net cn: arecords ou: dnsdata record: ns1, 193.247.80.43 record: ns2, 193.247.80.44 record: @,193.247.80.43 record: @,193.247.80.44
These are the A records, simply host, ip address mappings

dn: cn=cnames, ou=dnsdata, domain=4unet.net, o=ispman objectclass: top objectclass: domainRelatedObject domain: 4unet.net cn: cnames ou: dnsdata record: ftp, www record: mail, www record: *, www
And the CNAMES or Aliases for the hosts

All these scripts are available with the ISPMan package. Have a look at the screenshot to see how to modify a NS record

Configuring proftpd

You need to have proftpd with the LDAP module installed. If you are going to run ftp with virtual servers etc, and you think it will be quiet often busy then you should run it standalone instead of from inetd.
So comment the ftp lines from your inetd.conf and reload the inetd daemon.
Create a group with the gid 1000 called ftponly. We will give this gid to all domains.

edit your /etc/pam.d/proftpd as shown in LDAP authentification section above.
your /etc/proftpd.conf should look something like this.

To run proftpd, you can type /usr/sbin/proftpd and to kill it, killall /usr/sbin/proftpd

Setting up Cyrus

Compile and install Cyrus SASL and imapd, and the UW-IMAP client's c-sdk. The IMAP, client's sdk may already be installed on your system. Try first rpm -aq | grep imap. If in doubt, compile and install a newer version anyway. create a system user called cyrus and group mail, follow the installation instruction and test that you imap server is working. Once it is working simply set /etc/pam.d/imap as shown above in the LDAP authentification section.

you will need to create a user called cyrus or whatever admin your gave to /etc/imapd.conf in the LDAP directory.
For example if your cyrus admin is called "cyrus", then you may have an entry like the one below in your ldap directory.

dn: uid=cyrus, ou=admins, o=ispman
cn: Cyrus Admin
sn: Cyrus
objectclass: top
objectclass: systemadmins
uid: cyrus
userpassword: XXDDCCYY
ou: admins

ISPMan will make these entries for your during setup.

Setting up Postfix

Postfix is very friendly to LDAP. You can make virtual domain and virtual user lookups directly in LDAP so you dont have to specify which domains you should recieve mails for.
For example, if a mail arrives for domain "perl.ch", it should look in all domains and see it there is a domain called "perl.ch", if there is then it should accept the mail instead of returning "Duh MX for perl.ch loops back to myself".

my /etc/postfix/main.cf looks like this

queue_directory = /var/spool/postfix
program_directory = /usr/lib/postfix
command_directory = /usr/sbin
daemon_directory = /usr/lib/postfix
mail_owner = postfix
default_privs = nobody
myhostname = mozilla.developer.ch
mydomain = developer.ch
myorigin = $mydomain
inet_interfaces = all
default_transport = smtp

# the following map gets domain from LDAP to see
# if it shoud accept mails for this domain
mydestination = /etc/postfix/destination, ldap:ldapdomain
ldapdomain_server_host = localhost
ldapdomain_server_port = 389
ldapdomain_bind_dn = o=ispman
ldapdomain_search_base = o=ispman
ldapdomain_query_filter = (&(objectclass=dnsdomain)(domain=%s))
ldapdomain_result_attribute = domain

#these maps make a lookup to ldap to see if there is an
# entry called mailacceptinggeneralid that matched the address.

# for Example, mail comes in for atif@developer.ch, ldap server
# searched for mailacceptinggeneralid=atif@developer.ch
# If it finds it, then it requests the resulting attribute "maildrop"
# this could return a local or remote address.
# in my case it returns atif_developer_ch@mozilla, which means drop the
# mail in the mailbox called "atif_developer_ch" on host "mozilla"
virtual_maps = hash:/etc/postfix/virtual, ldap:ldapvirtual
ldapvirtual_server_host = localhost
ldapvirtual_server_port = 389
ldapvirtual_bind_dn = o=ispman
ldapvirtual_bind = no
ldapvirtual_timeout = 5
ldapvirtual_search_base = o=ispman
ldapvirtual_query_filter = (mailacceptinggeneralid=%s)
ldapvirtual_result_attribute = maildrop
ldapvirtual_lookup_wildcards = no

relocated_maps = hash:/etc/postfix/relocated
smtpd_sender_restrictions = hash:/etc/postfix/access

transport_maps = hash:/etc/postfix/transport

mailbox_transport = cyrus
best_mx_transport = local

virtual_domains=/etc/postfix/virtual
relay_domains = $mydestination, $virtual_domains

smtpd_banner = $myhostname ESMTP $mail_name ($mail_version)

local_destination_concurrency_limit = 2
default_destination_concurrency_limit = 10

Managing users with ISPMan

ISPMan makes it easy to manage users. Creating a user consists of two steps

creating an entry in the LDAP database
creating a mailbox on the IMAP server

Deleting a user, deleted both of these entries for you.

Scaling to more machines

ISPMan allows you to create user's mailbox to any machine in your mailfarm. For example, you may have mail1, mail2, mail3, mail4 etc, each managing 10,000 users. using the LDAP + Postfix + Cyrus combo, you can deliver mail to any of the internal machines.
For example, mail arrives for bob@developer.ch, Postfix asks ldap server to retun the maildrop for the entry that matches mailacceptinggeneralid=bob@developer.ch, and the LDAP server returns bob@mail5.developer.ch. The mail is then routed to the machine called mail5 in your mailfarm.

Currently I am working along with some cool developers on a IMAP/pop3 proxy that runs on the frontend mail servers on pop3 and imap port and transparently redireects requests to internal machines. So your users will only know one address for example mail.developer.ch or pop.developer.ch to connect to instead of knowing which mailserver their mail resides on.

providing web mail to users

IMP is a great product for providing web mail to users.
You can set an alias of mail.* in apache's httpd.conf to point to the central IMP installation.

For example the following is from my installation

I am working on a slightly modified version of IMP that will handle the following cases.

taked short usernames (aghaffar instead of aghaffar_developer_ch) and creates the full username for authentification according to the URL the user came to. For example, user aghaffar if logging to http://mail.developer.ch, then the username becomes aghaffar_developer_ch, and if user atif logs to http://mail.perl.ch then the username == atif_perl_ch.
redirection to internal mailserver according to the response from LDAP
Minor changes in the way usernames are stored in the address book etc to manage multiple hosts on same machine.

thus you can allow all your clients to read emails by simply going to http://mail.their.domain

Resources

PAM LDAP Module: The pam_ldap module provides the means for Solaris and Linux workstations to authenticate against LDAP directories, and to change their passwords in the directory.
openLDAP: The OpenLDAP Project is a collaborative effort to develop a robust, commercial-grade, fully featured, and open source LDAP suite of applications and development tools. The project is managed by a worldwide community of volunteers that use the Internet to communicate, plan, and develop the OpenLDAP Suite and its related documentation.
Postfix: The excellent SMTP server
Cyrus SASL library and imapd: The Cyrus Electronic Mail Project is continuing to build a highly scalable enterprise mail system designed for use in a small to large enterprise environments using standards based technologies. The Cyrus technologies will scale from independent use in small departments to a system centrally managed in a large enterprise.
ISPMan (the Exchange killer): The ISP management software I was plugging in this article
proftpd: A secure ftp server

Talkback form for this article

Every article has its own talkback page. On this page you can submit a comment or look at comments from other readers:

talkback page

Webpages maintained by the LinuxFocus Editor team
© Atif Ghaffar, FDL
LinuxFocus.org
Click here to report a fault or send a comment to LinuxFocus

2001-01-27, generated by lfparser version 2.8

dn: ou=dnsdata, domain=4unet.net, o=ispman domain: 4unet.net ou: dnsdata objectclass: top objectclass: domainrelatedobject objectclass: posixAccount uid: 4unet.net uidNumber: 2000 gidNumber: 1000 homeDirectory: /home/4unet.net userPassword: {crypt}XXffGGHH loginShell: /bin/true	The dnsdata branch definition. It also defined a posixAccount(a user) with the same name as the domain name. This user is sort of webmaster, who can log in via ftp and upload files to the designated areas etc.
dn: cn=soarecords, ou=dnsdata, domain=4unet.net, o=ispman cn: soarecords primary: ns1.4unet.net ou: dnsdata retry: 1800 rootmail: dnsmaster.4unet.net domain: 4unet.net minimum: 432000 objectclass: top objectclass: domainRelatedObject expire: 1209600 refresh: 21600	This defines the SOA records for the DNS for 4unet.net. A script is responsible to extract the values and format them into a DNS SOA record to be included in the zone file
dn: cn=nsrecords, ou=dnsdata, domain=4unet.net, o=ispman domain: 4unet.net cn: nsrecords ou: dnsdata objectclass: top objectclass: domainRelatedObject record: @,ns1.4unet.net record: @,ns2.4unet.net	And these are the NS records. A script will grab all the record attributes, and split them by character "," and get the target and nameserver to be added to the zonefile
dn: cn=mxrecords, ou=dnsdata, domain=4unet.net, o=ispman domain: 4unet.net cn: mxrecords ou: dnsdata objectclass: top objectclass: domainRelatedObject record: @,10, mx1.4unet.net record: @,100, mx2.4unet.net	Same as above, but MX records this time. These records also contains the priority field which is accordingly adjusted in the MX records of the zonefile
dn: cn=arecords, ou=dnsdata, domain=4unet.net, o=ispman objectclass: top objectclass: domainRelatedObject domain: 4unet.net cn: arecords ou: dnsdata record: ns1, 193.247.80.43 record: ns2, 193.247.80.44 record: @,193.247.80.43 record: @,193.247.80.44	These are the A records, simply host, ip address mappings
dn: cn=cnames, ou=dnsdata, domain=4unet.net, o=ispman objectclass: top objectclass: domainRelatedObject domain: 4unet.net cn: cnames ou: dnsdata record: ftp, www record: mail, www record: *, www	And the CNAMES or Aliases for the hosts