Sunday, July 30, 2006

Twisted Mail Server

Update: The saga now has a conclusion! See this blog post for details.

Well, these last two months have been hell. Working on sucky projects just plain sucks. It ate up so much of my time that I've got an unhealthy backlog of blog posts waiting for release into the wild.

The first post I am compelled to write is on my new mail server. In the past, I've used sendmail, qmail and postfix. I have not been happy with any of those (though I did really like qmail, until it got too cumbersome to keep it updated). I didn't have the level of control I wanted -- and this was solely because I couldn't fit my brain into those applications (though, again, qmail came the closest).

I decided to chuck it all, and write my own mail server in twisted, using the pre-built lego code that twisted offers for this sort of thing. I've been running the server for about a month now, and all I can say is "wow". Just WOW.

The level of control I have over the operation of my mail server is insane. I can get this thing to do exactly what I want, when I want. I've got a bazillion domain names for which I (or others) receive email. I was able to write the code that lets me handle that the way that makes sense for ME (and *not* the author(s) of Postfix, etc.).

Today, I needed to add support for aliases that were actually lists of recipients. One "if" statement and an additional implementation of smtp.IMessage later, it was operational. I don't know how I ever ran a mail server any other way.

I've been testing my mail server all month, and it's running beautifully. It has continued to be free of relay issues and spammer attacks. I couldn't be happier with the results.

Now that I am feeling more secure in the custom code, I'm ready to start adding additional features I need:
  • white listing: automatically updated with the address of people to whom I send email
  • black listing: I am starting to maintain a list of the most heinous offenders in my junk mail box; these will be regularly pushed to the mail server
  • greylisting: I have begun planning an implementation of greylisting, but this will take some time to get right
  • spammer databases: I am considering using one or more of these. My only problem with them is that I don't trust them. I don't want to block someone inadvertantly just because they were unlucky enough to have one of their boxes raped into becoming an open relay 3 years ago.
Having a mail server that runs on twisted seemed a little daunting at first. I feared maintenance and security burdens, however, I have already begun reaping the benefits and my fears have been shown to be baseless. I spend 1/10th the maintenance time on the twisted version. I have *fun* while updating the server or configurations. I can plug my own code into it instead of using third party applications I don't like or patches I don't understand.

My first exposure to twisted was late 2002 as I was writing my first "real" python script (a networking script, naturally). Since that time, twisted has integrated itself into my life such that I can't imagine living without it. I literally use it for all of my coding activities: my professional life depends upon it nearly 100%, and 50% of my entertainment is derived from programming activities, all of which incorporate some aspect of twisted.

I, for one, welcome our twisted overlords.

Now playing:
Bagpipes - Flow Gently Sweet Afton


  1. Do you have any plans on opening up the code so that other people can screw around with it?

  2. You bet, man. I'll put it up sometime this week. I'm catching a plane in a couple hours, otherwise I'd do it now ;-)

  3. What fun!

    I would also like to see some code :)

    I made a twisted mail server for an application at my old work a couple of years ago. It was pretty simple to do, and fun.

    I would really like to have a look at what you have, and use it myself if I like it.

  4. Duncan, I haven't tried too terribly hard to find where you might have put this up, but I was wondering if you had opened this up yet. I'm looking at writing a lightweight SMTP relay using twisted.mail but there's a dearth of good twisted-SMTP examples out there. Could you spare a link for a novice?

  5. Don, we keep missing each other on chat -- so sorry! Sadly, I haven't had a chance to do anything with the source code. Perhaps I can blog about the tricky bits I ran into when implementing it...

  6. Anonymous12:19 PM

    hi, duncan. please realease it! it's been over 2 years and I couldn't find it elsewere. I'm interested in writing one myself, and as I'm really new to twisted, having yours surely will help me. I'm even ready to write an article about how to do it.

  7. You should link to some example code - this is really high in google for 'twisted email server'. I'm dying to find out how this story ends :P

  8. Anonymous6:13 AM

    I'm also very interested in the code you used! I'm especially interested in the relay part, I got most of the ESMTP stuff sorted out myself:

  9. pepijndevos,

    Thanks for the message.

    In Jan 2009 I published a followup to this blog post, with notes about how to get the mail server code. You can check it out