Twisted Mail Server

Update: The saga now has a conclusion! See this blog post for details.

Well, these last two months have been hell. Working on sucky projects just plain sucks. It ate up so much of my time that I've got an unhealthy backlog of blog posts waiting for release into the wild.

The first post I am compelled to write is on my new mail server. In the past, I've used sendmail, qmail and postfix. I have not been happy with any of those (though I did really like qmail, until it got too cumbersome to keep it updated). I didn't have the level of control I wanted -- and this was solely because I couldn't fit my brain into those applications (though, again, qmail came the closest).

I decided to chuck it all, and write my own mail server in twisted, using the pre-built lego code that twisted offers for this sort of thing. I've been running the server for about a month now, and all I can say is "wow". Just WOW.

The level of control I have over the operation of my mail server is insane. I can get this thing to do exactly what I want, when I want. I've got a bazillion domain names for which I (or others) receive email. I was able to write the code that lets me handle that the way that makes sense for ME (and *not* the author(s) of Postfix, etc.).

Today, I needed to add support for aliases that were actually lists of recipients. One "if" statement and an additional implementation of smtp.IMessage later, it was operational. I don't know how I ever ran a mail server any other way.

I've been testing my mail server all month, and it's running beautifully. It has continued to be free of relay issues and spammer attacks. I couldn't be happier with the results.

Now that I am feeling more secure in the custom code, I'm ready to start adding additional features I need:

• white listing: automatically updated with the address of people to whom I send email
• black listing: I am starting to maintain a list of the most heinous offenders in my junk mail box; these will be regularly pushed to the mail server
• greylisting: I have begun planning an implementation of greylisting, but this will take some time to get right
• spammer databases: I am considering using one or more of these. My only problem with them is that I don't trust them. I don't want to block someone inadvertantly just because they were unlucky enough to have one of their boxes raped into becoming an open relay 3 years ago.

Having a mail server that runs on twisted seemed a little daunting at first. I feared maintenance and security burdens, however, I have already begun reaping the benefits and my fears have been shown to be baseless. I spend 1/10th the maintenance time on the twisted version. I have *fun* while updating the server or configurations. I can plug my own code into it instead of using third party applications I don't like or patches I don't understand.

My first exposure to twisted was late 2002 as I was writing my first "real" python script (a networking script, naturally). Since that time, twisted has integrated itself into my life such that I can't imagine living without it. I literally use it for all of my coding activities: my professional life depends upon it nearly 100%, and 50% of my entertainment is derived from programming activities, all of which incorporate some aspect of twisted.

I, for one, welcome our twisted overlords.

Now playing:

Bagpipes - Flow Gently Sweet Afton

Trac Spam

It seems I was the last to find out about the "trac spam" phenomenon. I've got a bunch of projects that use trac here, and as a result, I've been cleaning up the huge mess these bastards left on the wiki. Gotta love page history, though...

The pages that are most viewed are the ones with code examples on them, and the spammers were kind enough to delete that code, making the pages useless. The code is now restored and can be referenced again. In particular, the following projects were impacted most heavily:

• AmazonDBDriver


• NetCIDR


• PyRRD

I've now got the wiki set for auth users only. If you want to contribute to the wikis, just send me an email.

Technorati Tags: internet, spam, trac

Nevow Needs a Screencast

Nevow needs a screencast. Because it's so damned hot and people need to see it in action. They need to see how frickin' amazing it is.


Case in point: on a *whim* I decided to migrate a site I built last year in z3 to nevow. I had a spare hour, so what the hell. It took me *minutes*. And I don't mean 6487 minutes -- I mean 30 minutes. Here's what I did:

• Copied HTML output from the other site and used as the basis for an XHTML template


• Copied CSS and images from the other site


• Wrote a backend data store that uses RFC 2822 (python's email package)


• Wrote a ui.UI class, subclassing nevow.rend.Page (including children, data calls, etc)


• Added data and render calls in the XHTML template


• Wrote a .tac file


• Populated 5 RFC 2822-compliant files with data and headers for the content


• Started up the server and watched the new site scream

Let me emphasize a crucial point here: *none* of that was stubbed. No helper-scripts. No installers. There was pre-existsing HTML, CSS and images, and that's it. I did some dicking around and tweaking in the process, so I'm *sure* I could do the whole thing -- from start to finish -- in less than 20 minutes. Probably 15.

I don't understand why more developers aren't using Nevow. It just kicks ass. Yeah, every platform has its issues, and Nevow's no exception. But DAMN. I get record-setting development times with it, it runs fast, and it's insanely flexible.

Technorati Tags: divmod, nevow, python, twisted

Sets as an Elegant Alternative to Logic

python :: programming :: math

A few years ago I was working on a sub-project of CoyoteMonitoring
where we were writing a database "layer" for the file system,
essentially allowing us to query for files using standard SQL commands.
We needed to do some exclusionary logic for filtering bad user input.
Though mathematically equivalent, I deal with Set Theory and Logic very
differently: I'm a set guy, and I really hate using series of if/else
statements for problems that lend themselves easily to an approach that
is "set theoretic" (we're talking super-mundane set stuff -- nothing
really sexy).

On IRC, I was gently preparing a fellow project developer for some code
changes I had made that eliminated a series of ugly ifs:

 
[03-Oct-2004 02:02]   you know, we could do this really
quickly with sets :-)
[03-Oct-2004 02:02]   check this out:
[03-Oct-2004 02:02]   lets say our table has 4 fields:
[03-Oct-2004 02:03]   ['A','B','C','D']
[03-Oct-2004 02:03]   and we're doing a select for ['D', 'A', 
'C']
[03-Oct-2004 02:05]   and lets say we've got a bad query 
['Z', 'B', 'C']
[03-Oct-2004 02:07]   then, if we perform a set difference on
select
   tables against defs, the returned set will be zero
[03-Oct-2004 02:07]   and if we do the same with bad against
defs,
   we'll get a non-empty set, in this case a set with one element, 'Z'

I then shared some python 2.3 code (no built-in set object) that
basically illustrated the changes I had made. For me, the results were
much more readable. The code was certainly much shorter.

Since then, I've found all sorts of uses for this approach. I've
recently made use of this in a Nevow project that has to authenticate
off of a Zope instance. There are a series of group and role names that
can be qualified under one of three "group-groups" (tiers):

# setup sets for each class of user
admins = set(cfg.user_database.admin_roles)
libs = set(cfg.user_database.tiertwo_roles)
vols = set(cfg.user_database.tierone_roles)
# and then a convenience collection
legal_roles = admins.union(libs).union(vols)

Later in the code, we do checks like this:

if admins.intersection(avatarRoles):
     return IAdministratorResource

I like this much more than explicitly checking for the presence of
elements in lists.

Baz Camp, Day 1

hacking society :: programming :: colorado

Well, the folks at tummy.com have done
it again :-)
Baz camp is rocking. We're right on
the lake, with beaches and 90-degree weather in Loveland, CO... so you
can imagine that the view is most agreeable :-) We not only have
wireless access donated by the most gracious local provider
LP Broadband, but some of
the boys brought a big-screen projector for outdoor movies at night as
well as satellite TV. So yeah, we're *really* roughing it.

We are certainly roughing it more than the folks at Foo and Bar Camp!

We've got a camp cook who's making meals for us (including 1:00am
chili!) and the menu has been just great. We've got shaded gazebos up,
tents. Watched Primer and Hackers last night (Hackers was enjoyed a la
MST3K). Sean shared Icelandic treats he brought back from the Need for
Speed sprint, including fish jerky. Yummy.

I also happen to be getting a lot of work done, too :-) All offices
should operate like Hack Camp.

Hey Radix: why aren't you here? Evelyn Mitchell and I were talking
about fellow social geeks and agreed that this would be just your style
:-)

Twisted JSON-RPC TCP Proxy and Server

twisted :: python :: internet

Yesterday I finished the Twisted JSON-RPC server and proxy for TCP. I
decided to go with the
Netstring protocol
for its simplicity and the security of declaring and limiting string
length.

Usage is very simple and almost identical to the HTTP-based Twisted
JSON-RPC usage. Here is some server code:

from twisted.application import service, internet
from adytum.twisted import jsonrpc
class Example(jsonrpc.JSONRPC):
   """An example object to be published."""
     def jsonrpc_echo(self, x):
       """Return all passed args."""
       return x
factory = jsonrpc.RPCFactory(Example)
application = service.Application("Example JSON-RPC Server")
jsonrpcServer = internet.TCPServer(7080, factory)
jsonrpcServer.setServiceParent(application)

And for the client:

from twisted.internet import reactor
from twisted.internet import defer
from adytum.twisted.jsonrpc import Proxy
def printValue(value):
   print "Result: %s" % str(value)
def printError(error):
   print 'error', error
def shutDown(data):
   print "Shutting down reactor..."
   reactor.stop()
print "Making remote calls..."
proxy = Proxy('127.0.0.1', 7080)
d = proxy.callRemote('echo', 'hey!')
d.addCallbacks(printValue, printError)
d.addCallback(shutDown)
reactor.run()

For a slightly more complex example (with subhandlers and
introspection) see:
the wiki.

Next on the list? For a truly secure solution, I am exploring the use
of the
Twisted
Perspective Broker for JSON-RPC.

Now playing:

Yoko Kanno & The Seatbelts - Forever Broke

More Twisted and JSON-RPC

twisted :: python :: web

A recent comment on my
Twisted
JSON-RPC blog entry
put this near the top of my queue. I posted a comment in response to
the request for a download link (see my
comments
about that).

Please read those comments before downloading this, so you are aware of
the connection (or lack thereof) to official twisted code, as well as
my reservations on the matter. In addition to providing this link, I
will do the following: create a twisted.web2 version of the JSON-RPC
code this weekend, make an attempt to define a sensible twisted
protocol for JSON-RPC over TCP, and (time-permitting) begin work on a
twisted TCP JSON-RPC server and client. I have created a project space
for this work
here.

Update: There is now a twisted.web2 version of the JSON-RPC server available on the project page (all unit tests pass). There is an example server.tac file you can run as well as an example twisted client. Some time later tonight -- or rather, very early this morning (2006.05.06) -- I will put some eggs up and provide a link on the project page.

Now playing:

Glenn Gould - The Art of Fugue, BWV 1080: Contrapunctus XI (a 4)

Group Access in Twisted

This is a rant -- a positive one. twisted.cred is freaking brilliant. I've had to use it in the past to write my own credential checkers, so I've dabbled a bit. I was thrilled then because of the ease with which I was able to glue systems together. But tonight, I needed to add last-minute support for group access control to a twisted/nevow application and nevow resources that use JSON-RPC. The customer now wants different page views/menus for different classes of user; in addition, they have a new set of RPC

methods that should only be accessible to privileged users.

Typical nightmare situation, when it comes to last-minute tasks, right?

Not with twisted.cred, it isn't. Basically, all I had to do was create an interface for each group that needed to be represented. I then did the following:

• updated the function that instantiates the RPC parent and subhandlers, instantiating the right ones based on the passed interfaces
• updated the avatar realm to choose the correct interface for a given group type
• subclassed the root page for each group that needed a different page

I didn't have to touch the credential checker since it was already getting the group info (I *knew* the customer was going to ask for something like this, even though it wasn't in the reqs).

The interfaces, a few methods (implements/implementer, providedBy), and the amazing functionality provided by twisted.cred -- that's all that was needed. I've never written my own access control code before, and it took less time with cred to actually implement the thing than the "simple" mere configuration that other systems take. Really. It went so quickly and smoothly that I spent the time saved adding some nifty features that take advantage of these changes.

Python-esque Imports in JavaScript

programming :: javascript :: web

Caveat 1: It has been several years since I've been forced to work with
JavaScript. Caveat 2: This has either been done before and considered
stupid or done before and done better. If so, I'm sure I'll hear about
it :-)

And before I get started, I've been meaning to blog about
MochiKit. I've got all sorts of good
things to say about it, but I'm so busy using it that I'm not sure I'll
ever get around to it. Among the many reasons that people rave about
MochiKit, let me say that the first and foremost should be what the
MochiKit team has done most for JavaScript: coding convention. Even
though there are lots of stylistic inconsistencies in the various
MochiKit libraries, compared to JavaScript in the wild, it is a
completely unified whole. Wild JavaScript is generally pure shite.

Back to the topic: the project I am working on right now has multiple
"screens" that are loaded depending on user interaction. Pretty common
fare. But I really didn't want to load all the DOM manipulation stuff
in a series of *.js files on page load. There are WAY to many files for
this. So I wrote importLibrary().

My first attempt at writing this function was to append script tags to
HEAD with the proper source information in it. Which works...
eventually -- the js source just isn't available immediately. So here's
what I did:

     req = getXMLHttpRequest();
     req.open('GET', file, false);
     req.send('');
     js = req.responseText;
     eval(js);

Now, as far as I know, there is no reason to object to the "eval()"
call here, since this is just the same thing (again, as far as I know)
as when your browser downloads the file by itself. This filename is not
parsed from a URL or derived from any human input.

In addition, I wrote a little function for parsing the import
parameters so that

importLibrary('MyProject.ThisSection.ThisScreen')

maps to a file available at an arbitrary (pre-determined per-project)
location off of docroot on the web server.

The import itself has to be done synchronously to ensure that JS code
is available to everything after the called to import. Right now, I
have it doing simple checking to see if the js file has already been
imported, and if so, skips it. I would like to add some kind of "use
queue" as well, where most frequently clicked screens are bumped to the
top (or bottom), and those least visited are pushed off the queue and
then not maintained in HEAD.

This has really enabled me to organize the code for the project in a
sane way.

Now playing:

Glenn Gould - The Art of Fugue, BWV 1080: Contrapunctus IX (a 4, alla Duodecima)

The Joys of IRC

twisted :: python :: internet

And in particular, #twisted. Here's a great little story (and moral)
from Moshe Zadka:

[13:41]<moshez> dreid: did I tell you about the watchdog
[13:41]<dreid> moshez: no
[13:42]<moshez> dreid: oh, man it is awesome
[13:42]<moshez> dreid: ok, so it goes like that
[13:42]<moshez> dreid: there are a bunch of components
[13:42]<moshez> each component has any number of heart-beats
[13:43]<moshez> a heart-beat is a "heart" (opaque string) and a "beat" 
                (a number)
[13:43]<moshez> the heart-strings are sent, as UDP packets, to the EKG 
                port
[13:44]<moshez> the watchdog launches all the components, and then 
                watches the EKG
[13:44]<moshez> if any component has a heart which doesn't beat, it 
                starts fixing the problem
[13:44]<moshez> the first few times, it will shut down all components 
                and bring them back up
[13:44]<Tv> moshez: when does the dog eat the heart?
[13:45]<moshez> if it sees it needs to do that too many times, it will 
                decide "patient is dead" and reboot the system
[13:45]<moshez> if it sees it needs to reboot the system too many 
                times, it will decide "patient is stupid" and stop curing 
                efforts
[13:46]<dreid> hehe
[13:46]<dreid> i like that last part
[13:47]<foom> so stupid is worse than dead, eh?
[13:47]<dash> foom: there aren't any stories about voodoo curing stupid
[13:48]<moshez> foom: yes. you might be able to resuscitate, but you 
                can't cure stupidity

Now playing:

The Bothy Band - Julia Delaney