In Memoria: The Great Work

The OSCON Tuesday Night Extravaganza was just fabulous: awards, laughter, brain-bending, and affirmation. The primary speakers were Mark Shuttleworth, r0ml, and Damian Conway; but I'm going to be focusing on r0ml's talk right now :-) Well, in part, anyway.

Let's back up to Monday night, first: Alex Martelli and I had a chance to wax philosophical about programming and software. It was wonderful. Both because it revealed Alex's code-spirit and because of the sympatico I felt as his passionate idealism resonated with mine. While Alex talked of the holy architecture of mosques and cathedrals, of the contributions that such artisans as stonecutters, masons, sculptors, and calligraphers made, he emphasized how each individual played an essential role in bringing these wondrous works into being, that each act was an offering to the ideals that formed the basis of the respective belief system.

What's more, though, Alex extended the analogy from religion to mysticism, saying that even more than builders of such great structures, coders are alchemists engaged in the magum opus. We are the transmutators. In our crucibles, the opposites of function and beauty unite; performance and elegance are commingled to produce the perfection of our art. Alex was careful to point out that he intended perfection in both an abstract and practical sense. On one hand, being able to create and actually deliver code that others found useful, regardless of the sex appeal (or lack thereof), can be viewed as a form of perfection. It is accomplishment; attainment of the goal. On the other hand, it's just something that someone wanted us to write; it's not a proof of Fermat's Last Theorem. It's useful; it serves a specific function.

Before I get to r0ml's talk, I want to mention UQDS as employed by the Twisted and Divmod communities. I think it's phenomenal and I enjoy working with that system. It's a well thought-out and proven process that tends to produce code of an extremely high quality. However, it's not my natural tendency. I like quick and dirty prototypes; a little messy code goes a long way. I like to throw something out there and then fix it up and apply polish incrementally, as dictated by need.

This is why I've been enjoying the Twisted Community Code project/group on Launchpad. Not only do you have the benefits of using a tool like bazaar that lets one branch other projects on a whim, but you've got a community space to put these explorations, where others can easily see what you're doing, check it out, and try something of their own. (There's a whole 'nother blog post I have coming about that.) However, this finally brings me to r0ml's talk: a new spin on the development process.

For those of you that have seen his phenomenal rhetoric talks, you'd be delighted to see what he did :-) He established a nice mapping from both Microsoft's development process as well as the one defined by Rational. He used the five canons of classical rhetoric: inventio, dispositio, elocutio, memoria, and pronuntiatio. However, the really brilliant thing was where he started the process: smack in the middle, right where I like to do it :-) And he justified this beautifully. His mapping was the following:

• Memoria = Commit / Update
• Pronuntiatio = Run / Use
• Inventio = Bug Reporting / Patch Submission
• Dispositio = Triage
• Elocutio = Integration

The idea here being this: get what you've got done out there and in front of people's eyes. Everyone knows its crap; don't worry about it. Get it running and get others running it. Work on what matters most and integrate the changes. Repeat and continue.

I like to tease other Twisted devs that I tend not to do test-driven development, but bug-driven testing. What's interesting is that we both start with a requirements doc: for them, it's a development plan; for me, it's a bug/TODO list. The difference is that they then engage in Inventio whereas I start with Memoria. As r0ml said, with this model there is no development, there is only maintenance.

One of the other great things that r0ml mentioned about this process is that it not only gets you the developer started more quickly, it gets others started at the same time. Each programmer is engaged in a macroscopic genetic programming effort: everyone takes the source, mutates it, evolves it, reviews it, and the best implementations (or parts thereof) survive to become the basis for the next generation. Everyone gets to write at the same time; no one is blocked.

This development approach evokes images of philosophers from the Middle Ages sending letters to each other in cryptic alchemical symbols and diagrams, with all the implicit and explicit layers of meaning. I see this methodology as establishing the true foundation of the open source art: a gnostic, spirit-(of-open-souce)-ual transformation that brings us to improved states of mind and clarity.

The perfection of our art, whether sublime or mundane, can be merged in the mind of the developer as one... this union being our philosopher's stone. With each release of software engaged in this manner, we iterate the Great Work.

Async Batching with Twisted: A Walkthrough

While drafting a Divmod announcement last week, I had a quick chat with a dot-bomb-era colleague of mine. Turns out, his team wants to do some cool asynchronous batching jobs, so he's taking a look at Twisted. Because he's a good guy and I like Twisted, I drew up some examples for him that should get him jump-started. Each example covered something in more depth that it's predecessor, so is probably generally useful. Thus this blog post :-)

I didn't get a chance to show him a DeferredSemaphore example nor one for the Cooperator, so I will take this opportunity to do so. For each of the examples below, you can save the code as a text file and call it with "python filname.py", and the output will be displayed.

These examples don't attempt to give any sort of introduction to the complexities of asynchronous programming nor the problem domain of highly concurrent applications. Deferreds are covered in more depth here and here. However, hopefully this mini-howto will inspire curiosity about those :-)


Example 1: Just a DefferedList

This is one of the simplest examples you'll ever see for a deferred list in action. Get two deferreds (the getPage function returns a deferred) and use them to created a deferred list. Add callbacks to the list, garnish with a lemon.


Example 2: Simple Result Manipulation

We make things a little more interesting in this example by doing some processing on the results. For this to make sense, just remember that a callback gets passed the result when the deferred action completes. If we look up the API documentation for DeferredList, we see that it returns a list of (success, result) tuples, where success is a Boolean and result is the result of a deferred that was put in the list (remember, we've got two layers of deferreds here!).


Example 3: Page Callbacks Too

Here, we mix things up a little bit. Instead of doing processing on all the results at once (in the deferred list callback), we're processing them when the page callbacks fire. Our processing here is just a simple example of getting the length of the getPage deferred result: the HTML content of the page at the given URL.


Example 4: Results with More Structure

A follow-up to the last example, here we put the data in which we are interested into a dictionary. We don't end up pulling any of the data out of the dictionary; we just stringify it and print it to stdout.


Example 5: Passing Values to Callbacks

After all this playing, we start asking ourselves more serious questions, like: "I want to decide which values show up in my callbacks" or "Some information that is available here, isn't available there. How do I get it there?" This is how :-) Just pass the parameters you want to your callback. They'll be tacked on after the result (as you can see from the function signatures).

In this example, we needed to create our own deferred-returning function, one that wraps the getPage function so that we can also pass the URL on to the callback.


Example 6: Adding Some Error Checking

As we get closer to building real applications, we start getting concerned about things like catching/anticipating errors. We haven't added any errbacks to the deferred list, but we have added one to our page callback. We've added more URLs and put them in a list to ease the pains of duplicate code. As you can see, two of the URLs should return errors: one a 404, and the other should be a domain not resolving (we'll see this as a timeout).


Example 7: Batching with DeferredSemaphore

These last two examples are for more advanced use cases. As soon as the reactor starts, deferreds that are ready, start "firing" -- their "jobs" start running. What if we've got 500 deferreds in a list? Well, they all start processing. As you can imagine, this is an easy way to run an accidental DoS against a friendly service. Not cool.

For situations like this, what we want is a way to run only so many deferreds at a time. This is a great use for the deferred semaphore. When I repeated runs of the example above, the content lengths of the four pages returned after about 2.5 seconds. With the example rewritten to use just the deferred list (no deferred semaphore), the content lengths were returned after about 1.2 seconds. The extra time is due to the fact that I (for the sake of the example) forced only one deferred to run at a time, obviously not what you're going to want to do for a highly concurrent task ;-)

Note that without changing the code and only setting maxRun to 4, the timings for getting the the content lengths is about the same, averaging for me 1.3 seconds (there's a little more overhead involved when using the deferred semaphore).

One last subtle note (in anticipation of the next example): the for loop creates all the deferreds at once; the deferred semaphore simply limits how many get run at a time.


Example 8: Throttling with Cooperator

This is the last example for this post, and it's is probably the most arcane :-) This example is taken from JP's blog post from a couple years ago. Our observation in the previous example about the way that the deferreds were created in the for loop and how they were run is now our counter example. What if we want to limit when the deferreds are created? What if we're using deferred semaphore to create 1000 deferreds (but only running them 50 at a time), but running out of file descriptors? Cooperator to the rescue.

This one is going to require a little more explanation :-) Let's see if we can move through the justifications for the strangeness clearly:

1. We need the deferreds to be yielded so that the callback is not created until it's actually needed (as opposed to the situation in the deferred semaphore example where all the deferreds were created at once).
2. We need to call doWork before the for loop so that the generator is created outside the loop. thus making our way through the URLs (calling it inside the loop would give us all four URLs every iteration).
3. We removed the result-processing callback on the deferred list because coop.coiterate swallows our results; if we need to process, we have to do it with pageCallback.
4. We still use a deferred list as the means to determine when all the batches have finished.

This example could have been written much more concisely: the doWork function could have been left in test as a generator expression and test's for loop could have been a list comprehension. However, the point is to show very clearly what is going on.

I hope these examples were informative and provide some practical insight on working with deferreds in your Twisted projects :-)

Bazaar with Subversion and Combinator

For the past couple days, I've been experimenting with using Bazaar and Combinator more or less simultaneously. As you may know by now, Combinator is a tool that wraps some of Subversion's ugliness (mostly merging), helps manage branches, and sets Python paths for development environments. We use it extensively (almost exclusively) at Divmod.

One of my recent side projects has evolved into useful code more quickly than I had anticipated, so I thought I'd put it up on Launchpad in the Twisted Community Code. This, of course, led to questions about one-time imports, mirroring, and dual bzr/svn management. I eventually opted for the last, using the bzr plugin bzr-svn. Not having a lot of experience with Bazaar, I was at a bit of a loss, at first: there don't seem to be any dummy docs to get us beginners up to speed.

Through some painful, time-consuming trial and error and a couple dead ends, I arrived at a process that works for me, and codified it in a script. The comments in that script seemed generally useful, and given the dearth of docs, I thought I'd turn the comments into a blog post.

The Plugin

Once I figured out the right way to use bzr-svn, it was actually much easier than I thought it would be. Here are the basics: you need to have bzr installed and then you need to install bzr-svn, which is actually a bzr plugin and not a separate tool. When you have bzr-svn installed, you will have additional bzr commands at your disposal which, as you might guess, let you interoperate with an svn repository.

Two Become One

So here's how you get started: create your Subversion branch (we use Combinator) and get your working dir ready to code. You can either add dirs and files now, or do that later; it doesn't matter.

Then, in this working directory, perform a bzr checkout:

bzr co . bzrtest
cd bzrtest

This will create a Bazaar branch from your Subversion (Combinator) branch. 'bzrtest' (or whatever you name it) is your new bzr+svn branch and it is here where you'll be doing all of your work, committing, pushing to Subversion, and (in my case) pushing to Launchpad.

If your Subversion repository has a long history, you probably don't want to perform a 'bzr update' -- that'll just end in tears (it could take days to finish, use up lots of memory, require multiple restarts, and consume disk space by the gigaliter).

Launchpad

For my project, I had already registered a branch on Launchpad via the web interface, so I was ready to push the new Bazaar branch just created with the checkout command above:

bzr push lp:~oubiwann/txevolver/dev --use-existing-dir

I then logged into the web interface again, and set this newly pushed branch as the main development effort for the project. All future pushes (during this development phase) will now be done with the following command:

bzr push lp:txevolver

Future commit-push cycles just look like this:

bzr commit --local -m "My message"
bzr push lp:txevolver

Keep in mind that you can do multiple commits with Bazaar before you push to a server.

The Divmod Repo

Once you've done a local commit (or many local commits), you're ready to start pushing changes to your Subversion repository. This is where you use one of the commands that is provided by the bzr-svn plugin:

bzr svn-push svn+ssh://myRepo

And in my case, that's the following:

bzr svn-push \
svn+ssh://divmod.org/svn/Divmod/branches/genetic-programming-2620/Evolver

If you have done more than one local commit since your last push, you'll see a series of commits made to your svn repo after you issue the 'svn-push' command.

All Together Now

The script I mentioned at the beginning of this post is here. With it, I run a single command which extracts my commit message from the ChangeLog diff, commits locally, pushes to the Divmod svn repo and then pushes to Launchpad. A single command does everything I need, now: maintaining changes in both a bzr repo that can be easily branched by others on Launchpad as well as in my Subversion branch at work.

Once this project is ready to merge to trunk (if, in fact, it's final home is to be the Divmod svn repo), I'll do an svn up in the Combinator-created branch, unbranch, and commit to trunk. Upon the suggestion of JP, I'll probably also clean up the bzr-svn-created svn props, but other than that, overhead seems to be zero.

---

Subversion Update: I've been playing with this more, and here's another tidbit I didn't find documented anywhere: If you do a fresh bzr branch that had been associated with a svn repo in another working directory, you will need to rebind it to the svn repo you were working with before. You do that with the following command:

bzr bind svn+ssh://svn.yourhost.com/repo/YourProject/trunk

Google Code Update: If you are sync'ing a bzr branch with googlecode's subversion, you will need to prefix your initial push with svn:

bzr push svn+https://yourproject.googlecode.com/svn/trunk

Likewise, if you need to rebind, you'll use the following:

bzr bind svn+https://yourproject.googlecode.com/svn/trunk

The King is dead! Long live the Kinglets!

So what the hell does that mean? Well, there's sort of a back story there (but you can skip to the last paragraph of this post for the punchline).

Earlier this year, I blogged about the language patterns of twisted: Thinking in Twisted. The use of and relationships between such things as protocols, clients, servers, and various factories in Twisted was really starting to make its way into my bones. These and other core aspects of Twisted best practices form the basis of a pattern language... one that is of immense value when writing network code and applications. Here are a couple quotes from that blog entry:

The tricky thing about Twisted is that its patterns are an integral part of its "language" (perhaps it would be better to stop using quotes, and just call it a meta-language...). This is not true of most other languages -- though I would venture to say it is true of good frameworks in general. You don't need to know the patterns to use the grammar/syntax/control structures of most languages. Patterns in a language are usually optional. [However, w]ith Twisted, if you aren't using the patterns, you aren't really using Twisted.

When we learn to think in Twisted and write applications in Twisted, our solutions will be elegant and compact. Before that starts to happen, we try to use our "old" ways of python programming mixed in with the new one we are learning... and that, of course, defeats the purpose of the new "language."

I've recently hit another level of awareness in pattern abstraction, and it has its roots in Twisted. In fact, I never would have been working on what I am now working on if it weren't for Twisted and all the work done by Divmod and its crew. While at PyCon 2004, I met with Amir and Glyph and we discussed twisted's use in the industry at large, and in particular, opportunities for which the Divmod product line could be applicable.

What I liked so much (and immediately) about what Divmod was doing was that they were integrating useful components into a single application (Quotient). It stuck a positive chord with my instincts about where software should be going, and it kept me thinking for months while I worked on other projects. These thoughts culminated in a series of blog entries and software projects. First, a post called Dinosaurs and Mammals explored this. In the intervening time, the projects I have worked on have helped me evolve these ideas usefully. There is now direction, code, and product (more about that later in this post).

Then, in Everything as a Distributed Resource, I took a more sci-fi, radical view... but one that is born out by current trends and tendencies, particularly in the cell phone market. There's a great deal we haven't thought about in regards to the cell phone market. Not so much about cell phones and mobile technology, but rather the usage patterns and demands that are being made and (sometimes) met in this sector.

In an effort to follow through on the "mammal" paradigm, I did some exploration with URLdb and PyKon which I blogged briefly about in New Projects. The two projects mentioned there have since been merged into a project that we are calling Adytum's Twisted Collection. The latest addition to which is ImageDB.

ImageDB is a repository for static graphic media. The images that people upload here can be viewed via it's web interface, but it is more likely the ImageDB would be used as a part of something else. For example, when I get around to finishing PyKon, it will make use of
ImageDB. PyKon will not store any images itself. It's simply going to be an nicely designed, usable image gallery interface. Users won't know that, though. They will use PyKon and be able to create galleries and albums in that gallery by uploading images (which will be stored --
transparently to the user -- in an ImageDB instance), by selecting individual images (JavaScript ImageDB selector/browser), or by selecting "tags" a la del.icio.us (and therefore whole groups of
images).

Analogous to ImageDB is MessageDB, a repository for static text. It's next on the list to be developed. This micro-app will hardly ever be used directly, although it too will provide a basic web interface for viewing and editing stored content. MessageDB will be a storage app for RFC 2822-compliant "messages". For starters, various public mail list emails will get tagged and stored there.

Non-dynamic text for web sites and blogs will be stored as RFC 2822 messages. So the forth-coming tools "SimpleSite" and "WeBlog" (also part of the collection) will actually store and pull their content from MessageDB. But even more, when you want to use a photo, clipart, or logo in your site or on your blog, you will click a "select image" link and an image thumbnail browser will pop up. You will select one of the thumbnails, the popup will go away, and a link to that resource in your ImageDB instance will be inserted for you, in your blog or site text.

The idea is that by abstracting out general functionality into little micro-apps, you not only decrease the amount of work in maintenance and feature addition, you not only are reducing the amount of redundant work you have to do when building content-related applications, but you are providing an infinite amount of flexibility for using these various media in completely unplanned and unexpected ways.

Your media, it's storage, it's access (XML-RPC), now live in self-contained little units. They can "survive" on their own, without a heavy dependence upon some huge CMS that was written for a particular project. The apps are project-agnostic. They can be set up once and then reused repeatedly in different contexts by building new apps that obtain their data from the component micro-apps via simple links and/or XML-RPC calls.

And, again, the ways in which users and designers decide to combine these pieces and how they decide to use that is 100% flexible and thus fertile ground for the unexpected and creative :-)

I say "To Hell!" with large apps, and banish them forever! Let's build tiny, little, single-purpose apps once! And then for larger needs, simply glue together various micro-apps as needed per the project requirements.

As far as I'm concerned, the old King of the Bloated Application is dead. It's been replaced with tons of tiny, decentralized micro-apps. Long live the Kinglets!