Tuesday, February 28, 2006

Unit Tests: No really, you HAVE to use them

twisted :: programming :: testing

So, for the pycon sprint this year, I figured I'd work virtually
with some other developers who couldn't make it this year. There has
been some interest and requests on the pymon mail lists... I strongly
believe that patience and loyalty should be rewarded, and these guys
have put up with an awful lot of delays and bugs... and circumstances
beyond anyone's control.

The first thing I wanted to do was get a usable release candidate
available for them to download. After digging back into the code, I
realized that the best way to provide this was by patching up the
already-extant pymon RC2. With RC2 operable, I tried fixing some bugs
for later code changes, and I couldn't. The code was a mess. But not
anything most people would ever notice or even care about.

Here's what I mean: in the last big merge, we brought in all the
twisted code, and then proceeded to clean that up... but people wanted
to use it soon, and writing tests slowed us down, so we skipped it. It
is debatable whether that was the right thing to do, since it is really
important to weigh your user base when making such decisions. Being a
purist at heart, I'm really wishing it was possible to have pleased the
users and take the time needed to add the tests. Regardless, I have to
add them now. The project simply will not progress without it. I cannot
currently debug effectively without them.

But there is something more important here, the REAL reason I need to
add unit tests. When I tried to add them yesterday, I realized my code,
as it was, simply wouldn't allow it. This means my code is bad. Plain
and simple. It means that there really aren't "units" to test, and that
means the code is not well-enough thought out. After looking at the
problematic code yesterday, it appears pymon will be a good candidate
for componentization. This will:

  1. Provide clear units than will be much more easily testable as

  2. Clean the code up considerably

  3. Come more in line with the standards and best practices of the core
    twisted code

By adding unit tests early on in a project's life, you are giving
yourself the chance to benefit from a design litmus test. If your
twisted code lends itself more or less easily to twisted.trial, then
there is also a good chance that you have something that is workably
modular. For me, this will mean that I can accept all sorts of feature
additions and increased application sophistication without affecting
the quality, readability, or testability of the code. The code will
remain clean and simple. There will just be an increased number of

Now playing:
Yes - No Opportunity Necessary, No Experience 

Back to Subversion

version control :: software

Well, I experienced a spectacular failure when it came to distributing
darcs to multiple servers, and then pushing/pulling changes. I am
perfectly willing to accept that it was me, with a limited
understanding of darcs methodologies and an unwillingness to see the
problem through. But I just couldn't bring myself to loose two days on
it -- in the middle of projects with deadlines, one is enough, thank

So, back to subversion it is.

The nice thing, though, is that a few days before the move to darcs, I
had read the
wiki entry
on the
Divmod trac
as well as the
entry there. This is really good stuff and offers the
cleanliness I like to see in software development process. Though this
is not quite patch-based development, it certainly fits with my
experience and intuition. (I didn't work with darcs enough to really
say whether the patch-based approach fit my brain.)

The other thing is distributed content. Since I didn't have any issues
with read-only pulls from darcs repositories, this is what I am
considering: every three hours, I will run my tailor scripts against
the latest svn changes (trunk only), and distribute these via rsync to
other servers where the code will be accessible via http and darcs http

But all this will have to wait until the sprinting at pycon is

Saturday, February 25, 2006


twisted :: programming

I had a chance to do some paired programming with Itamar
Shtull-Trauring today. He was very patient with me, and I learned a
great deal just observing how he considered the problems at hand. I
will be processing what he shared and the implications of his approach
for several weeks. At least. It was a great experience, and I am
thrilled not only by what I learned but am genuinely excited about
digging deeper into twisted.web2 and HTTP in general. (But not because
HTTP is so great... more like it's a beast I want to beat now. Beat

Also, the company that Itamar
works for is doing some really awesome stuff. One could naively say
that they are doing "airline reservation searches", but it's so, so
much more than that. They are focused on a really interesting and
diverse problem space. Think about it: if they are keeping guys like
Itamar and Michael Salib interested and enthusiastic, there's
definitely more than meets the eye, and it's got to be interesting :-)
They are also looking for motivated, talented developers who would be
willing to live in the Boston area. Check out the site and then contact
them if you are interested. They've got new and significant investment
interest (real $), new offices, and big plans. If I could work remotely
for them, I'd do it in a second... alas, I can't leave my beloved
Rockies ;-)

Zope 3 Components

zope 3 :: twisted :: programming

I'm filing this under twisted as well as zope 3 because this is exactly
how I am going to use Zope 3 components. I was pretty excited/relieved
yesterday listening to Jim Fulton say that one of his top priorities is
working on the finer-grained architecture of Zope 3 such that
separation of components is possible. I would be using much more of z3
if I could use them as independent components right now.

Tuesday, February 21, 2006

Twisted Games

Every year or so, I search online for the latest CP/M emulators and the source code for my favorite childhood games: Star Traders (also Star Lanes), Ladder, Catchum, Hunt the Wumpus, Star Trek, Adventure, etc. These and others were either ASCII "graphics" or just text-only. Like a vulture waiting for the all-clear, I would regularly circle the den where my dad used the Kaypro II we had, and would "sneak" in as soon as it was available. So many fond memories. So fond, in fact, that when I was in school studying phsyics, I bought an old Kaypro on eBay as a fixer-upper. I still have it... but I haven't fixed her up yet ;-) (I got it booting, but I need to fix the wiring in the keyboard).

Recently on a private mail list, I was reminiscing with friends how I started programming as a result of these games: finishing them and not wanting them to end, so I'd write extensions for them and then play those. After that email, I went on my annual google binge, and this year's was the most successful yet. Here's some good stuff I found:

For those unfamiliar with the classics, here are the Linux ports of the famous BSD games -- if you're running Ubuntu or Debian, just do apt-get install bsdgames and enjoy :-)

Last August, JP Calderone (exarkun) blogged about some test code of twisted.conch.insults he was writing for potential twisted terminal-based apps. I only saw this recently, and it got me to thinking about my favorite old games:

  1. I could fairly easily port them, and
  2. some of them might do very well as multiplayer apps.

Ah, the thought of that was pure heaven: old friends, playing old games over the network, reminiscing and reliving old times... and doing it all together. I think my nostalgia circuits over-loaded, because I just sat there for a while with a wistful grin on my face... it could have been for hours, but I'll never say...

So, yet another set of projects has been initiated, and I am having an immense amount of fun -- nay, joy! -- working on them. I love the terminal. I live it. I wish all applications were terminal based, but it's a rare opportunity to have a project with those requirements. This is a wonderful childhood reprise :-)

As with my other projects, I will post updates here on the progress of the Twisted games I'm porting.

Saturday, February 18, 2006


This week I made some great progress on the accountdb twisted/nevow micro app. The prototype is finished, complete with screenshots. One of the nice things that happened while writing it was a personal issue resolution :-)

And, yes, as a programmer, most of my personal issues are with software and software implementation or design.

Here's the issue: I think Nevow's stan is fantastic and ingenious.  However, I tend to adhere pretty strictly to the principle that HTML doesn't belong in code, and should remain in templates. For that  reason, I even try to steer away from programmatically generated HTML.

Two things let me relax this position: 1) stan is *very* clean, and 2) when you have lots of "mini-templates", having them all in separate files *really* starts to affect performance negatively, especially if
you're creating objects to provide an interface for them.

This work has given me some nice insight on how I can optimize imagedb as well. I'm looking forward to that, since I've been using imagedb a lot more, lately.

Oh, I almost forgot: sales pitch for accountdb: this micro app is, as you can guess, responsible for storing users and providing web authentication as a service. Don't worry, there will be many more blog entries on this in the future, and I will describe how this is done and why this is so useful :-)

But for now, back to coding!

Tuesday, February 14, 2006

Subversion to Darcs

version control :: software

I have *finally* made the move now. I've been experimenting with
darcs for about 6 months now, on and off. At the hackathon in December
(sponsored by tummy.com), I was really into it and ran tons of tests. I
used tailor to convert several repositories, etc. I really loved the
patch approach that darcs uses, but I just couldn't motivate to migrate
the 47 repositories that I have in svn.

However, I have recently migrated all of my Subversion repositories
from one server to another, and it was the perfect opportunity. I spent
two days migrating to darcs and testing the results, and am delighted.
The scripts that I set up to do all the work can now be run easily at
any time (if the need arises for any reason). With darcs, the projects'
source code is now easily available for anonymous checkout, and what's
more, easy mirroring.

I have hacked trac to point to the darcsweb cgi, but I'm still pushing
darcs partials from the main server out to the first mirror... so there
will be some 404s for a little while, but I expect by the weekend to
have all repos viewable in darcsweb as well as available for anonymous
checkout. I like the look and functionality of darcsweb, though it is
slow. However, I find the query parameters obnoxious; I'll be rewriting
it as a stand-alone Nevow app...

Now playing:
Yes - It Can Happen

Sunday, February 12, 2006

Nevow Radio

nevow :: twisted :: music

As *everyone* knows, PyCon is just around the corner. I'm quite
looking forward to it, since I missed last year's. Not only that, but
I'll be visiting a good friend whom I haven't seen in 10 years.
However, my housemate is getting nervous -- she relies upon me
completely for all things technological. With respect to my forthcoming
two-week absence, the biggest source of her concern has been music.

Over the past 9 years, I have migrated all of my Audio CDs to hard
drive and then to to MP3 CDs/DVDs. However, this trend has slowly
undergone a partial reverse as my ancient file server (running a fresh
Ubuntu) has found new purpose in life: serving MP3s from it copious
free space. I also happen to be the only one around who understands and
can operate such things. My housemate really doesn't want to be without
music for two weeks.

Not wanting to leave my friend in the lurch during the conference and
sprints, I whipped up some python code to make it all better. First, I
wrote a little script that parses the iTunes Library XML file. This
allows me to now generate MuSE-readable playlist files for all of my
iTunes playlists with a single script (it seems that iTunes won't let
you batch-export all your playlists). With these generated, I wrote a
quick little Nevow app that globs these new playlist files and
interacts with MuSE (I wanted to run icecast, but it seems to only work
with ogg). All you have to do is select the desired playlist, hit
submit, and then open up the stream in iTunes. Selecting a new playlist
in the Nevow interface reloads MuSE and immediately starts streaming
the new choice.

This was absurdly easy to put together. And it's only two files: one
for the playlist generator, and an all-in-one .tac file for Nevow. It's
even got a virtual host monster plugged in, on the off chance that I
want to put this app on its own little local domain. It's little things
like this that really confuses me: what's the big deal about people
freaking out with Twisted and Nevow? This stuff is brilliant,
well-designed, elegant and makes perfect sense. When I read Guido's
comments about how Nevow/Twisted scares him, I was at a complete loss.
It can only mean that he (and others with the same views) simply
haven't taken the time to site down and actually *use* the stuff.

C'mon, guys -- tune in, already!

Now playing:
Yann Tiersen - Soir de fête