Unit Tests: No really, you HAVE to use them

twisted :: programming :: testing

So, for the pycon sprint this year, I figured I'd work virtually
with some other developers who couldn't make it this year. There has
been some interest and requests on the pymon mail lists... I strongly
believe that patience and loyalty should be rewarded, and these guys
have put up with an awful lot of delays and bugs... and circumstances
beyond anyone's control.

The first thing I wanted to do was get a usable release candidate
available for them to download. After digging back into the code, I
realized that the best way to provide this was by patching up the
already-extant pymon RC2. With RC2 operable, I tried fixing some bugs
for later code changes, and I couldn't. The code was a mess. But not
anything most people would ever notice or even care about.

Here's what I mean: in the last big merge, we brought in all the
twisted code, and then proceeded to clean that up... but people wanted
to use it soon, and writing tests slowed us down, so we skipped it. It
is debatable whether that was the right thing to do, since it is really
important to weigh your user base when making such decisions. Being a
purist at heart, I'm really wishing it was possible to have pleased the
users and take the time needed to add the tests. Regardless, I have to
add them now. The project simply will not progress without it. I cannot
currently debug effectively without them.

But there is something more important here, the REAL reason I need to
add unit tests. When I tried to add them yesterday, I realized my code,
as it was, simply wouldn't allow it. This means my code is bad. Plain
and simple. It means that there really aren't "units" to test, and that
means the code is not well-enough thought out. After looking at the
problematic code yesterday, it appears pymon will be a good candidate
for componentization. This will:

1. Provide clear units than will be much more easily testable as
   units


2. Clean the code up considerably


3. Come more in line with the standards and best practices of the core
   twisted code

By adding unit tests early on in a project's life, you are giving
yourself the chance to benefit from a design litmus test. If your
twisted code lends itself more or less easily to twisted.trial, then
there is also a good chance that you have something that is workably
modular. For me, this will mean that I can accept all sorts of feature
additions and increased application sophistication without affecting
the quality, readability, or testability of the code. The code will
remain clean and simple. There will just be an increased number of
components.

Now playing:

Yes - No Opportunity Necessary, No Experience 
Needed

Back to Subversion

version control :: software

Well, I experienced a spectacular failure when it came to distributing
darcs to multiple servers, and then pushing/pulling changes. I am
perfectly willing to accept that it was me, with a limited
understanding of darcs methodologies and an unwillingness to see the
problem through. But I just couldn't bring myself to loose two days on
it -- in the middle of projects with deadlines, one is enough, thank
you.

So, back to subversion it is.

The nice thing, though, is that a few days before the move to darcs, I
had read the
Combinator
wiki entry on the
Divmod trac
as well as the
branch-based
development entry there. This is really good stuff and offers the
cleanliness I like to see in software development process. Though this
is not quite patch-based development, it certainly fits with my
experience and intuition. (I didn't work with darcs enough to really
say whether the patch-based approach fit my brain.)

The other thing is distributed content. Since I didn't have any issues
with read-only pulls from darcs repositories, this is what I am
considering: every three hours, I will run my tailor scripts against
the latest svn changes (trunk only), and distribute these via rsync to
other servers where the code will be accessible via http and darcs http
checkout.

But all this will have to wait until the sprinting at pycon is
finished...

twisted.web2.xmlrpc

twisted :: programming

I had a chance to do some paired programming with Itamar
Shtull-Trauring today. He was very patient with me, and I learned a
great deal just observing how he considered the problems at hand. I
will be processing what he shared and the implications of his approach
for several weeks. At least. It was a great experience, and I am
thrilled not only by what I learned but am genuinely excited about
digging deeper into twisted.web2 and HTTP in general. (But not because
HTTP is so great... more like it's a beast I want to beat now. Beat
senseless.)

Also, the company that Itamar
(ITA)
works for is doing some really awesome stuff. One could naively say
that they are doing "airline reservation searches", but it's so, so
much more than that. They are focused on a really interesting and
diverse problem space. Think about it: if they are keeping guys like
Itamar and Michael Salib interested and enthusiastic, there's
definitely more than meets the eye, and it's got to be interesting :-)
They are also looking for motivated, talented developers who would be
willing to live in the Boston area. Check out the site and then contact
them if you are interested. They've got new and significant investment
interest (real $), new offices, and big plans. If I could work remotely
for them, I'd do it in a second... alas, I can't leave my beloved
Rockies ;-)

Zope 3 Components

zope 3 :: twisted :: programming

I'm filing this under twisted as well as zope 3 because this is exactly
how I am going to use Zope 3 components. I was pretty excited/relieved
yesterday listening to Jim Fulton say that one of his top priorities is
working on the finer-grained architecture of Zope 3 such that
separation of components is possible. I would be using much more of z3
if I could use them as independent components right now.

Twisted Games

Every year or so, I search online for the latest CP/M emulators and the source code for my favorite childhood games: Star Traders (also Star Lanes), Ladder, Catchum, Hunt the Wumpus, Star Trek, Adventure, etc. These and others were either ASCII "graphics" or just text-only. Like a vulture waiting for the all-clear, I would regularly circle the den where my dad used the Kaypro II we had, and would "sneak" in as soon as it was available. So many fond memories. So fond, in fact, that when I was in school studying phsyics, I bought an old Kaypro on eBay as a fixer-upper. I still have it... but I haven't fixed her up yet ;-) (I got it booting, but I need to fix the wiring in the keyboard).

Recently on a private mail list, I was reminiscing with friends how I started programming as a result of these games: finishing them and not wanting them to end, so I'd write extensions for them and then play those. After that email, I went on my annual google binge, and this year's was the most successful yet. Here's some good stuff I found:

• A java port of Ladder
• An ncurses C port of Ladder (which compiled fairly easily on Mac OS X)
• The BASIC source code for Star Trader
  
• CP/M games and more games (few years ago, this was hosted in a different domain... maybe walnut creek? I forget)
• CP/M OS Files
• Some cryptic instructions for running Kaypro CP/M in the MESS emulator

For those unfamiliar with the classics, here are the Linux ports of the famous BSD games -- if you're running Ubuntu or Debian, just do apt-get install bsdgames and enjoy :-)

Last August, JP Calderone (exarkun) blogged about some test code of twisted.conch.insults he was writing for potential twisted terminal-based apps. I only saw this recently, and it got me to thinking about my favorite old games:

1. I could fairly easily port them, and
   
2. some of them might do very well as multiplayer apps.
   

Ah, the thought of that was pure heaven: old friends, playing old games over the network, reminiscing and reliving old times... and doing it all together. I think my nostalgia circuits over-loaded, because I just sat there for a while with a wistful grin on my face... it could have been for hours, but I'll never say...

So, yet another set of projects has been initiated, and I am having an immense amount of fun -- nay, joy! -- working on them. I love the terminal. I live it. I wish all applications were terminal based, but it's a rare opportunity to have a project with those requirements. This is a wonderful childhood reprise :-)

As with my other projects, I will post updates here on the progress of the Twisted games I'm porting.

accountdb

This week I made some great progress on the accountdb twisted/nevow micro app. The prototype is finished, complete with screenshots. One of the nice things that happened while writing it was a personal issue resolution :-)

And, yes, as a programmer, most of my personal issues are with software and software implementation or design.

Here's the issue: I think Nevow's stan is fantastic and ingenious.  However, I tend to adhere pretty strictly to the principle that HTML doesn't belong in code, and should remain in templates. For that  reason, I even try to steer away from programmatically generated HTML.

Two things let me relax this position: 1) stan is *very* clean, and 2) when you have lots of "mini-templates", having them all in separate files *really* starts to affect performance negatively, especially if
you're creating objects to provide an interface for them.

This work has given me some nice insight on how I can optimize imagedb as well. I'm looking forward to that, since I've been using imagedb a lot more, lately.

Oh, I almost forgot: sales pitch for accountdb: this micro app is, as you can guess, responsible for storing users and providing web authentication as a service. Don't worry, there will be many more blog entries on this in the future, and I will describe how this is done and why this is so useful :-)

But for now, back to coding!

Subversion to Darcs

version control :: software

I have *finally* made the move now. I've been experimenting with
darcs for about 6 months now, on and off. At the hackathon in December
(sponsored by tummy.com), I was really into it and ran tons of tests. I
used tailor to convert several repositories, etc. I really loved the
patch approach that darcs uses, but I just couldn't motivate to migrate
the 47 repositories that I have in svn.

However, I have recently migrated all of my Subversion repositories
from one server to another, and it was the perfect opportunity. I spent
two days migrating to darcs and testing the results, and am delighted.
The scripts that I set up to do all the work can now be run easily at
any time (if the need arises for any reason). With darcs, the projects'
source code is now easily available for anonymous checkout, and what's
more, easy mirroring.

I have hacked trac to point to the darcsweb cgi, but I'm still pushing
darcs partials from the main server out to the first mirror... so there
will be some 404s for a little while, but I expect by the weekend to
have all repos viewable in darcsweb as well as available for anonymous
checkout. I like the look and functionality of darcsweb, though it is
slow. However, I find the query parameters obnoxious; I'll be rewriting
it as a stand-alone Nevow app...

Now playing:

Yes - It Can Happen

Nevow Radio

nevow :: twisted :: music

As *everyone* knows, PyCon is just around the corner. I'm quite
looking forward to it, since I missed last year's. Not only that, but
I'll be visiting a good friend whom I haven't seen in 10 years.
However, my housemate is getting nervous -- she relies upon me
completely for all things technological. With respect to my forthcoming
two-week absence, the biggest source of her concern has been music.

Over the past 9 years, I have migrated all of my Audio CDs to hard
drive and then to to MP3 CDs/DVDs. However, this trend has slowly
undergone a partial reverse as my ancient file server (running a fresh
Ubuntu) has found new purpose in life: serving MP3s from it copious
free space. I also happen to be the only one around who understands and
can operate such things. My housemate really doesn't want to be without
music for two weeks.

Not wanting to leave my friend in the lurch during the conference and
sprints, I whipped up some python code to make it all better. First, I
wrote a little script that parses the iTunes Library XML file. This
allows me to now generate MuSE-readable playlist files for all of my
iTunes playlists with a single script (it seems that iTunes won't let
you batch-export all your playlists). With these generated, I wrote a
quick little Nevow app that globs these new playlist files and
interacts with MuSE (I wanted to run icecast, but it seems to only work
with ogg). All you have to do is select the desired playlist, hit
submit, and then open up the stream in iTunes. Selecting a new playlist
in the Nevow interface reloads MuSE and immediately starts streaming
the new choice.

This was absurdly easy to put together. And it's only two files: one
for the playlist generator, and an all-in-one .tac file for Nevow. It's
even got a virtual host monster plugged in, on the off chance that I
want to put this app on its own little local domain. It's little things
like this that really confuses me: what's the big deal about people
freaking out with Twisted and Nevow? This stuff is brilliant,
well-designed, elegant and makes perfect sense. When I read Guido's
comments about how Nevow/Twisted scares him, I was at a complete loss.
It can only mean that he (and others with the same views) simply
haven't taken the time to site down and actually *use* the stuff.

C'mon, guys -- tune in, already!

Now playing:

Yann Tiersen - Soir de fête

NetCIDR 0.1 Released

networking :: python :: programming

As part of the on-going work with
CoyoteMonitoring,
NetCIDR
was written to allow for a clean and logical approach when analyzing
NetFlow
captures. Primarily, it's use is for determining whether a given IP
address is in a given netblock or a collection of netblocks. Here are
some quick example usages from the wiki and doctests:

     >>> CIDR('10.4.1.2')
     10.4.1.2
     >>> CIDR('10.4.1.x')
     10.4.1.0/24
     >>> CIDR('10.*.*.*')
     10.0.0.0/8
     >>> CIDR('172.16.4.28/27')
     172.16.4.28/27
     >>> CIDR('172.16.4.28/27').getHostCount()
     32

Here's how you create a collection of networks:

     >>> net_cidr = CIDR('192.168.4.0/24')
     >>> corp_cidr = CIDR('10.5.0.0/16')
     >>> vpn_cidr = CIDR('172.16.9.5/27')
     >>> mynets = Networks([net_cidr, corp_cidr])
     >>> mynets.append(vpn_cidr)

And now, you can check for the presence of hosts in various networks
and/or collections of networks:

     >>> home_router = CIDR('192.168.4.1')
     >>> laptop1 = CIDR('192.168.4.100')
     >>> webserver = CIDR('10.5.10.10')
     >>> laptop2 = CIDR('172.16.9.17')
     >>> laptop3 = CIDR('172.16.5.17')
     >>> google = CIDR('64.233.187.99')
     >>> home_router in mynets
     True
     >>> laptop1 in mynets
     True
     >>> webserver in mynets
     True
     >>> laptop2 in mynets
     True
     >>> laptop3 in mynets
     False
     >>> google in mynets
     False

CoyoteMonitoring 3

networking :: management :: software

For the past year, I have resisted speculation and development on
CoyMon 3. But in the past few months, many pieces have begun falling
into place -- each removing objections and concerns, and some providing
a clear path of development where before there was none.

CoyoteMonitoring
is a free, open-source network management tool. Specifically, it glues
together many open-source and difficult-to-configure subsystems for
obtaining and viewing
NetFlow
data used in network analysis. It is comparable to operating system
distributions in this regard: it is a NetFlow management distro. The
Department of Veterans affairs has actually been using CoyMon 2 for
about a year now, with astounding results. We are thrilled with the
success it has been for them, with the tremendous time and money it has
saved them.

The development of CoyMon 2 was sponsored in part by the VA, and due to
the timeline, much of the code was specific to their needs. However,
because of the time it would take to audit the code, CoyMon 2 has never
been publicly released. I have had many email conversations with WAN
managers and network engineerings very distressed at this. They too
have found a sad lack in the open source world when it comes to useful,
easy-to-use tools for querying, viewing, and analyzing NetFlow data.
Because of this continued interest, input, and moral support from these
hard-working individuals, I have been considering how this need could
best be addressed and met.

To be honest, my biggest technical concern has been with the current
NetFlow tools. The perl code that people have depended upon for this
(since the late 90s!) is krufty, hard to maintain, overly uses
almost-unnecessary and out-of-date perl modules, and as a group, were
not designed with maximal extensibility in mind. They have been genuine
gems in the field -- none of us could have done what we have done for
the past several years without this wonderful contribution. The
combination of these (and their dependencies!) with all the other
pieces that require configuration and glue is a difficult thing to
provide. It took an ENORMOUS amount of time and energy to get CoyMon 2
into a place where it could be deployed efficiently. CoyMon 2 runs like
a tank, though. It's a real champ and a testament to all that hard work
and organization.

However, it is time to fix the system at its roots. Enter CoyMon 3.

The biggest stumbling block to forward movement on CoyMon 3 has been
the available APIs for to tools upon which we depend, with one of the
most important being RRDTool. The python bindings for RRDTool are
archaic and decidedly non-OOP. Attempting to design a system that is
robust, effortless to maintain, and easily adapts new features but that
has composite components with difficult APIs is a recipe for
frustration, delay, and ultimately, non-delivery. The first step
towards addressing this problem came with the recent release of
PyRRD.
It is a fully OOP wrapper for RRDTool in python that removes this pain
from the equation.

The next biggest hurdle was the old perl code called FlowScan. After
much discussion and analysis, a clean and elegant way to provide this
functionality was arrived at, and we will soon have a new product to
show for it, freely available for download. CoyoteMonitoring will
depend heavily on this piece of software and related libraries. We are
making excellent progress on this, but ultimately, the problem to solve
here is one of modularity and configuration. How do you provide an
easy, non-programmatic method of extensibility for any number of
potential rules? We've got some great ideas, but only the natural
selection of actual use to prove the best approach. This is our current
top-priority.

With the advances that have been made in the past several months, we
are not only comfortable making an announcement about a new version of
CoyoteMonitoring, we are down-right confident :-) For the interested,
here are more detailed points on CoyMon 3:

• CoyMon 3 will be developed completely independently of any
  third-parties or businesses. This will mean slower development times,
  but cleaner more easily managed code. With the absence of sensitive
  customer code and/or configurations, you will see regularly available
  downloads.


• Supporting libraries will all have extensive unit tests for each
  piece of functionality.


• CoyMon 3 will have a 100% true component architecture.


• CoyMon 3 will continue to use flow-tools and will make full use of
  the python bindings for fast processing of NetFlow captures.


• CoyMon 3 will make use of Zope 3 technology for through-the-web
  management of such resources as collectors, protocols, queries,
  resulting data and graphics, as well as arbitrary content that is
  important to end users.


• CoyMon 3 will make use of the Twisted Python framework for all of
  its specialized networking needs, including (but not limited to) CMOB/X
  (a recently developed "object broker" for distributed collectors
  managed at a central location).


• CoyMon 3 will have completely re-factored supporting libraries,
  written and maintained by the CoyoteMonnitoring community. All the old
  Perl code will be replaced with light-weight, easy-to-maintain python
  libraries and scripts. These will include NetCIDR, PyRRD, and
  PyFlowCatch.


• CoyMon 3 will have consistent configuration across all its
  composite applications and it will make use of the famous, the useful,
  and the ever-concise Apache-style configuration files.


• And last, but not least, CoyMon 3 will abide by Chapter 4 of Eric
  Raymond's "The The Cathedral and the Bazaar": Release Early, Release
  Often. CoyMon 3 development snapshots will be available for download
  regularly.

Project spaces to keep your eyes on:

CoyMon

PyFlowCatch

PyRRD

NetCIDR