Recent Work on Various Open Source Projects

Over the last few months, I've been doing lots of work on various open source projects. I've been so burried in them that I haven't blogged (or microblogged) much about them. So much has been happening, though, that I needed to take a break from the coding and communicate some of this :-)

txAWS

Over the last few months, Robert Collins, Thomas Herve, Jamshed Kakar and I have been putting lots of effort into improving cloud support in the async (Twisted) Python Amazon EC2 library. It's been a lot of fun to see that part of the library take shape and start getting production use from Canonical. We have implemented the following functionality groups and their associated API methods:

• Instances
• Key Pairs
• Security Groups
• Elastic IPs
• Availability Zones
• EBS

There is a ticket for the following two groups of functionality, and branches in progress for both:

• Images (ticket #461136 on Launchpad)
  
• Image Attributes (ticket #461139 on Launchpad)
  

Once those are merged, EC2 support in txAWS will be complete.

As a bonus, we've added support for arbitrary endpoints and with that in place, have successfully tested txaws.ec2 against Eucalyptus clouds :-)

txJSON-RPC

There's a new release of txJSON-RPC out now, downloadable from PyPI. Work on the next version has been a great deal of fun. What started out as a conversation on IRC with Terry Jones, ended up as spec-driven doctest work on trunk for implementing support for multiple versions of the JSON-RPC spec (pre-1.0, 1.0, and 2.0).

With these changes in spec support, txJSON-RPC has really started to mature, and that's been fantastic. Even more, the jsonrpclib module that's included in txJSON-RPC (and can be used with non-Twisted projects) is getting spec version support as well.

SOOM in txULS

As some may know, one of my computing passions is ultra large-scale systems. After a phone conversation with Jamshed Kakar and some nice exchanges on the Python ULS-SIG mail list with Alex Drahon, I started working on a set of coding experiments in self-organizing objects. The Google Doc informally outlines the various stages and goals.

For now, the code is living in a txULS series on Launchpad. The reason for its inclusion in txULS is that ultimate goal of the SOOM (self-organizing object meshes) code is to produce an async API for building Twisted services that provide behaviours as outlined in the Google Doc (linked above).

I would to emphasize the networking-library-agnostic nature of the ULS-SIG: Twisted comes up since I spend a lot of time with Twisted, but ever networking library is welcome. I'm personally interested in exploring (or watching other developers explore) various Stackless Python experiments in the ULS systems space.

txSpore

This project was a spontaneous effort resulting from an evening of code review when I first discovered the official Python API from EA/Maxis for the game Spore. It's been a blast and something that Chris Armstrong and I have been working on together. The API is currently feature-complete, but Chris has some excellent ideas about improving usage as well as some additional API augmentations that will make life easier for game developers.

Already more featureful and usable than the official Spore Python API, there are great things in store for this library. Chris has come up with several very cool demo ideas that take advantage of the new API and will push it to the limits. We're both pretty excited :-)

Isomyr

I love isometric games. I'm a freak for the classic look. One night about a month ago, Chris and I discovered Isotope, an isometric Python game engine by Simon Gillespie. It was last updated in 2005 at version 0.9, so I started working on a branch that could be released as 1.0. I never heard back from Simon after an inquiry for his permission to release as 1.0, so I forked the code to a new project: Isomyr. I released the code rewrite work I had done to that point (plus some changes such as replacing some old code with Numpy) as 0.1. At which point things just went nuts...

Isomyr now has support for multiple worlds, customizable (per world) in-game time and calendars, and basic interactive fiction development. The latest chunk of code (that hasn't been pushed up to Launchpad yet) is adding support for general planetary simulation (e.g., axial title, varying daylight hours, seasons, and weather). As you might imagine, this has been a great deal of fun to work on!

PyRRD

PyRRD has gotten some recent community love, with requests for a mail list, new developer-oritnted features, etc. Currently at version 0.0.7, the 0.1.0 release isn't to far away. Folks have been using trunk for a while, which added support for the RRDTool Python bindings back in March of this year. (PyRRD's focus has primarily been for users/developers who didn't have the RRDTool Python bindings installed). In the next couple weeks or months, I expect that we'll be adding a few more features, and then preparing the new release.

PyRTF

Another fixer-upper project, PyRTF (mirroed on Google code as pyrtf-ng) has been on hiatus for a while, due to my diminished need to manipulate and interact with RTF files. However, a new developer has joined the project and the code-cleanup and unit test development now continues. Thanks Christian Simms!

A while ago, Simon Cusack (the original author of PyRTF) and I had some great discussions about the future development of PyRTF and his interest in merging the recent changes into trunk on SourceForge. I deferred on that action, wanting to wait until the code cleanup, unit tests, and API changes had been completed. With Christian's help, we may get there now :-)

Wrap-up

It's been about a year since I've been so active in open source development, and it feels really good to be at it again :-) Being back in Colorado seems to have helped in subtle ways, but mostly it's been the increased interaction and interest from developers in the community that I can thank for my increased activity (and thus enjoyment). You guys are awesome. You're the reason for any code I produce.

PyCon 2010 Talks Neeed!

Hey folks,

At my last count, we've only received 20 talks so far for PyCon 2010! There are only 14 days remaining for talk submissions... if you've had a great idea about a talk for PyCon, now's the time to make it happen!

Below I've pasted some links that folks might find helpful. The first one has pretty much everything you need to know about submitting a talk for PyCon.

• Proposal Submission Information
  
• Help and Tips for Speakers
  

txSpore: Twisted Spore

I just had a delightful weekend of coding :-) I spent the past two days porting the Spore Python API to Twisted. You can now incorporate Spore data (from static XML as well as REST requests) into your non-blocking Python applications/games!

This was a pretty easy task, really. The API just makes HTTP requests with twisted.web.client.getPage. There was a little bit of work involved in creating object models for the XML, and some head-scratching for the error-catching deferToThread unit test I tried to write (it's still buggy... need to figure that one out). Everything else was pretty much cake.

In fact, it was so much fun to kick back and write some playful code that I might overhaul the sync Python code as well and incorporate both into txSpore.

Do be aware, however, that the code still has some big improvements coming. The first thing I want to hit is actually create a client object. Right now, the client module contains a series of functions (since state's not currently needed). However, I want to start doing some basic object caching in order to limit the number of requests made to spore.com and increase the response time. That's the big item for 0.0.2. Update: 0.0.2 is now released!

Next I'd like to create some more demo apps that show off the API usage better. Right now, there's one demo (a .tac file). All it does is ask for a user name, renders a user page, and then links to a user "Spore assets" page (that's the thumbnail image above).

One thing that might be fun to do is write a script that checks for the latest achievements and publishes them to various microblog/status sites with the Twisted PyngFM client :-)

There's a project page up on Launchpad for txSpore, and I've posted a notice and some updates to the Spore developer forums. It's also been published on PyPI.

Enjoy!

Windows Media to MP3 Conversion for Mac OS X and Linux

For the past couple years, my girlfriend has been amazingly (astonishingly) patient about a whole slew of .wma files that we've got on the network drive... backups of her CD collection made when she was a Windows user. We managed to save them right before the computer died, but she hasn't been able to listen to them when she's booted into Ubuntu or Mac OS X.

Late last month, after getting back from two weeks abroad, Marjorie said that she'd really like to have access to her music collection again (the CDs are cumbersome and stored away in boxes for our impending move back to Colorado). With that said, I did some digging around, and found some immediately helpful links (two years ago, a few google searches had turned up results that indicated too much effort was involved).

I started out by trying a couple free Mac OS X GUI applications, but these ended up being quite horrible: either they did not offer the functionality I desired, they were buggy to the point of being unusable, or they rendered audio with unlistenable artifacts.

In the end, I had to use mplayer and lame in combination. After googling around and some trial and error, I discovered the combination of mplayer options that would successfully extract the audio data from .wma files and dump them as .wav files.

I started with a shell script, but quickly changed to Python, since there were several locations for the .wma files, and none of them on nice paths. I've used this script several times since then, when more .wma files were discovered, and have yet to encounter any issues in sound quality. Once nice-to-have would be to extract .wma metadata and save it in the new .mp3 files as id3 tags...

Anyway, here's the code:

#!/usr/bin/python
import os
import re
import subprocess
import sys


# script configuration
if sys.platform == "darwin":
    MPLAYER_PATH = os.path.join(
        "/Applications/Non-Standard/Audio and Video",
        "MPlayer OS X 2.app/Contents/Resources/mplayer.app/Contents/MacOS")
    # lame was manually installed into /usr/bin
    LAME_PATH="/usr/bin"
elif sys.platform == "linux2":
    MPLAYER_PATH = "/usr/bin"
    LAME_PATH="/usr/bin"
MPLAYER = os.path.join(MPLAYER_PATH, "mplayer")
LAME = os.path.join(LAME_PATH, "lame")
DUMP_FILE = "audiodump.wav"
BACKUP_DIR = "wma"
WORKING_DIR = "/tmp"


def make_audio_dump(filename):
    """
    Use mplayer to dump the audio contents of the .wma files as .wav files.
    """
    command = ("\"%s\" -nosound -vo null -vc dummy -af resample=44100 -aid 1 " 
               "-ao pcm:waveheader \"%s\"" % (MPLAYER, filename))
    subprocess.call(command, shell=True)


def convert_wav_to_mp3(input, output):
    """
    Use lame to convert the .wav files to .mp3 files. Remove the raw .wav file
    when done.
    """
    command = "\"%s\" -b 256 -h \"%s\" -o \"%s\"" % (LAME, input, output)
    subprocess.call(command, shell=True)
    os.unlink(input)


def convert_wma_to_mp3(wma_filename):
    """
    Given a .wma filename, get a filename for the new .mp3 file based on this,
    convert the original to a .wav and then that to an .mp3 file.
    """
    mp3_filename = re.sub("\.wma$", ".mp3", wma_filename)
    make_audio_dump(wma_filename)
    convert_wav_to_mp3(DUMP_FILE, mp3_filename)


def has_wma_files(filenames):
    """
    Given a list of filenames, check to see if any of them have the .wma file
    extension. If so, return a true value; otherwise, a false one.
    """
    for filename in filenames:
        if filename.endswith(".wma"):
            return True
    return False


def convert_wma_files(path):
    """
    Walk a given file system directory and all its child directories in order
    to find .wma files. If found, convert them to .mp3 files and backup the
    originals.
    """
    for dir, subdirs, filenames in os.walk(path):
        # we don't want to convert files that have already been converted
        if os.path.basename(dir) == BACKUP_DIR:
            continue
        # if there's nothing to do, move on
        if not has_wma_files(filenames):
            continue
        # define and create the backup dir, if it hasn't been already
        backup_dir = os.path.join(dir, BACKUP_DIR)
        if not os.path.exists(backup_dir):
            os.mkdir(backup_dir)
        for filename in sorted(filenames):
            # on mac os x samba shares, sometimes ._*.wma files are present;
            # skip these
            if filename.startswith("."):
                continue
            if filename.endswith(".wma"):
                print "Dumping audio for %s ..." % (filename)
                wma_filename = os.path.join(dir,  filename)
                wma_backup = os.path.join(backup_dir, filename)
                convert_wma_to_mp3(wma_filename)
                os.rename(wma_filename, wma_backup)


if __name__ == "__main__":
    path = sys.argv[1]
    os.chdir(WORKING_DIR)
    convert_wma_files(path)

Hope someone else finds this useful and their significant others don't have to wait 2 years for their music!

Twisted Ping.fm Client

I just merged async (Twisted) support into the Python ping.fm library today and have just taken it for a test drive. I do love Twisted :-) The Twisted pyngfm API usage is identical to the synchronous API, with the usual exception of using deferreds and callbacks.

Here's some example usage, the client I now use for command-line updates to Twitter, Identi.ca, Tumblr, Facebook, LinkedIn, Jaiku, and even Flickr (note that the keys are stored in an .ini-style config file):

#!/usr/bin/python
import sys
from ConfigParser import SafeConfigParser

from twisted.internet import reactor

from pyngfm.client import PingFMAsyncClient


def checkMessage(message):
    if len(message) > 140:
        print "Message is too long! (%s chars)" % len(message)
        sys.exit(1)


def getKeys():
    cred_file = "/etc/ping.fm.creds"
    config = SafeConfigParser()
    config.read([cred_file])
    api_key = config.get("default", "api-key")
    user_app_key = config.get("default", "user-app-key")
    return api_key, user_app_key


def pingIt(message):

    def check_result(status):
        print status

    def check_error(error):
        print error.getErrorMessage()

    def finish(ignored):
        reactor.stop()

    pinger = PingFMAsyncClient(*getKeys())
    deferred = pinger.user_post("status", body=message)
    deferred.addErrback(check_error)
    deferred.addCallback(check_result)
    deferred.addErrback(check_error)
    deferred.addCallback(finish)


if __name__ == "__main__":
    message = " ".join(sys.argv[1:])
    checkMessage(message)
    pingIt(message)
    reactor.run()

There's another example in the README that iterates through the recents posts made to ping.fm. If you do manage to use it and come across any issues, be sure to file a ticket.

Enjoy!

Possible Directions in Human Resource Management

This afternoon, while having a nice soak in the tub, I got lost in a Strossian reverie. Or perhaps it was more along the lines of one of his characters, rather than the author himself. Regardless, I was thinking about management styles in modern corporations... and ways in which one might improvise.

I was only half paying attention, until the stream of thought actually started getting interesting. At which point I sat upright in the tub and started taking mental notes. Quickly showering off, I couldn't stop the flood of ideas, hoping to get to a written medium as soon as possible, lest they be forgotten.

I forget how the day-dreaming started... perhaps my usual: pick a random topic where I have some experience, and start playing with it. Run simulations and tweak parameters until there's nothing more to look at or some interesting permutation has popped up. 

Ah! Now I remember. I was thinking about the increasing importance of QA in software as more and more marketshare is gained. Not from the usual perspective ("we gotta make sure users don't see bugs, or our shit's gonna flop"), but rather as part of the original problem domain.  It's very common for open source projects to suffer from a lack of sex appeal when considering QA. QA is essential for success; just as important as the code itself. As such, there should be ways of making QA as interesting, engaging, understood, and respected as the act of programming.

That train of thought spawned a couple more pathways for exploration, but the one that ended up being the most interesting was pondering the QA-interest problem at a human resource management level. What would it take to get talented and skilled engineers who would normally gravitate towards some other field of expertise interested in QA instead? How flexible would a company have to be to start attracting for this and other positions as new needs arose due to new pressures? How could it do so by growing and adapting from the inside?

Let's take these rather arbitrary initial conditions as part of a thought experiment following up on those questions:

• a company with a sufficient number of employees to support multiple departments
  
• a highly intelligent, motivated workforce
  
• flexible employees and teams, probably working in a distributed, remote environment

Imagine with this sort of company that employees are not locked into teams, departments, divisions, etc. Let's say there is some mechanism which allows for an easy re-shuffling of talent throughout the company. For budgeting and reporting purposes, corporate structures would remain, but the individuals that performed specific tasks were granted much more expansive, organic freedoms when it came to projects and tasks.

From the individual's perspective, an employee could choose to work for and closely associate with whatever team they wanted to spend some time with, given of course, that this team could make use of the newly reallocated employee's particular skills and abilities. This is somewhat analogous to the geographical freedoms one has when working remotely: you can move anywhere in the world, to whichever time zone you prefer, whatever culture you want to enjoy, given that you can continue to work effectively. 

Of course, checks and balances would have to be introduced. Otherwise, what's to prevent an employee from team-hopping, and never really getting any work done? Just playing social butterfly? Karma/experience points could be earned on per task, project, team, department, and division bases. Moving at any particular level to a different group would reset an employee's karma for that group and all other organizational units below it. For instance, changing to another department would zero out any team, project, and unfinished task karma that had been accumulated. Perhaps some of the patterns from role playing games could come in handy as sources of inspiration.

The ability to opt for a move could also be governed by an appropriate accumulation of karma points for a given organizational unit, to be determined at the discretion of that organizational unit. For example, hitting a certain karma "level" would allow one new movement opportunities. Obviously, in addition to having enough points, the target group would have to have a need and be willing to bring the requesting person on board.

Intuitively (i.e., without doing any research or checking sources), this feels like something modeled after biological systems, as opposed to the fairly static forms of systems organizations we currently see in many corporations. A company that was able to successfully implement such a human resource management approach could likely see additional, unexpected benefits. Biological systems tend to be well-suited to surviving shock, drastic changes, massive failures. We might see companies with longer-reaching vision, more innovation, less frequent employee turnover, and greater financial stability.

That aside, what appeals to me at a personal level is this: employees would be participating in their work at a new level. The management processes behind professional development would be opened up to them. Not only that, part of one's work would become a game with known rules and clearly defined markers for cumulative achievements. Rewards, though, would not be power-climbing, but rather lateral expansion and exploration. Deeper involvement in other areas of the company.

I'd love to hear from folks who can recommend some good reading/research materials on this topic...

Mac OS X: Execute Shell Commands via Icon-Clicks

My main development machine is a custom PowerBook running Ubuntu natively. I use it when I'm sitting on the couch, my office comfy chair, the futon, floor, etc. Every once in a while, though, I want to work at a desk from my 24" iMac. Just to mix it up a little. However, that box is my gaming and web-browsing machine: it runs Mac OS X and that's the way I want to keep it. So, if I'm going to do work on the iMac, I need to ssh into the machines that have the environments set up for development.

In the course of an average day of writing code, I'll connect to anywhere from 1 to 5 remote machines open up 5-10 ssh sessions in a terminal to each machine. If I'm at the iMac, this get's tedious. Today, it got tedious enough for me to do somthing about it. Here's what I want: to click on a Terminal icon and have an ssh connection automatically established to the box I need to work on. This it pretty easy on Linux and Windows, but I had no idea how to accomplish this on a Mac until tonight.

I thought I'd share my solution; others may like it... but I'm betting there are some pretty cool ways of doing this that didn't occur to me -- so feel free to share yours!


Profile Hack

From previous messing about with the open command, I knew I could open Terminal.app from the terminal:

open -n "/Applications/Utilities/Terminal.app"

This got me part way there... if only I could dynamically execute a command upon login... so, yeah, I did something nasty:

vi ~/.bash_profile

And then:

if [ ! -z "$REMOTE_CONNECTION" ]; then
    ssh $REMOTE_CONNECTION
    REMOTE_CONNECTION=""
fi

.command Files

I was stumped at that point, until some googling revealed a nifty trick I didn't know about:

• Create a new file in your favorite editor, using the .command extension
  
• Add the commands you want executed
  
• Save it and chmod 755
  
• Double-click it and enjoy

So here's what I added to rhosgobel.command:

REMOTE_CONNECTION=rhosgobel \
   open -n "/Applications/Utilities/Terminal.app"

The Obligatory Icon Tweak

I then used the standard "Get Info" trick of icon copying: "Get Info" for Terminal.app, copy icon, "Get Info" for all my .command files, paste icon.


Usage

Now, I just click my "Shells" menu, choose the destination, and start working on that machine. A new window or new tab opened with that instance of Terminal.app will give me a new session to that server, without having to manually ssh into it -- this is even more convenient than having an icon to double-click!

One bit of ugly I haven't figured out how to remove: when I open a shell to a remote server, there's another shell opened at the same time with a [Process completed] message.