The Schro is Back

Well, after year and a half of no posts, Schrödinger's Böx is back, with three new entries:

http://schrobox.blogspot.com/

The reason for the long gap is that feedster.com had stopped publishing their top 500 list. The list was pretty convenient, since they provided RSS links of each blog's full recent entries. Parsing them was cake. When that went away, so did posts for SchroBox.

I had a few minutes this morning to play, so I hacked on the really awful code to add support for parsing Technorati's top 100. This should make Greg very happy, as he has oft lamented the early and sad demise of The Box.

As for what this madness is and where it comes from, this post tells all (hint: it uses NLTK extensively).

Nevow Athena: AJAX/COMET with Twisted

Some of you may remember when, given 5 seconds to say something about Nevow's AJAX capabilities at the PyCon Web Framework panel, I said "we were there first." I was serious about that. If you take the time to dig through the dusty attics of Twisted's and Divmod's svn repositories, you'll see it too.

Not that I blame anyone for not knowing, though. If there are any two crimes that Twisted devs are guilty of (among potentially uncountably many) they would be:

1. being frighteningly clever, and
2. not sharing the useful bits of the cleverness with the world at large.

We're trying hard to really start sharing the goodness, though, honest!

Along these lines, we've got a series of tutorials about Athena that will be published "real soon, now." Initially, we're going to put out some intro text and two basic tutorials. Follow-on tutorials will cover advanced topics and/or basic ones in more detail.

It will be very exciting, though, to bring the power and elegance of Athena to the masses :-)

Personally, I've found that writing Athena apps (with its tightly bound Python and JS) provides ease of use and peace of deterministic structure in an arena (AJAX/COMET application development) that is usually littered with the bodies and casualties of poor planning, poorly established boundaries between application components (in requirements, architecture, and implementation), and poor education.

I look forward to community feedback on the tutorials: it's a complicated topic that will be condensed into only a little code in each fully functional example. The need for clarity and the defeat of confusion is paramount.

Technorati Tags: ajax, comet, divmod, documentation, howto, python, twisted

Merging New trunk Features to a Development Branch

When I was on the development team for Zenoss, I tried to get them to use branch-based development, including Divmod Combinator. They eventually did switch to using branches for any big changes, but they didn't embrace Combinator. When I'd taken over development of the Zenoss community code, there were only two developers, so I only implemented a very basic branch management process. But having used Combinator a lot recently, I wish that I'd taken that opportunity to get them using Combinator... man, their lives would be so much easier right now if I'd taken that step.

I am now managing all of my svn-based projects with Combinator. Even the ones were I am sole developer. The time saved and convenience gained is enormous.

Below is a perfectly good example of how Combinator makes svn management less painful. I don't know about you, but every time I have to manually to push new trunk changes into my current development branch, my brain does a little freak-out dance. I stress, thinking "Crap; I have to get this right. What are the exact commands I have to use again?" Thanks to Combinator, this is no longer a problem, and my brain does a happy dance instead.

These days, I am using Combinator on an almost hourly basis. It's very simple (to get setup with Combinator, see the tutorial and other example usage).


Background


Assuming you've got a Combinator-managed project called "Project", let's do some background first. Say you want to branch trunk for a new feature, one that you're tracking in ticket #836. Here's how:

$ mkbranch Project viking-feature-836

Now, just to make sure that you are no longer in trunk, check which branch you are in:

$ whbranch
Project: viking-feature-836
Divmod: trunk
Twisted: trunk

(Note that every project that you are managing with Combinator will be listed.)

Say you've got another ticket you're working on, #1066. You created the branch with this command, just like the other branch:

$ mkbranch Project norman-feature-1066

After starting this branch, you complete 836 and merge to trunk:

$ chbranch Project viking-feature-836
$ unbranch Project
$ svn commit -m "Completed 836."

unbranch updates trunk with the latest changes and then merges the branch it just came from (viking-feature-836) into trunk. Running svn diff will show you just how true this is.


Merging


So much for the background! Now for the merging I'd promised:

If the norman feature depends on the viking feature, we need to "merge" trunk to branch. Using Divmod Combinator, this is how that is accomplished:

$ chbranch Project norman-feature-1066
$ unbranch Project
$ mkbranch Project norman-feature-1066-2
$ cd ~/Project/branches/norman-feature-1066-2
$ svn commit -m "Merged changes from norman-feature-1066."

In other words, we:

1. merge our work to date from norman-feature-1066 to trunk (without committing!)
2. make a new branch, based on that trunk, which includes the norman feature to-date and all the latest changes made to trunk
3. commit the branch changes to our new branch

We're using trunk to get what we need by only making changes to trunk in the working directory, without making any changes in the repository. Keep in mind that the changes from the first branch were added (and not committed) to trunk before creating the second branch. Since the second branch was created from this modified trunk, we will need to commit those additions. Also, you will most definitely want to revert your tweaked trunk once you have finished creating your new, updated branch!


Paths


As a side note, Combinator is especially excellent for python projects, because of its python path management features. Any python tools, scripts, pluggins, or applications that use the code in your Combinator-managed projects will import from the current, active code (whatever shows up in whbranch). So when you're working in branch and you run your unit tests, you know that it's the branch code that's getting tested. When you switch back to trunk, all your code is running against trunk. Very nice :-)

Technorati Tags: branch-based development, divmod, python, twisted, version control

Hello, Divmod

As I've mentioned before, the first time I had face-to-face time with Glyph and Amir was not too long after they'd formed Divmod, at pycon 2004. I'd toyed with Twisted for about two years prior to that, and the stuff they were doing with Quotient was just amazing (better feature sets than gmail). So, when they reached out after I'd blogged about my Zenoss contract coming to an end, I was just thrilled. We had some exciting talks about current development and the future, and given my long-standing love of Twisted and Divmod products (such as Nevow), wild horses couldn't have dragged me away.

I consider the core Twisted developers (and Divmod programmers) to be some of the best in the world. To be working with these guys is a dream come true and I am giddy with anticipation of both what we will be producing and all that I will be learning.

And now, I would like to take a moment to give some love to the python community in general: you guys are awesome.

I know you already know this, but one can never say it enough :-) I had all sorts of people and companies offer really cool positions when they heard I was available. There are some exciting startups that need developers and fantastic non-profits (that pay salaries) that would be great fun to work for (and feel good about). Also, my favorite dev team at my favorite Linux distribution's parent company reached out; they totally rock and I want to thank them for that (you know who you are!).

Two things I learned: VMware uses python and even better, LucasFilm uses Twisted. How totally cool is that? Hopefully we'll be talking to them on the TwistedShow sometime in the near future...

Technorati Tags: divmod, work, twisted

Skater Heaven

So I completely flipped out this morning before breakfast: I looked out the bedroom window after a morning conference call, and there are skaters slaloming down the road behind the house. I've got a 3' Sector9 bamboo longboard and love to get me some slalom action. But I suck. These guys were awesome.

I grabbed Sengey (my girlfriend's Tibetan mastif) and took him out for a walk up the hill. The skaters were amazingly friendly and totally mellow; they jumped at a chance to tell someone about their event. Turns out today is day one of the Seismic Nationals which is being held in my back yard!

I took my phone with me and snapped some shots of the skaters, and they are here in this flickr set. Enjoy!

We've been renting this house month-to-month while waiting to move into the house in Valentine, NE. It's a McMansion and not cozy at all, so this skating event has made the past two months here totally worth while.

Update: a sweet video of the same event last year.

Technorati Tags: skating, colorado

One Number to Rule Them All and in the Phone Book Bind Them

Thanks to Grand Central, I've got a new perma-number. This service completely rocks. I've already got my current temporary house number connected to it, and when I move in a week, I'll be updating it with that number.

Here are the features I love:

• being able to chose you number
• no more hated phone voice mail -- I can check all voice mail on line
• I get an email for every new voicemail
• I can share voice mails with friends with a link

Yeah, I have voice mail issues...

I believe registration is open now, but ping me if you need an invite. Don't for get to use this once you've got your new number.

Ubuntu 7.04 on iMac Core 2 Duo, 2G

Well, I busted me arse this afternoon and evening trying to get Ubuntu set up on my new iMac. It installed without issue (using the "alternate" x386 iso), but has the following critical issues:

1. No wireless
2. No widescreen support
3. No sound
4. Doesn't recognize CD/DVD drive after install and reboot

I'm kind of bummed, since I'd planned on doing dev work on this box (can we even call the iMacs "boxes"?) in native Ubuntu, and now I'll have to use parallels until the driver support is developed. Or until someone sends me a link that I've managed to miss despite incessant googling today...

Once I'm moved into the new place, I'll have a GB switch in the office and will be able to work in Ubuntu over copper. The resolution bugs me more than the lack of sound and DVD, but I can live with those for now. I guess.

But, hey -- it plays WoW and UT2004 in OS X like a champ :-)

Technorati Tags: apple, ubuntu, mac os x, linux

publish-bot with logging capability

Last year sometime, I received several emails from people wanting me to release the supybot IRC plugins I wrote back in 2005 for svn and RSS updates. I'd actually written for several different IRC bots (both twisted-based, and non-twisted bots) but the code was particularly messy for supybot. During one conversation, I said that I'd really done it all wrong, and the last thing I wanted to do was clean up wrong code. If I got a chance, I would do it right:

• create an IRC client for communicating to servers+channels
• create a message server running on local host
• process/parse data from the local server to the IRC client (i.e., appropriate IRC channel)

Such a flexible design would suit every need I have for writing data to an IRC channel.

Turns out radix did just so earlier this year for Canonical, as he relates in this blog post. This is exactly what I needed this weekend while migrating my old IRC bot to a new server. Well, almost all I needed. There were a couple features I wanted to add...

Not having worked with bzr or launchpad much, I asked radix about branching his code, since I heard that Canonical actually provides developers a way of doing that without leaving the launchpad environment of trunk. His instructions were quite simple, and I summarize them here for the benefit of others:

• find the project you want to branch on launchpad
• checkout trunk (if that's what you're branching)
• commit (a local action) with a message about this being a new branch
• make your changes (with any number of commits)
• push all your commits to your branch on launchpad

The steps above assume that you have bzr installed and have uploaded your SSH keys to launchpad.

Here are the bzr commands that I used:

bzr branch http://bazaar.launchpad.net/~radix/publish-bot/trunk publish-logger-bot
cd publish-logger-bot
bzr commit --unchanged -m "Created branch of radix's publish-bot."
bzr push sftp://oubiwann@bazaar.launchpad.net/~oubiwann/publish-bot/publish-logger-bot

The changes I committed:

• enabled logging
• added log-rotation
• added an HTTP service for browsing the log files

The code is available here:
https://code.launchpad.net/~oubiwann/publish-bot/publish-logger-bot

Forthcoming features:

• a search form for the log files
• a parser for bot commands, thus enabling human -> bot communications (will probably dust off my old pyparsing code for this)
• arbitrary time log rotation (currently it's by day, ad midnight)

And for dessert:

[18-Aug-2007 17:56:50] [connected at Sat Aug 18 17:56:50 2007]
[18-Aug-2007 17:57:00] [wallflower (logger bot) has joined #supersecretchannel]
[18-Aug-2007 18:00:38] <keturn> what's a wallflower?
[18-Aug-2007 18:20:09] <PenguinOfDoom> It's a flower that grows out of a wall
[18-Aug-2007 18:20:52] <keturn> and has an irc client?
[18-Aug-2007 18:22:49] <keturn> is it related to an oubiwann?
[18-Aug-2007 18:34:00] <exarkun> I reckon it is

Technorati Tags: bzr, canonical, irc, python, software, twisted, web

Export iTunes Playlists as .m3u Files

I got really pissed at iTunes today: it crashed and then wouldn't start back up, throwing the infamous 'locked file' error. In exasperation, I tried Songbird. It looks promising, but still needs lots of work and didn't play my music-over-Samba very well. I then got the latest Cog release.

In my experience, Cog is simple, straight-forward, performs no voodoo and just works (Troll: "Stay away from the voodoo!"). I really like the fact that I can browse the file system in Cog, and drag the albums into the list pane, but I didn't want to start years worth of playlists all over again...

At which point I remembered that a couple of years ago I blogged about parsing iTunes playlists with ElemetTree. It looks like the effbot archives no longer point to the link I updated that post with, but google cache served me well and I found Fredrik Lundh's code, repasted here:

try:
    from cElementTree import iterparse
except ImportError:
    from elementtree.ElementTree import iterparse
import base64, datetime, re

unmarshallers = {

    # collections
    "array": lambda x: [v.text for v in x],
    "dict": lambda x:
        dict((x[i].text, x[i+1].text) for i in range(0, len(x), 2)),
    "key": lambda x: x.text or "",

    # simple types
    "string": lambda x: x.text or "",
    "data": lambda x: base64.decodestring(x.text or ""),
    "date": lambda x:
        datetime.datetime(*map(int, re.findall("\d+", x.text))),
    "true": lambda x: True,
    "false": lambda x: False,
    "real": lambda x: float(x.text),
    "integer": lambda x: int(x.text),

}

def load(file):
    parser = iterparse(file)
    for action, elem in parser:
        unmarshal = unmarshallers.get(elem.tag)
        if unmarshal:
            data = unmarshal(elem)
            elem.clear()
            elem.text = data
        elif elem.tag != "plist":
            raise IOError("unknown plist type: %r" % elem.tag)
    return data

Which was a great start, but I needed .m3u output. After reading how simple the format was, I was off and running. The end result was all of my iTunes playlists playable by Cog, which I am using now -- as I type -- to enjoy my music, free of pain. One thing worth exploring would be how to preserve the ordering of iTunes' playlist items.

Here's the code I used to "export" the iTunes playlists as .m3u:

import re

m3uList = "#EXTM3U\n%s\n"
m3uEntry = "#EXTINF:%(length)s,"
m3uEntry += "%(artist)s - %(album)s - %(song)s\n%(filename)s\n"

def phraseUnicode2ASCII(message):
    """
    Works around the built-in function str(message) which aborts when non-ASCII
    unicode characters are given to it.

    Modified from http://mail.python.org/pipermail/python-list/2002-June/150077.html
    """
    try:
        newMsg = message.encode('ascii')
    except (UnicodeDecodeError, UnicodeEncodeError):
       chars=[]
       for uc in message:
          try:
             char = uc.encode('ascii')
             chars.append(char)
          except (UnicodeDecodeError, UnicodeEncodeError):
             pass
       newMsg = ''.join(chars)
    return newMsg.strip()

class Playlists(object):

    def __init__(self, filename=None, destDir=None):
        self.lib = None
        if filename:
            self.lib = load(filename)
        if not destDir:
            destDir = './'
        self.destDir = destDir

    def processTrack(self, trackData):
        length = trackData.get('Total Time') or 300000
        song = trackData.get('Name') or 'Unknown'
        artist = trackData.get('Artist') or 'Unknown'
        album = trackData.get('Album') or 'Unknown'
        data = {
            'filename': trackData['Location'],
            'length': int(length)  / 1000 + 1,
            'song': phraseUnicode2ASCII(song),
            'artist': phraseUnicode2ASCII(artist),
            'album': phraseUnicode2ASCII(album),
        }
        return m3uEntry % data

    def processTrackIDs(self, ids):
        output = ''
        for id in ids:
            try:
                trackData = self.lib['Tracks'][str(id)]
                output += self.processTrack(trackData)
            except KeyError:
                print "Could not find track %i; skipping ..." % id
        return output

    def cleanName(self, unclean):
        clean = re.sub('[^\w]', '_', unclean)
        clean = re.sub('_{1,}', '_', clean)
        return clean

    def exportPlaylists(self):
        for playlist in self.lib['Playlists']:
            playlistName = self.cleanName(playlist['Name'])
            try:
                items = playlist['Playlist Items']
            except KeyError:
                print "Playlist seems to be empty; skipping ..."
                continue
            trackIDs = [x['Track ID'] for x in items]
            data = m3uList % self.processTrackIDs(trackIDs)
            fh = open("%s/%s.m3u" % (self.destDir, playlistName), 'w+')
            fh.write(data)
            fh.close()

def exportPlaylists(filename, dest=None):
    pls = Playlists(filename, dest)
    pls.exportPlaylists()

With usage like the following:

>>> from iTunesExport import exportPlaylists
>>> BASE = "/Volumes/itunes/__Playlists__"
>>> exportPlaylists('%s/Library.xml' % BASE, BASE)

Update: I've tweaked the code in this post a little bit, due to a reader's questions. To run this, copy both code blocks into a single file you should be good to go.

OpenOffice, Python and Plone... and Java

The last big project I'm working on for Zenoss is an OpenOffice-to-Plone publisher. Specifically, for the Zenoss Guide (combined admin and user manuals). The Guide is maintained in OpenOffice format, but in addition to .sxw/.doc and .pdf formats, they want to publish as HTML, where each section of the doc gets its own page in the Zenoss Community portal and users have the ability to comment on each section.
There are several things I'm using to implement this:

• Zope/Plone and Five
• OOoPy (for processing sections and creating .sxw files)
• lxml.etree (for preserving the original XML namespace names)
• writer2LaTeX (for converting generated .sxw files to HTML)

Sadly, I spent a great deal of time trying to figure out why w2l (writer2LaTeX) wasn't converting sections with images; I won't relay the horrors of debugging. Finally, I tossed in the towel and emailed the author, Henrik Just. He was phenomenally helpful... and he uncovered the underlying issue, one that I consider to be a dirty little secret (though perhaps those familiar with the combinations of Zip files, python and Java consider an openly acknowledged issue). From Henrik:

A little debugging showed that the java's zip classes causes the problem. Unfortunately they are not very tolerant with variations of the structure of zip files. With java 1.4, I cannot open the file at all. This, it turns out, is due to a bug that was fixed in java 1.4.2. But even with java 1.5 I can only read 5 of the 12 files in the zip file, which causes the odd behaviour you have seen.

I don't know who to blame here: Python or Java. Regardless, doesn't this seem absurd? That the standard compression format used to create .zip files is not implemented completely either in Python, Java or both? Though not completely surprised, I was surely flabbergasted. Despite my frustration, Henrik was helpful in offering some alternatives, as well as using w2l with an option for splitting a doc by headings.

Yet another example of the phenomenal goodness that is the community of open source developers (if not the languages used in that community...). Thanks Henrik!

Technorati Tags: community, five, openoffice, plone, java, python, zope