Tuesday, July 31, 2007

Export iTunes Playlists as .m3u Files

I got really pissed at iTunes today: it crashed and then wouldn't start back up, throwing the infamous 'locked file' error. In exasperation, I tried Songbird. It looks promising, but still needs lots of work and didn't play my music-over-Samba very well. I then got the latest Cog release.

In my experience, Cog is simple, straight-forward, performs no voodoo and just works (Troll: "Stay away from the voodoo!"). I really like the fact that I can browse the file system in Cog, and drag the albums into the list pane, but I didn't want to start years worth of playlists all over again...

At which point I remembered that a couple of years ago I blogged about parsing iTunes playlists with ElemetTree. It looks like the effbot archives no longer point to the link I updated that post with, but google cache served me well and I found Fredrik Lundh's code, repasted here:

from cElementTree import iterparse
except ImportError:
from elementtree.ElementTree import iterparse
import base64, datetime, re

unmarshallers = {

# collections
"array": lambda x: [v.text for v in x],
"dict": lambda x:
dict((x[i].text, x[i+1].text) for i in range(0, len(x), 2)),
"key": lambda x: x.text or "",

# simple types
"string": lambda x: x.text or "",
"data": lambda x: base64.decodestring(x.text or ""),
"date": lambda x:
datetime.datetime(*map(int, re.findall("\d+", x.text))),
"true": lambda x: True,
"false": lambda x: False,
"real": lambda x: float(x.text),
"integer": lambda x: int(x.text),


def load(file):
parser = iterparse(file)
for action, elem in parser:
unmarshal = unmarshallers.get(elem.tag)
if unmarshal:
data = unmarshal(elem)
elem.text = data
elif elem.tag != "plist":
raise IOError("unknown plist type: %r" % elem.tag)
return data

Which was a great start, but I needed .m3u output. After reading how simple the format was, I was off and running. The end result was all of my iTunes playlists playable by Cog, which I am using now -- as I type -- to enjoy my music, free of pain. One thing worth exploring would be how to preserve the ordering of iTunes' playlist items.

Here's the code I used to "export" the iTunes playlists as .m3u:

import re

m3uList = "#EXTM3U\n%s\n"
m3uEntry = "#EXTINF:%(length)s,"
m3uEntry += "%(artist)s - %(album)s - %(song)s\n%(filename)s\n"

def phraseUnicode2ASCII(message):
Works around the built-in function str(message) which aborts when non-ASCII
unicode characters are given to it.

Modified from http://mail.python.org/pipermail/python-list/2002-June/150077.html
newMsg = message.encode('ascii')
except (UnicodeDecodeError, UnicodeEncodeError):
for uc in message:
char = uc.encode('ascii')
except (UnicodeDecodeError, UnicodeEncodeError):
newMsg = ''.join(chars)
return newMsg.strip()

class Playlists(object):

def __init__(self, filename=None, destDir=None):
self.lib = None
if filename:
self.lib = load(filename)
if not destDir:
destDir = './'
self.destDir = destDir

def processTrack(self, trackData):
length = trackData.get('Total Time') or 300000
song = trackData.get('Name') or 'Unknown'
artist = trackData.get('Artist') or 'Unknown'
album = trackData.get('Album') or 'Unknown'
data = {
'filename': trackData['Location'],
'length': int(length) / 1000 + 1,
'song': phraseUnicode2ASCII(song),
'artist': phraseUnicode2ASCII(artist),
'album': phraseUnicode2ASCII(album),
return m3uEntry % data

def processTrackIDs(self, ids):
output = ''
for id in ids:
trackData = self.lib['Tracks'][str(id)]
output += self.processTrack(trackData)
except KeyError:
print "Could not find track %i; skipping ..." % id
return output

def cleanName(self, unclean):
clean = re.sub('[^\w]', '_', unclean)
clean = re.sub('_{1,}', '_', clean)
return clean

def exportPlaylists(self):
for playlist in self.lib['Playlists']:
playlistName = self.cleanName(playlist['Name'])
items = playlist['Playlist Items']
except KeyError:
print "Playlist seems to be empty; skipping ..."
trackIDs = [x['Track ID'] for x in items]
data = m3uList % self.processTrackIDs(trackIDs)
fh = open("%s/%s.m3u" % (self.destDir, playlistName), 'w+')

def exportPlaylists(filename, dest=None):
pls = Playlists(filename, dest)

With usage like the following:

>>> from iTunesExport import exportPlaylists
>>> BASE = "/Volumes/itunes/__Playlists__"
>>> exportPlaylists('%s/Library.xml' % BASE, BASE)

Update: I've tweaked the code in this post a little bit, due to a reader's questions. To run this, copy both code blocks into a single file you should be good to go.

Saturday, July 28, 2007

OpenOffice, Python and Plone... and Java

The last big project I'm working on for Zenoss is an OpenOffice-to-Plone publisher. Specifically, for the Zenoss Guide (combined admin and user manuals). The Guide is maintained in OpenOffice format, but in addition to .sxw/.doc and .pdf formats, they want to publish as HTML, where each section of the doc gets its own page in the Zenoss Community portal and users have the ability to comment on each section.
There are several things I'm using to implement this:

  • Zope/Plone and Five
  • OOoPy (for processing sections and creating .sxw files)
  • lxml.etree (for preserving the original XML namespace names)
  • writer2LaTeX (for converting generated .sxw files to HTML)
Sadly, I spent a great deal of time trying to figure out why w2l (writer2LaTeX) wasn't converting sections with images; I won't relay the horrors of debugging. Finally, I tossed in the towel and emailed the author, Henrik Just. He was phenomenally helpful... and he uncovered the underlying issue, one that I consider to be a dirty little secret (though perhaps those familiar with the combinations of Zip files, python and Java consider an openly acknowledged issue). From Henrik:
A little debugging showed that the java's zip classes causes the problem. Unfortunately they are not very tolerant with variations of the structure of zip files. With java 1.4, I cannot open the file at all. This, it turns out, is due to a bug that was fixed in java 1.4.2. But even with java 1.5 I can only read 5 of the 12 files in the zip file, which causes the odd behaviour you have seen.
I don't know who to blame here: Python or Java. Regardless, doesn't this seem absurd? That the standard compression format used to create .zip files is not implemented completely either in Python, Java or both? Though not completely surprised, I was surely flabbergasted. Despite my frustration, Henrik was helpful in offering some alternatives, as well as using w2l with an option for splitting a doc by headings.

Yet another example of the phenomenal goodness that is the community of open source developers (if not the languages used in that community...). Thanks Henrik!

Technorati Tags: , , , , , ,

Twisted JSON-RPC on Google Code/Projects

I've been getting lots of emails from folks asking for the Twisted JSON-RPC code that I had put up a while ago. They weren't able to find it due to my having to bring down the trac instances I was running. Good news: I've moved the repository up to Google:
The front page has a link to the wiki which in turn has all the usage info. You can get recent eggs here:
If you want to contribute to this code base and bring it up to date, email me.


Update: Primary development has moved to Launchpad. Documentation is maintained on the Ubuntu.com wiki page here.

Monday, July 23, 2007

Farewell, Zenoss

Well, python community, I am back on the market.

After a year and half of consulting for Zenoss, where I contributed to the Zenoss code base and was lead dev on the community development effort, I am on my own again.

It's been pretty amazing having a front-row seat on the growth of Zenoss as a company. When I fist started consulting for them, there were only three employees and the general view was that most of the employees would end up being remote developers (this didn't end up being the case, as they found a fair number of local python developers). I was actually on-site when they interviewed Drew "the Docs guy", who ended up being employee #4. Since then, the number of consultants and employees has increased as greater and greater focus has been placed on non-dev business like marketing, sales and customer support. It was kind of strange being there this week... while at a party, I remarked to Erik Dahl that I barely recognized the place anymore.

I do feel badly that I have poured so much time and work into the recent Zenoss community effort and not had the time to contribute to the Twisted and Divmod code bases (as well as other projects) like I had wanted to this Summer. Hopefully my next gig will provide for more support in that arena. At the very least, I could very well have some spare time in the next few months to tune back in and maybe even crank out some tickets :-)

Well, I'm off to pick up some red curry or pad thai at the Lemon Grass!

Technorati Tags: , , ,

Tuesday, July 10, 2007

Open Source Community Interaction: Is More Less?

I have a friend who needs our help. In fact, there are potentially disturbing ramifications in the trend mentioned below, so *we* may need our help.

Here's the deal: my friend is a research assistant at McGill University in Quebec. He is working with his professor with regard to the following topic:
"Open source software (OSS) developers who interact MORE with leaders (administrators) apparently contribute LESS (e.g., write less codes). We need more insights into this finding. Basically, we want to know why this is happening. We have speculated that open source software (OSS) communities are public, voluntary-based and self-organized communities. Therefore, if OSS leaders are too controlling, then developers don't contribute as much. This was one of our speculations. I am wondering if any OSS developers agree with this."
Sadly, very few of the developers their research team has contacted have given feedback on this matter. Below is the short list of questions they are asking developers. Note that though the example of contribution was code in the above statement, these questions assume contribution to take any of the following forms:
  • source code
  • documentation
  • answering questions and providing assistance to other users via mail lists, IRC channels, forums, etc.
  • grass-roots marketing
  • other forms of community involvement that contribute positively to an open source project
Okay, so here are the questions:
  1. Does the open source project's leader/lead developer have a huge impact on your contribution?
  2. What reasons would make developers who interact more with core project contributors apparently contribute less to the open source project?
  3. How would you feel if the leaders of the open source projects to which you contributed became too controlling?
I responded when Meral sent me his email with the two examples of Twisted and Python communities, but these guys need many more points of view than a few... and they aren't getting much traction. I feel that this research is important, and when published, could affect how open source projects and contributions to said projects are viewed by a wider population.

If this is also important to you, I encourage you to send an email to meral dot hussein at mail dot mcgill dot ca. Give it the subject of "McGill University Survey" and let 'er rip. The sooner you can do this, the better -- the deadline for the first round data is July 13th.

Technorati Tags: , , , , , ,

Tuesday, July 03, 2007

It's Been a Long Time

To use the words of Rakim (track 4), though without their import: it's been a long time. There's so much that's going on in my non-technical life, just thinking about blogging has seized up all my joints and limbs. Until now, that is.

To summarize:
  • the good folks at Zenoss keep me mercilessly busy where our community efforts are release-a-rific
  • I'm no longer living in Loveland, CO, but leasing month-to-month in other parts of this great State... and wondering where home base will end up being (I'm thinking of buying my first-ever actual property)
  • I've got a girlfriend for the first time in 4 years; her magical qualities defy enumeration, though I will say that she is a highly accomplished practitioner of the healing art of Jin Shin Jyutsu (studied with Mary Burmeister herself) as well as being a fellow meditator
  • I'm preparing to support two people at an up-coming Lakota Sundance in South Dakota
  • In addition, I'm preparing for my first shedra classes this fall (specifically, Madhyamaka)
Since I now need a nap after deciding which (of the millions of) things to recount/summarize, the rest of this post will be dedicated to game music, in particular System Shock 2 :-)

Check out this thread. In particular, scroll down to "Reply #6 on: 23. April 2007, 13:05:18" and witness the goodness:

"These are the real arrangements created by Eric Brosius himself, authored as full soundtrack songs and burned as an album. Crafted from the original audio, unlike the 22khz audio files from the shipped game, this release is CD quality."

Technorati Tags: , , ,