OSCON 2014 Theme Song - Andrew Sorensen's Live Coding Keynote

Andrew Sorensen live-coding at OSCON 2014

Keynote

Shortly after Andrew Sorensen began the performance segment of his keynote at OSCON 2014, the #oscon Twitter topic began erupting with posts about the live coding session. Comments, retweets, and additional links persisted for that day and the next. In short, Andrew was a hit :-)

My first encounter with Andrew's work was a few years ago when I was getting back into Lisp. I was playing with generative music with Overtone (and then, a bit later, experimenting with SuperCollider, Hy, and Twisted) and came across his piece A Study in Keith. You might want to take a break from reading this port and watch that now ...

When Andrew started up his presentation, I didn't immediately recognize him. In fact, when the code was displayed on the big screens, I assumed it was Clojure until I looked closely and saw he was using (define ...) and not (defun ...).  This seemed very familiar, and then I remembered Impromptu, which ultimately lead to my discovery of Extempore (see more links below) and the realization that this is what Andrew was using to live code.

At the end of the performance a bunch of us jumped up and gave a standing ovation. (In fact, you can hear me yell out "YEAH" at the end of his presentation when he says "And there we go."). It was quite a show. It seemed that OSCON 2014 had been given a theme song. The next step was getting the source code ...

Andrew's gist (Dark Github Theme)

Sharing the Code

Andrew gave a presentation on Extempore in the ballroom right after the keynote. This too was fantastic and resulted in much tweeting.

Afterwards a bunch of us went up front and chatted with him, enthusing about his work, the recent presentation, the keynote, and his previously published pieces.

I had Andrew's ear for a moment, and asked him if he was interested in sharing his keynote source -- there had been several requests for it on Twitter (that also got retweeted and/or favourited). Without hesitation, he gave an enthusiastic "yes" and we were off and running for the lounge where we could sit down to create a gist (and grab a cappuccino!). The availability of the source was announced immediately, to the delight of many.


Setting Up Extempore

Sublime Text 3 connected to Extempore

Later that night in my hotel room, I had time to download and run Extempore ... and discovered that I couldn't actually play the keynote code, since there was some implicit setup I was missing. However, after some digging around on the docs site and the mail list, music was pouring forth from my laptop -- to my great joy :-D

To ensure anyone else who is not familiar with Extempore can also have this pleasure, I've put together the all the prerequisites and setup necessary in a forked gist, in multiple parts. I will go through those in this blog post. Also: all of my testing and live coding was done using Ben Swift's Extempore Sublime Text plugin.

The first step is getting all the dependencies. You'll want to start the downloads right away, since they are large (the sample files are compressed .wavs). While that's going on, you can install Extempore using Homebrew (this worked for me on Mac OS X with no additional tweaking/configuration necessary):

With Extempore running, let's do some setup. We're going to need to:

• load some libraries (this takes a while for them to compile),
• define some samples, and then
• define some musical note aliases for convenience (and visual clarity).

The easiest way to use the files below is to clone the gist repo and load them up in Sublime Text, executing blocks of text by hi-lighting them, and then pressing ^x^x.

Here is the file for the fist two bullets mentioned above:

You will need to edit this file to point to the locations where your samples were downloaded. Also,

at the very end there are some lines of code you can execute to make sure that your samples are working.

Now let's define the note aliases. You can just hi-light the entire contents of this file in Sublime Text and then ^x^x:

At this point, we're ready to play!


Playing the Music

To get started on the music, open up the fourth file from the clone of the gist and ^x^x the root, scale, and left-hand-notes-* constants.

Here is the evolution of the left hand part:
Go ahead and start that first one playing (^x^x the definition as well as the call). Wait for a bit, and then execute the next one, etc. Once you've started playing the final left hand form, you can switch to the wider range of notes defined/updated at the bottom.

Next, you'll want to bring in the right hand ... then bassline ... then the higher fmsynth sparkles for the right hand:

Then you'll increase the energy with the drum section:

Finally, you'll bring it to the climax, and then start the gentle fade out:

A slightly modified code listing for the final keynote form is here:


Variation on a Theme

I have recorded a variation of Andrew's keynote based on the code above, for your listening pleasure :-) You can listen to it in your browser or download it.

This version plays part of the left hand piano an octave lower. There's a tiny bit of clipping in places, and I accidentally jazzed it up (and for too long!) with a hi-hat change in the middle. There are also some awkward transitions and volume oddities. However, these should be inspiration for you to make your own variation of the OSCON 2014 Theme Song :-)

The "script" used for the recording can found here.


Links of Note

Some of these were mentioned above, some haven't been. All relate to Extempore :-)

• A Study in Keith.
• The Disklavier Sessions - 2013
• Andrew on Github: https://github.com/digego
• Twitter: https://twitter.com/digego
• Extempore:
• Site: http://extempore.moso.com.au/
• Docs: http://benswift.me/extempore-docs/index.html
• Mail list: https://groups.google.com/forum/#!forum/extemporelang
• Source code: https://github.com/digego/extempore

Python and Rhythmbox

I've got a fairly large collection of digital music on a networked drive, and I access it from multiple machines on the network. I consolidated it 5 years ago when I started using iTunes, and over the past few years picked up about 400 songs from the iTunes store. This is something I avoid now, since Amazon offers songs at higher bitrates and without the crippling, non-Fair Use of DRM. (Apple seems to have recently changed it's policy, though the pain their DRM crap has caused me doesn't make me a very loyal customer.)

Since I started working at Canonical, I've been using Ubuntu for more than just development -- it's my main-use machine. I still use iTunes every once in a while, but my primary media player is Rhythmbox. There are a couple of issues with the old library, though.

Obviously, Rhythmbox can't play Apple's encrypted .m4p files or a couple audio book files I have. What's more, there are about 200 files that iTunes is able to locate but which Rhythmbox cannot (this may be due to the differing case sensitivities of the respective OSs). In order to track all these issues down conveniently, I wanted to export the import errors and missing files as a text file. Sadly, Rhythmbox doesn't have this functionality.

Fortunately, it comes with a Python console :-)

The missing files export was fairly easy, after some digging around and poking at the Python objects:

Try as I might, I was completely unable to obtain similar data for the import errors. After looking at the C code, I was able to determine that though the import errors were treated generally as a media source, due to their nature (not being able to provide the actual media itself), the related meta data was handled differently. Yet I wasn't able to decipher how, exactly.

So, I hopped on their mail list and asked for help :-) After a few quick exchanges, I was pointed in the right direction by one of the developers, who said that I needed to make use of the db object and some constants. After a quick test, this advice resulted in the following:


Note that the shell and rhythmdb objects are exposed by Rhythmbox in Python console sessions.

I now have a complete list of files that either need some file name updates or need to be burned to CD in iTunes and ripped to OGG.

So far, I've been pretty pleased with Rhythmbox. Thanks to their use of Python, I find I'm now becoming somewhat of a fan :-)

Export iTunes Playlists as .m3u Files

I got really pissed at iTunes today: it crashed and then wouldn't start back up, throwing the infamous 'locked file' error. In exasperation, I tried Songbird. It looks promising, but still needs lots of work and didn't play my music-over-Samba very well. I then got the latest Cog release.

In my experience, Cog is simple, straight-forward, performs no voodoo and just works (Troll: "Stay away from the voodoo!"). I really like the fact that I can browse the file system in Cog, and drag the albums into the list pane, but I didn't want to start years worth of playlists all over again...

At which point I remembered that a couple of years ago I blogged about parsing iTunes playlists with ElemetTree. It looks like the effbot archives no longer point to the link I updated that post with, but google cache served me well and I found Fredrik Lundh's code, repasted here:

try:
    from cElementTree import iterparse
except ImportError:
    from elementtree.ElementTree import iterparse
import base64, datetime, re

unmarshallers = {

    # collections
    "array": lambda x: [v.text for v in x],
    "dict": lambda x:
        dict((x[i].text, x[i+1].text) for i in range(0, len(x), 2)),
    "key": lambda x: x.text or "",

    # simple types
    "string": lambda x: x.text or "",
    "data": lambda x: base64.decodestring(x.text or ""),
    "date": lambda x:
        datetime.datetime(*map(int, re.findall("\d+", x.text))),
    "true": lambda x: True,
    "false": lambda x: False,
    "real": lambda x: float(x.text),
    "integer": lambda x: int(x.text),

}

def load(file):
    parser = iterparse(file)
    for action, elem in parser:
        unmarshal = unmarshallers.get(elem.tag)
        if unmarshal:
            data = unmarshal(elem)
            elem.clear()
            elem.text = data
        elif elem.tag != "plist":
            raise IOError("unknown plist type: %r" % elem.tag)
    return data

Which was a great start, but I needed .m3u output. After reading how simple the format was, I was off and running. The end result was all of my iTunes playlists playable by Cog, which I am using now -- as I type -- to enjoy my music, free of pain. One thing worth exploring would be how to preserve the ordering of iTunes' playlist items.

Here's the code I used to "export" the iTunes playlists as .m3u:

import re

m3uList = "#EXTM3U\n%s\n"
m3uEntry = "#EXTINF:%(length)s,"
m3uEntry += "%(artist)s - %(album)s - %(song)s\n%(filename)s\n"

def phraseUnicode2ASCII(message):
    """
    Works around the built-in function str(message) which aborts when non-ASCII
    unicode characters are given to it.

    Modified from http://mail.python.org/pipermail/python-list/2002-June/150077.html
    """
    try:
        newMsg = message.encode('ascii')
    except (UnicodeDecodeError, UnicodeEncodeError):
       chars=[]
       for uc in message:
          try:
             char = uc.encode('ascii')
             chars.append(char)
          except (UnicodeDecodeError, UnicodeEncodeError):
             pass
       newMsg = ''.join(chars)
    return newMsg.strip()

class Playlists(object):

    def __init__(self, filename=None, destDir=None):
        self.lib = None
        if filename:
            self.lib = load(filename)
        if not destDir:
            destDir = './'
        self.destDir = destDir

    def processTrack(self, trackData):
        length = trackData.get('Total Time') or 300000
        song = trackData.get('Name') or 'Unknown'
        artist = trackData.get('Artist') or 'Unknown'
        album = trackData.get('Album') or 'Unknown'
        data = {
            'filename': trackData['Location'],
            'length': int(length)  / 1000 + 1,
            'song': phraseUnicode2ASCII(song),
            'artist': phraseUnicode2ASCII(artist),
            'album': phraseUnicode2ASCII(album),
        }
        return m3uEntry % data

    def processTrackIDs(self, ids):
        output = ''
        for id in ids:
            try:
                trackData = self.lib['Tracks'][str(id)]
                output += self.processTrack(trackData)
            except KeyError:
                print "Could not find track %i; skipping ..." % id
        return output

    def cleanName(self, unclean):
        clean = re.sub('[^\w]', '_', unclean)
        clean = re.sub('_{1,}', '_', clean)
        return clean

    def exportPlaylists(self):
        for playlist in self.lib['Playlists']:
            playlistName = self.cleanName(playlist['Name'])
            try:
                items = playlist['Playlist Items']
            except KeyError:
                print "Playlist seems to be empty; skipping ..."
                continue
            trackIDs = [x['Track ID'] for x in items]
            data = m3uList % self.processTrackIDs(trackIDs)
            fh = open("%s/%s.m3u" % (self.destDir, playlistName), 'w+')
            fh.write(data)
            fh.close()

def exportPlaylists(filename, dest=None):
    pls = Playlists(filename, dest)
    pls.exportPlaylists()

With usage like the following:

>>> from iTunesExport import exportPlaylists
>>> BASE = "/Volumes/itunes/__Playlists__"
>>> exportPlaylists('%s/Library.xml' % BASE, BASE)

Update: I've tweaked the code in this post a little bit, due to a reader's questions. To run this, copy both code blocks into a single file you should be good to go.