Psychohistory

natural language :: semantics :: science fiction

I am delighted with the silliness issuing forth from
Schrodinger's Box, it
continues to amuse and I am always eager to see what the next post will
be. However, as I alluded to at the end of my previous post, there is
more potential here than satisfying the high-minded call of absurdity.

For example, imagine yourself at work, trapped in a project with a
bunch of tired people just wanting to go home. You've been tasked by
the boss with "thinking outside the box" and you're making no progress.
Everyone is grumpy, creativity is harder and harder to imagine. Simply
gather a bunch of text for the topic at hand, feed it into webXcreta,
and viola -- instant brainstorming material. Proceed to discuss the
sentences that webXcreta pops out, easily dig yourself out of the rut,
get off the hook with the boss, and run home to pursue the many other
distractions of a mundane existence.

This could also be used effectively to alleviate writer's block. You're
writing a historical romance in ancient Gaul, with a plot that
stretches over Teutonic tribes in Western Europe, through to Rome and
into Asia Minor? Well, gather some source material, feed it into
webXcreta, and bing-bango! Innumerable ideas and sources of inspiration
to get that ink flowing again.

So, yes -- there is some practicality involved. On a slightly more
radical note, I'm exploring as possible use of the weighting
"algorithm" I used in webXcreta for representation of minorities. The
square of the log could be a very effective means of ensuring that no
voice is completely suppressed, that no majority ever gains absolute
control. I'd like to hear what people with political science
backgrounds have to say about that sort of thing.

And then there's the potential role for this to be used in assessing
public opinion, popular trends, and predictive analysis. Now we're get
to the subject of this blog entry... psychohistory ;-) I'm talking
Isaac Asimov and science-fiction: the psychohistory of Hari Seldon in
the famous Foundation series. Or at least a part of it. webXcreta makes
use of the
Natural Language Processing Toolkit which is a great
tool, but we'd need something more to make this science-fiction a
reality. We'd need a "semantic processing toolkit". I image that the
corpora for such a toolkit would not be tagged parts of speech, etc.,
but rather semantic tags. Perhaps domain-specific tags for contextual
meaning. Then, instead of a grammatical average, you would take a
semantic average. Now *that* would be REALLY interesting...

Now playing:

Yes - And You And I (live version)

Crazy Truth

software :: python :: webxcreta

I recently emailed a friend about the webXcreta project, and to give
him some background on it, I described Eigenradio:

A few years ago, I came across one of the most bizarre software
projects I had seen on the net: a guy at MIT had created an internet
radio station that "consumed" Top-40 songs being played live on other
internet radio stations, pushed them through massive statistical
commutation and custom software, and then spit out "new" music from
these computations. The music generated was a sonic, statistical
average of what was popular and getting air-play. Most people found the
resulting "music" disturbing. I, however, couldn't stop laughing --
literally. You could actually "hear" the statistics of the thing, if
you listened carefully. It was stunning. And hilarious.

That bit about hearing the statistics is key. And it perfectly
describes how it felt to listen to that music. It felt like an
epiphany. One of Eigenradio's taglines was brilliant:

What you hear on Eigenradio is the best of the New Music, distilled and
de-correlated. One song on Eigenradio is worth at least twenty songs on
old radio.

Now, with webXcreta, I find myself in a similar situation: I read the
posts, and I convulse with laughter. It's not the content so much that
makes it so irrepressibly funny to me, but rather what's under the
covers. To give you a quick sense of my humor, I laugh at truth. Truth
is endlessly amusing to me. I remember reading James Gleik's "Chaos"
book in high school, and laughing for about 15 minutes after I read his
description of Sierpinski Gaskets: they have zero area and infinite
points. It wasn't so much like a light going off in my mind, as a
bomb. The truth of it turned my mind upside down, and I had new eyes.
It was an ecstatic experience -- thus the laughter.

There's something similar happening in my mind with reading theses
Schrodinger's Box
blog posts. There's something hidden, under the covers that is the
true source of my laughter; the quote above from the Eigenradio site
points to the answer. To explore this further, consider this: what if
you absolutely had to read 1000 pages of text in less than a minute,
what would you do? What's the cheapest alternative to a
massive/complete data set? A random sampling! Read a shotgun-spread of
statistically sampled textual data from those 1000 pages.

And that's it. That's what's making me laugh. When I read
Schrodinger's Box,
part of my mind is actually aware that it is seeing parts of thousands
of data sources simultaneously, and the truth of that inspires a
quasi-ecstatic hilarity. Crazy truth.

In my experience (and, arguably that of the entire world of science),
Crazy Truth is a gold-mine for discovery. It will be interesting to see
how this code evolves and what strange uses it gets put to...

Now playing:

Yes - Close to the Edge

Madness Ensues...

weblogs :: programming :: python :: natural language

Almost exactly a year ago, I proposed to do something akin to
eigenradio but with text. It was in
this
blog entry.

Well, tonight I decided to take a break from this week's regular coding
tasks, and pick webXcreta back up. I started from scratch again, and it took off... I couldn't stop programming :-) This was a LOT of fun. The
Natural Language Toolkit
(NLTK)
has been greatly simplified since I used it last year. It's a dream -- everything I'd always hoped it would be.

The first entry in this new "random blog" is forth-coming. I expect it
to be ready sometime Saturday evening. I've scrapped data from
bloglines' top-100 and have performed all the grammatical analysis on
it. Titles are analyzed and given weights. In fact, the title code is
ready for the first run and has produced the first title. Wanna see it?
;-) The first blog entry will be titled "A Show Of Writer -
Right". All these entries of insanity will be published here:

Schrodinger's Box.

The last bit I have to do is the weighting for sentences and construct
full entries (it's a little more complicated than generating the
titles).

I may end up enjoying this more than "real" blogging...

Update: The first one is up! Check it out here

.

Now playing:

Depeche Mode - The Sweetest Condition

mods

music

I've been catching up on some of my blog reading today, and came across
this
hilarious and insightful post by fzZzy. For those of you that don't
know about mods, you are seriously missing out. Some of my favorite
music is in mod format (.mod, .s3m, .xm, etc.). In particular, this bit
from fzZzy's post is a perfect summation of all that I have felt about
mods and the mod scene:

There is a psychic difference between the product that one creates for
love versus money. When someone really, truly loves something and does
their absolute best, it bleeds out between the lines, oozes all over
your hands and sticks to you and won't wash off.

Perfectly said. He goes on to talk about programming as his passion and
the joy of doing what you love. Bingo. Exactly how I feel, up to and
including the programming bit ;-)

For anyone that's curious, the song he mentions first is
here.
But don't stop there ;-) Check out that guy's site and his music.
Phenomenal. In particular, "blastoff"... it reminds me of a mix between
Yes and Rush (both awesome) and then with classic game music thrown in
(even *more* awesome!).

For those interested, here are my
mod links at del.cio.us.
In particular,
The Mod Archive is where I have gotten
almost all my mod music. Many good memories of downloading mods during
lunch while at USi, back in the day...

Enjoy!

All Hail the Wayback Machine!

weblogs :: internet

Where backups (or lack thereof), hardware, and all other means had
failed me, the
Wayback Machine
has come through -- in shining colors. I had lost a tremendous amount
of data in a hard drive crash last year, and in the long term, the most
painful personal part of that was the loss of my weblog entries from
the summer until my move to Colorado.

Immediately after the hard drive crash, I tried the Wayback Machine's
archives in an panic. Apparently, not enough time had passed for the
posts to be archived because nothing was there. Well, there has now,
and I am *delighted*! I am now adding these lost entries to the my blog
republisher on blogspot.

All Hail the Wayback Machine!

SonicVault

music :: software :: python :: twisted

On 13 Nov 2005, I posted to my blog about a "new project" but haven't
followed up with any further info. Well, I made some phenomenal
progress lately, and I can take a break to chat about it. The new
project is
SonicVault, devised and
commissioned by
Neosynapse (a long-time
partner company with mine). As the stub on the wiki says, it is

a networked, encrypted MP3 library that supports fingerprinting
technology (to be used in determing file organization, detecting
duplicates, etc.)

Originally, we were going to do full audio fingerprinting (using
spectral densities, Mel Cepstrum Coefficients, etc.), but found that
there is an open source version of the fingerprinting technology
TRM and since TRM is used with
MusicBrainz, we decided to go
with that. I have since come across another open source solution called
fdmf, and am exploring it for
future use.

We encountered some issues with the python bindings provided by
MusicBrainz, so we ended up writing our own client using the wonderful
RDFLib python package. We put up some
sample code on the
RDFLib wiki that you can check
out. (More on RDF later...)

We now have a functional upload server that encrypts all
communications, an XML-RPC API for querying stored files, and a
built-in MusicBrainz client exposed to the XML-RPC API that provides
users with a facility for easily modifying MP3 metadata/ID3 tags.

Silverton, CO

colorado

I've just emailed my family about an old, small town in Colorado I have
just fallen in love with: Silverton. I have yet to visit it, but I've
been doing tons of research this week, and Silverton has made it to the
top of a short list.

I've leeched a bunch of photos off the net and put them here, in case
anyone is curious:

http://images.adytum.us/tags/silverton

I am very excited about the possibility of heading further into the
Heart of the Mountains, higher up in elevation, and deeper into the
snow :-) I love small towns, low population counts, and pristine
forests -- all of which Silverton provides. I've heard a bunch of good
things about Durango (1 hour to the south), so I'm hoping similar
things are true of its neighbors.

I've reached out to a couple of the Silverton community members and
have started hearing back from them. As dreams begin to take shape in
plan form, I will post updates. Be sure to check it out on google maps,
here:

http://tinyurl.com/exfbb

Wish me luck!

Automated Blogging: Advogato & Blogger

blogging :: atom :: python

Well, today I decided I might as well start posting my blogs to
blogger.com in addition to my Advogato posts. A couple friends of mine
are on blogger, so I imagine I'll start posting more comments there,
and I thought it was a shame for the lack of content in my blogger
account (I was using it as a placeholder).

I figured it would take no extra work to update it, really -- my
Advogato posts are all via email. All I needed to do was write a script
to post to blogger too. Well, working with the blogger Atom API was a
little frustrating. It's very sensitive, only accepting ASCII-encoded
strings, needs certain characters escaped, and returns HTTP 500 with no
clue as to what's going on. Anyway, I finally got everything tweaked
just right, and figured I would post the python code for doing it, as
many of the examples on the net are old.

The code below is copied and pasted from a script of mine, with most of
the guts ripped out (I parse from email and do authentication, to make
sure I'm the only one sending blog emails). There might be something
left out, but this should be enough to get you started.

import time
import base64
import urllib2
from xml.sax.saxutils import escape

username = 'your_blogger_name'
password = 'your_pass'
bloggerid = 'some_number'
blogger = "https://www.blogger.com/atom/%s" % bloggerid
title = 'blog entry title'
blog_entry = 'your blog entry'

created = time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())
body = """<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<entry xmlns="http://purl.org/atom/ns#">
   <title mode="escaped" type="text/plain">%s</title>
   <issued>%s</issued>
   <generator url="http://your_url/atom">Your Client
Name</generator>
   <content type="application/xhtml+xml">
     <div xmlns="http://www.w3.org/1999/xhtml">%s</div>
   </content>
</entry>""" % (title, created, blog_entry)

req= urllib2.Request(url=blogger)
base64string = base64.encodestring('%s:%s' % (username,
password)).strip()
req.add_header("Authorization", "Basic %s" % base64string)
req.add_header("Content-type", "application/atom+xml")
req.add_data(body)
f = urllib2.urlopen(req)

Now playing:

Yoko Kanno & The Seatbelts - Gotta Knock A Little Harder (Friday,  9:59pm MST)

Test Post via Email

atom :: python :: blogging

This is a test post from email -> postfix -> python -> atom

Let's see how it goes...

Now playing:

Yoko Kanno & The Seatbelts - Call Me Call Me (Friday,  2:57am MST)

Future Blogging Changes

blogging :: software :: python :: twisted

Advogato has served me well over the past year, and I will continue to use it for blogging. I email a special account on one of my servers with my blog entries, and it gets published within seconds on Advogato. I have nightly cronjobs that run XML-RPC queries against Advogato and save backups (in addition to the originals which are in my "Sent" folder). This makes me very happy, as I suffered a traumatic server crash over a year ago, wherein I lost over a year's worth of world-changing blog posts.

However, I believe I am ready to take blogging back into my own hands. I managed to rescue some of the data on the drive, including early posts from 2003 and part of 2004. These are currently archived here, but I would like to unify all old posts, Advogato posts, and future posts in a single framework, look-and-feel.

As a result, I have finally broken down and started writing a collection of user-land networked applications:

• ImageDB, a small web micro-app which provides a backend upon which one could write an image gallery application like this one; uses image-tagging like del.icio.us;
• AccountsDB, which will provide distributed user and group management, authentication, as well as fine-grained permissioning with ACLs (access control lists) and ACL management for applications;
• MessageDB, which will store and index RFC 2822-compliant messages, will allow users to tag them like ImageDB and del.icio.us, as well as providing many convenient means for users and parent applications to add messages;
  
• WeBlog, which will be composed of the micro-applications MessageDB, AccountsDB, and ImageDB and provide a nice user interface for blog posts, blogs, and groups of blogs (community spaces).

This is just a handful of the full suite we are working on here.

All of these applications are small, networked, light-weight, and run independently of heavy software like Apache, MySQL, PostgreSQL, etc. They are python- and twisted-powered. We have plans of implementing peering (using Q2Q/Vertex) so that, for example, a trusted network of ImageDB installations could provide images for each other (like a private, distributed, images.google.com). There are so many possible ways that this could go, it's just insanely exciting. We're talking new paradigm here, folks.

So, stay tuned. This blog-space and my account on Advogato will eventually become re-publishers of ElectricDuncan running on WeBlog ;-) And then I'll have two more backups of these techno-babble ramblings...