Twisted JSON-RPC

My clients/partner companies use web services a great deal. To be honest, WS have made my life easier in many respects... but they can be a real pain. And implementations can be pretty lacking...

For reasons I won't get into (business, politics, and AJAX), I discovered today that I needed to convert an XML-RPC server to a JSON-RPC server. I looked at several implementations and they were either not general enough for my use, or they were horrible.

So, I hacked a copy of twisted.web.xmlrpc and turned it into jsonrpc using the simplejson library. Right now, it's doing JSON over HTTP, but I fully intend to write a TCP implementation as well. The problem, though, is this: as I was putting this together today, all I could think about was ways to make the code general enough to provide a common basis for use in implementing *-RPC. Ah, down that path lies madness. And it's one of those things you just can't avoid thinking about...

I'm currently writing some twisted.trial tests for it, but I also need to add some more stuff to make this generally useful (not to mention easier to debug). Hmmm, I'm actually really looking forward to doing a twisted TCP implementation of JSON-RPC. That should be fairly fast. And clean. Maybe.

And, of course, I'm sure I'll do a twisted.web2 implementation as well.

I have an amazing headache now, and need to get some food.

Update: All twisted.trial tests are passing and I am running a twisted JSON-RPC server now.

Update: For those wanting to use this code, please read the comments on this entry (dated May 5, 2006) and then see this post.

Technorati Tags: jsonrpc, json, python, twisted, web, xmlrpc

Audio Fingerprinting

music :: technology

Here are a collection of links that I thought might be useful for folks
exploring audio finger printing technology. I've had this sitting
around in my drafts for a while, thinking I might do more with the
info, but... not. So here it is ;-)

MusicBrainz and TRM

   http://blog.musicbrainz.org/archives/2005/10/acoustic_finger.html
   http://blog.musicbrainz.org/archives/2005/09/general_update.html

Alternative to TRM

   http://www.w140.com/audio/

MusicBrainz developing TRM Replacement

    
http://lists.musicbrainz.org/pipermail/musicbrainz-devel/2005-October/ 
001388.html
    
http://lists.musicbrainz.org/pipermail/musicbrainz-devel/2005-October/ 
001432.html
   http://chatlogs.musicbrainz.org/2005/2005-10/2005-10-10.txt
    
http://lists.musicbrainz.org/pipermail/musicbrainz-devel/2005-October/ 
001440.html
    
http://lists.musicbrainz.org/pipermail/musicbrainz-devel/2005-October/ 
001447.html
    
http://lists.musicbrainz.org/pipermail/musicbrainz-devel/2005-October/ 
001452.html
    
http://lists.musicbrainz.org/pipermail/musicbrainz-devel/2005-October/ 
001466.html
    
http://lists.musicbrainz.org/pipermail/musicbrainz-devel/2005-October/ 
001478.html

Now playing:

Mannheim Steamroller - The Second Door

twisted.web2.xmlrpc How-To

twisted :: programming :: documentation

Last night, Sean Reifschneider and I sat together at
hacking society. He got a new
release of
vPostMaster
ready, and I started converting the
twisted.web.xmlrpc How-to to
twisted.web2.

It was pretty easy, really. The major difference in implementation is
that xmlrpc_*() methods now take the request object in their sigs.
Another minor change: we're dumping the .rpy stuff from the old docs,
and instead showing how to generally plug an XML-RPC resource into
twisted.web2 web server (i.e., other resources). It looks like dried's
http_auth branch
will soon be reviewed for inclusion in trunk, so we will be adding
support and docs for authentication in t.w2.xmlrpc as well.

Clay Shirky at ETech

Tim O'Reilly (at least, I think it's Tim) took some notes during Clay Shirky's talk at ETech. Here's one of my favorites:

Social software is the experimental wing of political philsophy, a
discipline that doesn't realize it has an experimental wing. We are
literally encoding the principles of freedom of speech and freedom of
expression in our tools. We need to have conversations about the
explicit goals of what it is that we're supporting and what we are
trying to do, because that conversation matters. Because we have
short-term goals and the cliff-face of annoyance comes in quickly when
we let users talk to each other. But we also need to get it right in
the long term because society needs us to get it right. I think having
the language to talk about this is the right place to start.

pymon Shell Sprint Overview

pycon :: pymon :: twisted

Well, this is about a week late, but I've been sick as a dog for that
same period, so perhaps I can be forgiven ;-) Having been gone from
Colorado for two weeks (the most extended absence since I moved here),
I have to say that Evelyn Mitchell is right: there is *nothing* like
the water here at the foothills of the Rockies. Nothing. I couldn't
believe it, actually. I had to keep drinking to make sure it wasn't
some strange mental effect. After a week, I'm finally getting used to
the pure goodness of it again, but I will rest very happy knowing how
amazing the water is that I take in every day. Oh, and the air too.
When I stepped off the plane in Denver, I wanted to kiss the ground.
The air was a flood of clean sanity in an insanely polluted world.

Back to the sprint summary. Right before I packed up to leave on the
last day of the sprints, I sent a similar message to the pymon mail
list. With a few edits, here it is again.

What's done? Architecture.

• initial coding for second round of grammar


• initial tie-ins for higher-level application logic


• shellparser.Grammar() has moved out of the sandbox and into it's
  own module at pymon.grammar.Grammar()


• shellparser.ShellParser() has moved out of the sandbox and into
  pymon.parser.Shell()

What needs to be done? A little architecture touch-up and fairly
straight-forward implementation.

• refine the grammar (introspection?) so that we can at least do
  auto-completion with tab


• add extensive docstrings to the classes and methods for use with
  "help"


• refine grammar to get "help" for various commands (tied closely to
  the previous two bullets)


• add support to ZConfig schema and pymon.conf for named nodes (named
  services?) -- very easy


• querying in-memory configuration -- very easy


• querying in-memory service status -- a little more involved, but
  not bad

To finish the move out of the sandbox and into the lib, the following
has to happen:

• shell.tac gets *completely* rewritten as Shell() or
  LineReceiverShell() in pymon.services() with shell-specific
  configuration added to schema.xml/pymon.conf and then plugged into
  pymon.tac.

The grammar and shell parsing seem to be very flexible, so we should be
able to use them with services like XML-RPC, jabber, and email with few
or no changes/additions.

More Schrodinger's Box

webxcreta :: software :: weblogs

This is the second month of operation for the
Schrodinger's
Box experiment. It's quite fun, and I am enjoying the end-of-month
self-referencing posts. It's quite a blast to see whacky themes emerge
from the grammatical averages. I'll probably be generating posts from
the previous ones for another day or so, and then the crunching on the
top 500 blogs will resume...

pymon Sprinting at PyCon 2006

pycon :: pymon :: networking :: programming

For the past two days, I've been working on a Cisco-inspired shell for
pymon. The purpose being: provide a user interface familiar to network
engineers, allowing them to update/control a running pymon instance.
Ravi and I worked on the shell last year, with Ravi writing a
pyparsing
grammar. However, since then, the specs have evolved considerably: they
are now much better defined and understood. As a result of these
changes, I had to chuck the old code and start over. Which isn't a bad
thing, especially since we didn't get very far. Things have gone very
well and I am excited about the direction and rapid movement.

Here's the current status of the current pyparsing grammar work:

complete

• node (add, update, del, etc.)


• memory (write, clear)


• show (nodes, services, lists)

almost done

• services (add, update, del, etc.)

not started

• lists (email/notification list management)

During lunch today,
Sean
Reifschneider from
tummy.com
had some great input and had the following requests:

• acknowledgments via web gui (and maybe also email? irc?)


• scheduled downtime (existing feature)


• dependencies (e.g., if the router is down, only notify about the
  router and not all the hosts behind the router) -- because of they way
  I have designed the rules engine, I don't think this will be too
  difficult... and I may even be able to use the NetCIDR package I wrote
  for CoyMon to do this very cleanly


• read-only SNMP (gets for remote load average and disk usage); this
  should be fairly trivial. We might be able to push this out with the
  next release. It will just use the local agent, like ping does


• escalations (partially implemented)


• zsh-like functionality in pymon shell (tab-completion at least)

This gives the project some nice near-term focus which we will be
working on once I restructure the factories and clients and add unit
tests for twisted.trial.