The King is dead! Long live the Kinglets!

So what the hell does that mean? Well, there's sort of a back story there (but you can skip to the last paragraph of this post for the punchline).

Earlier this year, I blogged about the language patterns of twisted: Thinking in Twisted. The use of and relationships between such things as protocols, clients, servers, and various factories in Twisted was really starting to make its way into my bones. These and other core aspects of Twisted best practices form the basis of a pattern language... one that is of immense value when writing network code and applications. Here are a couple quotes from that blog entry:

The tricky thing about Twisted is that its patterns are an integral part of its "language" (perhaps it would be better to stop using quotes, and just call it a meta-language...). This is not true of most other languages -- though I would venture to say it is true of good frameworks in general. You don't need to know the patterns to use the grammar/syntax/control structures of most languages. Patterns in a language are usually optional. [However, w]ith Twisted, if you aren't using the patterns, you aren't really using Twisted.

When we learn to think in Twisted and write applications in Twisted, our solutions will be elegant and compact. Before that starts to happen, we try to use our "old" ways of python programming mixed in with the new one we are learning... and that, of course, defeats the purpose of the new "language."

I've recently hit another level of awareness in pattern abstraction, and it has its roots in Twisted. In fact, I never would have been working on what I am now working on if it weren't for Twisted and all the work done by Divmod and its crew. While at PyCon 2004, I met with Amir and Glyph and we discussed twisted's use in the industry at large, and in particular, opportunities for which the Divmod product line could be applicable.

What I liked so much (and immediately) about what Divmod was doing was that they were integrating useful components into a single application (Quotient). It stuck a positive chord with my instincts about where software should be going, and it kept me thinking for months while I worked on other projects. These thoughts culminated in a series of blog entries and software projects. First, a post called Dinosaurs and Mammals explored this. In the intervening time, the projects I have worked on have helped me evolve these ideas usefully. There is now direction, code, and product (more about that later in this post).

Then, in Everything as a Distributed Resource, I took a more sci-fi, radical view... but one that is born out by current trends and tendencies, particularly in the cell phone market. There's a great deal we haven't thought about in regards to the cell phone market. Not so much about cell phones and mobile technology, but rather the usage patterns and demands that are being made and (sometimes) met in this sector.

In an effort to follow through on the "mammal" paradigm, I did some exploration with URLdb and PyKon which I blogged briefly about in New Projects. The two projects mentioned there have since been merged into a project that we are calling Adytum's Twisted Collection. The latest addition to which is ImageDB.

ImageDB is a repository for static graphic media. The images that people upload here can be viewed via it's web interface, but it is more likely the ImageDB would be used as a part of something else. For example, when I get around to finishing PyKon, it will make use of
ImageDB. PyKon will not store any images itself. It's simply going to be an nicely designed, usable image gallery interface. Users won't know that, though. They will use PyKon and be able to create galleries and albums in that gallery by uploading images (which will be stored --
transparently to the user -- in an ImageDB instance), by selecting individual images (JavaScript ImageDB selector/browser), or by selecting "tags" a la del.icio.us (and therefore whole groups of
images).

Analogous to ImageDB is MessageDB, a repository for static text. It's next on the list to be developed. This micro-app will hardly ever be used directly, although it too will provide a basic web interface for viewing and editing stored content. MessageDB will be a storage app for RFC 2822-compliant "messages". For starters, various public mail list emails will get tagged and stored there.

Non-dynamic text for web sites and blogs will be stored as RFC 2822 messages. So the forth-coming tools "SimpleSite" and "WeBlog" (also part of the collection) will actually store and pull their content from MessageDB. But even more, when you want to use a photo, clipart, or logo in your site or on your blog, you will click a "select image" link and an image thumbnail browser will pop up. You will select one of the thumbnails, the popup will go away, and a link to that resource in your ImageDB instance will be inserted for you, in your blog or site text.

The idea is that by abstracting out general functionality into little micro-apps, you not only decrease the amount of work in maintenance and feature addition, you not only are reducing the amount of redundant work you have to do when building content-related applications, but you are providing an infinite amount of flexibility for using these various media in completely unplanned and unexpected ways.

Your media, it's storage, it's access (XML-RPC), now live in self-contained little units. They can "survive" on their own, without a heavy dependence upon some huge CMS that was written for a particular project. The apps are project-agnostic. They can be set up once and then reused repeatedly in different contexts by building new apps that obtain their data from the component micro-apps via simple links and/or XML-RPC calls.

And, again, the ways in which users and designers decide to combine these pieces and how they decide to use that is 100% flexible and thus fertile ground for the unexpected and creative :-)

I say "To Hell!" with large apps, and banish them forever! Let's build tiny, little, single-purpose apps once! And then for larger needs, simply glue together various micro-apps as needed per the project requirements.

As far as I'm concerned, the old King of the Bloated Application is dead. It's been replaced with tons of tiny, decentralized micro-apps. Long live the Kinglets!

pymon Update

twisted :: python :: web

There are now pymon web ui screenshots up on the project trac
site. These are two status views (summary and detailed) for monitored
services. They are 100% Nevow and load super fast. I'm very happy with
the progress so far. For this first release, I think we'll just make
some adjustments to the admin landing page, and call it ready to go.

Now playing:

Deep Forest with Peter Gabriel - While The Earth Sleeps (Sunday, 11:48pm MST)

pymon, ZConfig, Nevow, and pyparsing

twisted :: python :: web

As I have published recently on the pymon
site, we're getting very close to a pre-release of pymon. The
latest changes involved migrating from a custom, home-grown
configuration scheme and API to ZConfig. I am very happy with the
Apahce-style config files of ZConfig. It has drastically simplified the
configuration files, thus lightening the work-loads of systems and
network administrators who use pymon. But even more, I like the ZConfig
python API for accessing configuration data. Our home-grown solution
had an API that was similar to, but not as clean as ZConfig's.

After getting pymon running with ZConfig, I turned to the web UI for
viewing monitored services states. I haven't touched Nevow since
version 0.4, and it was really nice to dive back in. I'd forgetten what
a joy Nevow is to work with. Now some significant progress is being
made on state data display, and some design choices need to be made
about Nevow render patterns and slots... fortunately neither rocket
science nor mind-numbing ;-) There's a good chance that the web UI will
be ready in time for the release.

Unfortunately, I don't think the first steps towards the pymon OS will
be. We've got some code prototyped in my sandbox. This will be a great
feature. You telnet/ssh into pymon, and run commands that control the
monitoring configuration, display monitored service states, etc., just
like you would on a firewall. We're using pyparsing to generate the
grammar and parse user commands. It's in a very early stage, but it is
one of the more exciting aspects of pymon, as it is interesting code
and there's nothing out there even remotely like it (to our knowledge).
As soon as the first release is out, though, we'll be merging the
prototype pymon OS into trunk, providing at the very least a glimpse
into pymon's bright future ;-)

As a couple of us have been using older versions of pymon for a couple
years now, and I have been using the new version for a year, we're
ready to start sharing with others, getting feedback, and developing a
user and contributor base. What's more is, even though I can't say much
now, we are demoing pymon for a couple companies, one of which
specializes in security solutions for the defense industry.

Look for a pymon release very soon!

Twisted Network Programming Essentials

twisted :: programming :: books

The title says it all. Go buy it
now! I've only just managed to buy a copy yesterday. I'm fairly
familiar with twisted and have used it extensively over the past
several years, so I was going to buy the book just to support Abe and
the Twisted community. It turns out, though, that he has written this
in an amazingly lucid style, one that I will actually use to teach
twisted to junior developers -- a task I have not attempted yet as a
result of not having a good approach or proper materials to draw from.
Until now, twisted has been for self-learners only. With this book, you
will be able to teach twisted to interested and motivated middle- and high-schoolers.
One particularly wonderful nugget was the description of twisted.cred. I've
always been a little foggy with the the architecture and use of
twisted.cred (credentialing and authentication in twisted)... mostly
because I've never needed to use it. Turns out I am working on a new
project that will utilize it. Just in time to get the crash-course on
cred from Abe's book :-)

And this is definitely and by far the most cool, insane, intense cover of
any O'Reilly book. Ever. Future books included. You just can't beat the
writhing, twisted ball of mating snakes. No joke. Read about it on the
last page.

Now playing:

Coldplay - Swallowed in the Sea (Tuesday,  4:50pm MST)

Ubuntu

software :: linux

Well, I finally caved. After the past three years of running OS X at
home, I have need for a file server again. I tend to run SuSE in
production, as many of my customers like the administrative interface
of YaST, but after using Fink (apt-get) on the Mac, I have been very
curious to try a Debian distro. Ubuntu is the top of many peoples'
lists these days, and it's won several recent awards, so I thought I
would give it a shot. The short and long of it: Ubuntu as a server
rocks.

I use that qualification because I only used it briefly as a desktop
OS, running the LiveCD on my 17" PowerBook for a night. A most
enjoyable experience... if only there was support for the Broadcom
builtin wireless :-(

As a server, it is phenomenal. This is due not so much to Ubuntu, but
rather as a result of it being a Debian distro. Getting servers
installed, locked down, and running the network applications I need is
my top priority when selecting a platform these days. My second
priority is time: all of that has to happen within 1-3 hours of
beginning the process. I'm not a sysadmin anymore, and I can't spend my
time on that when it is better spent on development and R&D.

We'll have to see how SuSE fares at Novell over the next year (due to
recent changes in the SuSE team and the Novell layoffs, I'm watching
closely). If things start looking bad -- or if Ubuntu gets a set of
killer admin UI apps -- future server upgrades may experience a distro
switch.

Oh, and as a side note: LVM is phenomenal. I'm running it on the file
server, and am just amazed at the progress that has been made in the
last 4 years. I see no reason to use Veritas Volume Manager anymore
(for any of my needs or any of my customers' current needs). If you've
got a ton of drives laying around, not pulling their weight, dump them
in a box and put them to dynamic use with LVM.

Now playing:

Jon Brion - Monday (Sunday,  3:34pm MST)

Coldplay - Fix You

music :: entertainment

I almost *never* listen to the radio. I don't think I even have an FM
tuner (not counting the car; I almost never drive, either!). However, I
did see the special preview concert (on MTV?) that Coldplay gave for
the release of their new album X&Y. It was a while ago, and I hadn't
heard any radio play since then. When I was visiting Greg and Ladan in
San Francisco, I heard "Fix You" a couple times on the radio. Twice I
think. Since then, it has been in my head constantly.

I actually really like the music, and I love Coldplay's vocal style.
But I've had problems with the lyrics of that particular song since I
saw them perform it on MTV. But the music is so nice... consequently, a
tension was setup in my mind between it and the meaning of the lyrics.

My objection is due to a vehement position I hold that no one can "fix"
anyone else. In my opionon, one has to take responsibility for all
one's actions: past, present, and future. And through the continued and
disciplined application of awareness and compassion for oneself, the
"fixing" can occur. No one is to blame for our own suffering. If we are
suffering, we have the power -- through changing our minds and
perceptions -- to turn that suffering around. Our own minds/hearts are
the core of the issues, not someone else's. No one else can make us
happy; no one else can make us suffer. This is admittedly contrary to
your average opinion on the matter. In my opinion, this is because
people are very, VERY uncomfortable with taking responsibility of their
own actions. And perhaps even more uncomfortable with the frightening
prospect of being the makers of their own universes.

However, I digress. I think I have now found a way to enjoy the lyrics,
and it ties in nicely with an interpretation of the "X&Y" title as
chromosomal :-) If you consider the inheritance we receive from our
parents -- genetic, emotional, behavioral, etc. -- and you agree that
there is plenty of work (and a responsibility!) to improve on this
inheritance, then "fixing you" could be an address to our parents, or
rather, our parents inside us. Something along the lines of "we promise
to take care of you in us, to do the most with what you have given us,
to fix the broken pieces and go further..."

With that perspective, I am now listening to the song without
misgivings, fully enjoying both the lyrics and the music :-)

Now playing:

Sting - Sister Moon (Wednesday,  6:45am MST)

Last.fm and MusicBrainz

internet :: music :: software

While working on a related project, I came across MusicBrainz (with an associated Mac
OS X application with an awesome "i*" name: iEatBrainz) and
AudioScrobbler /
Last.fm. Let me just say that Last.fm
-- for whatever reason it was named that -- is truly and aptly named:
this is the last radio you will ever need. If this technology hasn't
been incorporated into XM radio and it's radio-a-likes, it soon will
be. Otherwise someone else will do it, and they'll be out of business.

Let's say that you don't want to listen to your own music collection;
you want to spice things up and hear something new... but familiar.
Let's say you're in a particular mood and you really want to hear
something like the fantastic folk stylings of Shawn Colvin. Well, open
up the Last.fm player, type "Shawn Colvin" and you are now listening to
personalized radio with a random playlist built by Last.fm to provide
music that is similar or the same as Shawn Colvin.

Already, I have discovered two new female vocalists I'd never heard of
who I now adore, as well as an old-style punk band that I want to hear
more of.

This is the best thing that has ever happened to radio, and quite
frankly, I'm surprised it's taken so long to get this started. It's
going to be a hell of a ride from here on in, with this technology...
this is just the beginning of end-user-tailored broadcast media.

I've been using iEatBrainz and the MusicBrainz db to go through the
hundreds of old MP3s I have that I ripped before rippers connected to
CDDB/FreeDB... so many files with no ID3 tags. This is enabling me to
perform organization tasks in my music library that I have been putting
off for ages. It was just too painful. But now that iEatBrainz, I have
music superpowers.

All this has made the new project I am working on even more exciting :-)

Now playing:

Ben Harper - She's Only Happy in the Sun (Tuesday,  1:41pm MST)

San Francisco

friends :: travel

Two dear friends of mine, Greg and Ladan, whisked me away to San
Francisco to stay with them for almost two weeks, and I've just
returned. I had the most amazing time there -- including (but not
limited to!) meeting new friends, having long and deep conversations
with old friends, eating *amazing* Persian food, and seeing Depeche
Mode in concert. Though it wasn't a vacation -- I coded my ass off all
day while they were at work and school -- it was just what I needed.

You know, good friends are just amazing. The generosity of open hearts
can really be life changing. In "modern" life, so much attention gets
paid to success, acquisition, consumption... that we can get distracted
by either participating in that or fighting it. Stepping outside of
that battlefield, one finds the truly great things in life. The support
that friends provide is one of these things. Thanks Greg and Ladan!

Now playing:

Ben Harper - Waiting on an Angel (Tuesday,  1:09pm)

New Project

software :: music :: mathematics

So, I can't really say anything about it yet, but I'm really excited it
:-) It's going to be open source, so others will be free to contribute.
I'm pretty sure it hasn't been done yet in the open source world, and
it's only just being done commercially now.

So... stay tuned, and as soon as all the details are worked out, I'll
post about it again here. Okay, I'll give you a hint: Hidden Markov
Models...

;-)

z3 Project Starter Script Update

software :: zope3

I just thought I'd give a quick update in regards to this script. I
have tweaked a few things and submitted a new version on zope.org.
There's a news item about it here.

Information and details on the script are here.

Enjoy!

Structure Generation and z3

software :: zope3

Carlos de la Guardia posted a recent
blog entry that mentions the need for "rails-like" code-stubbing,
or the automatic generation of directories and files in z3. I think
there was one comment that wasn't spam, and then I posted one in
addition to that. So, I thought I'd post a long over-due entry here
myself.

A tool like that exists; I wrote it a while ago, but didn't get a
chance to clean it up (a little) and post it to the zope.org site until about a month ago. Here is the
z3 project starter.

It's alpha, and I need to encode the 1px.gif image so that a proper
binary file can be created. But that's pretty easy, I just haven't
gotten around to it. The basic css tabs are really only usable with
vhost. It has only been tested repeatedly in a narrow set of use, so it
needs more people playing with it. I'd love it if folks wanted to
contribute, helping to get it into shape suitable for entry into the z3
repository. It'd be great to bring it in line with Benji's quickstart
guide.

/back

general

It's been far, far too long since my last post. I can lay the blame for
this soundly at the feet of economic hardship and business suckage. One
of my oldest business partners was acquired by LEDS last year, and this
guy continues to be the most amazing friend, resource, and bread-winner
for my company. It really blows me away. Lots of other "partner"
companies make promises out the arse, with 80% of them falling through.
Not Allen, though. He just blows my mind... if it weren't for him, I
wouldn't even be able to write this blog entry.

Okay, so that's why I've been missing in action. In my silence, though,
I have done a tremendous amount of coding. I've been working on two
different versions of twisted object brokers for CoyMon; been updating
pymon to run from python eggs; been doing an insane amount of z3
development; teaching z3; been working on some custom astronomy python
utilities; been doing python image processing again; been exploring
darcs, ubuntu, MochiKit, Dojo Toolkit, Axiom; catching up on python
blogs; catching up on DivMon, Quotient, Nevow; just signed up for http://clickchronicle.com; been
catching up on experiments in gravitational physics as well as my
5-year cycle for M-Theory refreshing.

There's also a small group of friends that have started a mail list for
very cool explorations in practical... hmmm, don't even know what to
call it... metaphysics, I guess. Some of the most amazing discussions
and conversations I've had in a while.

I expect that I'll be posting here again, soon. It feels good to be
crawling out from under the rock where I've been hiding :-)

New Zope3 Skin

Today I developed a skin for Zope3. As is evident from an earlier post, I had been exploring the Django web app dev platform, and after some intensive use and investigation, I returned to zope3. It was like a breath of fresh air. However, I sorely missed the slick interface of Django.

So, I decided to make my own :-)
AdytumAdmin z3 skin

All of my z3 projects will be using this skin (or a derivative) as the default admin skin. This will give clients something they can be happy with/proud of when they visit the ZMI to manage their site, and it will keep me and other developers happy, as we have to look at the ZMI for hours on end each day. If you've worked with z3 at all yet, you know how basic and ugly the default (non) skin is... and thus understand the need for people to start doing this kind of thing.

I would like to note and share my gratitude of the fact that this was based on the ZopeTop skin for Zope3 (I think written by Paul Everitt). Without that work and prior development, this would have been an arduous task. With the work, it was a facile process of tweaking colors and shuffling page/zpt parts around. Simply fantastic.

Once again, z3 shines as a rapid development platform. This was all done in less than a day :-)

Update: This blog post originally pointed to resources that are now no longer available. However, thanks to the Wayback Machine, I've been able to rescue the old instructions from an archived version of the web page.

Setup

There are no docs included with the tarball. You just need to do the following:

• checkout the adytumadmin directory from svn
• move the adytumadmin directory into you lib/python directory of your z3 instance
• create a the file etc/package-includes/adytumadmin-configure.zcml in your z3 instance directory with the following contents:

  <include package="adytumadmin"></include>

• restart your z3 instance

Access

Access the skin by using ++skin++AdytumAdmin in your URL, e.g.:

http://127.0.0.1:8080/++skin++AdytumAdmin/@@contents.html

If you like this skin enough to make it your default skin and don't want to add it to the URL manually, just add this line to your etc/overrides.zcml file:

<browser:defaultskin name="AdytumAdmin">

Note that it is the skin name and not the skin package that is used.

Screenshots

• Add ZPP page
• Folder listing
  

Django vs. Zope3, or "The Right Web App Dev Platform"

development :: python :: zope 3 :: django

First of all, let me start off by saying that this is not a flame nor a
crusade :-) The title of this post needs to be taken in context: this
is about making the right choices for you, the developer. We are not
machines, we are quirky people, each with our own uniquenesses. The
choice of platform needs to be as much about what technology best fits
the needs of the project as well as that which fits the personality,
the strengths, and the weaknesses of the developer.

I just spent an interesting two days with django. Let me just say,
right off the bat: I like it, it's fun, it's cool. I like their site,
their growing docs (I've contributed a little to the wiki), the homey
feel of the project. I got my first little app up and running in no
time. Piece of cake, just as advertised.

However, for my second project, I wanted to do something a little
strange. Oddly enough, just the kind of irregular thing a client would
ask for. Thus began the headaches. I spent about 30 hours with my head
up against the wall. I will say that my primary problem was not
understanding the not-so-documented subtleties of django. Additionally,
I'm really not that good at database modeling. Object relational
mappers are cool (much better than having to write SQL), but to me, the
logical next step is to give up the ghost and use an object-oriented
database. For situations that require hard-core databases, hire the
dba/db designer, and do what's needed.

It was tough going. I really wanted to do some strange things, and
django didn't seem to be so flexible in different areas. And I think
that's actually something that the django developers have openly
stated. They have written and are targeting their software for a narrow
problem domain, a domain that brings them a lot of business and for
which there is great need.

I want to emphasize that I'm not a python or software noob. I've been
writing code, in one language or another, since 1981. I've been in the
industry since 1994. I've worked with twisted for a couple years, more
so recently. I regularly delve into its internals. I have a love/hate
relationship with Zope: I love z3 and hate Zope2. I spend a fair amount
of time digging through z3 code and find it (almost entirely)
delightful. However, I don't have any of that experience with django.
The source code doesn't make architectural sense to me. I guess I like
and use frameworks, and django's actually really an application and not
a framework. This is reflected in the code, and I have difficulty with
that.

So, as I slugged through the issues of bending django to my will, I
found myself wishing for things: what would it take to add ZPT support
to django's templating? Could a wrapper/adaptor be written for the ORM
so that ZODB could be used? Could the code be refactored to use
interfaces, adaptors and components? Yup. I was unconsciously trying to
rewrite djando into z3. However, this is also flattering to django: I
was trying to hang on as long as I could. Why? Well, it's got some
really cool things:

* a fantastic look and feel
* a slick admin interface generation (z3 has auto-generation of this
too, but it's not as pretty)
* really cool URL dispatcher for custom URLs perl application

Overall, they did some great work. They sell the point that you can
build apps in record time with django. And they're right -- you can.
But you know what else? I was able to build a project from the ground
up with z3 in two days -- and that *included* learning curve. No
kidding. Talk about rapid. I was stunned. In fact, after that
experience, I emailed several close developer friends of mine as well
as a few partner companies we consult for. I did a write-up on the
project and an analysis of the z3 development process and how it
benefits our timelines. The reason things went so smoothly with z3 is
because it's designed the way I think. By using a framework that
matches me so well, I move through code with zero resistance and get a
tremendous about of work done.

I'm not trying to sell z3 here. But, what I did discover with django
was that it's not my platform for web app dev: z3 is. This is because
of the kind of developer I am, the kind of code I like to work with,
and the kind of tools that allow me to be as productive as possible.
This combination of factors is different for different developers. As
many others in the community have pointed out, pick the right tools for
you and the job. There's a lot of hype about Rails, Django, etc., these
days; all the different web platforms available for python. People are
complaining about proliferation, but that's what drives evolution.
Software evolution has provided me with great business tools: twisted
and z3. What's it provided you?

Phillip Eby has changed my life... again.

python :: programming

First it was PEAK, then it was PyProtocols/generic functions (which has
now been split from PyProtocols in to its own package called
RuleDispatch). Now, it's EasyInstall. Everyone has been raving about
it, but I hadn't tried it out until tonight. Holy crap. This is crack
for python programmers. I built my first python egg when Bob Ippolito
first started writing about the alpha code he and Phillip Eby were
working on. Pretty cool stuff, but it didn't really hit me as to how
great eggs were until I actually saw it in action today with my upgrade
of setuptools, my first use of EasyInstall, and my upgrade of
PyProto/RuleDispatch (each creates an egg and dumps the file in
site-packages).

It blew my mind when I saw these two luminaries getting together. I
knew that it didn't matter what the hell these guys did together, it
would rock the world. Awesome. And thanks :-)

Solaris Returns?

operating systems :: solaris

Since moving to Colorado, I have had many friends visit... and now
family members are signing up. This is actually very nice for me, as I
have always been the one visiting. People *like* coming to Colorado,
hiking in the mountains, breathing the incredible air at high
altitudes. However, every single commodity-based workstation I have
(and I have over 10 of these boxes) has failed. These two topics are
actually related.

Most of the x86 machines I have, I did not purchase new. All of them
have had replacement parts, after the originals had burned out or
become obsolete. And all have since died, in some fashion or another.
In fact, the only exception to this are two machines I have: a Gateway
PC from 1997 with most of it's original hardware running FreeBSD (with
3 extra NICs) as my firewall/router and a Gateway laptop from 1998.

As a result of all these dead machines, there are no machines available
for use by all the people visiting -- there are two Mac OS X laptops
for use, but I am on my 17" almost 90% of the time I am awake, and
there are often more than 3 people in the house. This means that 2 or
more people have to juggle one 15" PowerBook. I have considered buying
a Mac mini, but with the changes of architecture at Apple, I am
exploring other options (I *will* be buying a 17" x86 PowerBook).

One of those is Sun. In a past life, I was a systems administrator and
worked a great deal with Sun hardware and was pretty familiar with
SunOS and intimately familiar with Solaris 2.5 and up. Last night, I
discovered (a month late) that Solaris has been open sourced. I
remember discussing this possibility with fellow admins and developers
in the late 90s, where a few of us optimists were called crazy for
thinking this was a possibility. I had since given up hope, and haven't
worked with Solaris in about two years, and then only in passing.

I spent several hours yesterday reading developer blogs and browsing
the site
(http://www.opensolaris.org/os/),
and I'm getting pretty
excited about the potential of this move by Sun. It is very likely that
this could move me back into not only using Solaris machines again, but
brushing up on my Solaris sysadmin skills, particularly catching up
with all the work they've done on virtual server technology. Running
python application servers in Solaris virtual machines will provide
many excellent solutions for application architectures.

Anyway, Sun is offering a "free" Opteron Sun box, with a 3-year support
subscription for about $30/month, and I'm heavily leaning towards
purchasing this box to fill the machine gap in the house:

http://tinyurl.com/cf4j2

If that goes well, and I like it, I may get another one for a local
subversion repository/backup development server... If their new
hardware is as solid and reliable as their old stuff (I have a
still-working Sparc server from 1995 that I bought from a DiGEX
employee -- it was the original web server for Dole.com), then I will
be most happy and they will have a new hardware customer for developer
platforms.

Tibetan Village Project

tibet :: culture :: society :: internet

A good friend of mine, Tamdin Wangdu, started and currently manages a
non-profit called the
Tibetan Village
Project. I have talked with Tibetan friends and Westerners (Injees)
about a concept that is tangentially related: the preservation of
Tibetan culture in the West by means of establishing a functional,
thriving, "organic" Tibetan village. The architecture we are planning
is 100% authentic traditional. Tibetan monks and laity are providing
time, drawings, ideas, planning, and resources. It's getting exciting.

Tremendous resources would have to be poured into this. We're looking
at all kinds of options, from forming several companies that would
employ members of the village and whose profits would be put into
village infrastructure and support, to artisans and farmers.

The village would be comprised of Tibetans and Injees, both. Monks and
laity. Babies growing up, learning Tibetan and English fluently; the
culture and history of Tibet and the West being taught simultaneously.
The idea being that Tibetans would evolve from
refugees to a people with a new home, an old, preserved culture, and a
new integration.

But a group of people living together tends to have limited wide-spread
impact. We want this to be something that can be easily duplicated; we
want this idea to catch hold in the hearts and minds of others. We are
therefore going to publish ALL documents related to this project and
organize them for easy navigation and quick access of particulars.

This is a start... but we need to actively encourage and participate in
the wide-spread preservation of Tibetan culture. The first idea I had
regarding this was rather silly (but I still like it): to provide
streaming images and an application that could make these streamed
images the desktop background image on Mac OS X, Linux/*BSD, or even
*shudders* Windows workstations. Similarly for screensavers. The
streamed images would be taken from massive Tibetan digital archives
that detail every aspect of Tibetan life and culture.

The second idea I had is 100% inspired by BBC Radio Scotland. And yes,
we Scots are refugees ourselves -- just ask us! (Especially the
MacGregors...). BBC Radio Scotland is an amazing online resource for
both real-time and pre-recorded Scottish programs. Very well organized,
very simple to navigate, and a wide selection of modern Scottish
culture. It provides a perfect template or starting point for doing
this with any culture.

Imagine a Tibetan village in rural, mountain America. A village where
several clever and community-owned companies employ villagers; where
there is a focus on sustainability and cooperation; where there is a
very good broadband connection, with regular shows broadcast on the
internet live and archived for later listening. Shows of traditional
Tibetan music, world, national, local, and village news in Tibetan;
Tibetan news in Tibetan and English; shows discussing the cutting edge
politics, political research, self-governance, sustainable living, and
the Tibetan Village project; history shows; fictional story shows
presenting Tibetan folklore; various dharma talks and instruction by
monks and nuns; Tibetan language classes that are broadcast, as well as
English-for-Tibetans classes. Every member of the village would be
actively involved in preserving culture and spreading it; greatly
decreasing any possibility of the current trend: its complete
annihilation.

The list could go on and on...

And this is technically and exciting project: I see visions of twisted
python streaming media code :-) Twisted Oog Radio player anyone? Raise
your hand if you're sick of RealMedia and WMP? Yes... this could be
very cool.

Update

I didn't make it to pycon this year, and I just found some good links
on streaming media with twisted, including a presentation at PyCon2005:

http://twistedmatrix.com/pipermail/twisted-python/2005-March/009730.html ([Twisted-Python] streaming producer)

http://twistedmatrix.com/pipermail/twisted-python/2004-October/008846.html
([Twisted-Python] First public release of Flumotion Streaming
MediaServer)


http://www.flumotion.net/


http://www.flumotion.net/doc/flumotion/presentation/pycon2005/html/img0.html

Thinking in Twisted

I've been emailing with a fellow developer and friend about learning to write applications and parts of applications using the Twisted networking framework. Even though Twisted is written in Python, conceptually -- and from an abstract point of view -- it is like a language of its own, with a foundation, syntax, and grammar that is Python.

My friend was trying to make use of basic Twisted stuff like deferreds, combine that with other parts of Twisted, but traverse the problem-solving process from an essentially ground-up, Python approach. I had tried to provide the shift in perspective needed in order to write Twisted apps with much less effort than he was exerting, but I fear that I failed.

Then, this morning, I thought of the perfect analogy. Actually, it's more than an ananlog -- it is the thing itself. Since really starting to learn Twisted, my programming has changed. Well, that and since absorbing such books as Patterns of Enterprise Application Architecture and Refactoring to Patterns. The deal with writing applications in Twisted is that you're no longer programming in the same language (Python); you're learning a new language. A more abstract one.

When we learn to think in Twisted and write applications in Twisted, our solutions will be elegant and compact. Before that starts to happen, we try to use our "old" ways of Python programming mixed in with the new one we are learning... and that, of course, defeats the purpose of the new "language."

The tricky thing about Twisted is that its patterns are an integral part of its "language" (perhaps it would be better to stop using quotes, and just call it a meta-language...). This is not true of most other languages -- though I would venture to say it is true of good frameworks in general. You don't need to know the patterns to use the grammar/syntax/control structures of most languages. Patterns in a language are usually optional.

With Twisted, if you aren't using the patterns, you aren't really using Twisted.

I am midway through this learning process. Therefore, I can't point to the solutions for many Twisted problems immediately or directly. I have, however, learned to see most of the critical components of a Twisted implementation. I still miss a few, though, and I have to keep reviewing code until I catch them all. Seeing these components clearly is what leads to the solution. As the time between "component vision" and "solution" gets shorter and shorter, we come closer and closer to thinking in Twisted.

pymon Nearing 0.3.3 Release

software

pymon is nearing its second public release (but first packaged
release). We've spent the past couple months doing a refactor by
examining the patters we were using or not using. In the process, we
have made MUCH better use of twisted's internals and are increasingly
more impressed with it as a networking application development
framework. To be honest, I would be very happy doing full-time twisted
development now.

Email alerts are turned on; ping hosts are running; HTTP status, HTTP
text checks, DNS dig checks, and MySQL admin queries are currently
being developed. We're going to try to knock out FTP and SMTP in the
next couple weeks, as well -- and then we'll bundle it up and release
it :-)

A tentative road map is up on the wiki, and we've got the beginnings of
developer docs on how to write "monitors" (like ping checks, HTTP
status checks, etc.).

http://pymon.sf.net

Vacation

leisure :: nature

So, even though it's been a week, I still haven't blogged about my
mini-vacation. I didn't go anywhere and I still had to do a little
work, but I did get to relax and hang out with a really good friend who
was visiting from out of town. She had her Spring break from
acupuncture school, and for the second half she came out to Colorado to
chill in the Rockies. We had the absolute best time; a much needed
vacation for both of us.

I swear, I will never get tired of the Rocky Mountains. The most
amazing vistas, trails, weather, etc. Just incredible. We went
horseback riding through a place in Estes Park, CO that does rides 365
days a year. It was such a blast. We were out in the afternoon sun on a
gorgeous day for over two hours. The best views I have seen since I've
been here (with the possible exception of a couple amazing sights inside
Rocky Mountain National Park).

For most of the trip, we vegged, watched movies, went window shopping
in Boulder, listened to music and talked. A genuine vacation. Not one
of those "run around and do all you can do till you pass out and need
another vacation" vacations. Rather, a time to really relax and
breathe. Also, her last night here, I introduced her to my Tibetan
friends at the weekly momo-making party. She had the absolute best
time. All of us laughed, ate, drank, and laughed some more. In fact, we
laughed so hard we were actually in pain. She got to listen to lots of
Tibetan and build up her ear for it. I taught her a little and
translated a little, but my understanding is so limited that it's not
much use. On the ride back to Loveland, we chatted away like a couple
prairie dogs about the Tibetan culture, their deep understanding of and
commitment to community, about how this impacts our own experiences,
understanding, and limitations, etc. It was an awesome time :-)

Accounting

accounting :: software :: society

So, I'm preparing for taxes (after my extension, of course). I've been
using this open source software called SQL-Ledger (written in perl,
unfortunately) to do my books this year; it's a double entry accounting
package. It's ugly as hell (I've made improvements -- css and python),
but I love it. I emailed my accountant with some double entry questions
(checking to see if the lights really are dawning), and I decided to do
some checking on the internet, where I found this gem I thought I'd
share ;-)

This is the first thing I've ever read that has made me not only feel
good about accounting, but want to participate in the process :-)
We'll, the second thing, actually. The first thing was starting to use
SQL-Ledger to it's capacity.

It's kinda fun:

3.1 The history of double entry accounting

People had been tracking money (to pay bills, collect taxes) for
thousands of years, but it wasn't till the 1400's that the Italians
invented double entry accounting. This made banking reliable, enhanced
trade and commerce and very quickly Italy became the banking capital
and wealthiest country in Europe. It was the 1400's equivalent of the
invention of the internet and the dotcom boom. They had buckets of
money to spend. They spent it on paintings of Adam talking to God done
on black velvet. Well they would have but velvet hadn't been invented
yet. They made do with what they had, which was plaster ceilings. They
spent money on painters, sculptors and people like Leonardo da Vinci.

In grade school this period is called the Renaissance, a flowering of
art and intellect, that appeared in Italy for no obvious reason and
then spread to the rest of Europe.

Quite why all these geniuses suddenly appear without any warning is not
explained by your history teachers (who spend their life pondering deep
questions like this), but Jared Diamond (Guns, Germs and Steel, Pub:
Norton 1997) is happy to tell you. He says that geniuses like
Michealangelo and Einstein are rather commonplace. Most of them are
oppressed and are living in abject poverty, and are busy surviving if
they even do that. Give them a good feed, treat them well, put them in
the company of peers and pretty soon they'll be coming up with all
sorts of things you hadn't dreamed of.

The Renaissance then was a result of the invention of double entry
accounting, not (as we've been told) a flowering of intellect and art
that happened for no reason at all.

Reference:
http://www.austintek.com/gnucash/ncsa-
gnucash-talk-3.html

pymon

python :: programming :: software

As so often happens, when working/playing on one thing, the miraculous
inner lights of the frail but mysterious human mind shed their
brilliance on another. Okay, so perhaps in this instance the areas are
not so far separated, nor the lumens so brilliant as those in a dying
plastic camping flashlight. Nonetheless, the fancy is tickled, and
besides actual development, that's what matters!

So what the hell am I talking about? APIs. I was working on the PyKon
Nevow code the other day, and some things I had written and prototyped
fermented in a corner of my brain, and popped into a completely new
perspective today, one that I could use with
pymon. I am addicted to good APIs. I
study twisted's internals as well as that of Phillip Eby's PEAK
framework(s). I hunger for elegant, efficient, and clever design
strategies. It's very Newtonian of me, but I love well-ordered systems.
The discussions that have been underway on the pymon maillist boiled
down (very quickly) to the following:

* The need for several mini-APIs to be used internally by pymon


* The need to define user- and/or enternal-APIs


* The need to get rid of a little cruft that had accumulated while in
branch (both old an new)


* The need to utilize twisted component-based approach (adaptors) to
accomplish these goals

We're really excited about this, because we're trying to be true to the
art of programming. QualityRUs, not time-contraint, budget-driven,
happenstancery. I wrote some more about it on the 2005.04.06 news item
at http://pymon.sf.net. It's going to
be fun writing all the adaptors we will need :-)

PyKon

I got some more work done on PyKon last night after finishing a deadline for a client. I'm excited about having a modern, functional, python-based image gallery application. The first version will allow both:

• manual image uploads (ftp/scp, i.e., changes you would make on the file system with a shell account) and the daemon that runs the web user interface will also regularly check for gallery/album/image changes and process added/changed files), as well as
• through-the-web uploads.

For the second release, I want to enable some work we've been doing on an object registry. This would essentially make PyKon like iPhoto for Mac OS X: a global repository for images, and the capability to create albums with multiple overlapping images (duplicates) with no extra space taken up by the images (as they would only be references/links to the image tagged in repository).

We're hoping to make this better than IDS (written in perl), Gallery (in PHP), and JAlbum (in java). We really want this thing to kick ass and be very easy to use. We're a ways off, but progress so far is exciting.

Oh, and Nevow (for Twisted) rocks.

Pete Yorn

music :: leisure

I am really behind the times. I discovered Pete Yorn only yesterday. I
was watching "Me, Myself, & Irene" with my mother last night. We were
both in a punchy mood, just having watched Bruce Campbell's "Alien
Invasion", so we laughed ourselves silly at Jim Carey's schizoid
antics. At some point in the movie near where they met Whitey, the
coolest song started playing. I've owned this movie since it came out,
and had never heard this song before. Well, it turned out it was the
song "Strange Condition" by Pete Yorn.

So I got a couple of his CDs, and holy crap -- this guy's good! He
reminds me of Elliot Smith, some classic REM, Counting Crows and a
little Grant Lee Buffalo. And sometimes you can hear a little Robert
Smith (The Cure). I feel like one of those dorks in "High Fidelity"
now... "She's kind of a
Sheryl-Crow-meets-a-post-Partridge-Family-pre-LA
Law-Susan-Day...only...black."

Nevow: URLdb

Well, I am really, really impressed with Nevow. It really grows on you as you code. I've been posting little tidbits that I learn on the Nevow wiki, the kinds of things that beginners should have easy access to.

Today, I dug really deep into the nevow libs and followed the path of form posts (configurables, validation, etc.). Very interesting stuff. Not only do I like the way Nevow behaves and handles for developers, I like its guts, too. That's actually fairly uncommon when it comes to web app frameworks.

Anyway, today was very successful: my TinyURL knockoff (URLdb) is "fully" functional. It has a form interface for posting/creating shorter URLs as well as a URL you can post to via javascript, etc. The reason that I put "fully" in quotes, is that there is no persistent backend storage yet: it's all in memory. This was for simplifying the development/learning process only, and I have already started on writing the Atop classes.

I look forward to using Nevow in the future -- immediately, on the PyKon project; but in general, for any web app that doesn't require the pre-built functionality of a full-blown CMS.

Ruby on Rails vs Nevow

nevow :: web :: programming :: python :: ruby

I've just read the post at Blue Sky on Mars, and
I've got to agree 100%. In fact, earlier today, I posted a couple small
projects I am working on in the Nevow
wiki, and mentioned a personal preference for Nevow over Ruby on
Rails. Personally, I believe one has more power and capabilities at
one's finger tips with Nevow. And this in no way takes away from the
simplicity and elegance of Nevow.

First, let me say that I am a huge Zope/Plone, twisted/Nevow,
and PEAK fan. Each has its strengths. But as far as general web
development frameworks go in python, nothing compares to Nevow for
building custom applications from the ground up. I know there are lots
of tools to choose from (Quixote, WebWare, among others), but the
simplest, cleanest -- and in my view -- the most powerful, is Nevow.

Nevow
has a thriving, active community, so I propose we give David
Heinemeier Hansson and Ruby on Rails a run for their money. Nevow
and it's community can beat them hands-down. Let's get some amazing and
easy-to-read docs up. Let's get some videos. Let's show the world that
python has even more to offer than Ruby, and a clean web app dev
framework in Nevow is one of them.

Here are some Nevow resources:

nevow.org

Nevow Manual


Nevow at Divmod

Nevow Wiki

Code Repository

Nevow mail list archives

Nevow at del.icio.us

Nevow at PyWebOff

Nevow examples - these are
the examples that come with the Nevow distribution

Python Eggs

python :: programming

Okay, two developers that I highly regard are working on
something very, very cool: a package/utility that works around
distutils limitations and will be able to provide the same
functionality as java .jar files do, but for python. And then some.
They are called python .egg files, and you can read about them here:

Python Eggs

DirtSimple

New Projects

As if things haven't been busy enough, with work, etc., I've got a couple python/twisted/Nevow projects in the works.

The first one is PyKon (Python Ikon), a gallery application for managing galleries, photo albums, and images. It's just getting started, so there's nothing to show yet, but it uses Nevow and the Python Imaging Library. This project actually comes at the request of many friends who have been bugging me about this for quite some time. You know who you are ;-) This will be the first non-technical, user-oriented app I have built with Nevow, and am therefore quite looking forward to it. I will also be delighted to not have to install, run, and manage any more PHP-based galleries.

The second project is a complete whim: I want to be able to provide fellow developers on mail lists and irc channels with a means of using something like TinyURL, but with the additional ability of being able to add an optional, distinguishing string at the end. This will make it more clear when reading a post to a mail list or seeing a bunch of URLs pop up on IRC. I'm calling it URLdb. I started by copying the Nevow pastebin example app (which totally rocks, btw). The code is here.

supybot IRC Channel Bot and Subversion Commits

programming :: python :: internet

Last night at the Hacking Society meeting in Fort Collins, I finished
my latest utility. I use supybot on our dev channel and the other day,
I learned how to write plugins for it. After writing one, I realized
that wasn't exactly what I needed. What I really needed was to add my
own daemon to run inside supybot's daemon. Then I learned the complete
internals of supybot. Quite an interesting experience. I wrote a
daemon/notification server last year that monitors arbitrary
directories and files for changes to contents, time stamps, etc. I
wanted my daemon to monitor a directory where our subversion hooks post
updates, see which files (updates) have been added, and then parse the
updates for information that is useful to post on an IRC channel. *And*
I wanted this daemon to run inside the supybot daemon. In less than 24
hours, supybot was learned and my daemon was buried inside of it nicely
;-)

At the Hacking Society meeting, I finished up the parsing for the
subversion commit updates... today, the results were viewable as
everyone made their code changes. Ah, data is wonderful.

Living in an Epic Tolkien Setting

leisure :: nature

We worked pretty hard at Adytum last week, so Friday and the weekend
were filled with little rewards... like pizza, going to the movies, and
most importantly, hiking in the Rockies. I have been in the Alps;I have
been in the Himalayas. Each are impressive and beautiful. The Rockies,
though... mmm. That's home.

I am now the proud owner of a season pass to the US National Parks.
This sponsorship ranks among the highest causes I contribute to -- just
as important as preserving indigenous cultures and human rights.
Environment, ecologies, biosystems -- these are insanely precious
resources. And, you know, one of the most potent ways to recharge,
de-stress, and get reconnected to the stuff that's really important in
life: living.

We drove up into Rocky Mountain National Park, just West of Estes Park,
CO. It was less than an hour from my front door, a fact I am
continually in awe of and grateful for. (Yes, those're *two* dangling
prepositions.) Driving down Routes 34 and 36, seeing these Immense
Mountains and then More Immensity behind those Immensities was just
extraordinary. I was just giggling all yesterday. My mind simply went
on overload and I reverted to the state of a 4-year-old who is fresh to
the world, has absorbed all he can handle, and simply babbles in
synaptic twitching.

The Rockies. This is my home, now. I just can't get over it. Whenever I
am hiking or driving or bicycling in those mountains, I feel like those
enormous, snow-capped peaks are my big brothers, or uncles... like
they're sheltering and taking care of me. It feels like an excerpt from
the Silmarillion or The Two Towers, where -- like the Ents -- the
mountains will, at any moment, begin speaking in deep, booming voices.

I simply couldn't be happier with living in Colorado. I see now that
I've been waiting to live here all my life. I'm home. And I'll let you
know if the mountains start talking about little people or feisty
rings.

Constantine

entertainment

I haven't seen any reviews, and I don't want to. I just got back from
the movies where I saw this little gem, and I've got to say that it was
awesome! I had such a great time watching it. I laughed at all the
jokes, enjoyed the fun occult references, and thought that Keanu did a
great job. My biggest fear was that this was going to be a poor attempt
at Matrixhood (like the first half of Underworld), and it totally
wasn't. There were a couple humorous nods to the Matrix, but the feel
was entirely different. Not having read the original comic, I have no
idea how true they stayed to the story, but whatever the case may be,
they told a great story ;-)

Very good fun.

iTunes Playtlist with cElementTree

python :: programming :: music

When we moved to Colorado from Maryland in January, a Tibetan friend of
ours who is a monk helped us pack, drive, unpack, and setup house when
we got here. During the drive, we spent many hours practicing English,
Tibetan, and listening to "injee" music on my iPod. There were certain
songs that he could listen to more easily than others. It was sometimes
surprising to me the songs where lyrics were enunciated well enough to
be mimicked without understanding the content. The one that made me
laugh the hardest was "Cold As Ice" by Foreigner... there is something
totally hilarious about driving cross-country with a Tibetan monk, both
of you singing "She's as cold as ice!" at the top of your lungs. It's
one of those moments you know will stick with you forever. Another good
one was KC & the Sunshine Band's "Shake Your Booty" -- it turns out
English "shake" is cognate with a Tibetan colloquial word of similar
pronunciation and meaning.

Anyway, I was supposed to build CDs of photos and music from our
several weeks together and send them back home to Lama Gyaltsen. iTunes
exports playlists in several formats, including XML. This was a perfect
excuse to play with ElementTree some more, while the CDs burned (that's
another cool thing about iTunes: got a play list that's 3 and a half
hours worth of music and you want to burn to CD? No problem! It splits
them up for you. Save your brain cells for the important stuff like
playing with python).

Here's the code:

import cElementTree as et
filename = '/Users/oubiwann/Desktop/Mix - Lama Gyaltsen.xml'
tree = et.parser(filename)
root = tree.getroot()
template = '%s: %s'
for i in root.find('./*/dict').getchildren():
     if i.tag == 'dict':
         data = i.getchildren()
         for j in range(0,len(data)):
             if data[j].tag == 'key':
                 if data[j].text == 'Name':
                      print template % ('Song Name', data[j+1].text)
                 if data[j].text == 'Artist':
                      print template % ('Band Name', data[j+1].text)
                 if data[j].text == 'Total Time':
                      converted = '%s:%.2s' % 
divmod(int(data[j+1].text), 60000)
                      print template % ('Length', converted)

Update: Fredrik Lundh has just blogged about this :-) He has a
added simpler loader for for iTunes playlists to ElementTree. Read
about it here.

Nevow + Twisted, or mod_python?

python :: web :: programming

To twist or not to twist, this is always the question.

Or, rather, I always want to use twisted, but can I produce the numbers
and the tests for customers that show it performs as well as
mod_python?

Or, even better: how does one architect a killer twisted web app server
that performs insanely well with high concurrency and blinding speed?
Hmm, It's been a while since I checked out the Nevow code base, so I
just did an svn update, and I see there there is now a wsgi cgi for
mod_python in the examples directory. This is cool, I shall have to
test... Perhaps I can have my cake and eat it too.

Well, I just installed it on a server of mine and have been running
apache bench against the following:

nevow wsgi mod_python code
a nevow example running on the twisted daemon
the AdytumSolutions web page (custom mod_python/publisher mini-cms)
a tomcat example jsp page.

I ran 5000 requests, 20 concurrent requests for each. Here are some
results:

nevow wsgi - Requests per second: 48.96 [#/sec] (mean)
nevow twisted - Requests per second: 64.85 [#/sec] (mean)
simple mod_python app - Requests per second: 824.22 [#/sec] (mean)
tomcat - Requests per second: 288.01 [#/sec] (mean)

I've just spent about an hour reviewing the Nevow example apps (started
up the examples by cd'ing to the examples directory and
running 'twistd -noy examples.tac'). I must say: Donavan,
Matt, and company have done an amazing job in the last year. Truly
impressive. Lots of really good examples that get the creative juices
flowing and jump start one right into seeing how to solve all kinds of
problems with Nevow. In fact, I am so impressed that, if I can learn
how to get as much performance out of Nevow, I would gladly sacrifice
some raw running/processing speed for ease of development and
maintenance. And extensibility. And joy.

I wonder if I can get my clients to see things this way?

Perhaps it's time to hop back on #twisted.web and start asking
questions I'm sure they are quite tired of answering...

Advogato Diary Archiver

web :: python :: programming

Last night I finished the script to pull (XML-RPC) my diary entries on
Advogato into a custom archive format. I had to use an ugly sys.stdout
hack to use the cElementTree.dump() method... but discouraging it's
misuse was probably a motivating factor for Fredrik Lundh to send its
output to stdout in the first place. I just wanted the entire contents
of a particular div, and I didn't want to have to do all that text/tail
crap.

The really nice thing thing about this, though, is piece of mind: a
single server crash anywhere will no longer result in catastrophic loss
of blog entries. Because, you know, blog entries are so damned
important ;-)

Update: Duh. cElementTree.tostring(aElement). Say no more.

America Beyond Capitalism -- Posters!

economics :: governance :: politics

Today, I got an email from Ted Howard (director of the Democracy
Collaborative) letting me know that there are posters based on the
book America Beyond
Capitalism available for download here. Very cool stuff. Some very good facts to know and
be aware of. One that really pisses me off is in 1945 corporate
taxes provided 35.4% of federal revenues and now corps are only paying
7.4%. That shit just ain't right. I also like this one -- and in to answer their question,
no! it's not what I want. In fact, since working for myself, I have
changed my work week to less than 40 hours a week. I strive for 30, but
would like to get it down to 25 hours/week. With more time off, I get
to spend my time relaxing, hiking, volunteering for projects I care
about -- improving my quality of life. As a result, during my work
hours, I am much more productive and energetic, happy about life and
working.

BTW, if you haven't read the book America Beyond
Capitalism, you really ought to check it out. It's a fast read with
some intense (but accessible) data. And it's not a doom and gloom thing,
either - there is all kinds of information in it about what various
groups, researchers, corporations, communities, etc., are doing to
change the immense imbalance of power and wealth.

Advogato and Bad HTML

web :: programming :: python :: html

I don't know if I'm the only one who's trying to use Advogato in
non-ordinary ways (and I'd guess that at least Titus Brown is), but
I'm having a hell of a time getting mechanize to
work with the site. I love vi and I won't apologize for it ;-) I love
using it as my IDE as well as my CMS. I want to be able to edit
valuable information in a vi/vim terminal window. Sean Reifschneider wrote a nifty little python
script for editing messages in vi (I forget which RFC to which they
were compliant) and posting to websites. Alex Levy did something related, where you
could edit pickled data files via command line editors. I just totally
dig that stuff. I've written three mini-CMSs in PHP, perl, and python
(over the past 5 years) that use the command line as the primary means
of adding and editing data. Now, I want to do the same thing with my
Advogato account.

Alas, this is proving very difficult so far. Advogato does not seem to
be using proper html! So far, none of the text input fields in the
forms I have looked at are labled type="text" -- they are missing the
"type" attribute all together. As a result (at least, I think the lack
of "type" is the cause), mechanize does not see these inputs as
controls, and I can therefore not manipulate them. This means that I
cannot login and I cannot update my account settings. Very troublesome.
Additionally, it would be wonderful if the forms could be given
name attributes for easy access. I would love for someone to point out
a way around this or my misunderstanding of mechanize.

Of course, there is always the hope that the Advogato XML-RPC API will
be extended to allow for the managing of account data as well as for
publishing articles. Then I wouldn't care how the html gets abused ;-)

Update:
It seems that I can, in fact, get controls for all the account form
elements in mechanize, even though the text boxes are missing the
"type" attribute. However, I am still unable to process the login form
via mechanize. I am able to login, but that requires passing the
passphrase for the account in a GET, the consequence being that the
account password will turn up in log files.

pymon

python :: programming :: networking

Well, after several months, I've finally had a chance to sit down and
work on pymon again. We're plugging
away, slowly but surely, on the next alpha release. Every time I dig
into twisted code, I am just delighted. Those guys just rock. The
flexibility it provides in writing network-oriented apps is just
amazing. I'm really excited to see how this project matures.

However, I'm getting tons of Zope/Plone work now, and that's stuff's
currently paying the bills, so pymon might have to wait for a few more
weeks. Until I get crazy with working with Zope/Plone and need a real
coding break ;-)

Jefferson in "The Economy of Ideas"

history :: open source

I was checking out Ian Bicking's recent del.icio.us additions (that's
not stalking!), and I got high on hypertext crack... yeah, I couldn't
stop. Finally, I hit the wall here:

The Economy of Ideas.

And the intro to the article is a thought-provoking quote by Thomas
Jefferson. It's been ages since I read this, and its eloquence (though
the phrasing is dated, i.e., not PC) is still enough to take the breath
away:

"If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an
idea, which an individual may exclusively possess as long as he keeps
it to himself; but the moment it is divulged, it forces itself into the
possession of everyone, and the receiver cannot dispossess himself of
it. Its peculiar character, too, is that no one possesses the less,
because every other possesses the whole of it. He who receives an idea
from me, receives instruction himself without lessening mine; as he who
lights his taper at mine, receives light without darkening me. That
ideas should freely spread from one to another over the globe, for the
moral and mutual instruction of man, and improvement of his condition,
seems to have been peculiarly and benevolently designed by nature, when
she made them, like fire, expansible over all space, without lessening
their density at any point, and like the air in which we breathe, move,
and have our physical being, incapable of confinement or exclusive
appropriation. Inventions then cannot, in nature, be a subject of
property."


- Thomas Jefferson

I have been and continue to be involved in many open source projects;
as developer, contributer, user, etc. I have also been involved in many
closed-source, proprietary projects. There are trade secrets and code I
will take to the grave with me, but this is from a sense of honor and
ethics; of agreements between entities. Given my preference, though, I
would love for all software to be open source... the idealist beauty of
Jefferson's words makes promises too sweet to resist.

Experiment in Community

community :: society :: governance

I live with three very different people; that's three, including
myself. We are all different, and we decided to try living together as
a social experiment. Personally, I am interested in decentralized
social and governmental systems: distributed power, distributed
decision making, etc. I am an optimist in that I believe that a
"sufficiently large" collection of people "educated" on a particular
topic will collectively make statistically wise decisions on that
topic, no matter the variations in individual decisions. To me, this
seems one (of possibly uncountably many) functional definitions of
democracy. It may come off as elitist (and to some, that's okay) due to
the use of the term "educated" but this in no way refers to social,
economic, or any other form of privilege. I use the term very generally
to connote experience, knowledge, and understanding of a certain topic.
Of course, there is that ambiguity "sufficiently large"...

Anyway, I digress. Social experiment. Understanding of social systems
via practical experience, with willing participants that are aware of
the process, the goals, etc. We have hit the "growth" stage now, i.e.,
problems. And in a discussion with one of the members today, I was
asked how I define "community." This was an interesting question... I'm
always talking about this concept, this abstraction; I'm always
listening to others talk about it. We all seem to agree that we're
talking about the same thing, but here was my chance to see, in fact,
if that's the case. I paced around the room for a while, like an
autistic child about to make a break through. Here's how I defined it:

A community is a group of entities that interact for the common benefit
of all entities in the group through exchange of information between
entities and the compensation for deficiencies and/or efficiencies of
individual entities.

I'd never thought of it like this before, and it really took me by
surprise: community dynamics is an engineering problem! Of course, I
say this with all the humor it deserves ;-) And this revealed the
nature of the problem between two of the individuals in our social
experiment: two of us are unwilling to compensate for each other's lack
for the common benefit of the group. We are therefore failing as a
community, in that we cannot satisfy the definition. This awareness was
a painful moment for me, as I have been greatly looking forward to this
adventure. It's early still, so there is hope. The biggest difficulty
we may encounter is motivation: if we don't feel in our bones that we
have to work with these differences in order to survive, we may
not put in the required effort.

Community interaction is not an endeavor of self. In the "West", we've
got plenty of that! We are all about the individual. I find this
particularly true of Americans, and have been guilty of such for much
of my life. We feel that many (all!) sacrifices somehow lessen "the
self" and are therefore tantamount to the complete violation of our
most precious principles. Practical experience in communities that meet
the requirements of the functional definition above show us something
quite the reverse: personal sacrifice for the benefit of all improves
the quality of life for all members of the group, and sometimes
especially for the one who sacrificed.

In these "modern", post-tribal times and for the most part, we are no
longer bound to each other by the glue of survival needs, forced to
live in communities and tribes, getting along out of necessity. As
such, it may be that Quality of Life has become the new definition of
Survival. And in fact, our survival as a sane and happy society may
require that quality of life be perceived as such.

Curious now, I have looked up the definitions of community on WordNet:

* community -- (a group of people living in a particular local
area; "the team is drawn from all parts of the community")

* community -- (a group of people having ethnic or cultural or
religious characteristics in common; "the Christian community of the
apostolic age"; "he was well known throughout the Catholic community")

* community -- (common ownership; "they shared a community of
possessions")

* community -- (a group of nations having common interests; "they
hoped to join the NATO community")

* profession, community -- (the body of people in a learned
occupation; "the news spread rapidly through the medical community")

* community, community of interests -- (agreement as to goals;
"the preachers and the bootleggers found they had a community of
interests")

* residential district, residential area, community -- (a district
where people live; occupied primarily by private residences)

* community, biotic community -- ((ecology) a group of
interdependent organisms inhabiting the same region and interacting
with each other)

I particularly like the last one, from the ecological point of view.
Being an adherent to a Tibetan Buddhist school of philosophy, I enjoy
the point of interdependence.

Generic Functions and Rule-Based Systems

Well, I've been pretty bummed that business is keeping me too occupied to attend PyCon 2005 this year. Last year's was so great, and I had a chance to sprint with Chris McDonough and Donovan Preston and learned a great deal from them both.

In fact, I was decompressing the data I took in during the python sprints for the following 6 months. I guess it shows what a geek I am too, that I bemoan too much business, as it's preventing me attending this year.

I was checking out the presentations here, and I came across Phillip Eby's presentation here. This really hit home, since one of the projects I am working on is for processing *massive* amounts of data produced while testing equipment at manufacturing plants. There are sets of sets of sets of ... tests. Thousands of the buggers. Many of them are very similar in nature, and they all have associated rules for pass and failure.

I can't wait to tryout Phillip's generic functions in PyProtocol's... in fact, I haven't waited ;-) I'm already writing some prototype code for it...

Python rocks.

Clearance

personal :: moving

I
really wanted to get a picture of it, but I kept forgetting. It's
really too bad, too. Alas, I shall have to describe it, hoping to
capture some of the comedic impact.

I have just moved to Colorado with some friends. We did a caravan
cross-country, and it was a blast. However, there were, shall we say,
some "low points" along the way...

(rest of entry truncated/lost)

Sorting, Packing, and Throwing Out

personal :: moving

This
is the worst part of moving for me, and it always has been. The joy of
a move to a new place is tainted only by the arduous series of tasks
required in getting there ;-)

In the process of sorting through all the printed manuals, web
pages, etc. held together with binder clips, I found my first python
reference manual. So, today I had to say good-bye to something I didn't
even know I had: a printed hard-copy of the Python 1.5.2 reference
manual.

Yes, I shed a tear. A little Pickled tear. But I used cPickle, since it was faster.

Paul Graham - Value and Productivity in the New Economy

internet :: techology :: economics

In Paul Graham's essay What the Bubble Got
Right, he makes mention of several things that many people are
discussing all over the net in technological, political and economic
circles. Of particular interest are his discussions on value and
productivity. Of particular interest to me is how these insights mesh
with the visions that others are having of economics, business,
management and society in the coming decades.

The following bears greater reflection:

Recognizing an important trend turns out to be easier than figuring out
how to profit from it. The mistake investors always seem to make is to
take the trend too literally. Since the Internet was the big new thing,
investors supposed that the more Internettish the company, the better.

In fact most of the money to be made from big trends is made
indirectly. It was not the railroads themselves that made the most
money during the railroad boom, but the companies on either side, like
Carnegie's steelworks, which made the rails, and Standard Oil, which
used railroads to get oil to the East Coast, where it could be shipped
to Europe.

Gar and Ted's Excellent Adventure

economics :: politics

Okay, and Duncan too ;-)

Well, today I had the immense good fortune to meet with Gar Alperovitz, author of America Beyond Capitalism and Ted Howard, Director of The Democracy Collaborative. We had a fantastic meeting this afternoon in Washington, D.C.

You know, there are times when you meet people and the
personalities, atmosphere, communication and relating go beyond all
expectations. This was one of those times. They are both exceptional
individuals with lofty goals coupled with a down-to-earth approach for
attaining these goals; two very welcoming and natural or "real" people
with a great sense of humor and deep interest in open, honest,
thoughtful discussion. It was an honor to share conversation with such
scholars and activists, but even more so, it was an honor be sharing
food and drink with two wonderful human beings.

These are the types of interaction that have been slowly and queitly transforming our society. I look forward to more :-)

(rest of entry truncated/lost)

A Random Blog

I think I will call it "Schrodinger's Box" ...

I was reading the paper "Emergence of Scaling in Random Networks", and part of it sparked a thought: it shouldn't be that difficult write a blog that generates random gibberish... but not just any gibberish. The parts of speech should be correct (for the most... part); sentences should represent complete "thoughts." In the spirit of that most bizarre of internet radio stations, Eigenradio, the source material will be analogous to radio stations: blogs, news items, popular google searches. But how, you ask? Easy!

[rest of entry truncated/lost]

Technorati Tags: programming, python, random, weblogs

Traceroute Games

This post is actually taken from and email that I wrote to some friends (late last December) who like this kind of stuff. I went searching through my email, looking for what it was that I did with pydot and graphviz just a while back, and I realized I never blogged about it, and I should have ;-)

A few days after the Winter Solstice, I was screwing around with pydot (graphviz) and a custom traceroute python wrapper I threw together (Update: this is now available here). I wanted to graphically visualize multiple traceroutes and see the paths shared in common. This is part of a proof-of-concept for some monitoring ideas I have.

The end result was rather cool, the images for which are available here. Here's the Python code that generated the images (though all the heavy lifting is done by the traceroute module, linked to above):

import pydot

from adytum.net import traceroute


hosts = ["www.google.com", "www.amazon.com", "www.aol.com", "www.cnet.com",
       "www.yahoo.com"]
linkedHosts = []

for host in hosts:
  trace = traceroute.Trace(host, useDNS=True, queries=5)
  trace.run()
  domains = trace.results.getLinkedDomains(subdomainLimit=2)
  linkedHosts.extend(domains)

graph = pydot.graph_from_edges(linkedHosts, directed=True)
graph.write_png("traceroutes.png", prog="dot")

Beyond Capitalism

economics :: politics

Today,
while running errands for the move, I was listening to NPR and an
interview with Gar Alperovitz on his new book "America Beyond
Capitalism: Reclaiming Our Wealth, Our Liberty and Our Democracy." It
was awesome. He had some very interesting stats on wealth in this
Nation, the division wealth by race, the political inheritance of the
South (from the Confederacy) as it has recently become a Republican
stronghold, the decline of labor unions and the labor politics, and the
rise of the very wealthy.

Yet, he had some excellent and encouraging
things to say about the amount of work that has been going on at the
grassroots level. However unbalanced things have become politically and
economically, there has been a whole new focus at very practical levels
-- right where we live, work, and play.

(rest of entry truncated/lost)