Friday, December 30, 2005

The King is dead! Long live the Kinglets!

So what the hell does that mean? Well, there's sort of a back story there (but you can skip to the last paragraph of this post for the punchline).

Earlier this year, I blogged about the language patterns of twisted: Thinking in Twisted. The use of and relationships between such things as protocols, clients, servers, and various factories in Twisted was really starting to make its way into my bones. These and other core aspects of Twisted best practices form the basis of a pattern language... one that is of immense value when writing network code and applications. Here are a couple quotes from that blog entry:
The tricky thing about Twisted is that its patterns are an integral part of its "language" (perhaps it would be better to stop using quotes, and just call it a meta-language...). This is not true of most other languages -- though I would venture to say it is true of good frameworks in general. You don't need to know the patterns to use the grammar/syntax/control structures of most languages. Patterns in a language are usually optional. [However, w]ith Twisted, if you aren't using the patterns, you aren't really using Twisted.

When we learn to think in Twisted and write applications in Twisted, our solutions will be elegant and compact. Before that starts to happen, we try to use our "old" ways of python programming mixed in with the new one we are learning... and that, of course, defeats the purpose of the new "language."
I've recently hit another level of awareness in pattern abstraction, and it has its roots in Twisted. In fact, I never would have been working on what I am now working on if it weren't for Twisted and all the work done by Divmod and its crew. While at PyCon 2004, I met with Amir and Glyph and we discussed twisted's use in the industry at large, and in particular, opportunities for which the Divmod product line could be applicable.

What I liked so much (and immediately) about what Divmod was doing was that they were integrating useful components into a single application (Quotient). It stuck a positive chord with my instincts about where software should be going, and it kept me thinking for months while I worked on other projects. These thoughts culminated in a series of blog entries and software projects. First, a post called Dinosaurs and Mammals explored this. In the intervening time, the projects I have worked on have helped me evolve these ideas usefully. There is now direction, code, and product (more about that later in this post).

Then, in Everything as a Distributed Resource, I took a more sci-fi, radical view... but one that is born out by current trends and tendencies, particularly in the cell phone market. There's a great deal we haven't thought about in regards to the cell phone market. Not so much about cell phones and mobile technology, but rather the usage patterns and demands that are being made and (sometimes) met in this sector.

In an effort to follow through on the "mammal" paradigm, I did some exploration with URLdb and PyKon which I blogged briefly about in New Projects. The two projects mentioned there have since been merged into a project that we are calling Adytum's Twisted Collection. The latest addition to which is ImageDB.

ImageDB is a repository for static graphic media. The images that people upload here can be viewed via it's web interface, but it is more likely the ImageDB would be used as a part of something else. For example, when I get around to finishing PyKon, it will make use of
ImageDB. PyKon will not store any images itself. It's simply going to be an nicely designed, usable image gallery interface. Users won't know that, though. They will use PyKon and be able to create galleries and albums in that gallery by uploading images (which will be stored --
transparently to the user -- in an ImageDB instance), by selecting individual images (JavaScript ImageDB selector/browser), or by selecting "tags" a la (and therefore whole groups of

Analogous to ImageDB is MessageDB, a repository for static text. It's next on the list to be developed. This micro-app will hardly ever be used directly, although it too will provide a basic web interface for viewing and editing stored content. MessageDB will be a storage app for RFC 2822-compliant "messages". For starters, various public mail list emails will get tagged and stored there.

Non-dynamic text for web sites and blogs will be stored as RFC 2822 messages. So the forth-coming tools "SimpleSite" and "WeBlog" (also part of the collection) will actually store and pull their content from MessageDB. But even more, when you want to use a photo, clipart, or logo in your site or on your blog, you will click a "select image" link and an image thumbnail browser will pop up. You will select one of the thumbnails, the popup will go away, and a link to that resource in your ImageDB instance will be inserted for you, in your blog or site text.

The idea is that by abstracting out general functionality into little micro-apps, you not only decrease the amount of work in maintenance and feature addition, you not only are reducing the amount of redundant work you have to do when building content-related applications, but you are providing an infinite amount of flexibility for using these various media in completely unplanned and unexpected ways.

Your media, it's storage, it's access (XML-RPC), now live in self-contained little units. They can "survive" on their own, without a heavy dependence upon some huge CMS that was written for a particular project. The apps are project-agnostic. They can be set up once and then reused repeatedly in different contexts by building new apps that obtain their data from the component micro-apps via simple links and/or XML-RPC calls.

And, again, the ways in which users and designers decide to combine these pieces and how they decide to use that is 100% flexible and thus fertile ground for the unexpected and creative :-)

I say "To Hell!" with large apps, and banish them forever! Let's build tiny, little, single-purpose apps once! And then for larger needs, simply glue together various micro-apps as needed per the project requirements.

As far as I'm concerned, the old King of the Bloated Application is dead. It's been replaced with tons of tiny, decentralized micro-apps. Long live the Kinglets!

Sunday, December 18, 2005

pymon Update

twisted :: python :: web

There are now pymon web ui screenshots up on the project trac
site. These are two status views (summary and detailed) for monitored
services. They are 100% Nevow and load super fast. I'm very happy with
the progress so far. For this first release, I think we'll just make
some adjustments to the admin landing page, and call it ready to go.

Now playing:
Deep Forest with Peter Gabriel - While The Earth Sleeps (Sunday, 11:48pm MST)

Saturday, December 17, 2005

pymon, ZConfig, Nevow, and pyparsing

twisted :: python :: web

As I have published recently on the pymon
, we're getting very close to a pre-release of pymon. The
latest changes involved migrating from a custom, home-grown
configuration scheme and API to ZConfig. I am very happy with the
Apahce-style config files of ZConfig. It has drastically simplified the
configuration files, thus lightening the work-loads of systems and
network administrators who use pymon. But even more, I like the ZConfig
python API for accessing configuration data. Our home-grown solution
had an API that was similar to, but not as clean as ZConfig's.

After getting pymon running with ZConfig, I turned to the web UI for
viewing monitored services states. I haven't touched Nevow since
version 0.4, and it was really nice to dive back in. I'd forgetten what
a joy Nevow is to work with. Now some significant progress is being
made on state data display, and some design choices need to be made
about Nevow render patterns and slots... fortunately neither rocket
science nor mind-numbing ;-) There's a good chance that the web UI will
be ready in time for the release.

Unfortunately, I don't think the first steps towards the pymon OS will
be. We've got some code prototyped in my sandbox. This will be a great
feature. You telnet/ssh into pymon, and run commands that control the
monitoring configuration, display monitored service states, etc., just
like you would on a firewall. We're using pyparsing to generate the
grammar and parse user commands. It's in a very early stage, but it is
one of the more exciting aspects of pymon, as it is interesting code
and there's nothing out there even remotely like it (to our knowledge).
As soon as the first release is out, though, we'll be merging the
prototype pymon OS into trunk, providing at the very least a glimpse
into pymon's bright future ;-)

As a couple of us have been using older versions of pymon for a couple
years now, and I have been using the new version for a year, we're
ready to start sharing with others, getting feedback, and developing a
user and contributor base. What's more is, even though I can't say much
now, we are demoing pymon for a couple companies, one of which
specializes in security solutions for the defense industry.

Look for a pymon release very soon!

Monday, December 05, 2005

Twisted Network Programming Essentials

twisted :: programming :: books

The title says it all. Go buy it
! I've only just managed to buy a copy yesterday. I'm fairly
familiar with twisted and have used it extensively over the past
several years, so I was going to buy the book just to support Abe and
the Twisted community. It turns out, though, that he has written this
in an amazingly lucid style, one that I will actually use to teach
twisted to junior developers -- a task I have not attempted yet as a
result of not having a good approach or proper materials to draw from.
Until now, twisted has been for self-learners only. With this book, you
will be able to teach twisted to interested and motivated middle- and high-schoolers.
One particularly wonderful nugget was the description of twisted.cred. I've
always been a little foggy with the the architecture and use of
twisted.cred (credentialing and authentication in twisted)... mostly
because I've never needed to use it. Turns out I am working on a new
project that will utilize it. Just in time to get the crash-course on
cred from Abe's book :-)

And this is definitely and by far the most cool, insane, intense cover of
any O'Reilly book. Ever. Future books included. You just can't beat the
writhing, twisted ball of mating snakes. No joke. Read about it on the
last page.

Now playing:
Coldplay - Swallowed in the Sea (Tuesday,  4:50pm MST)

Saturday, December 03, 2005


software :: linux

Well, I finally caved. After the past three years of running OS X at
home, I have need for a file server again. I tend to run SuSE in
production, as many of my customers like the administrative interface
of YaST, but after using Fink (apt-get) on the Mac, I have been very
curious to try a Debian distro. Ubuntu is the top of many peoples'
lists these days, and it's won several recent awards, so I thought I
would give it a shot. The short and long of it: Ubuntu as a server

I use that qualification because I only used it briefly as a desktop
OS, running the LiveCD on my 17" PowerBook for a night. A most
enjoyable experience... if only there was support for the Broadcom
builtin wireless :-(

As a server, it is phenomenal. This is due not so much to Ubuntu, but
rather as a result of it being a Debian distro. Getting servers
installed, locked down, and running the network applications I need is
my top priority when selecting a platform these days. My second
priority is time: all of that has to happen within 1-3 hours of
beginning the process. I'm not a sysadmin anymore, and I can't spend my
time on that when it is better spent on development and R&D.

We'll have to see how SuSE fares at Novell over the next year (due to
recent changes in the SuSE team and the Novell layoffs, I'm watching
closely). If things start looking bad -- or if Ubuntu gets a set of
killer admin UI apps -- future server upgrades may experience a distro

Oh, and as a side note: LVM is phenomenal. I'm running it on the file
server, and am just amazed at the progress that has been made in the
last 4 years. I see no reason to use Veritas Volume Manager anymore
(for any of my needs or any of my customers' current needs). If you've
got a ton of drives laying around, not pulling their weight, dump them
in a box and put them to dynamic use with LVM.

Now playing:
Jon Brion - Monday (Sunday,  3:34pm MST)