Blogging via Email with Python

python :: programming :: weblogs

I've been playing with Abe Fettig's twisted application Hep
for a couple months now, and I really like it: some nifty integration
of concepts and utility. Playing with Hep also got me looking at Yarn, and that got me thinking: what would it take to roll my own little app that would let me post to MovableType via email?

(rest of entry truncated/lost)

Mount Vernon

history

So, for my Winter Solstice present, my dear friends Ellen and Megan took me on a tour of George Washington's Mount Vernon Estate.
It was really cool... not only a thoughtful present (since I've been
doing some reading if the pre- and post-revolutionary times), but as a
physical frame of reference I will be able to use for years to come.

In the 8 years that I lived in the D.C. area, I've never toured a
site or monument that had as much meaning and substance as the grounds,
mansion, slave quaters, greenhouses and tombs/cemetary of Mount Vernon.
I would encourage anyone interested in early U.S. history to check it
out if they ever get a chance. Not life-changing, but certainly very
useful... providing an excellent context for the letters and writings
of the time period.

(rest of entry truncated/lost)

Syntagma

python :: programming :: natural language

This
is a bizarre word. So strange, in fact, that I have a great deal of
trouble remembering it. Not it's meaning, but the actual word itself. I
figure if I blog about it, my troubles are over.

I spent probably three hours today scouring the internet looking for it, using such phrases as

vowel consonant groups


vowel consonant statistics OR heuristics

vowel consonant frequency distribution

consonant vowel letter statistics corpus

computational linguistics

natural language processing

(rest of entry truncated/lost)

Support for Mundialization

open :: business

I've just discovered Mundialization thanks to evangineer's post on #esp regarding hex's recent addtions to del.icio.us.

The part that really speaks to me is this:

Firstly, people say that the 21st century will be an age
of cities and towns and, in other words, an age of regional governance.

That just sums it all up for me. It's along the same lines as the recent column in Scientific American, Common Sense - Surprising new research shows that crowds are often smarter than individuals.
Decentralization is healthy, smart, and has the potential (with
excellent communication) to be a powerful resource. I wonder if there's
such a thing as "distributed law" yet?

Anyway, in the interest of supporting change, there's nothing like
putting one's money where one's mouth is. So, I have drafted a
Mundialization Declaration for AdytumSolutions, and am waiting to get
feedback/approval. Here it is:

(rest of entry truncated/lost)

New Home for the OBF and OBMN

open :: business

The Open Business Foundation and Open Business Materials Network now have a new home at Adytum.
Despite being a couple years old, they are still nascent and not very
active. Seeing how I didn't renew their domains a couple months ago and
they were definitively destroyed two days ago, it's time I found a
place to put them ;-) If they ever grow into anything beyond a small,
conceptual project, they may find their own namespace again, but for
now, Adytum will suffice.

freetype2 in fink (Mac OS X) is hosed

macosx :: software

However, there is a quick fix. This is all I did, and matplotlib compiled and installed without issue:

sudo cp -rp /sw/lib/freetype2/include/* /sw/include

Vio la.

Vim and Subversion as a CMS

web :: software

Like nearly everyone on the planet who has ever had to write
HTML, I abhore it. When I started working for the University of
Maryland as a developer in 1998, I learned to hate it more than ever. I
has responsible for multiple, complex sites, all with regularly
changing content. Maintaining all that HTML was a nightmare.

A few months later, I discovered Zope and began my long love affair
with Python and an abiding interest in Content Management Systems. Yet,
all the "long" years (1 human year is 7 years in software ;-) ) since
then, there hasn't been a great deal of progress with open source CMSs.
Sure, there has been a great deal of refinement and improvement in user
interface, in feature sets, etc., but when it comes right down to it,
what does it mean?

I started my search for a CMS at UMCP because I need to save time
more than anything else. I needed a way to easily edit files, to not
have to play browser compatibility wars with every change to the site,
to easily generate new content.

For large, complex sites, I now use Zope and Plone, especially when
customers will need to manage their own content. However, I have found
that through a combination of simple technologies, I can acheive all
the speed and convenience I have ever needed without any of the tedium
-- including HTML.

I have many friends who are emacs lovers, and I have heard all the
arguments, yet I have been a vi/vim user for many years and do not ever
see that changing. Additionally, I am a huge fan of python and the
software version control system called Subversion. These three things
have become merged into a seamless CMS that fits many of my content
management needs.

Here are the components that I use:

* HTML templates

* Ini-style config files for menus, stories, and other content (very easy to create/update)

* Very simple python code to apply the content to templates and render HTML output

* Subversion to manage site changes, versions, rollbacks, etc.

* vim as a command-line tool to manage the content

The thing that really did it for me was a feature of vim I had never
played with until recently: windows. Windows in vim are panes. they can
be arranged vertically, horizontally, or any combination. They are easy
to create (:vne, :new) and easy to maniuplate (^xw, etc.). I typically
have 4-5 panes in a single terminal window open at any given time, and
I can access all the buffers in any pane. This means that I can compare
two python libraries side-by-side, edit the project web site (all in
ini-file format!), update the ChangeLog and add to the TODO file all
without haveing to change mental modes.

And that's the big key and biggest show stopper for any software
from a user perspective: does the tool you use cost you time in mental
adjustment while operating the standard tools of your trade? If so,
it's time to re-evaluate.

For me, I discovered that I was saving time. Huge chunks of lost
time were regained and I was able to start doing many more tasks
simultaneously, since I didn't have to shift mental modes. I have an
irc session open on the development channel, an interactive python
session for debugging and generating doc tests, and my ultra-vim
terminal for code and content.

It's a beautiful thing.

I did try to run interactive python sessions in vim -- and with
success, too. A very nice plugin called vimsh developed by Brian Sturk.
However, I found that this actually slowed me down, so I switched back
to an independant terminal window.

I guess the moral of the story is explore and don't listen to The Man; use the tool that's best for you ;-)

This is not where I keep my stuff

I keep my stuff here

The Breakdown of the Tribe and Its Anticipated Rebirth

community :: society

A good friend who has very little contact with the open source
movement asked me to offer ideas that could assist a non-technical
organziation for which we have both volunteered much time and energy.

As a result, I summarized (at length) ideas that other friends of
mine and I have been discussing. Though focus here is put on the open
source community, our deepest concern has been for society as a whole
and useful models of which small groups can take advantage and from
which they can learn.

I've been doing some serious analysis concerning societal paradigms,
especially in regard to evolving definitions of "tribe" over time.
Related, is the issue of group dynamics and group health. The
efficiencies and "health" of the massive open source community has been
especially puzzling until recently.

Let me start with background:

It is my current view that tribes existed as cohesive units due to
violent "external" conditions (including predators, enemies, and other
hinderances to survival). If the forces at play were external only,
we'd still have tribes today. However, I think there was a balance of
force at play: external threat vs. internal mutual dependance. Tribes
formed to increase chances of survival; they stayed together because,
even though others are a pain in the ass, they needed the lecherous
cobler, the violent warrior, the reclusive black smith, etc.

So why did the tribes start falling apart? I think it was
technology... in the form of roads, improved communication, better
farming techniques, whatever. With improved technology, comes increased
independence of individuals, greater power of each person to do things
on their own and be less dependent. Each advance in technology has
further empowered those capable of wielding it.

Technology also tends to abstract things. For instance, the modern
grocery store is an abstraction of a much longer, real-world process:
tilling, planting, harvesting, going to market. These things still
occur, but just not in the consciousness of the modern consumer... the
process has been replaced with a 5 minute zip down the road in an iron
"horse."

Furthermore, it is my opinion that the extended family is an eroded
form of the tribe, and the nuclear family an even further decay. This
has concerned me for some time, especially if you believe that society
tends to represent lower common denominators (I'm thinking of average
level of consciousness). These are related in that if group
consciousness is in the process of evolving, and we have "just now" (in
the past 2000 years) experienced a dissociation of the tribe, then we
are very far from evolving a healthy replacement. It could be anywhere
from several decades to hundreds of years before we see a large-scale
integration of whatever form the next healthy "tribe" will take.

However, like so many other things in nature that evolve slowly over
time, we may actually be right in the middle of it. More digression
first:

We drive on superhighways, lights cover the midnight city like a
spaceship... or an artist's rendition of electrons in a computer chip.
We can send people up into space... where they actually experience
relativistic changes in time (minute, but measurable). This is a world
of science fiction... become fact. In the 30s, 40s, and 50s, wacko
writers were envisioning crazy worlds that didn't do half the stuff we
can do now. One of those wackos was Asimov, whose early "predictions"
for the "far future" were outstripped within 20 years of some of his
short stories.

Our world is not the one our kind has known for millennia. It's
changing at every level. It was a crushing blow to me personally, when
this really hit home: there is no going back, there is only accepting
what we have become and making this work for us... us homo... what?
perhaps homo connectivus.

All this change sneaks up on us, though. Somehow we still function
as though parts of our minds and being were in the mid-1800s... or
800s... or 2800 B.C.E. With our changing world and the increased amount
of abstraction we are forced to deal with, I think we tend to seek safe
harbor in the human past. However, with each generation, artists,
poets, and musicians have been egging us onwards. Trying to tune our
ears, eyes, hearts, and minds to the demands that will be increasingly
made upon us and future generations.

Which brings me closer to the point of this whole thing. At the
beginning, I said how I think I am beginning to understand the open
source movement, a movement that, as one of its mottos, says "Use the
Source, Luke." The only way I was able to start getting it was through
abstraction.

Now mind you, open source was condemned to failure by some of the
biggest business minds in the world. It has baffled people for years:
how can it continue to exist? How does it grow? There is no money, what
is the economic model for this headless behemoth?

The secret of the open source community is that it works like a
brain. There is no ultimate hierarchy in a collection of billions of
neurons (perhaps an emerging pre-consciousness though?). More
importantly, it doesn't matter how many sick, unhealthy cells there are
in it (within obvious limits). If a set of neurons that were a major
conduit for certain signals break down, the signals get rerouted. The
brain is a highly redundant, fault-tolerant network.

The open source community is composed of millions of individuals all
over the globe. I work on projects with team members having multiple
PhDs, members who are university students, high school students and
middle schoolers, musicians, mathematicians, psychologists, blue-colors
workers, etc. The community is becoming all-pervasive. It is one of the
most tolerant, accepting, healthy and helpful communities I have ever
been involved in.

But that didn't make sense, when I thought about it. I *know* some
of those people, and many are not healthy, and some are down-right
mean. After thinking about it, I realized that the best way to consider
it was via an abstraction: Graph Theory from mathematics. Graph Theory
has nothing to do with pictures: many graph theoretic problems focus on
network connections and traversals. Weighted values. Navigating through
a "problem space" with the least expenditure of energy. Companies like
FedEx use graph theory calculations to determine the
fastest/cheapest/most productive delivery routes.

Communities with lots of members form vast networks. People are
nodes on the network and any possible means of interacting with those
people are the connections between the nodes. If someone in the open
source community treats me like shit, I don't get bent our of shape. I
make a mental note, and maybe tell my friends what a dick that person
is (thus giving that node a negative weight with less chance of
traversal). But I then try to find an alternate path to the information
I need or the information I need to share. Over time, I discover
faster, more efficient, path ways of interaction that improve my
quality of life while interacting with other members of the community.

This may seem like a cold distillation, but I am not trying to
capture or limit anything. I'm just trying to find accurate or useful
*approximations* of behavior. In highly trafficked networks, patterns
of usage begin to emerge. In networks of humans and their projects,
mini-networks tend to emerge in areas of higher-than-nornal usage.
These, mini-networks are little communities... or, shall we say,
"tribes". They are dynamic in nature and tend to develop patterns of
internal flux (immigration, emigration) as well as movement/migration
of the community/tribe as a whole. These communities often don't have
set leaders. However, the ones without leaders have very well-defined
goals and mechanisms for determining consensus. The ones with leaders
tend to have a very democratic approach to their leadership. The most
effective communities exchange an extraordinary amount of information:
constant communication and feedback is critical to success.

In networks, there is no "ultimate dependance": people and lines of
communication (nodes and connections) are only used to their abilities.
There is no round-hole-square-pegging -- that just takes too much time.
In public communication, people observe what others say and how they
say it. From that, they make initial guesses at who is good at what.
Initial guess plus trial and error. Sounds like differential
equations... and artificial intelligence ;-) You can have complete
losers, slackers, etc., in any group. They tend to add flavor ;-) They
just don't end up being used as part of the network when stuff has to
get done. They get used for whatever they are good at... like making us
laugh or letting us vent.

Once I saw this, it became clear how this could be a replacement for
the tribe. First a recap. Tribes (maybe) fell apart for the dual
reasons of:

1) increased chances of survival outside the structure of the tribe, and


2) increased convenience of obtaining amenities without having to depend on tribe members.

Since then, we've been wandering about, trying to rediscover the tribe,
trying to reinvent that which has already become extinct for the simple
fact that the causes and conditions no longer exist. Yet we evolved
with the need for the tribe; its lack is therefore part of the modern
human experience of longing.

As our educated population becomes increasingly more so, we are
finding it more and more difficult to be lead by people who represent
the lower common denominators. Perhaps not so much for what they are,
as what they represent: an outmoded form of governance. Intelligent
networks don't have to tell each node what to do. They just need a
purpose and the resources to accomplish their goals.

The need for Survival has evolved into the need for improved Quality
Of Life. In the same way the amenities support survival, Joy supports
Quality of Life. So, tribes may be able to reform under the conditions
that members experience improved quality of life and joy in their daily
activities. Joy is an amazing antidote to daily human annoyances. Joy
can bind groups with a common vision together where nothing else would
have.

The people working on open source projects do so out of love and
joy. We love what we do. Those of us that are truly fortunate get to
work on open source *as* our jobs. The rest, though, do their jobs, and
when off work, they increase their quality of life by working on that
which brings them joy... and they do so in an open, non-hierarchical
system, with excellent communication, working with others to accomplish
things that could not be done by one, lone person.

Star Wars

entertainment

There has been some complaining about the changes George Lucas
has made to the new DVD release of StarWars. And you know what? I don't
care! The spirit of it remains in tact, and unless you're the Comic
Book Guy from the Simpons, the minute changes that have been made
shouldn't matter to you.

I'm watching the DVD of A New Hope right now, and holy shit -- it's
the first time I feel like I'm back in the San Diego theater as a
little kid, watching it for the first time. Awseome!

Thanks George!

We love you!

Modeling Human Behavior

In my recent post about generating personality types with python, I discussed a partial implementation of a Myers-Briggs module, using linear algebra to perform various operations on vectors that represent one's "Jungian" temperament.

I have finished all of the basic code and the module can be used to return the best matches for any given temperament. However, I would like to be able to provide a calculable method for a sliding-scale of compatibility. This could be used in games to simulate interactions between a player and an NPC. In a room full of NPCs, it could be used to establish flows of communication, charisma gradients throughout the room... it could be used to augment 2-dimensional random walks of NPCs (dispersion/Brownian Motion) to preferences for certain gradients established by field lines of compatibility between individuals.

Time to break out the books and dig back into multivariate calculus...

These thoughts arose after I wrote a simple implementation of personality.generator.BasePerson. That made it clear that the interaction of instantiated person objects is not too far away (conceptually; programatically, there will most likely involve a great deal of code and time).

This was the catalyst to start investigating what research has been done on the modeling of human behavior...

The first paper I read was a true gold-mine: "Human Behavior Models for Game-Theoretic Agents: Case of Crowd Tipping." It sent me off on tangents into alluring, siren-like world of http://mathworld.wolfram.com/ and http://scienceworld.wolfram.com/physics/. It also referenced other excellent papers I was able to find online. Below I have listed some of the interesting papers I came across as well as some excerpts.

Human Behavior Models for Game-Theoretic Agents: Case of Crowd Tipping (http://tinyurl.com/7xpqq)

• Their emotional system uses the OCC model (originally formulated by Ortony, Clore and Collins, an emotional appraisal model).
• Their cognitive framework is a modifed MDP (Markov Decision Process) that incorporates a BDI model (Belief-Desire-Intention).
• They have a stress and physiological subsystem that "initially reacts to a set of stimuli that are perceived from and/or experienced in the environment." They "model eight physiological reservoirs or stressors, including: energy, sleep, nutrients, noise and light impacts, and other physical capacities..."
• They discuss a simple motor subsystem for interaction in a microworld.

More Realistic Human Behavior Models for Agents in Virtual Worlds (http://tinyurl.com/5fy6p)

"There are a number of specific agent research issues which arise such as, among others, building and deploying agents that can:

• Acquire and maintain knowledge of the environment, of other agents actions (human or synthetic), and of their own experiences, successes and failures.
• Create tactical plans and carry them out in a believable manner covering both reactive and deliberative behaviors in the presence of other players.
• React appropriately to stress, fatigue, and anxiety and reflect their integrative impact on judgment and performance...
• Construe emotional reactions and mood as stimuli to personal behavior and choice...
• Cope with multi-stage activities (e.g.,campaigns), strategic plans, and survival and decide when to make tactical sacrifices (or not) for the betterment of larger contexts (meta-games)..."

Excellent comment, and one of my biggest problems with the current state of games:

"The coding of the software by game-makers and military wargame modelers is usually created in the simplest manner and in response to a market or project deadline. Historically there has been little capability for deliberative reasoning by these agents and one doesn't encounter theoretically rigorous constructs that can be counted on to perform according to mathematical theory. Due to these constraints, perhaps the most popular construct for game creation and other simulations is the finite state machine (FSM) approach. FSMs offer the rudiments needed to implement Markov chains and MDPs, and to organize agents into iterative meta game-playing participants within a multi-stage, hierarchical network. However, the vast majority of FSM systems implemented to date are programmed from the bottom up, with little agent reasoning and without the concept of a larger theory to validate them against."

"Most of the games and artificial lifeforms out there are artistically and stylistically impressive (very impressive), but not entirely faithful to real human behavior. Usually, the game makers hire a psychologist to verify the behaviors seem plausible, but they rarely get down to actual fine-grained details and rarely implement models based on first principles (e.g., reflexes, reaction times, effects of stress, fatigue and adrenalin effects, judgment rates, etc.)."

And this puts a fine point on game theoretic issues:

"Traditional game theory is mathematically rigorous but overly simplistic. The games that are most commonly studied involve fixed, highly constrained payoff tables; intellectually hobbled opponents; and single layer of play where metagaming and systemic thinking is not allowed."

I could just go on and on about this paper! I highly recommend reading it.

Integrating the OCC Model of Emotions in Embodied Characters (http://tinyurl.com/67xcr)

This paper has a fantastic flow chart of the original OCC model; it's a perfect start for programmers. The author discusses five phases of emotional processing: Classification, Quantification, Interaction, Mapping, Expression.

A Model for Personality and Emotion Simulation (http://tinyurl.com/42qf5)

From the abstract:

"This paper describes a generic model for personality, mood and emotion simulation for conversational virtual humans."

• Discusses mathematics for PE model (Personality and Emotional State) as well as PME model (Personality, Emotional State, and Mood).
• Discuss OCEAN model: Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism.
  

Conversational Agents for Game-Like Virtual Environments (http://tinyurl.com/44yrg)

This paper has a well-developed architecture for its conversational agent software. I will definitely be adapting their insights...

A Reliable Computational Model For BDI Agents (http://tinyurl.com/5bzvj)

From the Introduction:

"The BDI approach is based on the study of mental attitudes... and tackles the problems arising when trying to use traditional planning in situations requiring real-time reactivity. The Beliefs represent the informational state of a BDI agent, that is, what it knows about itself and the world. Desires or goals are its motivational state, that is, what the agent is trying to achieve. A typical BDI agent has a so-called procedural knowledge constituted by a set of Plans which
define sequences of actions and tests (steps) to be performed to achieve a certain goal or react to a specific situation. The Intentions represent the eliberative state of the agent, that is, which plans the agent has chosen for eventual execution."

In the abstract and later in the paper, they mention the "...many problems concerning concurrency control and recoverability..." and this made me think of Twisted (python) as a possible framework to address these issues. Twisted was a result of what is now called Imagination, a framework for building playable virtual realities. I haven't seen them
focus on behavior modeling, but may well have kept such things in mind.

Reading about Markov/BDI agents got me thinking, so I went to the wolfram math site. I started at the Random walk page:

• http://mathworld.wolfram.com/RandomWalk.html
• http://mathworld.wolfram.com/RandomWalk2-Dimensional.html
  

then the Brownian Motion page (possible inspiration for NPC movement in a crowd):

• http://scienceworld.wolfram.com/physics/BrownianMotion.html
  

then to Percolation Theory from fluid dynamics (Greg! we're getting close to your expertise here! possible inspiration for movement of memes, moods, etc., in a crowd of NPCs)

• http://mathworld.wolfram.com/PercolationTheory.html

These two don't mean much to me, as I have no idea how to implement them practically. I'll have to find some source code that explores them:

• http://mathworld.wolfram.com/MarkovSequence.html
• http://mathworld.wolfram.com/MarkovChain.html
  

Technorati Tags: software, games, mathematics, ai, python

Python, Linear Algebra, and Character Development in Games

So, a few things:

1. I was bored last night working on the AOL project (QA phase... blech!)
2. I therefore played some NWN with my new level 15 samurai (12 fighter, 3 weapons master prestige class).
   
3. I wanted to relax after that, so I played some simcity4 and ran my city into serious backruptcy.
4. Then I wanted to challenge my mind :-) So, back to python!

Though not an avid player (too little time and too few friends into old-school RPGing), my favorite part of D&D is character development. I absolutely love it. As a by-product of this, I would also like to see much greater complexity of interaction and response་between players and NPCs. Thus, a while ago I decided to start thinking་about that, and one of the results was the seasonal system for Rook:

Rook Seasons and Planets

and then system of astrology used on Rook:

Rook Astrology

Categorizing that stuff and breaking it down for compatibilities and་temperments is going to be quite a task. However, it will be a fun one.

I think a cool side affect of writing that code is that we will then be་able to translate between Rook signs and Earth signs, and thus we could་actually "use it" or at the very least, refer to it meaningfully. (More་than just I'm an Igou, Karthik is a Kvuang, and Greg is a Rookwuni.
Heh, that's still pretty cool.)

The code I did write:

Last night, I wrote the first code. I decided to start with something་known and easily classifiable, temperaments as well as compatibilities:་Myers-Briggs. The initial delvings are now in the subversion repository.

This stuff is really cool, because I decided to use some tricks from་quantum mechanics math (the stuff we did with eigenvectors/values,་etc.) and use matrices and vectors composed of signed values (+/-1).་Under certain operations, patterns begin to emerge (nullifications and
doublings). I haven't explored this fully yet, but the dot products་with one's personality vecotor and a compatible type seem to result in་'2'. I have to exhaustively show all dot products to really establish a་pattern. But at the very least, this is interesting information and
methodology.

The fact that it may be reducible to operations in linear algebra is་good! Computable = predictable = appropriate system for gaming and་computations.

Technorati Tags: software, games, mathematics, python

The File System as a Database and the Last Post of Summer

software :: python

Heh, that title reads like the title of a Victorian-era
pseduo-science journal article. However, I really do feel that August
31st is the last day of Summer... that come September first, Everything
Changes. But not in a bad way -- Fall is amazing. It's exhilarating in
a funny, erie kind of way.

So, this is the last blog entry of the Summer, 'nough said.

As for the rest of the title, I've been discussing this topic with
friends of late. Basically, how can we access the file system in
python, treat it like a database, search on it like a database, and
write code for it in such a way that when we are ready, we can
migratate to a database with no code changes (only config or module
import changes)?

Well, we've debated the issue(s) back and forth. I even asked
Phillip Eby about it on the PEAK mail list (we will be using PEAK for
this project... once we learn it!). But I think we've all been trying
to make the problem and solution too general. A very simple and rigid
API would work for us right now. It means less time spent on R&D,
and since this is for a paying customer, that means more money for us
in the long run.... as long as what we implement leaves enough
flexibility for future change.

So, without further ado, an adaptation from a post to the mail list today:

If we have a file on the filesystem, then the full path + the file
name uniquely identifies that file. In my limited knowledge of OODBs,
this is pretty standard (path-to-object = UID). Then there's the file
itself, which contains some data. Additionally, however, is the path:
it contains data that is just as important as the data inside the file.

UID: full path + filename

Data: stored in file at /fullpath/filename

Data: stored in path and filename

How do we think about this problem? If this were a table, we might be looking at a schema like this:

Table

-----

id: full_path + filename

blob: rrd file/text file/ini file/xml file with DTD/whatever

additional field 1:

additional field 2:

additional field 3:

...

additional field n:

I'm not proposing an OR mapping here: that's complicated shit. Way
beyond me. Some of the biggest brains in the software development world
are working on ways to do that which make sense and work right.

I'm just talking about doing something simple and straight-forward.
Something that's easy to configure and easy to migrate from a
filesystem to an RDBMS.

This shouldn't be as hard as I was thinking originally. The only
issue is that for every implementation, there will need to be a
configuration. This is because, by their nature, database tables and
fields defined therein are fixed; directory structures aren't/don't
have to be. A configuration would "lock" a directory structure... you'd
have to have an API (or something) that defined what each level of the
directory structure indicated, as these would have to be mappable to
fields defined in a table (for migration to a SQL framework).

Additionally, if you wanted to move the data stored in you blob
field out of its own little format into SQL, you'd have to define an
additional config/API for mapping its data to more fields in the
table...

So what you'd really have here is a directory structure schema and
then a file storage schema. Using the two together, you'd get what I
originally asked Phillip Eby about...

PEAK already has 'peak.storage.files' which lets you interact with
text files transactionally. We could do something like this for other
types of data-containing files at the end of whatever directory tree.
The combination of this with an implementation of a queryable filsystem
data interface should leave us with a fairly powerful tool for many of
our projects.

Questions to ask:

* What constitutes a database? (root dir and below?)

* What constitutes a table? (all directories at the first level, inside the root dir?)

* What constitutes a row? (every branch from root? this means all paths
from the root dir have to have the same number of dirs, subdirs, etc.)

* Can there be no file at the end of a path?

* Can there be empty files at the end of a path?

* Can there be multiple files at the end of a path?

* Can there be files in intermediate directories? (dirs that aren't the end of a path; good place for metadata?)

Freemind

software

It's been a couple years since I checked out freemind (a free "mind mapping" software for oganizing text/notes in loose relationships), and I just downloaded the latest version for Mac OS X. I must say, I'm impressed.

The software is very stable, provides *exactly* what I need, and is free :-) I highly recommend interested parties check it out:

http://freemind.sourceforge.net/.

Now, if they could only develop a live wiki version of the same thing ;-) Well, it's java, so I imagine it wouldn't be too hard to turn this into a servelet... of course, I'm not a java programmer and I have no right to make that statement!

Python port, anyone?

PyHTMLWidgets

python :: html :: web

Well, I've been working with some ancient code base (1995) to produce HTML for PyHTMLWidgets.... and thanks to a friend of mine (Paul Taney), I have realized that the proper tool has been in front of me the whole time: HTMLgen.

Once I am finished with the looming projects (two weeks, I hope!), I
will update the code base for PyHTMLWidgets and provide a download.
It's not rocket science... in fact, it's not science at all: it's
convenience. In the world of awesome templating systems and complete
tool sets for projects, there are customers out there that want things
in a particular way... sometimes, this precludes the use of some of the
better tools sets out there, and the need arises for something
completely independant and without dependancies on a larger system.

Thus, the ever-present need for separate tools with the ability to loosely bind them together...

Update:

A benchmark of different xml tools in python led me to the ElementTree web site and I am excited to see what this tool can do. There are a couple possibilities for PyHTMLWidgets:

* Use HTMLgen

* Use ElementTree instead


* Rewrite HTMLgen to use ElementTree

I'll have to spend more time thinking about potential features (mmm,
candy... mmm, scope creep) as well as running some benchmarks, but this
could actually be an interesting project...

Update:

Well, I had a chance to look at the source for HTMLgen, and it's
really not what I need. On the other hand, I've been playing with
ElementTree and that is a fantastic tool set. I am very pleased with
it's potential, and I think that it will provide *very* significant
increases in speed over what we are currently using.

My Dream Home Robot

technology

Two years ago, I had the desire to build a small flying "robot"
that I could control remotely and connect to an X10 home automation
setup. Turns out that I probably couldn't have done it at the size I
wanted, seeing how Seiko Epson has just produced it and it has the
smallest, lightest gyro sensor in the world. (See http://news.bbc.co.uk/1/hi/technology/3579232.stm.)
Hell, I probably couldn't have done it no matter what, seeing how I've
never built a robot in my life! To be useful for what I wanted, it had
to be wirelessly remoteable and capable of sending images (either
static snap shots or live video). This little critter meets such
requirements.

This is especially cool to me, far more cool than wheeled robots. It
receives messages via bluetooth, and with its micro camera, you could
remote it through your house, checking things like ovens, toasters,
irons, etc. It can operate for 3 hours before running out of juice in
its on-board batteries (rechargeable?). I would imagine that once you
had a device like this plugged up to your home automation system, new
ideas for useage would crop up by the hundreds...

! Not Blogging

technology

Yup, that's a double negative. I have no time to do it, so I
can't... and I'm not... not... blogging. In fact, I'm supposed to be
writing code for AOL...

Anyway, I think a friend of mine got mentioned in Clay Shirky's latest article
about Situated Software (of course, when I saw that title, all I could
think about was ATHF's Meatwad singing MC Peepants'/MC Chris's lyrics
"...get re-situated..."; yeah, I know I'm fucked ;-) ). Paul Barry is
my friend's name; he's working for NYU and taking classes. Really smart
dude. He built an LMS from the ground up for NYU. Unless my memory and
email addresses have failed me, Clay typo'ed Paul's name. Or it might
not be the same Paul... still waiting verification from Paul...

Insanely Busy...

technology

I have absolutely ZERO time to blog these days. Python
programming is keeping me busy 120% of the day. The other 3.2 hours of
my 32 hour days (yeah, that's my circadian rythm... or however the hell
you spell it) are spent learning programming concepts and theory I am
only becoming aware of through amazing and ungodly uberhackers in the
open source community.

Business is good, but I would love to have more time to ruminate on
the things the sundry collection of neurons in my body find amusing.
These days, it seems to be matters sociological and economic that
really do it for me. Of particular interest: need to comment on

* one of Tim O'Reilly's excellent essays

* The Dyson's at OSCon

* The social and catalytic impact conferences can have


* architect Christopher Alexander's three-volume set

* Conversation with Alan Thompson about software glue and business trends

We shall see...

Until then, I leave the place-holder and personal reminder ;-)

Everything as a Distributed Resource

In my blog entry Dinosaurs and Mamals, I mentioned small application servers that were capable of being highly distrubuted. In conversations on this topic with friends, we discussed various scenarios where such things could have an impact on our daily lives. For instance, I'm sure that many people have heard about the potato powered web server joke and then the real thing that it inspired. This demonstrates the possibility of low-power servers; conjoin that with wareable computing, and mammal application servers, and you have a whole new world...

There was this great game I used to play called "Alpha Centuari" -- the best part was choosing the research your culture would engage in. I find a similar "rush" from the communities one can join in the Orkut online friends network. I know lots of people that love that kind of thing, too... possibly pointing to something inherent in human nature about social groups, hobbies, aquisition, or just bright, shiney objects.

Even now, people have taken this one step closer to "reality": from simple association, to actual contribution. We may not all be able to get PhDs in astrophysics or molecular biology and do research that actively contributes to the human body of knowledge, but we can all run client applications on our workstations that borrow idle CPU cycles to work on problems in which we are interested.

So what happens when web servers can fit in the colar of a shirt, when shoes pumps generate electricity as you walk, when fibres woven throughout your clothes act as wireless antenae? What happens when processing power is literally everywhere and highly moble? Universities, companies, governments, special interest groups, neighborhoods, etc., can all lobby you for your computing resources. They don't have to ask you for your time or your money; you don't have to change the way you do things or when you do them. All you have to do is choose to associate with a group and then give them permission to use your cycles. They get to use CPUs that they don't have to power, insure, air condition, troubleshoot, network, maintain, etc. There is a potentially HUGE amount of money that can be saved for each organization.

But with increased mobility and with large numbers of projects to which you can contribute your unused cycles, comes such ubiquitous evils as marketing, advertising, lobbying, etc. If a company can influence you to change your mind and go with them instead of their competitor, then they can do it with thousands of other potential contributors as well. As quickly as you see a "give-us-your-spare-cycles" advertisement or have a conversation with a friend, you can change your mind and give to someone else: the CPU isn't waiting at home, it's there with you.

The consumer becomes a commodity broker; organizations, the consumer. Decisions are immeidate;
impact on a company's R &D is instantaneous. Profits are now. All of a sudden, the entire technology-enabled population is a rich philanthropist and every organization out there is hoping to get your "funding," hoping you'll let them add one more CPU to the thousands already volunteered. This could give birth to a whole new "processing"economy...

Evolving Networks

Note: This is from an email I had with Itamar Shtull-Trauring, who was kind enough to let me blather for a bit...

There was a very interesting article in this year's April issue of Scientific American. It basically discussed a non-neural network in the brain, its own complete entity, but one that is connected to the neural network we know and love. It's a slower network that seems to provide feedback and additional processing/learning power. It uses mechanics and chemistry to communicate along its own network and respond/react to the neural one.It's a great article -- I recommend it (I have severely abused it with that synopsis above).

It got me to thinking... first: what if the original pre-brains were cells that communicated electrically and chemically, but with no neural network capabilities? What if the original multicellular ancestor of the brain was just a simple highly-connected, bi-directional network? What if the chemical and neurological networks grew in tandem and developed together, with the chemical network acting as a "meta-network," providing feedback? And, to really stretch things, what if this feedback is what caused the electrical network to develop into a neural network?Now I'm sure you see where I am going with this and how fruity it is...

Like primitive primordial cellular organisms, our society also has a primitive electrical network: the Internet. How can we develop a functional analog to the biological system (the biological system being proven to work and being insanely efficient... for a neural network...)?

Okay, so that's the visionary's question; no need for reasons or justification -- can we do it? Then let's make it! But why would we want to? Do we really want to turn the insanity of the Internet into something resembling a neural network? Perhaps, but that's not the goal; the ultimate goals might be something like these:

• We know the Internet is slow; how can we make it fast? efficient? (fastest paths, memory/learning, ...)
• We know that the Internet is filled with hostiles (spam, viruses, worms, etc.) - how can we safeguard this?
• There will always be something new that the Internet hasn't seen before - how can we integrate anticipation?

Could we create a virtual separate network inside the internet? Could it use a completely different protocol than anything else in existence? Could it be very simple, very specialized, and very secure? Could it provide protection services? Or perhaps act as some kind of feedback mechanism for Internet devices? A subscription service with intelligence?
This leads to other questions I'm sure people a lot smarter than me have already examined: can we use nature as an analog for developing network communications into something more mature?

• How do complicated organic networks regulate themselves?
• How do they protect themselves from violence?
• How do they self-organize? adapt?
• Are there ways in which we could model these behaviors with special designs (protocols, applications, services)?

So that's it, nothing with much technical content... just some ideas floating around...

Political Views

politics

This blog entry is posted without the premission of the person
with whom I had the conversation, so I do not include the name. It was
a comment made in an email discussion we were having about the
political test mentioned in the blog entry "What is a Libertarian."
However, their point of view is very compelling and educated through
the experiece that I think is most important:

Local.

It is my firm belief that if you cannot implement something locally
in a real and meaningful way, it either can't be done, has no meaning
to "real" people, or is a concoction of smoke and mirrors. Likewise, if
something globally is implemented in a local manner, I don't believe
one can dismiss a problem about its local implementation with a
statement like "oh, yeah... those people in that place act that way,
but that's not the *approved* way; we encourage our members to do this
instead."

That has no meaning. It boils down to experiment vs. theory,
practice vs. preaching. Say what you mean, and do what you say. Here's
what my friend had to say:

Over the years, I've been drawn to more radical ideas, particularly
libertarianism, "lite" socialism, and Ivan Illich's writings on
anarchy. But I find it's really enlightening to actually spend time
with the *people* who follow the ideas...

My meetings with the libertarian group in [some US city], for
example, really left me cold. I shared their respect for intelligence,
but I wasn't comfortable with their obvious willingness to let dumb
people just take it in the teeth from cradle to grave.

So I veered back toward more planned and buffered visions for
society (socialism, technocracy) for awhile. But that stuff is hard to
swallow, unless you either believe that people will rise to meet
society's expectations (don't count on it), or you're willing to accept
a lot of people cheating the system (welfare cheats, etc).

At the moment, I'm sort of resigned to the latter, so I'm leaning
toward a mild nanny state with reasonable safeguards against
parasitism. My guts are telling me that our alternatives are far more
risky--a tough, win/lose society that will come apart at the seams once
the majority comes to the conclusion that they'll never get the brass
ring through competition.

But I could be wrong. Check out this column by David Brooks.

What's a Libertarian?

politics

That's not a rhetorical question, and nor is it sarcastic. I
honestly didn't know until a few days ago when I took the time to look
it up.

I would not describe myself as a political person, and so I have
always ignored much of politics and political definitions. But after
recently taking a poltical mini-quiz (http://www.theadvocates.org/quiz.html),
l became curious as to how the political positions were defined. After
some reading, it turned out I really am a left-libertarian, just like
the quiz said.

I have beeen passionate about freedom and liberty my whole life. As
I entered high school, learned about our current government in horrible
detail, learned about the genius of the founding fathers, it became
very clear to me I wanted as little of the Federal Government
interferring with my life as possible. Years of watching one Government
scandal after another just reinforced this view. With the birth of the
internet and Open Source Software, these strong opinions evolved
naturally into passions. They have become pillars in everything from
how I teach programming and software development to how I help run a
business.

I started poking around tonight (this morning) and found some interesting things I'd like to share:

From the libertarian FAQ (http://www.theadvocates.org/library/libertarian-faq.html):

"Libertarians believe that, on every issue, you have the right to
decide for yourself what's best for you and to act on that belief so
long as you respect the right of other people to do the same and deal
with them peacefully and honestly.

"Self-government is the combination of personal responsibility and
tolerance. Responsibility means you govern yourself. Tolerance means
you don't force your values on peaceful, honest people."

"Actually, true conservatives tend to be libertarian on economic
issues, and true liberals tend to be libertarian on social issues. The
consistency is to the extent to which one believes in individual
liberty..." --Sharon Harris

Here are some good quotes from http://theadvocates.org/celebrities.html:

I'll put my favorite quote first :-)

"How does one get involved [in libertarianism]? Think about freedom,
and act on your thoughts. Spend your dollars wisely. Oppose the
expansion of state power. Promote 'bottom-up' solutions to public
problems, solutions that empower individuals rather than demanding
intervention by force of government. Give to private charity. Join a
libertarian organization; the Libertarian Party, or the Advocates for
Self-Government, or the Reasonn Foundation. Start your own
business; create wealth and celebrate others who create wealth. Support
voluntary cooperation." --Eric Raymond

"I'm a libertarian. I think a lot of people are libertarians and are afraid to admit it -- or don't know." --Kurt Russel

"The less [government] the better. As far as your personal goals are
and what you actually want to do with your life, it should never have
to do with the government. You should never depend on the government
for your retirement, your financial security, for anything. If you do,
you're screwed... That's all the government should be: Army, Air Force,
Navy, Marines." -- Drew Carey

"Libertarian... Everyone leaves everyone else alone." -- Clint Eastwood, when asked how he saw himself politically

"Milton Friedman is one of this century's most distinguished and
influential economists... His tremendously influential 1979 best-seller
Free To Choose (co-authored with his wife Rose) is a clear and lively
examination of economics and politics. It shows how our freedom has
been eroded and our prosperity undermined through an explosion of laws,
regulations, agencies and government spending, and makes a profound and
convincing argument for repealing most government interference in the
economy... Free to Choose was presented to millions of Americans as a
10-episode series shown on (of all things) PBS."

I wonder if I can still get a hold of that? Sounds like an awesome series.

"Fortunately, we are waking up. We are again recognizing the dangers
of an overgoverned society, coming to understand that good objectives
can be perverted by bad means, that reliance on the freedom of people
to control their own lives in accordance with their own values is the
surest way to achieve the full potential of a great society." --Milton
Friedman

"We are entering a new era of more representative rule-making. As
just one example, internet technology is changing the way government
works..." --Ron Paul

The Ron Paul FREEDOM PRINCIPLES:

* Rights belong to individuals, not groups.

* Property should be owned by people, not government.

* All voluntary associations should be permissionable -- economic and social.

* The government's monetary role is to maintain the integrity of the monetary unit, not participate in fraud.

* Government exists to protect liberty, not to redistribute wealth or to grant special privileges.

* The lives and actions of people are their own responsibility, not the government's.

Related Links:

http://www.americanlibertyfoundation.org/

http://www.mises.org/

http://www.defendyourprivacy.com/

http://www.fee.org/

http://www.cfif.org/

http://www.eff.org/

http://www.ti.org/

Games for the Future

interactive fiction :: games :: technology

What is a game? Why do people play them? What are people's past
experiences with games? What do they want from them in the future?

I'm sure the gaming industry has poured tons of money and research
into answering these very questions. I have no idea what the public may
want from games; but I can share my own views on the matter.

It is my personal opinion that games exercise that desperate human
element that still longs to be running across the plains with it's
closet tribe members, providing food from the hunt for the community,
being a vital part of the never ending cycle: life and death and life.
We want to touch magic that is "real"; we want an immersive environment
that takes us away from the every-day find-the-cheese-in-the-maze
doldrums.

Playing games let us touch this in ways that are either not possible
or at best very difficult to attain practically. We've played things
like "Pong" or "Donkey Kong"... then there was "Doom" and "Quake" --
all of these games changed our lives in one little way or another. Then
there were more recent knock-outs like "System Shock 2", "Deus Ex", and
"Morrowind." These were much more involved and took the gamer deeper
into another world.

However, the games that I return to over and over again, what little
time I have left to game, are the ones with great story lines: when on
the train with my laptop and no connection to the net, I like to fire
up Zork or Nethack. When I do have an internet connection, I like to
play MUDs and MUSHes. I have even built MUSHes of my own. I even used
one for business -- a place where partners and co-workers could chat
online in a unique, friendly environment. Community-based and living,
these strike me for two reasons:

1) I am a voracious reader. I love words and good stories. I love my imagination;

2) I'm a sucker for that old feeling of sitting around with a bunch of friends playing pen-and-paper RPGs.

Along these lines, I've been working with some friends over the past
year building a universe (history, story-line, etc.) that encompases a
huge time frame with lots of interesting possibilities for game play.
We strive to capture everything from science and politics to mysticism
and space travel; linguistics and psycology to economics and
anthropology. Writing the history and developing the languages has been
the best part of this for me. Playing the game will simply be icing on
the cake.

Though some of us have dreams of making 3D environments that echo
this "reality", my aspirations are far simpler: I want to write fun
code, and I want to play with others in my imagination and I want them
to play with me in theirs. A good MUD or MUSH allows for this kind of
interaction with others in a story environment that brings what I like
about gaming together. When it comes down to it, graphics are great but
they are not the reason I play. It's the story that drives me. It's the
people and creative setting that make it fun. The better the story and
the more immersive the environment, the closer I come to capturing that
feeling of running across the plains with my tribe mates ;-)

Dinosaurs and Mammals

Large software corporations like Microsoft and Oracle in addition to groups of software developers like elements in the Open Source community are giving the world new versions of the same thing: monolithic software for a world that no longer needs or wants such products.

Take for example Microsoft Exchange Server: this is a heavy-duty, if often ridiculed, system of software. This software was meant to handle hundreds to thousands of users and their information. What about the company that wants a Microsoft solution but only has 50 users? They have to pay for Exchange. Yeah, you can get a great discount, etc., but I'm not talking about price -- I'm talking about need. And designing software for need.

Oracle is more guilty of this sin. Let's say I need to have my data highly distributed on remote machines, but that I don't have tons of data and I don't need the horsepower Oracle can provide. Yet, if I use their product or any other large RDBMS like PostgreSQL, SQL Server, or even MySQL, I am using the wrong tool for the job.

Another example: Apache is the world's most popular Web server, but let me commit heresy and ask "Why?" How many people need it's speed? Or its capacity to handle millions of requests? Think of the hundreds of thousands of web sites out there that are visited once a day or once a month. Using something like apache for these is engineering overkill.

Microsoft and Oracle as well as organizations like the Apache Foundation, MySQL, and PostgreSQL are producing software dinosaurs. The have lived, they have thrived, they have helped define our information universe. That universe is changing. With the introduction of programming languages and development processes that foster rapid development, for the first time in the history of modern technology, we are in a place to design enterprise software around the needs of the end user (individulas and organizations), not around the needs of licensing structures and coporate philosophies. Imagine that the motivation behind software development wasn't market domination, but rather market diversity. The right tool for the right job.

I haven't counted the lines of code in the apache web server, but let's just say it's 100,000 lines. What if I only need to serve a couple pages per second and a max of 100 concurrent sessions? What if I could use Web server software that could provide what I need in only 200 lines of code? 100? 6? Why should I use Apache for projects that don't require Apache's strengths? The same thing for my email server and calendar application. What if I could do the same thing for all my networked applications? What if in addition to being small, this software was highly distributable as well? What if I could build office applications perfectly suited for a firm of 40 employees, and they didn't have to pay for licensing of the cliched software products? In addition, as the firm grew, this software could support 100, 1,000, or 100,000 users just as easily simply by adding the required light-weight components.

Software like this is easier to setup and maintain. It's easier to get involved with and understand. It's easier to modify, add to, customize, and enhance. It's easier to provide your firm with what they need.
Enter the new software and application engineering model: the mammal. This is where we are heading. Imgine application servers so light-weight they could run on a 9-volt battery for a week. Imagine them embedded in hundreds of devices all over the country or even the globe... so cheap and distributed, it didn't matter if one went down or even if twenty of them went down. They are quick, nimble mammals, not lumbering dinosaurs.

There are already software developers working on projects with views and goals along these lines. Watch for them: they are small and furry. They move quickly and quietly. Pay attention to them; they will one day rule the world.

The Printing Press, Literacy, Education, and Unalienable Rights

community :: technology :: education

There seems to be some very interesting things afoot these
days... subtle revolutions in the works... large-scale evolution taking
place. We are seeing this in all areas of society: technology, law,
government, education, religion and spirituality, and most importantly,
in the community. What follows are some outlines of things my friends
and I have been discussing.

Version 1.0

* Clergy hold monopoly on the written word

* Guttenburg's Press

* Literacy

* Higher Education

Version 2.0

* Corporations and their teams of programmers hold monopoly on software

* Open Source Software movement, improved high-level languages that everyone can learn

* Computer literacy grows

* Information for everyone; power of software generation for everyone

More thoughts on this later...

A Revolutionary's Cookbook

community

So you want to start a revolution? Well, hey, we all want to
change the world. Who are the world changers? Some of them are like
great machines that chew up everything in front of them. Others are
charismatic, fervor inspiring fanatics. There are all kinds. My
favorites, though, are the ones I model my behavior after: parents.

Take it from the Buddha: life sucks. Then we die. Then we get reborn
and have to learn all over again that life sucks. People hate this; it
makes them crazy and irritable. So, when people come in contact with
others who don't seem so upset about the fact that life sucks, it's
attractive.

We're all screwed up; it's the nature of humanity (biochemical
analog programming). We learn crap at an early age that we spend the
rest of our lives dealing with. But when we meet people who embrace
this craziness, things seem okay. We breath a little lighter. Hell, we
might even smile ;-)

Sometimes these sunny folk are even a little stable; sometimes
they're not nomadic hippies; sometimes they seem kinda normal. When
these people stick around in one area, they start to make friends --
people want to be near them. Their positive outlook is not some
bullshit facade, but rather a living reality; one that is upheld as
genuine by proof of these people's actions.

Watching them makes me realize that they treat every part of the
world as if it was their precious child. Every interaction with
existance and all its suffering is both a learning experience and a
teaching experience. I like being around this and I want to be more
like them. These people are community seedlings. They are like the
initial structure and catalyst around which crystals form. They are the
revolution.