lfest: A Routing Party for LFE Web Apps

The Clojure community (in particular, James Reeves) has produced one of the best APIs I've seen for creating RESTful services, or any application using path dispatching (routing), really: Compojure.

I've been really missing this in LFE. Despite the fact that creating RESTful apps in LFE with YAWS is so simple it doesn't even need a framework, the routes can be a bit difficult to read, due to some of the destructuring that's done (e.g., URL paths).

Take the following route declaration from the yaws-rest-starter project:

Note that this example delegates additional routing to other functions, since all the routing in one function was pretty difficult to read. This, however, detracts from the overall readability in a different way: there's not one place to look to see what functions map to the complete URL-space for this service.

I wanted to be able to do something like what you can do in Clojure:

LFE is a Lisp with macros, so why not? Also, nox or mheise (I forget who) on the #erlang-lisp channel had previously noted all the lfe* projects (and these too), and suggested that someone create the inevitable "lfest" repo on Github.

Enter lfest. After a weekend of hacking and debugging some tiny macros, surprisingly little code now supports routes definitions like the following in LFE+YAWS web apps:

If you're curious to see what this actually expands to before getting compiled in a .beam file, you can view it here.

Bonus for LFE webdevs: this project also has a module of HTTP status codes, for easy re-use in other projects.

The recent work I've been doing with macros has really helped shape the section I've been planning to write in the LFE User Guide. I haven't wanted it to be yet another dry description of macros in a Lisp-2. Instead, my hope is to help jump-start people into how to think about macros, how to start writing them immediately (without years of study first), and how to debug them.

The trick, as with all complicated subjects, is how to remove the barrier to entry and get folks that necessary hands-on experience without dumbing-down the material. It's almost ready...

Also, it's Bay to Breakers today, and Ocean Beach is bumpin. The synchronicity is just eery. May you party just as hard with routes in LFE.

Rendering ReST with Klein and Twisted Templates

In a previous life, I spent about 25 hours a day worrying about content management systems written in Python. As a result of the battle scars built up during those days, I have developed a pretty strong aversion for a heavy CMS when a simple approach will do. Especially if the users are technologically proficient.

At MindPool, we're building out our infrastructure right now using Twisted so that we can take advantage of the super amazing numbers of protocols that Twisted supports to provide some pretty unique combined services for our customers (among the many other types of services we are providing). For our website, we're using the Bottle/Flask-inspired Klein as our micro web framework, and this uses the most excellent Twisted templating. (We are, of course, also using Twitter Bootstrap.)

Here's the rub, though: we want to manage our content in the git repo for our site with ReStructured Text files, and there's no way to tell the template rendering machinery (the flattener code) to allow raw HTML into the mix. As such, my first attempt at ReST support was rendering HTML tags all over the user-facing content.

This ended up being a blessing in disguise, though, as I was fairly unhappy with the third-party dependencies that had popped up as a result of getting this to work. After a couple false starts, I was hot on the trail of a good solution: convert the docutils-generated HTML (from the ReST source files) to Twisted Stan tags, and push those into the renderers.

This ended up working like a champ. Here's what I did:

1. Created a couple of utility functions for easily getting HTML from ReST and Stan from ReST.
2. Wrote a custom IRenderable for ReST content (not strictly necessary, but organizationally useful, given what else will be added in the future).
3. Updated the base class for "content" page templates to dispatch, depending upon content type.

Afterwards I was rewarded with some nicely rendered content on the staging MindPool site :-) (once the content text has been completed, we'll be pushing it live).

Kudos to David Reid for Klein and (as usual) to the Twisted community for one hell of a framework that is the engine of my internet.

txSpore: Twisted Spore

I just had a delightful weekend of coding :-) I spent the past two days porting the Spore Python API to Twisted. You can now incorporate Spore data (from static XML as well as REST requests) into your non-blocking Python applications/games!

This was a pretty easy task, really. The API just makes HTTP requests with twisted.web.client.getPage. There was a little bit of work involved in creating object models for the XML, and some head-scratching for the error-catching deferToThread unit test I tried to write (it's still buggy... need to figure that one out). Everything else was pretty much cake.

In fact, it was so much fun to kick back and write some playful code that I might overhaul the sync Python code as well and incorporate both into txSpore.

Do be aware, however, that the code still has some big improvements coming. The first thing I want to hit is actually create a client object. Right now, the client module contains a series of functions (since state's not currently needed). However, I want to start doing some basic object caching in order to limit the number of requests made to spore.com and increase the response time. That's the big item for 0.0.2. Update: 0.0.2 is now released!

Next I'd like to create some more demo apps that show off the API usage better. Right now, there's one demo (a .tac file). All it does is ask for a user name, renders a user page, and then links to a user "Spore assets" page (that's the thumbnail image above).

One thing that might be fun to do is write a script that checks for the latest achievements and publishes them to various microblog/status sites with the Twisted PyngFM client :-)

There's a project page up on Launchpad for txSpore, and I've posted a notice and some updates to the Spore developer forums. It's also been published on PyPI.

Enjoy!

Twisted JSON-RPC on Google Code/Projects

I've been getting lots of emails from folks asking for the Twisted JSON-RPC code that I had put up a while ago. They weren't able to find it due to my having to bring down the trac instances I was running. Good news: I've moved the repository up to Google:

http://code.google.com/p/twisted-jsonrpc/

The front page has a link to the wiki which in turn has all the usage info. You can get recent eggs here:

http://twisted-jsonrpc.googlecode.com/svn/releases/

If you want to contribute to this code base and bring it up to date, email me.

Enjoy!

Update: Primary development has moved to Launchpad. Documentation is maintained on the Ubuntu.com wiki page here.

Nevow: URLdb

Well, I am really, really impressed with Nevow. It really grows on you as you code. I've been posting little tidbits that I learn on the Nevow wiki, the kinds of things that beginners should have easy access to.

Today, I dug really deep into the nevow libs and followed the path of form posts (configurables, validation, etc.). Very interesting stuff. Not only do I like the way Nevow behaves and handles for developers, I like its guts, too. That's actually fairly uncommon when it comes to web app frameworks.

Anyway, today was very successful: my TinyURL knockoff (URLdb) is "fully" functional. It has a form interface for posting/creating shorter URLs as well as a URL you can post to via javascript, etc. The reason that I put "fully" in quotes, is that there is no persistent backend storage yet: it's all in memory. This was for simplifying the development/learning process only, and I have already started on writing the Atop classes.

I look forward to using Nevow in the future -- immediately, on the PyKon project; but in general, for any web app that doesn't require the pre-built functionality of a full-blown CMS.

Dinosaurs and Mammals

Large software corporations like Microsoft and Oracle in addition to groups of software developers like elements in the Open Source community are giving the world new versions of the same thing: monolithic software for a world that no longer needs or wants such products.

Take for example Microsoft Exchange Server: this is a heavy-duty, if often ridiculed, system of software. This software was meant to handle hundreds to thousands of users and their information. What about the company that wants a Microsoft solution but only has 50 users? They have to pay for Exchange. Yeah, you can get a great discount, etc., but I'm not talking about price -- I'm talking about need. And designing software for need.

Oracle is more guilty of this sin. Let's say I need to have my data highly distributed on remote machines, but that I don't have tons of data and I don't need the horsepower Oracle can provide. Yet, if I use their product or any other large RDBMS like PostgreSQL, SQL Server, or even MySQL, I am using the wrong tool for the job.

Another example: Apache is the world's most popular Web server, but let me commit heresy and ask "Why?" How many people need it's speed? Or its capacity to handle millions of requests? Think of the hundreds of thousands of web sites out there that are visited once a day or once a month. Using something like apache for these is engineering overkill.

Microsoft and Oracle as well as organizations like the Apache Foundation, MySQL, and PostgreSQL are producing software dinosaurs. The have lived, they have thrived, they have helped define our information universe. That universe is changing. With the introduction of programming languages and development processes that foster rapid development, for the first time in the history of modern technology, we are in a place to design enterprise software around the needs of the end user (individulas and organizations), not around the needs of licensing structures and coporate philosophies. Imagine that the motivation behind software development wasn't market domination, but rather market diversity. The right tool for the right job.

I haven't counted the lines of code in the apache web server, but let's just say it's 100,000 lines. What if I only need to serve a couple pages per second and a max of 100 concurrent sessions? What if I could use Web server software that could provide what I need in only 200 lines of code? 100? 6? Why should I use Apache for projects that don't require Apache's strengths? The same thing for my email server and calendar application. What if I could do the same thing for all my networked applications? What if in addition to being small, this software was highly distributable as well? What if I could build office applications perfectly suited for a firm of 40 employees, and they didn't have to pay for licensing of the cliched software products? In addition, as the firm grew, this software could support 100, 1,000, or 100,000 users just as easily simply by adding the required light-weight components.

Software like this is easier to setup and maintain. It's easier to get involved with and understand. It's easier to modify, add to, customize, and enhance. It's easier to provide your firm with what they need.
Enter the new software and application engineering model: the mammal. This is where we are heading. Imgine application servers so light-weight they could run on a 9-volt battery for a week. Imagine them embedded in hundreds of devices all over the country or even the globe... so cheap and distributed, it didn't matter if one went down or even if twenty of them went down. They are quick, nimble mammals, not lumbering dinosaurs.

There are already software developers working on projects with views and goals along these lines. Watch for them: they are small and furry. They move quickly and quietly. Pay attention to them; they will one day rule the world.

In Praise of Evolvable Systems

Shirky: In Praise of Evolvable Systems first appeared in the ACM's net_worker, 1996, and there are some particularly thought-provoking comments that Clay Shirky makes:

...the Web's poorer engineering qualities seem not merely desirable but essential.

The effects of this ease of implementation, as opposed to the difficulties of launching a Gopher index or making a CD-ROM, are twofold: a huge increase in truly pointless and stupid content soaking up bandwidth; and, as a direct result, a rush to find ways to compete with all the noise through the creation of interesting work. The quality of the best work on the Web today has not happened in spite of the mass of garbage out there, but in part because of it.

Finally, Orgel's Rule, named for the evolutionary biologist Leslie Orgel -- "Evolution is cleverer than you are". As with the list of the Web's obvious deficiencies above, it is easy to point out what is wrong with any evolvable system at any point in its life... However, the ability to understand what is missing at any given moment does not mean that one person or a small central group can design a better system in the long haul.

Centrally designed protocols start out strong and improve logarithmically. Evolvable protocols start out weak and improve exponentially. It's dinosaurs vs. mammals, and the mammals win every time. The Web is not the perfect hypertext protocol, just the best one that's also currently practical. Infrastructure built on evolvable protocols will always be partially incomplete, partially wrong and ultimately better designed than its competition.

As with the early Web, the 'glue' protocol subsumes the other protocols and produces a kind of weak integration, but weak integration is better than no integration at all, and it is far easier to move from weak integration to strong integration than from none to some. In 5 years, DVD, HDTV, voice-over-IP, and Java will all be able to interoperate because of some new set of protocols which, like HTTP and HTML, is going to be weak, relatively uncoordinated, imperfectly implemented and, in the end, invincible.