lfest: A Routing Party for LFE Web Apps

The Clojure community (in particular, James Reeves) has produced one of the best APIs I've seen for creating RESTful services, or any application using path dispatching (routing), really: Compojure.

I've been really missing this in LFE. Despite the fact that creating RESTful apps in LFE with YAWS is so simple it doesn't even need a framework, the routes can be a bit difficult to read, due to some of the destructuring that's done (e.g., URL paths).

Take the following route declaration from the yaws-rest-starter project:

Note that this example delegates additional routing to other functions, since all the routing in one function was pretty difficult to read. This, however, detracts from the overall readability in a different way: there's not one place to look to see what functions map to the complete URL-space for this service.

I wanted to be able to do something like what you can do in Clojure:

LFE is a Lisp with macros, so why not? Also, nox or mheise (I forget who) on the #erlang-lisp channel had previously noted all the lfe* projects (and these too), and suggested that someone create the inevitable "lfest" repo on Github.

Enter lfest. After a weekend of hacking and debugging some tiny macros, surprisingly little code now supports routes definitions like the following in LFE+YAWS web apps:

If you're curious to see what this actually expands to before getting compiled in a .beam file, you can view it here.

Bonus for LFE webdevs: this project also has a module of HTTP status codes, for easy re-use in other projects.

The recent work I've been doing with macros has really helped shape the section I've been planning to write in the LFE User Guide. I haven't wanted it to be yet another dry description of macros in a Lisp-2. Instead, my hope is to help jump-start people into how to think about macros, how to start writing them immediately (without years of study first), and how to debug them.

The trick, as with all complicated subjects, is how to remove the barrier to entry and get folks that necessary hands-on experience without dumbing-down the material. It's almost ready...

Also, it's Bay to Breakers today, and Ocean Beach is bumpin. The synchronicity is just eery. May you party just as hard with routes in LFE.

After the Cloud: So Far

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

Systems Engineering in a Box

The recent redefinition of "the cloud" as a service and commodity is a brilliant bit of frugal resource management (making use of idle resources in an expensive data center) coupled with flawless marketing. Yes, from a business perspective, that's an amazing coup. But it's the 30,000 foot technical perspective that really impresses me:

In the same way that software frameworks, their libraries, and best practices have, through the trials of last 40 tears, productized application engineering, the cloud has started to experience something similar. What everyone is now calling the cloud is really the productization of systems engineering.

Systems engineering (and the management of related resources) has proven to be an expensive, time-consuming endeavor best left to the experts. Sadly, those that need it are often in the unenviable position of having to determine who the experts are without having the proper background to do so effectively. When the planning, building, and management of large systems works well, it's a labor of sweat and blood. When it doesn't, it's the same thing, with a nightmare tinge about the whole thing coupled with an odd time-dilation effect.

It seems that in applicable circumstances, some businesses are spared that nightmare by using a cloud service or product.

Bionic CGI

As someone with a long history and interest in application development, I was particularly keen on Google App Engine when it came out. This was a different take on the cloud, one that Mosso also seems to be embracing: upload an application that is capable of having it's data access and views distributed/load balanced across multiple systems (virtual or otherwise).

This is essentially CGI's grandchild. You have an application that needs to be started up by any number of machines in response to demand. A CGI app in Mosso will probably need very few (if any) adjustments required in order to run "in the cloud." Google is a special case, since developers are using custom, black-box infrastructure built by Google (for insights into this, check out these papers), but I'd be willing to bet someone lunch that there is room for a CGI analogy at some level of Google App Engine.I guess with Google, we kind of have both application and systems engineering in a box, in so far as the systems support your application.

At any rate, it's CGI better than it was before. Better, stronger, faster.

The Rub

However fascinating these cloud offerings may be, I find myself not getting what I need. As a developer of Twisted applications, I'm interested in small apps. Hell, I don't even like running databases and full-blown web servers. A while ago, I spent a couple years working on some Twisted-based application components that could be run as independent services (thus load-balanceable) and completely replace the standard web server + database + lots of code routine for application development.

So what about developers out there like me, who want to run tiny apps? We don't need "classic" web hosting, nor CGI in the cloud, nor cloud-virutualized versions of large machines.

As a segment of the population, business consideration for developers such as myself might seem like a waste of time. But before dismissing us, consider this:

1. Exploring small niche's like this one often lead to interesting revelations.
2. Market segments that have proven quite vibrant may be able to expand into even greater territories (e.g., the iPhone apps phenomena).

Next up: Tiny > *