The Future of Programming - An Overview

Art by Philip Straub

There's a new series of blog posts coming, inspired by on-going conversations with peers, continuous inspection of the development landscape, habitual navel-gazing, and participation at the catalytic OSCON 2014. As you might have inferred, these will be on the topic of "The Future of Programming."

Not to be confused with Bret Victor's excellent talk last year at DBX, these posts will be less about individual technologies or developer user experience, and more about historic trends and viewing the present (and near future) through such a lense.

In this mini-series, the aim is to present posts on following topics:

• An Overview
• Themes at OSCON 2014
• Adopting the Functional Paradigm?
• Retrospective on Paradigms
• The Rise of Polyglotism
• Preparing for the Future

I did a similar set of posts, conceived in late 2008 and published in 2009 on the future of cloud computing entitled After the Cloud. It was a very successful series and the cloud industry seems to be heading towards some of the predictions made in it -- ZeroVM and Docker are an incremental step towards the future of distributed processes/functions outlined in To Atomic Computation and Beyond. 

In that post, though, are two quotes from industry greats. These provide an excellent context for this series as well, hinting at an overriding theme:

• Alan Kay, 1998: A crucial key to growing large systems is effective communications between components.
• Joe Armstrong, 2004: To effectively model and solve problems in a distributed manner, we need concurrency... this is made easier when we isolate processes and do not share data.

In the decade since these statements were made, we have seen individuals, projects, and companies take that vision to heart -- and succeeding as a result. But as an industry, we continue to struggle with the definition of our art; we still are tormented by change -- both from within and externally -- and do not seem to adapt to it well.

These posts will peer into such places ... in the hope that such inspection might guide us better through the tangled forest of our present into the unimagined forest of our future.

Up next: Themes at OSCON 2014 ...

Some Thoughts on the Mobile Device Interface

Smartphones: Yesterday's News


As some of you know, I started 2010 by working in a new position at Canonical: Ubuntu Project Manager. I've been having an absolute blast; working my butt off has never been more fun, challenging, or interesting. I'm finding that nearly every side-interest I've had in the past several years is coming to the forefront in my project management work.

There were all sorts of adjustments I needed to make before PM'ing again, and one of those was catching up on communication technology. I now live by email, calendars, IRC, Skype, and phone conversations. Gone are the days of going heads-down into some code for a week or two. I need to stay connected, 100% of the time. I needed to get a smartphone.

What I really wanted was a Nokia N900. Sadly, T-Mobile's not offering one, so I got an Android phone with a physical keyboard instead: a new G1. Yeah, out of date, but considering that I was still using a Razr, the G1 is cutting-edge ;-)

I gotta tell you, this little phone has changed my life. The craziest thing is not the apps, the Market, the features, etc.; it's the touchscreen that has made me a believer.

You Can Touch This


I can't believe how radically the touchscreen phone has changed my computing habits and preferences. When I sit down at my laptop or desktop to do something quickly, I don't want use the keyboard or mouse. I want to point, swipe, and tweak with my fingers. When I'm on the phone, my brain has my speaking and text-processing faculties tied up. To easily multitask while talking, I need to be able to use a different part of my brain: that part involved in motor control.

I like using the touchscreen so much that I will often use my G1 for tasks that are better suited for my laptop, merely because of the joy I get from using the interface.

Touch, tap, drag, push, swipe. I love it. Can't get enough of it.

The best thing of all? This is really silly: I love the virtual desktops and being able to navigate between them with a swipe. Whose idea was this? That designer or engineer needs to be promoted! I have never experienced a more intuitive way of switching virtual desktops. I didn't even know how much this was important to me until I used the G1. I want this for my laptop!

Having such a positive bias towards tactile technology, you can imagine my joy when I saw this Ubuntu blueprint. And then when I was asked to work with Bryce and Andy on the PM portion of multi-touch support, I was quite delighted. This will give application developers and device engineers what they need in order to start creating new exciting stuff for the Ubuntu world. You will be able to have an iPad-like experience on your Linux devices (that have the proper hardware).

The thing is, as much fun as tactile interfaces are, I want way more now. I've been given a taste... now I want the banquet.

An Interface for the Future


So the iPad has been getting lots of press. It's bigger than a phone, I like that: I could replace my pen-and-paper medium-sized Moleskine with one. The thing I like the best though? Yeah, you guessed it: the interface. Curling pages and apps that have been re-worked specifically for the new format/size. I love it when a device lets me use it in a way that is natural and intuitive, and provides visual (or other sensory) feedback on my use.

Apple has made, in my opinion, a good and interesting product. But certainly not a revolutionary one. It's a natural progression from what folks are already doing with smartphones and netbooks.
So let's talk about revolutionary :-)

Imagine you've got a crazy new shirt, one whose fibers convert your movements to electrical current and can power your devices. Perhaps it comes with a "battery net" for literally flexible power storage. Now imagine that some clever sod has equipped your shirt with sockets for micro-SD cards (or something similar). If you've read my "After the Cloud" posts, then you probably know where I'm going with this :-)

Now add some embedded micro-controllers, and bluetooth, a smart phone with cloud-controller software, and you've got a personal S3 with potentially terabytes of light-weight, wearable storage. And your phone controls the nodes, redundancy, failover, etc. Maybe your phone runs Ubuntu and you're pushing backups of your personal cloud onto your U1 account.

But how do we interface with all this great storage? Here comes the banquet I mentioned :-)

I want to be able to reach "into" my phone, grab an icon (application, file, contact, whatever) and put it where I want. But I mean really put it: I can pick up some data off my desktop, and throw it over my shoulder or at my feet. With appropriate sensing equipemnt on the power shirt's sleeves, my arms and hands are now the perfect "mice."

In fact, there's no more virtual desktop (that's old skool). One of the new primary functions of your mobile device is to peer "into" the halo of data that surrounds you now. You can either spin your virtual storage space around you like an inverted, 3D lazy susan, our you can physically move your phone around, like a diving mask peering into the water.

Closing Thoughts


The more data I have, the more I feel that I live in it. The problem is that our current tech forces us into tiny sardine cans and we have to consume our data with the equivalent of a single chop-stick. If I'm going to live in my data, I want to have the best possible experience of immersion that I can. I want an interface that can handle my future.

I could go on and on... I love this sort of thing. The important thing to know now is that the community is working on the building blocks for our technological future. The first steps are being made in open source software that will allow us to take giant, insanely cool steps in the not-to-distant future.

txAWS 0.0.1 Released!

The first version of txAWS just made it out the door, thanks to prodding from Scott Moser, who is helping to get txAWS into Ubuntu Lucid Lynx. It's been uploaded to PyPI now too, so you can do the usual (easy_install txAWS), or you can download it from Launchpad.

For those interested in writing async code for the cloud, txAWS is the library for you :-) What's more, if you're interested in contributing to a Twisted-based project, this could be just the thing to get you started. The use of Twisted is pretty basic in txAWS (though we do adhere to various coding idioms pretty strongly, to enhance maintainability), and would be a nice introduction. What's more, there are lots of exciting features and work still to do, and you could really make a difference.

Recent Work on Various Open Source Projects

Over the last few months, I've been doing lots of work on various open source projects. I've been so burried in them that I haven't blogged (or microblogged) much about them. So much has been happening, though, that I needed to take a break from the coding and communicate some of this :-)

txAWS

Over the last few months, Robert Collins, Thomas Herve, Jamshed Kakar and I have been putting lots of effort into improving cloud support in the async (Twisted) Python Amazon EC2 library. It's been a lot of fun to see that part of the library take shape and start getting production use from Canonical. We have implemented the following functionality groups and their associated API methods:

• Instances
• Key Pairs
• Security Groups
• Elastic IPs
• Availability Zones
• EBS

There is a ticket for the following two groups of functionality, and branches in progress for both:

• Images (ticket #461136 on Launchpad)
  
• Image Attributes (ticket #461139 on Launchpad)
  

Once those are merged, EC2 support in txAWS will be complete.

As a bonus, we've added support for arbitrary endpoints and with that in place, have successfully tested txaws.ec2 against Eucalyptus clouds :-)

txJSON-RPC

There's a new release of txJSON-RPC out now, downloadable from PyPI. Work on the next version has been a great deal of fun. What started out as a conversation on IRC with Terry Jones, ended up as spec-driven doctest work on trunk for implementing support for multiple versions of the JSON-RPC spec (pre-1.0, 1.0, and 2.0).

With these changes in spec support, txJSON-RPC has really started to mature, and that's been fantastic. Even more, the jsonrpclib module that's included in txJSON-RPC (and can be used with non-Twisted projects) is getting spec version support as well.

SOOM in txULS

As some may know, one of my computing passions is ultra large-scale systems. After a phone conversation with Jamshed Kakar and some nice exchanges on the Python ULS-SIG mail list with Alex Drahon, I started working on a set of coding experiments in self-organizing objects. The Google Doc informally outlines the various stages and goals.

For now, the code is living in a txULS series on Launchpad. The reason for its inclusion in txULS is that ultimate goal of the SOOM (self-organizing object meshes) code is to produce an async API for building Twisted services that provide behaviours as outlined in the Google Doc (linked above).

I would to emphasize the networking-library-agnostic nature of the ULS-SIG: Twisted comes up since I spend a lot of time with Twisted, but ever networking library is welcome. I'm personally interested in exploring (or watching other developers explore) various Stackless Python experiments in the ULS systems space.

txSpore

This project was a spontaneous effort resulting from an evening of code review when I first discovered the official Python API from EA/Maxis for the game Spore. It's been a blast and something that Chris Armstrong and I have been working on together. The API is currently feature-complete, but Chris has some excellent ideas about improving usage as well as some additional API augmentations that will make life easier for game developers.

Already more featureful and usable than the official Spore Python API, there are great things in store for this library. Chris has come up with several very cool demo ideas that take advantage of the new API and will push it to the limits. We're both pretty excited :-)

Isomyr

I love isometric games. I'm a freak for the classic look. One night about a month ago, Chris and I discovered Isotope, an isometric Python game engine by Simon Gillespie. It was last updated in 2005 at version 0.9, so I started working on a branch that could be released as 1.0. I never heard back from Simon after an inquiry for his permission to release as 1.0, so I forked the code to a new project: Isomyr. I released the code rewrite work I had done to that point (plus some changes such as replacing some old code with Numpy) as 0.1. At which point things just went nuts...

Isomyr now has support for multiple worlds, customizable (per world) in-game time and calendars, and basic interactive fiction development. The latest chunk of code (that hasn't been pushed up to Launchpad yet) is adding support for general planetary simulation (e.g., axial title, varying daylight hours, seasons, and weather). As you might imagine, this has been a great deal of fun to work on!

PyRRD

PyRRD has gotten some recent community love, with requests for a mail list, new developer-oritnted features, etc. Currently at version 0.0.7, the 0.1.0 release isn't to far away. Folks have been using trunk for a while, which added support for the RRDTool Python bindings back in March of this year. (PyRRD's focus has primarily been for users/developers who didn't have the RRDTool Python bindings installed). In the next couple weeks or months, I expect that we'll be adding a few more features, and then preparing the new release.

PyRTF

Another fixer-upper project, PyRTF (mirroed on Google code as pyrtf-ng) has been on hiatus for a while, due to my diminished need to manipulate and interact with RTF files. However, a new developer has joined the project and the code-cleanup and unit test development now continues. Thanks Christian Simms!

A while ago, Simon Cusack (the original author of PyRTF) and I had some great discussions about the future development of PyRTF and his interest in merging the recent changes into trunk on SourceForge. I deferred on that action, wanting to wait until the code cleanup, unit tests, and API changes had been completed. With Christian's help, we may get there now :-)

Wrap-up

It's been about a year since I've been so active in open source development, and it feels really good to be at it again :-) Being back in Colorado seems to have helped in subtle ways, but mostly it's been the increased interaction and interest from developers in the community that I can thank for my increased activity (and thus enjoyment). You guys are awesome. You're the reason for any code I produce.

After the Cloud: Epilogue

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

Though wildly exciting to imagine a future of computing where ubiquitous devices are more deeply integrated into the infrastructure we use to power our applications, the real purpose of these posts has been to explore possibilities.

Let's have crazy thoughts. Let's build upon them, imagining ways in which they could become a reality. Let's not only munch on the regular diet of the technical "now"; let's plant seeds in many experiments for the future.

This post is about thinking of small, mobile devices and cloud computing. But it's also a rough template. Let's do the same thing for a the desktop: how might it evolve? Where will users be spending their time? Let's do it for the OS and the kernel: what radical changes can we envision there? The technology behind health care. Education. Our new patterns of behaviour in a constantly changing world. All of these and more deserve our attention.

The more we discuss such topics in a public forum, the more thought will be given to them. Such increased awareness and attention might spark the light of innovation years ahead of time, and do so in the context of an open exchange of ideas. Let's have Moore's law for the improved quality of life with regard to technology; let's take it out of the chip and into our lives.

After the Cloud: Heaps of Cash

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

A One-Two Punch

Let me give you the punchline first, this time. Imagine a service that:

1. Combines the cloud management features of EC2 for any system (in this case, mobile devices), the monitoring and update management of Canonical's Landscape, the buying/selling power of an e-commerce application, the auctioning capabilities of eBay, and the data of marketing campaigns.
2. Seamlessly integrates all this into a cloud (Open Heap!) provisioning/acquisition system or as part of your mobile provider's billing and information web pages.

So where does the cash come in?


Power to the People

As usual, revenue would depend upon market adoption. That would depend upon appeal (addressed with marketing), usefulness (addressed by software engineering and usability), and viability. That last one's particularly interesting, as it's where people and the cash intersect.

A product suite and service, all built around open heaps, could have a long and fruitful life if implemented with the end user in mind. Users would have the opportunity to become partners in an extraordinary way: they would be consuming a service, while at the same time, being given the opportunity to resell a portion of that service for use in the cloud-like architectures of open heaps.

The first company that does this really well would have a continuously growing following of users. This company would be helping consumers earn immediate cash back on their property. This is something I believe deeply in; it's a positive manifestation of the continuing evolution of the consumer's role in the market. I'm convinced that the more symbiotic a relationship between consumer and producer, the healthier an economy will be.


Providers

In an open heap scenario, there are two providers: the mobile phone provider and the heap provider. Phone companies get to make money from the deal passively: through a partnership that provides them with a certain percentage of the revenue or indirectly through heap-related network use.

The heap provider (e.g., someone like Amazon or RightScale), would stand to make the most of everyone. Even though they wouldn't own the devices themselves (in contrast to current cloud providers), they would be able to assess fees on various transactions and for related services.

Imagine application developers "renting" potential CPU, memory, storage and bandwitdth from a heap that included 100s of 1000s of mobile users. The heap provider would be the trusted third party between the device owner and the application developer. In this way, the provider acts like an escrow service and can asses feeds accordingly.

Imagine a dynamic sub-market that arises out of this sort of provisioning: with millions of devices to choose from, a user is going to want to make theirs more appealing than 1000s of others. Enter auctions. Look at how much money eBay makes. Look at the fractional fees that they asses... fees which have earned them billions of dollars.

Throw in value-adds like monitoring and specialized management features, and you've got additional sources of revenue. There's a lot of potential in something like this...


Review
Obviously, all of this is little more than creative musing. The technology isn't quite there yet for a lot of what is required to make this a reality. Regardless, given the shere numbers of small, networkable devices in our society, we need to explore how to best exploit untapped resources in mobile computing, providing additional, cheaper environments for small applications, decreasing the dependency we have upon large data centers, and hopefully reducing the draw on power grids. We need decentralized, secure storage and processing. We need smarter, more fair, consumer-as-beneficiary, economies.

Next, we develope a new segment of the market, where any user or company with one or more networked devices would be able to log in to an open heap provider's software and offer their machine as another member in that cloud. There's a lot of work involved in making that happen, much of it focused on the design and implementation of really good software.

If we can accomplish all that, we will have reinvented the cloud as something far greater and more flexible than it is today.

After the Cloud: Open Heaps

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

Refresher

Up to now, we've considered technical explorations and possible related future directions for the technology surrounding the support of distributed applications and infrastructure. This post takes a break and returns to thoughts of provisioning resources on small devices such as mobile phones. As stated in To Atomic Computation and Beyond:

This could be just the platform for running small processes in a distributed environment. And making it a reality could prove to be quite lucrative. A forthcoming blog post will explore more about the possibilities involved with phone clouds...

But first, I'm so tired of the term "cloud," so I did some free-association... from cloud to clouds to "tons of little clouds" to "close to the ground" to cumulus to heap (Latin for "cumulus"). Heap! It's irresistible :-)

"Open" is such a terribly abused word these days (more so than cloud), but using it as an adjective for a wild collection of ad-hoc, virtualized process spaces satisfies some subtle sense of humor. Open Heaps it is.


Starting Points

Let's think about the medium in our example: cellular telephony. Is there a potential market here? Here are some raw numbers from Wikipedia:

By November 2007, the total number of mobile phone subscriptions in the world had reached 3.3 billion, or half of the human population (although some users have multiple subscriptions, or inactive subscriptions), which also makes the mobile phone the most widely spread technology and the most common electronic device in the world.

I think we can count that as a tentative "yes."

Can we do this the easy way and just use TCP/IP? In other words, what about using WiFi phones or dual-mode mobile phones as the communication medium for devices in our open heaps? Well, that would certainly make many things much easier, since everything would stay in the TCP/IP universe. However, the market penetration of standard mobile phones is so much greater in comparison.

That being said, how many currently operating phones are capable of serving content on the internet, running background processes, etc.? Maybe only a small fraction, perhaps even enough to justify supporting only devices such as handhelds, smartphones, MIDs, UMPCs, and Netbooks.

Two possibilities for ventures here might be:

1. A startup that developed an Open Heap offering for any Internet-connected device.
2. A company that formed a partnership with one or more mobile carriers, acting as a bridge between the carrier-controlled network/device-management capabilities and the Internet.

The Business Problem

So, let's say we've got the technology ready to go that will allow users to upload a process hypervisor to their phones, and that this technology provides the ability for users to allot process resources (e.g., RAM, CPU, storage). There are still a couple basic principles to address to justify a business in this area:

1. How will people be better off with this than without it?
2. How will this technology generate revenue?

In general, I believe that consumers are always better off with more choices. I also believe that balanced systems run better than those that are rigged to benefit just one group. As such, I have an idealist's interest in things like Open Heaps, as they will empower interested consumers to earn revenue (however small) on their own property (mobile phones and other devices with marketable resources). What's more, if there are billions of devices available as nodes in Open Heaps, and there is a computing demand for those resources, then there will inevitably be competitors aiming to capitalize on them. Generally speaking, I also believe that increased competition provides a better chance for improved quality of service.

Conversely, imagine that Open Heaps don't happen, that the idle resources of mobile devices (or any other eligible equipment) either remain untapped of their potential or, worse, are put to use by corporations that only desire the end consumer to have limited power over their own property and how it's used. Dire scenarios aren't difficult to imagine, thanks to various examples of anti-consumer behaviour we've seen from large corporations and special interest organizations in the recent past.

So, yes -- I think we can make a case for this being of benefit to consumers (and thus a marketer's dream!). The more prevalent mobile devices become, the more they will integrate into our daily lives... and the more important it will be that these are managed as the rightful property of the consumer, people that have the right to rent or lease and profit from their property as they see fit.

But, how could this generate revenue?

Next up: Gimme da cash!

After the Cloud: To Atomic Computation and Beyond

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

To restate the problem: we've got cloud for systems and we've got cloud for a large number of applications. We don't have cloud for processes (e.g., custom, light-weight applications/long-running daemons).

Personally, I don't want a whole virtual machine to myself, I just need a tiny process space for my daemon. When my daemon starts getting slammed, I want new instances of it started in a cloud (and then killed when they're not needed).

What's more, over time, I want to be writing my daemon better and better... using less of everything (memory, CPU, disk) in subsequent iterations. I want this process cloud to be able to handle potentially significant changes in my software.

Dream Cloud

So, after all that stumbling around, thinking about servers in the data center as the horsepower behind distributed services, and then user PCs/laptops as a more power-friendly alternative, the obvious hit me: phones. They are almost ubiquitous. People leave them on, plugged in, and only use them for a fraction of that time. What if we were able to construct a cloud from cell phones? Hell, let's throw in Laptops and netbooks, too. And Xboxes, Wii, and TiVos. Theoretically, anything that could support (or be hacked to support) a virtual process space could become part of this cloud.

This could be just the platform for running small processes in a distributed environment. And making it a reality could prove to be quite lucrative. A forthcoming blog post will explore more about the possibilities involved with phone clouds... but for now, let's push things even further.

When I mentioned this idea to Chris Armstrong at the Ubuntu Developer Conference last December, he immediately asked me if I'd read Charles Stross' book Halting State. I had started it, but hadn't gotten to the part about the phones. A portion of Stross' future vision in that book dealt with the ability of users to legally run programs of other's phones. I really enjoyed the tale, but afterwards I was ready to explore other possibilities.

Horse-buggy Virtualization

So I sat down and pondered other possibilities over the course of several weeks. I kept trying to think like business visionaries, given a new resource to exploit. But finally I stopped that and tried just imagining the possibilities based on examples computing and business history.

What's the natural thing for businesses to do when someone invents something or improves something? Put new improvements to old uses, potentially reinventing old markets in the process. That's just the sort of thing that could happen with the cloudification of mobile devices.

For examples, imagine this:

• Phone cloud becomes a reality.
• Someone in a garage in Silicon Valley buys a bunch of cheap phones, gumstix, or other small ARM components, rips off the cases, and sells them in rack-mountable enclosures.
• Data centers start supplementing their old hardware offering with this new one that lets them use phone cloud tech (originally built for remote, hand-held devices) to sell tiny fractions of resources to users (on new, consolidated hardware... like having hundreds of phone uses in a single room with full bars, 24/7).
• With the changing hardware and continuing improvements in virtualization software, more abstraction takes place.
• Virtualization slowly goes from tool to prima materia, allowing designers not to focus on old-style, horse-drawn "machines" like your grandpa used to rack, but rather abstract process spaces that provide just what is needed, for example, to enable a daemon to run.

Once you've gotten that far, you're just inches from producing a meta operating system: process spaces (and other abstracted bits) can be built up to form a traditional user space. Or they can be used to build something entirely different and new. The computing universe suddenly gets a lot more flexible and dynamic.

Democritus Meets Modern Software

So, let's say that my dream comes true: I can now push all my tiny apps into a cloud service and turn off the big machines I've got colocated throughout the US. But once this is in place, how can we improve our applications to take even better advantage of such a system, one so capable of massively distributing our running code?

This leads us to an almost metaphysical software engineering question: how small can you divide an application until you reach the limits of functionality, where any further division would be senseless bytes and syntax errors? In terms of running processes, what is your code atom?

Prior to a few years ago, the most common answer would likely have been "my script" or "my application". Unless, of course, you asked a Scheme programmer. Programming languages like Scheme, Haskell, and Erlang are finding rapidly increasing acceptance as solutions for distributed programming problems because functional programming languages lend themselves easily to the problem of concurrency and parallelism.

If we had a massive computing cloud (atmosphere, more likely!) where we could run code in virtual process spaces, we could theoretically go even further than running a daemon: we could split our daemon up into async functions. These distributed functions could be available as continuously running microthreads/greenlets/whatever. They could accept an input and produce an output. Composing distributed functions could result in a program. Programs could change, failover, improve, etc., just by adding or removing distributed functions or by changing their order.

From Atoms to Dynamic Programs

Once we've broken down our programs into distributed functions and have broken our concept of an "Operating System" down into virtual process spaces, we can start building a whole new world of software:

• Software becomes very dynamic, very distributed.
• The particulars of hardware become irrelevant (it just needs to be present, somewhere).
• We see an even more marked correlation between power consumption and code, where functions themselves could be measured in joules consumed per second.
• Just for fun, let's throw in dynamic selection of fuctions or even genetic algorithms, and we have ourselves one of the core branches of the predicted Ultra-large Scale Systems :-)

I mention this not for cheap thrills, but rather because of the importance of having a vision. Even if we don't get to where we think we're going, by looking ahead and forward, we have the opportunity to influence our journey such that we increase the chances of getting to a place equal to or better than where we'd originally intended.

From a more practical perspective: today, I'm concerned about running daemons in the cloud. Tomorrow I could very well be concerned about finer granularity than that. Why not explore the potential results of such technology? Yes, it my prove infeasible now; but even still, it could render insights... and maybe more.

A Parting Message

Before I wind this blog post down, I'd like to paste a couple really excellent quotes. They are good not so much for their immediate content, but for the pregnant potentials they contain; for the directions they can point our musings... and engineerings. These are two similar thoughts about messaging from two radically different contexts. I leave you with these moments of Zen:

On the Erlang mail list, four years ago, Erlang creator Joe Armstrong posted this:

In Concurrency Oriented (CO) programming you concentrate on the concurrency and the messages between the processes. There is no sharing of data.

[A program] should be thought of thousands of little black boxes all doing things in parallel - these black boxes can send and receive messages. Black boxes can detect errors in other black boxes - that's all.
...
Erlang uses a simple functional language inside the [black boxes] - this is not particularly interesting - *any* language that does the job would do - the important bit is the concurrency.

On the Squeak mail list in 1998, Alan Kay had this to say:

...Smalltalk is not only NOT its syntax or the class library, it is not even about classes. I'm sorry that I long ago coined the term "objects" for this topic because it gets many people to focus on the lesser idea.

The big idea is "messaging" -- that is what the kernal of Smalltalk/Squeak is all about... The key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be. Think of the internet -- to live, it (a) has to allow many different kinds of ideas and realizations that are beyond any single standard and (b) to allow varying degrees of safe interoperability between these ideas.

If you focus on just messaging -- and realize that a good metasystem can late bind the various 2nd level architectures used in objects -- then much of the language-, UI-, and OS based discussions on this thread are really quite moot.

Resources

• Petri Nets
• π-Calculus
• Ambient Calculus

Next up: The Business of Computing Atmospheres

After the Cloud: The New Big

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

Intermission

I've made a few hints so far about what cloud service I'd like to see come into being, and at the end of this post, we'll get closer to discussing that. Hang in there: the post after this one will describe that in more detail. Then, after that, there will be at least one post which will take a peek at some of the many business opportunities that could come from this.

A Passing Comment

At PyCon 2006 in Dallas, TX, an after-hours event was held in a local bookstore. At one point during that evening, Itamar, Moshe and I got into a discussion about miniaturization and Moshe went off on a hilarious rant that Itamar and I just sat back and enjoyed. His whole tirade was based on the beauty and perfection of gumstix. This was the first I'd heard of them; I had no idea a product like that was on the market, and it hit me like a ton of bricks.

For the next day or so all I could think about was buying a boxload of gumstix computers and doing something with them -- anything! And not just because they were the coolest toys ever, but because there was something about them that I could just feel was a part of the future of computing (see my 2004 post on Dinosaurs and Mammals). It seemed that these miniture devices could help prototype what was destined to be one of the most exciting fields in the coming years for both systems and application engineers.

Sadly, I never did get that box :-) But I neither did I stop thinking about them. Confronted with the problem of small distributed services sitting on big, barely-used iron, gumstix haunted my musings.

Tiny Apps in the Cloud?

When at Divmod, one of the strategies that Glyph and I were working on concerned Twisted adoption in web hosting and cloud environments. The differences between CGI and Twisted applications are magnified when one considers a cloud environment like Mosso and one that would suitably support Twisted design principals. I spent a lot of time pondering the ramifcations of that one, let me tell you. A potential merger permanently postponed those business possibilities, but a nice side benefit was the forking of Python Director into a pure-Twisted conversion, txLoadBalancer (with the beginnings of native, in-app load-balancing support).

Thoughts of adjusting tiny apps to be able to run on big cloud hardware still grated, though. It felt dangerously close to pounding round pegs into square holes. What I really wanted was something closer to the future hinted at by Ultra Large-Scale Systems research: massively distributed, fault-tolerant services running on everything :-) Until then, though, I would have been satisfied with tiny apps on tiny hardware, consuming only the resources they need in order to provide the service they were designed for.

This brought up ideas of distributed storage, memory, and processing as well as the need for redundacy and failover. But tiny. All I could see was tiny hardware, tiny apps, tiny protocols, tiny power consumption. For me, tiny was big. The easiest "tiny" problem to address with small devices was storage. And I already knew the guys that were working on the problem.

Distributed Storage Done Right

There's an odd, rather abstract parallel between EC2 and Tahoe (a secure, decentralized, fault-tolerant filesystem). EC2 arose in part from a corporation acting out of its best interests: turn a liability into an asset. For Tahoe, the "body" in question isn't a corporation, but rather a community. And the commodity is not bottom lines, but rather data owned and treasured by members of a data-consuming community.

Here's a quick description of Tahoe from a 2008 paper:

Tahoe is a storage grid designed to provide secure, long-term storage, such as for backup applications. It consists of userspace processes running on commodity PC hardware and communicating with [other Tahoe nodes] over TCP/IP.

Tahoe is written in Python using Twisted and a capabilities system inspired by those defined by E. But what does this mean to a user? It means that anyone can setup and run a storage grid on their personal computers. All data is encrypted and redundant, so you don't need to trust members of the community (your data grid), you just need to set aside some disk space on your machines for them.

In a message to the Tahoe mail list, I responded to an associate who was exploring Tahoe for in-memory use by Python mapreduce applications. I wanted in-memory distributed storage for a different use case (tiny apps on tiny devices!) but our interests were similar. It turned out one of the primary Tahoe developers was working on related code; something that could be used as the basis for future support for distributed, solid-state devices.

Here's some nice dessert: Twisted coder David Reid was reported to have gotten Tahoe runnig on his iPhone. Now we're talking ;-) (Update: David has informed me that Allmydata has a Tahoe client that runs on his iPhone).

Processing in the Right Direction

But what about the CPU? Running daemons? Can we do something similar with processing power? If a whole virtual machine is too much for users, can we get a virtual processing space? I want to be able to run my process (e.g., a Twisted daemon) on someone else's machine, but in such a way that they feel perfectly safe running it. I want Tahoe for processes :-)

As part of some recent experiments in setting up a virtual lab of running gumstix ARM images, I needed to be able to connect mutliple gumstix instances in a virtual network for testing purposes. In a search for such a solution, I discovered VDE. Then, unexpectedly, I ran across a couple fascinating wiki pages on the site of related super-project Virtual Square Networking. Their domain is currently not resolving for me, so I can't pull the exact text, but here's a blurb from a sister project on SourceForge:

View OS is a user configurable, modular process virtual machine, or system call hypervisor. For each process the user is able to define a "view of the world" in terms of file system, networking, devices, permissions, users, time and so on.

Man, that's so close, I can almost taste it!

Where is all this techno-rambling going? Well, I'm sure some of you have long since guessed by now :-) Regardless, I will save that for the next post.

Oh, and yes: tiny is the new big.

Next Up: A Passing Message

After the Cloud: So Far

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

Systems Engineering in a Box

The recent redefinition of "the cloud" as a service and commodity is a brilliant bit of frugal resource management (making use of idle resources in an expensive data center) coupled with flawless marketing. Yes, from a business perspective, that's an amazing coup. But it's the 30,000 foot technical perspective that really impresses me:

In the same way that software frameworks, their libraries, and best practices have, through the trials of last 40 tears, productized application engineering, the cloud has started to experience something similar. What everyone is now calling the cloud is really the productization of systems engineering.

Systems engineering (and the management of related resources) has proven to be an expensive, time-consuming endeavor best left to the experts. Sadly, those that need it are often in the unenviable position of having to determine who the experts are without having the proper background to do so effectively. When the planning, building, and management of large systems works well, it's a labor of sweat and blood. When it doesn't, it's the same thing, with a nightmare tinge about the whole thing coupled with an odd time-dilation effect.

It seems that in applicable circumstances, some businesses are spared that nightmare by using a cloud service or product.

Bionic CGI

As someone with a long history and interest in application development, I was particularly keen on Google App Engine when it came out. This was a different take on the cloud, one that Mosso also seems to be embracing: upload an application that is capable of having it's data access and views distributed/load balanced across multiple systems (virtual or otherwise).

This is essentially CGI's grandchild. You have an application that needs to be started up by any number of machines in response to demand. A CGI app in Mosso will probably need very few (if any) adjustments required in order to run "in the cloud." Google is a special case, since developers are using custom, black-box infrastructure built by Google (for insights into this, check out these papers), but I'd be willing to bet someone lunch that there is room for a CGI analogy at some level of Google App Engine.I guess with Google, we kind of have both application and systems engineering in a box, in so far as the systems support your application.

At any rate, it's CGI better than it was before. Better, stronger, faster.

The Rub

However fascinating these cloud offerings may be, I find myself not getting what I need. As a developer of Twisted applications, I'm interested in small apps. Hell, I don't even like running databases and full-blown web servers. A while ago, I spent a couple years working on some Twisted-based application components that could be run as independent services (thus load-balanceable) and completely replace the standard web server + database + lots of code routine for application development.

So what about developers out there like me, who want to run tiny apps? We don't need "classic" web hosting, nor CGI in the cloud, nor cloud-virutualized versions of large machines.

As a segment of the population, business consideration for developers such as myself might seem like a waste of time. But before dismissing us, consider this:

1. Exploring small niche's like this one often lead to interesting revelations.
2. Market segments that have proven quite vibrant may be able to expand into even greater territories (e.g., the iPhone apps phenomena).

Next up: Tiny > *

After the Cloud: Prelude

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

These days, it seems that no matter where we go, we hear something about "the cloud." It's not really buzz anymore... it has become far more accepted and widely discussed to be that. For some organizations it's actually part of their current, every-day infrastructure. For others, it soon will be. As far as I'm concerned, now's the perfect time to start discussing what's next :-)

If you've spent any time reading some of the blog content I've managed to post over the past several years, you've probably noted that I like to explore the long view (if rather informally). Well, that's what I've got in store for you now: a series of blog posts that explore the long view of a post-cloud industry. Hopefully, with some new twists and turns along the way.

First off, I want to cover some basic ground, so the first couple posts might be a little less interesting that those that follow. Fortunately, I've been pondering these particular ideas since my month sabbatical last August -- this means I've already got most of the material written and ready to go!

These posts are going to take a peek at practical, hands-on ideas regarding the ways in which one might make use of current nascent tech to build prototypes for tomorrow's infrastructure, what that infrastructure might be, business ideas about what do do with that tech, and even future possibilities for information-based markets.

Hope you enjoy it as much as I've enjoyed thinking about it :-)

Native LoadBalancing for Twisted Apps

Yesterday, right before midnight, I tagged the 1.1.0 release of txLoadBalancer on Launchpad after completing the last of the planned features. There are some pretty radical changes that have been developed for this release... and the coolest part is this is just the beginning :-) (See the TODO if you don't believe me!)

You can checkout from lp:~oubiwann/txloadbalancer/1.1.0 or download from PyPI. If you're a PyPI expert, I've got some questions for you at the end of this post... Been having some sucky experiences with PyPI lately :-(

So here's what's going on with txLoadBalancer:

Improved API

The biggest thing you'll notice if you've switching from PythonDirector is the massive overhaul the API has undergone. Things are cleaner and generally more modern, with a concise and well-defined module layout.

New Load Balancing Algorithm

I've added support for a weighted host scheduler. Given a weight that represents the frequency a host should be used, a host will be randomly selected, based on it's weight. For example, with two hosts, one having a weight of 1 and the other having a weight of 3, host 2 will be chosen about 75% of the time and host 1 will get about 25% of the requests.

Right now, this algorithm has to make several calls to other parts of the code in order to get all the data it needs (it also builds some crazy iterators). As such, it's rather slow and performs poorly when compared to the very light-weight least-connections algorithm. That being said, the next release will include optimizations for the weighted scheduler that make use of a Twisted timer and caching.

Native Twisted Load-Balancing

Here's the sexiest part: you can now load-balance your Twisted application by using the txLB API; you don't even need to run the load-balancer as a separate app! This evolved as a feature after a conversation with an as-yet unnamed cloud hosting provider, a follow-up discussion with the Divmod team, and then some quiet pondering about ways in which Twisted applications could be supported in cloud/grid/massively-multi-core architectures.

The "self load-balancing" API in txLB is not a comlete solution for grid-hosting, but it is a first step in one direction (we've been discussing lots of others, too, including the use of our deployment tool).

Before I show you how to use the self load-balancing API, let's take a quick look at a normal Twisted application service:

You start that with the command twistd -noy myweb.tac. For use with the next example, you can also start two more, one on port 7002 and the other on port 7003.

Now here's what you do to make a self load-balanced app:

As you would expect, you need to indicate the proxy host:port, the algorithm to use, and the hosts that are to be balanced. The host setup assumes that you have three services running on localhost ports 7001, 7002, and 7003. All that's needed now is to just run that code with the usual twistd -noy myapp.tac. Also, for demonstration purposes, this is a somewhat simplified example of what is possible.

This may seem like a lot of extra work when compared to the simple web host above, but think about it: we're load-balancing here :-) This saves you from having to manage yet another application. With a few extra lines of code, you can keep it all in one place and have it manage itself.

Note that this API is in development and continuing to improve. The example above is from code running in trunk. For the more verbose configuration that is in the 1.1.0 release, be sure to see ./bin/txlbWeb.tac from the source tarball. To play with the latest and greatest, you'll want to checkout the code here: lp:txloadbalancer.

Other Goodies

Here is some other good stuff in the release:

• You can now ssh into a txLB instance and mainipulate the load-balancer in real time from an interactive Python interpreter.
• You can change the proxy to listen on a different port while the application is running (no restart requred!).
• Changes made to the configuration while running are no longer volatile; they are saved to disk (and your old config gets backed up).
• Work from Apple, Inc. was included in this release, too (they use the old PythonDirector in their Calendaring server). This includes a bug fix and management socket feature.
• There is a significant jump in performance between this release and the previous one. I believe this to be due to the separation of concerns in the API, but haven't yet confirmed that.

Coming Work

There are a lot of exciting features coming for txLB. Just to name a few:

• improved weighted algorithm
• resources-based algorithm (a scheduler that determins the weight of a proxied host by memory, CPU, etc., utilization)
• smarter proxied host failover and recovery
• a heartbeat manager
• txLB-powered application cloning (when started, an app will determine if it needs to run the clone as the managing load-balancer or simply as a proxied host)
• auto-discovery of balanced hosts
• proxy fail-over (a balanced host taking over as manager in the event that the manager goes down)
• ApacheMQ/Stomp integration
• LDAP/RADIUS authentication

Additionally, I'll be putting together some basic performance metrics contrasting Apache and load-balanced Twisted apps. I will also be comparing previous versions of txLB/PythonDirector with the latest release(s).

Problems with PyPI

I will close this post on a sad note: PyPI used to be an amazing experience for me (a couple years ago, when it was still being called "cheeseshop"). Everything worked as it was supposed to. This hasn't been the case when I've used it recently (over the past few months).

For all that I say about PyPI, I allow for the fact that I may just be missing something, and it may be entirely my fault. That being said, I spent about 3 hours online last night combing though the SIG mail list, the bug list on sourceforge, and blog posts about setuptools and PyPI, and could find no answers to my questions. Well, with the possible exception of a bug report, but it doesn't look like it was confirmed by a PyPI team member, so I'm not sure if it's valid or not.

Here are my issues:

• When I upload my project using python setup.py [sdist|bdist_egg] upload, no metadata defined in my setup() function is presented on my package's PyPI page. When I click the metadata link, it's only got three sparse lines.
• When I manually upload from the package's PKG-INFO itself, all the metadata is presented on the page as it should be, with the exception of the long description. It is in plain text instead of ReST (I am checking that it is valid ReST using distutils settings of reporter.halt_level = 5, reporter.report_level = 1, settings.pep_references = False, and settings.trim_footnote_reference_space = None; these are the same settings that Zope Corp uses to verify the ReST that it uploads to PyPI).
• When I manually edit the long description in the form, I get the same thing: plain text, no ReST.
• When I upload a package that is displayed properly on PyPI (such as zc.twist; uploaded as one of my projects by chaning the name), I get the same problem (this is why I think it might be something that I'm doing wrong...): no metadata, and when I upload the PKG-INFO manually, no ReST.

Why, oh why, cruel fates, does this not work any more? I used to be able to upload to PyPI without any of these issues...

txLoadBalancer

Well today was a flurry of activity... pulled an all-nighter whipping a python load balancer into shape after some late-afternoon discussions on #divmod.

At Divmod, we're going to be labbing out some distributed services experiments with twistd servers, and one set of those experiments involves "developer friendly" load balancing. JP suggested that I take a look at how PyDirector works and see if we could use that. Which was actually interesting in a full-circle kind of way: I worked on PyDirector when I was at PBS, ages ago, where I wrote a weighted lb algorithm for it.

Jumping into the code again after a 5-year hiatus was like seeing an old friend :-)

All tonight, I worked on the following branches:

• https://code.launchpad.net/~oubiwann/pydirector/1.0.0 - A copy of the 2004 release of PyDirector
• https://code.launchpad.net/~oubiwann/pydirector/more-twisted - Full Twisted support, development
• https://code.launchpad.net/~oubiwann/pydirector/1.1.1 - Full Twisted support, release
• https://code.launchpad.net/~oubiwann/txloadbalancer/0.9.1 - PyDirector compatible; Identical to PyDirector 1.1.1 (with the exception of the obvious name change)
  
• https://code.launchpad.net/~oubiwann/txloadbalancer/1.0.1 - Twisted-only (no threading nor asyncore modules); name space was changed to txlb
  
• https://code.launchpad.net/~oubiwann/txloadbalancer/main - current development

txLoadBalancer 0.9.1 and 1.0.1 are up on PyPI in the usual place.

I did lots of manual functional testing for each branch tonight, but I didn't do any TDD. While I'm still playing with it, I'll probably start adding tests as bugs crop up (BDT), and as it gets more serious I'll go fully into TDD and fill in what's missing at that point.

Tonight's mad rush was actually a great deal of fun. It's been a while since I've had the opportunity to plow through a bunch of code like that, and I enjoyed myself to near exhaustion :-) I don't think I'll be able to get to sleep tonight (er, this morning), due to the endless thinking about all the ways in which I want to use this code, mutate it, and... well, I better leave some surprises for later!

Update: I've edited the links for the latest micro-releases that fixed some issues with setup.py.

Update 2: Thanks to the heads-up in the comments from Kapil, I've patched txLoadBalancer trunk with the changes from Apple (David Reid and Wilfredo Sanchez).

App Engine Haiku

I've been playing with Google App Engine a bit more tonight, and I've come to some additional conclusions: 

1. For the best results, you gotta drink the whole pitcher of koolaide; and
2. It's really quite tasty koolaide.

One of the things I really wanted to test was imaging. I use Flickr far more than Picasa, but Flickr's authentication API is kind of a pain in the ass (especially when used with App Engine). Google auth is (for obvious reasons) easier to work with. This, of course, led me to explore Picasa. To my surprise, it was delightful to use. I chopped up gdata for only the parts needed to support Picasa and was uploading images within mere minutes. And not only was the effort minimal, but the performance was outstanding.

One of the things that makes application development a more efficient process is having a unified platform on which to work. This is why working with the Divmod platform is so nice when I do extensive Twisted application development: all the infrastructure is already built and ready for me to use. Today, I got to really taste that same experience with the Google platform. And I liked it.

I did notice one interesting psychological side effect of working with the file count limitation: I inadvertently treated it like a game. Not unlike the rules in poetry, it provided a structure and bounds within which I was forced to operate, adjust for, and test my creativity against. I rather enjoyed this microgame, and that was a surprise :-)

The Problem with and Solution to Google's App Engine

I know everyone is all aglow with the new web development offering from Google, but let me do the unpopular thing and put some things into perspective: there are limitations.

In fact, the limitations that exist will prevent me from using App Engine with all of my projects, save one (that one being a very simple web site). First, the limitations that prevent me from using App Engine (from one of their FAQs):

• Sockets are disabled with Google App Engine
• The system does not allow you to invoke subprocesses, as a result some os module methods are disabled
• Threading is not available

This means that I can't write a deferred wrapper for their data layer, I can't use Twisted for such things as XML-RPC or AMP-based communications, and I can't use an async templating system (like Nevow). I'm stuck with CGI and blocking code. And for all but the simplest projects, that's a big "No Thank You" from me.

This doesn't mean that I won't use it -- I will. I have one project that this will be perfect for... but it's for someone else, not me.

However, these limitations are actually good news :-) Here's the silver lining:

As Glyph as alluded to in his recent blog post (and in our tweets), we've recently completed a massive week-long BizDev Divmod sprint in Boston. One of the results of this is based on community feedback we've had over the last year, and which culminated at PyCon 2008 in Chicago with multiple requests for particular services from The Twisted Company. That result is a set of tools, features, and management options folks will be able to use with our software (app server, smart object db, network services, etc.). People really want to start using our stuff in cloud/grid computing environments. They need support for multiple and diverse network services, inter-store communications, massive deployments, etc. Two months before PyCon, we started working on tickets to support this, and we're making excellent progress toward providing the requested features.

We're still unclear as to which parts of this will be open source, as that will be driven by a combination of business and community demand. Regardless, Google's lack of support for this stuff has (for now) left the field wide open for us. And that, folks, is a big "Thank You Google!" :-)