The Future of Personal Data: A Followup

The Next Step

A few years ago, I wrote a post about the Future of Personal Data as a result of all the ultra large-scale systems reading and exploration I was doing. Google was foremost in my mind when writing that, but Apple has since come into the spotlight here as well.

Recently, Matt Zimmerman has decided to leave Canonical and join forces with Singly, an exciting startup company focused on secure user data storage and the socialization of (and development around) that data. In this blog post, the following core values were given about the software underlying Singly:

• I own my personal data
• I want my data to be useful to me
• I make the decisions to protect or share my

I would like to see the following added:

• I make the decisions on how my data is used
• If my data is sold, I should get a return for this

This is not petulance speaking :-) This comes from a historical perspective on social and economic fairness. Witness the changes in individual rights and personal finance since the industrial revolution...

Fair Market Value

There are probably many solvent entities out there who would claim users are already getting a return for the use of their data: free email and office docs from Google, free cloud services from Apple, etc. But I would imagine that there are massive margins being made on user data, and services (as valuable as they may be) are a paltry return for such a gold mine. I cannot help but be reminded of the selling price of Manhattan Island or Alaska (in 2011 values, the Lenape Indians got around $29.61/square mile; the Russians got about $150.77/square mile).

Money makes a good point, but personal data (and this post) isn't about the almighty coin. This is about clearly defining who owns what and ensuring that those who don't want to be taken advantage of, aren't. This is about identifying exploitation, and building something better and longer-lasting in its place.

Who's Going to Pay for What?

In David Pakman's "Disruption" blog post about Singly, he makes the following comment:

"I cannot see consumers getting into the business of selling their data to marketers so they can see personalized advertising. Instead, I believe marketers will be encouraged to offer value to us in exchange for access to our data."

I have to agree with him... but I can only offer a qualified agreement. True, I find it hard to believe that users will be selling their data directly to marketers. From the user-side, the pain of inconvenience would not likely be worth the payoff; at the marketer end, individual data is useless, and munging an in-house-built collection would incur a lot of overhead not part of their core business.

However, users' data stored in lockers, updated regularly, pre-processed, has enormous value in the market. Right now, Apple and Google are making eye-crossing amounts of money from data just like this. Again, I would imagine that this data is only really valuable in large quantities and for interesting, identifiable demographics

If users provide their data, but in exchange only get a "nicer app" or a "useful utility" I'm going to cry "foul!" (unless someone can show me the actual numbers involved and unequivocally prove that fair exchange is occurring).

You Say Disruption, I Say Revolution

Instead, if a service such as Singly, were to offer a co-op style dividend payment system to all of its users, that would seem to be much more fair. Not only that, it would be the beginning of a market revolution. This is not to say that co-ops are some perfect economic model, but rather that the data we, as users, generate is of immense value. The more that Singly has, the greater potential for revenue. The more buzz that builds around Singly users generating revenue from their data, the more users they get. With mass-adoption, a new sub-economy is born.

Perhaps a better model than co-op is that of a mutual fund investment firm. Each Singly user has a portfolio of data. Depending upon each user's preferences, any or all of that data could be used by Singly to generate revenue. Some groups of users will generate more than others, and users in these groups would get greater returns.

Whichever analogy you prefer, with a little exploration it seems fairly clear that opportunities for a large payout are present. For instance, I like to imagine a world where entities like Apple and Google can't harvest user data, but must go through brokers whom users have given their permission to sell their data for the most profit. I also imagine there's a lot of lawmaking that would have to take place... and even more lobbying.

Even with that, Google and Apple would still make money hand over fist (or they'd become brokers themselves) -- enough to continue providing free services. Yet at the same time, users would be in control of their data; they'd be financing (or financially augmenting) their data-consumption lives with said data.

With the right press coverage, Singly could find themselves not only swamped with a massively growing user base, but at the very center of a new economy. With the right level of negotiation and coordination, businesses could buy into this new paradigm without losing their shirts in the disruption.

A Plea

In summary, I applaud the goals and vision of Singly. I, for one, would deeply appreciate writing applications against their data locker (to any Facebook or any of its dubious applications), where a user's rights are clear and protected. That being said, Sinlgy would have my eternal allegiance if they also took up the cause of rights for user data in the market place; if they helped transform the current nascent data economy into a world economy capable of achieving as-yet unimagined financial heights.

And if not Singly, my loyalty would be given to whomever did do this.

Recent Work on Various Open Source Projects

Over the last few months, I've been doing lots of work on various open source projects. I've been so burried in them that I haven't blogged (or microblogged) much about them. So much has been happening, though, that I needed to take a break from the coding and communicate some of this :-)

txAWS

Over the last few months, Robert Collins, Thomas Herve, Jamshed Kakar and I have been putting lots of effort into improving cloud support in the async (Twisted) Python Amazon EC2 library. It's been a lot of fun to see that part of the library take shape and start getting production use from Canonical. We have implemented the following functionality groups and their associated API methods:

• Instances
• Key Pairs
• Security Groups
• Elastic IPs
• Availability Zones
• EBS

There is a ticket for the following two groups of functionality, and branches in progress for both:

• Images (ticket #461136 on Launchpad)
  
• Image Attributes (ticket #461139 on Launchpad)
  

Once those are merged, EC2 support in txAWS will be complete.

As a bonus, we've added support for arbitrary endpoints and with that in place, have successfully tested txaws.ec2 against Eucalyptus clouds :-)

txJSON-RPC

There's a new release of txJSON-RPC out now, downloadable from PyPI. Work on the next version has been a great deal of fun. What started out as a conversation on IRC with Terry Jones, ended up as spec-driven doctest work on trunk for implementing support for multiple versions of the JSON-RPC spec (pre-1.0, 1.0, and 2.0).

With these changes in spec support, txJSON-RPC has really started to mature, and that's been fantastic. Even more, the jsonrpclib module that's included in txJSON-RPC (and can be used with non-Twisted projects) is getting spec version support as well.

SOOM in txULS

As some may know, one of my computing passions is ultra large-scale systems. After a phone conversation with Jamshed Kakar and some nice exchanges on the Python ULS-SIG mail list with Alex Drahon, I started working on a set of coding experiments in self-organizing objects. The Google Doc informally outlines the various stages and goals.

For now, the code is living in a txULS series on Launchpad. The reason for its inclusion in txULS is that ultimate goal of the SOOM (self-organizing object meshes) code is to produce an async API for building Twisted services that provide behaviours as outlined in the Google Doc (linked above).

I would to emphasize the networking-library-agnostic nature of the ULS-SIG: Twisted comes up since I spend a lot of time with Twisted, but ever networking library is welcome. I'm personally interested in exploring (or watching other developers explore) various Stackless Python experiments in the ULS systems space.

txSpore

This project was a spontaneous effort resulting from an evening of code review when I first discovered the official Python API from EA/Maxis for the game Spore. It's been a blast and something that Chris Armstrong and I have been working on together. The API is currently feature-complete, but Chris has some excellent ideas about improving usage as well as some additional API augmentations that will make life easier for game developers.

Already more featureful and usable than the official Spore Python API, there are great things in store for this library. Chris has come up with several very cool demo ideas that take advantage of the new API and will push it to the limits. We're both pretty excited :-)

Isomyr

I love isometric games. I'm a freak for the classic look. One night about a month ago, Chris and I discovered Isotope, an isometric Python game engine by Simon Gillespie. It was last updated in 2005 at version 0.9, so I started working on a branch that could be released as 1.0. I never heard back from Simon after an inquiry for his permission to release as 1.0, so I forked the code to a new project: Isomyr. I released the code rewrite work I had done to that point (plus some changes such as replacing some old code with Numpy) as 0.1. At which point things just went nuts...

Isomyr now has support for multiple worlds, customizable (per world) in-game time and calendars, and basic interactive fiction development. The latest chunk of code (that hasn't been pushed up to Launchpad yet) is adding support for general planetary simulation (e.g., axial title, varying daylight hours, seasons, and weather). As you might imagine, this has been a great deal of fun to work on!

PyRRD

PyRRD has gotten some recent community love, with requests for a mail list, new developer-oritnted features, etc. Currently at version 0.0.7, the 0.1.0 release isn't to far away. Folks have been using trunk for a while, which added support for the RRDTool Python bindings back in March of this year. (PyRRD's focus has primarily been for users/developers who didn't have the RRDTool Python bindings installed). In the next couple weeks or months, I expect that we'll be adding a few more features, and then preparing the new release.

PyRTF

Another fixer-upper project, PyRTF (mirroed on Google code as pyrtf-ng) has been on hiatus for a while, due to my diminished need to manipulate and interact with RTF files. However, a new developer has joined the project and the code-cleanup and unit test development now continues. Thanks Christian Simms!

A while ago, Simon Cusack (the original author of PyRTF) and I had some great discussions about the future development of PyRTF and his interest in merging the recent changes into trunk on SourceForge. I deferred on that action, wanting to wait until the code cleanup, unit tests, and API changes had been completed. With Christian's help, we may get there now :-)

Wrap-up

It's been about a year since I've been so active in open source development, and it feels really good to be at it again :-) Being back in Colorado seems to have helped in subtle ways, but mostly it's been the increased interaction and interest from developers in the community that I can thank for my increased activity (and thus enjoyment). You guys are awesome. You're the reason for any code I produce.

A Sinfonia on Messaging with txAMQP, Part I

A Sinfonia on Messaging:

1. The Voice of Business
   
2. The Voice of Architecture
3. A RabbitMQ and txAMQP Interlude

Prelude

A complete messaging solution is a three-part invention of business need, architecture, and implementation. In its final form, these three voices blend in harmony, with each one taking a dominant role depending upon which part of the solution one examines.

Neither have I the ability nor skill to seamlessly weave three concepts together while clearly explaining their roles. Therefore, I will separate out the voices from each other and leave it as an exercise for the reader to construct an application and practice the principles involved, thus experiencing well-earned contrapuntal pleasures first-hand.

Introduction

As computing and data exchange systems increased in complexity over the past 30 years, so has the need for improvements -- and where possible, simplifications. Some of these efforts have been focused on decentralization of communications (shared, distributed load) and decoupling of messaging from applications (removing redundancy and increasing delivery speed/throughput). The first steps towards this were made in the 1990s with explorations in "middleware" application universe.

Messaging, as we now refer to it in the industry, arouse from those middleware adventures: out of the business drive to refactor old software as new services to wider, more sophisticated audiences. With many new services replacing a single, monolithic application, formal and well-architected solutions were needed for creating, editing, and deleting shared data.

From Wikipedia:

Message-oriented middleware (MOM) is infrastructure focused on message sending that increases the interoperability, portability, and flexibility of an application by allowing the application to be distributed over multiple heterogeneous platforms. It reduces the complexity of developing applications that span multiple operating systems and network protocols by insulating the application developer from the details of the various operating system and network interfaces.

AMQP (Advanced Message Queuing Protocol) is one of these protcols.

In a recent blog post about Ubuntu as a business server, Vaughan-Nichols provides evidence for Ubuntu's and Canonical's commitment to enterprise, saying "... the new [version of] Ubuntu also includes AMQP [...] support. AMQP is an important set of middleware and SOA [...] protocols."

AMQP has demonstrated itself as a compelling protocol for messaging solutions, even to the point of being included in two Linux distributions. The code included in this blog series is txAMQP, an asynchronous Python AMQP library built with Twisted.

Note that last year I had planned to write a blog series on messaging with Twisted and XMPP, but was unable to as a result of time constraints. These days, I'm working with AMQP instead of XMPP, but I still hold some hope that I'll be able to write an anolog for this series from the perspective of XMPP.


The Voice of Business

Dipping into the future, one of the goals for AMQP as developed by a special interest group is the following:

Decentralized, Locally Governed Federated Mesh of AMQP Brokers with standardized Global Addressing. The killer application for AMQP is transacted secure business messages between corporations - e.g. send a banking confirmation message to confirms@bank.com [...]

I find this rather exciting due to my interest in ultra large-scale systems; scenarios like the one described above are the seeds for tomorrow's ULS systems :-)

For now, though, let's look at a more immediate use for AMQP: a messaging protocol for shared services between departments in a small store. In this exercise, the voice of business is the primary melody; everything else (architecture and implementation) is done in support of this theme.


An Example

Fhionnlaidh MacGrudder creates hand-made kilts to order. He's got a sales guy who works with movie costume design shops and the like. He's got a web girl who wrote and maintains a custom store front app. He's got a friend who does shipping and billing for him (as well as some other local Glen Orchy artisans). Until now, these three business "groups" associated with the kilt shop have been maintaining their own records, sometimes copying and updating them manually from each others' various export files.

Fhionnlaidh's niece Fíona is programmer, business student, and is dating shipping guy's son. Horrified by the inefficiencies in her uncle's business processes (and tired of her boyfriend's father's complaints), she has proposed the following:

• sales guy will maintain customer contact info offer this as a service to the store and billing
• the web store will dynamically update displayed data when sales guy changes it
• the CRM will dynamically update displayed data when a web store customer updates their info
• MacGrudder will have a new web page he goes to where all pending orders are presented with their full details; changes can be updated by a customer in real-time until MacGrudder has started working on the order
  
• billing/shipping guy will be notified instantly as soon as MacGrudder marks a kilt order as completed
  

This setup has the following benefits:

• Contacts will be maintained in a single data store
• There is zero latency between customer-driven updates and sales guy-driven updates
• Customers have increased post-purchase flexibility with their orders
• Shipping guy can plug into MacGudder's messaging and be notified when packages are ready for pickup
• Everyone has more time for buttered scones and tea (especially shipping guy, who will no longer be making unneeded trips down the glen)
  

The following changes will be made to the current software:

• Contacts will need to be merged into the CRM
• A read/write data service for the contacts will need to be created
• The CRM front-end will need to be upgraded to an AJAX-enabled version
• The web store app will need to be updated to support AJAX
• A new page will be created which displays the status of all orders and allows MacGrudder to change an order from "pending" to "in-progress" to "completed"
• The current "new order" email notification code in the web app will need to be changed so that it uses the same messaging as MacGrudder's status page
  
• A new service needs to be created for shipping guy so that he can choose to be notified about pending pickups by email or he can check a web page or even make a query directly to the service, thus preventing unnecessary trips to MacGrudder's isolated little shop
  
• After all the work is done, somone's going to need to order more scones

This example is not meant to fully justify messaging for businesses, but rather to provide a simple use case for which we can write some simple (and less than robust) code. It is a toy, but a conceptually useful one with a solid, concrete foundation.

In the next installment, we'll review the business process (with diagrams!) and the explore the architecture of the system, before and after. Another post will take that architecture and combine it with MacGrudder's already extant infrastructure, reusing as much as possible. With that in place, we will have the opportunity to look at some RabbitMQ basics and some actual txAMQP code.


References

• SOA in Practice
  
• Message-oriented middleware on Wikipedia
  
• AMQP on Wikipedia
  
• The AMQP Spec
• A synchronous Python library for AMQP
• The async Python library for AMQP
• A txAMQP example
• An intro to messaging and py-amqplib
  

After the Cloud: To Atomic Computation and Beyond

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

To restate the problem: we've got cloud for systems and we've got cloud for a large number of applications. We don't have cloud for processes (e.g., custom, light-weight applications/long-running daemons).

Personally, I don't want a whole virtual machine to myself, I just need a tiny process space for my daemon. When my daemon starts getting slammed, I want new instances of it started in a cloud (and then killed when they're not needed).

What's more, over time, I want to be writing my daemon better and better... using less of everything (memory, CPU, disk) in subsequent iterations. I want this process cloud to be able to handle potentially significant changes in my software.

Dream Cloud

So, after all that stumbling around, thinking about servers in the data center as the horsepower behind distributed services, and then user PCs/laptops as a more power-friendly alternative, the obvious hit me: phones. They are almost ubiquitous. People leave them on, plugged in, and only use them for a fraction of that time. What if we were able to construct a cloud from cell phones? Hell, let's throw in Laptops and netbooks, too. And Xboxes, Wii, and TiVos. Theoretically, anything that could support (or be hacked to support) a virtual process space could become part of this cloud.

This could be just the platform for running small processes in a distributed environment. And making it a reality could prove to be quite lucrative. A forthcoming blog post will explore more about the possibilities involved with phone clouds... but for now, let's push things even further.

When I mentioned this idea to Chris Armstrong at the Ubuntu Developer Conference last December, he immediately asked me if I'd read Charles Stross' book Halting State. I had started it, but hadn't gotten to the part about the phones. A portion of Stross' future vision in that book dealt with the ability of users to legally run programs of other's phones. I really enjoyed the tale, but afterwards I was ready to explore other possibilities.

Horse-buggy Virtualization

So I sat down and pondered other possibilities over the course of several weeks. I kept trying to think like business visionaries, given a new resource to exploit. But finally I stopped that and tried just imagining the possibilities based on examples computing and business history.

What's the natural thing for businesses to do when someone invents something or improves something? Put new improvements to old uses, potentially reinventing old markets in the process. That's just the sort of thing that could happen with the cloudification of mobile devices.

For examples, imagine this:

• Phone cloud becomes a reality.
• Someone in a garage in Silicon Valley buys a bunch of cheap phones, gumstix, or other small ARM components, rips off the cases, and sells them in rack-mountable enclosures.
• Data centers start supplementing their old hardware offering with this new one that lets them use phone cloud tech (originally built for remote, hand-held devices) to sell tiny fractions of resources to users (on new, consolidated hardware... like having hundreds of phone uses in a single room with full bars, 24/7).
• With the changing hardware and continuing improvements in virtualization software, more abstraction takes place.
• Virtualization slowly goes from tool to prima materia, allowing designers not to focus on old-style, horse-drawn "machines" like your grandpa used to rack, but rather abstract process spaces that provide just what is needed, for example, to enable a daemon to run.

Once you've gotten that far, you're just inches from producing a meta operating system: process spaces (and other abstracted bits) can be built up to form a traditional user space. Or they can be used to build something entirely different and new. The computing universe suddenly gets a lot more flexible and dynamic.

Democritus Meets Modern Software

So, let's say that my dream comes true: I can now push all my tiny apps into a cloud service and turn off the big machines I've got colocated throughout the US. But once this is in place, how can we improve our applications to take even better advantage of such a system, one so capable of massively distributing our running code?

This leads us to an almost metaphysical software engineering question: how small can you divide an application until you reach the limits of functionality, where any further division would be senseless bytes and syntax errors? In terms of running processes, what is your code atom?

Prior to a few years ago, the most common answer would likely have been "my script" or "my application". Unless, of course, you asked a Scheme programmer. Programming languages like Scheme, Haskell, and Erlang are finding rapidly increasing acceptance as solutions for distributed programming problems because functional programming languages lend themselves easily to the problem of concurrency and parallelism.

If we had a massive computing cloud (atmosphere, more likely!) where we could run code in virtual process spaces, we could theoretically go even further than running a daemon: we could split our daemon up into async functions. These distributed functions could be available as continuously running microthreads/greenlets/whatever. They could accept an input and produce an output. Composing distributed functions could result in a program. Programs could change, failover, improve, etc., just by adding or removing distributed functions or by changing their order.

From Atoms to Dynamic Programs

Once we've broken down our programs into distributed functions and have broken our concept of an "Operating System" down into virtual process spaces, we can start building a whole new world of software:

• Software becomes very dynamic, very distributed.
• The particulars of hardware become irrelevant (it just needs to be present, somewhere).
• We see an even more marked correlation between power consumption and code, where functions themselves could be measured in joules consumed per second.
• Just for fun, let's throw in dynamic selection of fuctions or even genetic algorithms, and we have ourselves one of the core branches of the predicted Ultra-large Scale Systems :-)

I mention this not for cheap thrills, but rather because of the importance of having a vision. Even if we don't get to where we think we're going, by looking ahead and forward, we have the opportunity to influence our journey such that we increase the chances of getting to a place equal to or better than where we'd originally intended.

From a more practical perspective: today, I'm concerned about running daemons in the cloud. Tomorrow I could very well be concerned about finer granularity than that. Why not explore the potential results of such technology? Yes, it my prove infeasible now; but even still, it could render insights... and maybe more.

A Parting Message

Before I wind this blog post down, I'd like to paste a couple really excellent quotes. They are good not so much for their immediate content, but for the pregnant potentials they contain; for the directions they can point our musings... and engineerings. These are two similar thoughts about messaging from two radically different contexts. I leave you with these moments of Zen:

On the Erlang mail list, four years ago, Erlang creator Joe Armstrong posted this:

In Concurrency Oriented (CO) programming you concentrate on the concurrency and the messages between the processes. There is no sharing of data.

[A program] should be thought of thousands of little black boxes all doing things in parallel - these black boxes can send and receive messages. Black boxes can detect errors in other black boxes - that's all.
...
Erlang uses a simple functional language inside the [black boxes] - this is not particularly interesting - *any* language that does the job would do - the important bit is the concurrency.

On the Squeak mail list in 1998, Alan Kay had this to say:

...Smalltalk is not only NOT its syntax or the class library, it is not even about classes. I'm sorry that I long ago coined the term "objects" for this topic because it gets many people to focus on the lesser idea.

The big idea is "messaging" -- that is what the kernal of Smalltalk/Squeak is all about... The key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be. Think of the internet -- to live, it (a) has to allow many different kinds of ideas and realizations that are beyond any single standard and (b) to allow varying degrees of safe interoperability between these ideas.

If you focus on just messaging -- and realize that a good metasystem can late bind the various 2nd level architectures used in objects -- then much of the language-, UI-, and OS based discussions on this thread are really quite moot.

Resources

• Petri Nets
• π-Calculus
• Ambient Calculus

Next up: The Business of Computing Atmospheres

ULS-SIG: New Python Special Interest Group

After several discussions at the end of last summer and an incubation in Meta-SIG, Steve Holden, Jim Baker and I are pleased to announce the Python special interest group for ultra large-scale systems. For more about ULS system, you might want to read this post or this one. The SIGs page has been updated, so you can find us there with the rest of them (subscribe and archive links, too), and we've even got our own page :-)

The initial group of interested parties (and thus the first members of the list) represent an interesting cross-section of the Python community, including the following:

• Jython hackers
• Twisted hackers
• Stackless hackers
• XMPP experts
• MMPORG developers
• SOA and Business Process consultants
• General technology and software companies

The technological umbrella of ULS systems covers a vast array of topics and interests, but basic principle unifying all of these is their potential contribution to making massive, highly distributed systems a functional reality.

One overview of ULS systems research states that we currently don't have an effective understanding of software (it's nature, development, and management) at the scale anticipated for ULS systems. These "fundamental gaps" will hinder the development of such systems until they can be crossed. Doing so will require breakthroughs in many fields with insights and experience gained over time.

Python programmers represent an extraordinary segment of the population: creative, curious, motivated, communicative, and deeply intelligent individuals. Our community is filled with minds that continuously produce solutions for a vast array of problems across a great many disciplines.
If there's any one group out there that could pull this off, I think it's ours :-)

A future post will provide a sketch of areas of interest in Python that are already sneaking up on the gaps outlined in ULS systems reports. There is a lot of software and supporting libraries that have an obvious connection to ULS systems, but even more fun are the ones that don't... and isn't always the darkest corners that yield the most unlooked for surprises?

While you're waiting for that, though, feel free to join us on the mail list!

A Champion for Services Architecture Done Right

I've heard a series of discussions lately and have read some articles that I've found rather frustrating, all on the topic of SOA. I've taken a few notes about my objections and had several blog posts planned where I would attempt to articulate the historical failures of software, services, and/or architecture makeovers performed by any number of consulting firms for large IT shops. I was going to discuss the horrible irresponsibility of basing business decisions on buzzword flinging cowboys, and the inevitable price that is paid by companies wooed and seduced by flash.

I was going to start this off by launching a strong criticism of Anne Thomas Manes' views on the "death of SOA", one that I'd read about in a recent article. It seemed that such a view was sensationalist and entirely missed the point of the essence of service oriented architectures.

But before I did that, though, I needed to do some research and present a solid argument, not just take some trade rag's version of the story with a superficial smattering of misquoted details. And holy cow... not only am I glad that I did this, but I am now an ardent fan of Ms. Manes, Vice President and Research Director at the Burton Group.

Her post SOA is Dead; Long Live Services is brilliant. The death of which she speaks is not the mortal (and figurative) soul of SOA itself -- any individual or magazine claiming as much has either completely misunderstood her or has been making its money on the misunderstandings and hype that she would like to see die. She asserts that the very heart of SOA continues to be badly needed, that "the requirement for service-oriented architecture is stronger than ever." She then goes on to say:

The acronym got in the way. People forgot what SOA stands for. They were too wrapped up in silly technology debates (e.g., “what’s the best ESB?” or “WS-* vs. REST”), and they missed the important stuff: architecture and services.

And finishes with this bit of excellence:

The latest shiny new technology will not make things better. Incremental integration projects will not lead to significantly reduced costs and increased agility. If you want spectacular gains, then you need to make a spectacular commitment to change.

And you know what? Her other posts are even better. While reading all of them, I was completely dumbfounded that she said so well and clearly what the industry has been misinterpreting for several years. Why are more people not listening to her? How did I not hear of her until now?! I'll tell you what, I'm now subscribed to Burton Group's Application Platform Strategies blog.

IT shops need to pay close attention to what this woman and her team have to say. She really knows what she's talking about and articulates her views exquisitely. That whole series of blog post I wanted to make? Yeah... she already did that :-) (and way, way better than I could have). Please, I beg you: go read her posts. If you are the director of your IT shop or someone to whom your director listens and respects and you have any interest in distributed services, messaging, and related architectures, go memorize her statements, summaries, arguments and positions. Let her experience and wealth of knowledge guide judicial and wise decisions at your company.

And remember, it is the good, well thought out, and cost-effective architecture that survives. It is this architecture -- with countless peers -- that serve as the basis for the next generation of improvements and advances. Technologies produced in this manner are the ones that propel us into the future and prepare us for a world which we are incapable of imagining or implementing today.

Current and Future Happenings

Sorry there's been so much radio silence at this end lately... a lot has been going on, and it looks like it's going to stay that way for a while. I just need to get used to it and start posting again :-)

Canonical

The big news is that Canonical quite took me by surprise :-) I had planned on doing consulting work again, but I was made an offer of camaraderie, to come join a team at Canonical that I know well, and I couldn't resist. I'm now working on the Landscape team at Canonical, the same folks who brought you the much beloved Storm ORM :-)

Already, I've been working there for two weeks and it's been a delight. They use a lot of the same processes that we did at Divmod and in the Twisted project (in fact, three of us on the team are Twisted developers), so that was very smooth. Another thing that made the transition very easy was manner in which they engage in a beautiful mix of group discussion and rapid development. The open source community roots at Canonical are very deep... and you can see them very clearly without digging :-)

At Canonical, I've repeated run across old friends from my Zope days, from Hacking Society in Colorado, and other places/associations from my past. I am somewhat stunned at the job Canonical has done in acquiring a talented and dedicated workforce. I've never seen a company embrace open source at the level and to the degree that this company does, while at the same time retaining all of the most excellent qualities of the community within the corporate culture. Someone should do a socio-technological/business PhD thesis on these guys...

In preparation for the many (and intense) marathon sprints that this team runs in a year, I've purchased a new laptop. It's the first dedicated Ubuntu dev machine/Desktop I've had... I've been running all my Ubuntu instances as virtual machines in Parallels and VMWare Fusion (or as remote servers at colos and virtual host providers). My love for the Evolution mail client continues to grow and I've found the only reason I miss the Mac is for the automatic handling of sound and to play Spore :-) 

SOA Conference

Now on to some future stuff. I've been invited to speak about dynamic languages (Python) and ultra-large scale (ULS) systems at SOA-India this year in Bangalore. The industry that has grown up around service oriented architectures (SOA) overwhelmingly tends towards Java, so this is a really great sign. I think the efforts that the Java Mothership has made in building bridges with dynamic languages such as Ruby and Python is having a tremendous impact throughout the programming world. I've got an eye on Ted Leung and the Jython team :-)

Anyway, the conference promises to be quite interesting, with speakers from around the world and with diverse backgrounds. I'm expecting to return from Bangalore with a multitude of new ideas and lots of new avenues to explore.

Blog 

Speaking of SOA, I am still working on the second part of the book review for Josuttis' book SOA in Practice. Perhaps before I finish that one, though, I will blog about another O'Reilly title I have been enjoying immensely: Van Lindberg's  Intellectual Property and Open Source. Note that Van has been quite active in the Python community and is contributing his expertise at many levels for the benefit of us all. Regardless, the book is very well written and I will have nothing but good to say about it :-)

After that, I'm going to finish up the draft I have for a blog post on metaclasses, based on notes I took while working with Incredible Pear on the PBS DTV project.

And finally, there have been more requests for me to write about setting up a Twisted Mail server... so, as one reader puts it, I will conclude the telling of that tale in an up-coming post as well :-)

SOA in Practice: A Handbook for Early-Stage ULS Systems (Part 1)

The ULS Series

• Required Reading: Ultra Large Systems
• Ultra Large Systems: An Example
• SOA in Practice: A Handbook for Early-Stage ULS Systems (Part 1)

A Book Review

First off, this is an O'Reilly publication. What's more, if O'Reilly had something like a "criterion collection," this work would be in it. This title is what it says it is, "SOA in Pactice: The Art of Distributed System Design." Authored by Nicolai M. Josuttis, this is one of the best written technical overview works I have ever read, both for writing style and content. For anyone interested in ULS and/or SOA, I have one thing to say: buy this book immediately, with expedited shipping.

I'm not going to write a formal review with pros, cons, deep analysis about message, etc. However, what I will do is spend some time discussing the crossover from SOA to ULS, covering details with quotes from "SOA in Practice." I will not cover the book in detail and reveal all of its precious nuggets, but I will give a taste of what it has to offer and how it applies to ultra large-scale systems.

Divergence

Since most of what I want to discuss is about what we can gain by taking lessons learned from SOA and applying them to efforts in exploring or prototyping ULS systems, I want to initially outline the stark differences between those systems and SOA.

The most obvious difference is scale. To put things in perspective, imagine implementing a large SOA for a large organization. Imagine the requirements, the project planning, the logistics, the code, the bugs, the setbacks, the short-term failures, and finally, the successful delivery. Now multiply that: two related but semi-autonomous SOA projects. And again, with four. How about a third time for eight?

Any reader with experience in working with large projects is probably having heart palpitations right now (and for that, I apologize). You have first hand experience of the difficulties and the pain: with a linear increase in the size of a project, there is an exponential increase in the difficulty of managing that project (people, code, timelines, etc.), asymptotically approaching 100% unmanageability, regardless of the amount of resources you throw at the project.

The point just past the asymptote is where ULS systems and SOA meet. In other words, a ULS system as a whole -- by definition -- cannot be built. Such a system can accrete over time, but is simply too large to be designed, built and managed. Rather, it is emergent. Efforts being made in ULS systems research right now are focused on how we can best facilitate that emergence.

Convergence

One of the profound problem solving skills that maths like analytic geometry teach us is understanding potentially intractable problems by examining discrete and meaningful chunks. It's easy to chop something up; it's quite a different matter to chop such that the pieces are useful and provide further insight.

If working with ULS systems is like integrating over the volume of a complex solid in 11-space, then SOAs provide us with the tools of breaking part of that work up into a manageable chunk, one that we can wrap our heads around. Many of the same problems that technicians are anticipated to encounter when working with ULS systems exist at a smaller scale and are well understood within the context of SOA.

And this is where our friend Nicolai's book comes in.

ULS Systems Review

Before we continue, let's take a quick look back at some of the ULS basics laid out by the report of the Software Engineering Institute (SEI) of Carnegie Mellon:

• ULS systems are systems of systems at internet scale.
• ULS systems will be interdependent webs of software-intensive systems, people, policies, cultures, and economics.

In order to become a functional reality, these systems will require extensive research in the following areas:

• Human Interaction
• Computational Emergence
• Design
• Computational Engineering
• Adaptive System Infrastructure
• Adaptable and Predictable System Quality
• Policy, Acquisition, and Management

This means exploring for use in ultra-large scale systems such things as potential mechanisms for user interfaces, genetic algorithms/programming, new patterns in systems design, behavioral simulations of systems components in varying compositions, decentralized infrastructure, ultra-high availability, and integration with countless third-party support systems. Just to name a very bare minimum.

Intersection

Of those research areas, lessons learned from SOA can be applied to ULS systems research most predominantly in the following areas:

• Human Interaction
• Design
• Adaptive System Infrastructure
• Adaptable and Predictable System Quality

In Part 2, it is with an eye towards these that I will comment on Nicolai Josuttis' excellent work.

New ULS Systems Blog

I'm currently drafting two new ultra large-scale systems blog posts, with one in particular being almost ready to go. While writing more on one of them today, a very cool thing happened: I received an email from the ULS systems team at the Software Engineering Institute of Carnegie Mellon University letting me know that they've got a new ULS blog site up. You can check it out here:

http://ulsblog.wordpress.com/

Be sure to read all the articles and go back often! As you can imagine, I'll be spending a lot of time there :-) I have a feeling this is the beginning of an emerging ULS community...

As for my forthcoming ULS systems blog posts, one concerns SOA and the other discusses currently extant code bases in the Python and Twisted Python communities that can be used for building ultra large-scale systems (or prototypes thereof) quickly and efficiently.

The Future of Personal Data

In a recent post about ULS systems, I said this:

The balance of power, from individuals all the way to the top of whatever organizations exist in the future will rest in information. Not like it is today, however. The "information economy" of today (+/- 10 years) will look like kids' games and playgrounds. The information economies this will evolve into will be so completely integrated into human existence that they will resemble the basic necessities like water and food.

I'm not going to focus on the ULS systems topic in this post, but there is a very deep connection between privacy, personal data and all things ULS. Any thoughts of a ULS system should be coupled with how this will impact the system's users and their data. Any thought of our personal data's future existence should include the anticipated future of computing: ULS systems.


Inside and Out

In a nutshell, here's how things look:

• Yesterday: Paid Services - You want something, you buy it. Demographic research is expensive and mostly outsourced.
• Today: Free Services - You want something, companies give it to you for free... in exchange for your demographic data.
• Tomorrow: Information Economy - You want something, you leverage the value of your information in brokering the the service deals that mean the most to you.

What do we have right now? Companies are fighting for each other over who gets to have our data for free. Yay, free stuff! We used to have to pay for that sort of thing! But paying for people to hold your data was the old, old world. Having them do it for free is the old world. Here's the new world: They pay you.

Why would they do that? Why would things shift from the current status quo? The value of personal information.

There are many ways to assess the value of personal information, but let's look at a few from the perspective of large organizations (entailing everything from government to business). Simplistically, we can assign value to a single individual's data based on the value of a large collection of many individuals' data. The more participants, the greater the value of the whole, and therefore the greater the value for each individual's data. This perspective is limited because it treats data very staticly. The data may change, but in relation to the system it's "acquired" and inside as opposed to "for sale" and outside.


We Are the Markets

But the value of our data is not defined simply by the presense of bits or membership in a valued data conglomerate. Our data is not just our emails, our medical records, our purchasing trends, nor our opinions about local and national politics. Like an organism moving through an ecosystem, our data is dynamic and living; it is the very trace we leave in the world around us, be it digital or otherwise.

Any part of our lives that is ever recorded in "the system" provides data and comprises part of our movements through this system. Our traces through this digital ecosystem impact it, change it, shape its future direction. The collective behaviours (not just collective data) are immensely valuable to organizations. Their value is on-going and growing, with accrued, compounded interest.

Static data bits seem like property to us: you can buy them, you can sell them, you can store them somewhere. But moving, living data... that's a different story. That's not a buy-once commodity; ownership of that might be tantamount to slavery in a future, information-based economy. However, organizations might opt to lease it, or individuals might turn the past back on the future and offer license agreements to organizations.

More likely, though, individuals will form co-ops or communities (we have already seen this happend extensively in today's Internet) with shared mutual interest. Seeing how a group entity with shared values has a larger effect on the system than single individuals, data from such groups would likely be much more interesting and number-crunch-worthy. The greater power a group has to perterb systems' ecomonic or political trends, the more valuable that group's data will be to other groups.

In addition, I'm sure there'd be all sorts of tiered "offerings" from individuals and groups: the juicier/more detailed the data, the higher the premium offered. The changes this will introduce to markets (global and local), legal systems, and politcal organziations are probably barely imaginable right now. But what would it take to get us there? What would it take for my data and your data to be valuable enough to transform the world and make Wall Street look like an old-time, irrelevant boys club?


Privacy

One thing: a fanatical devotion to privacy, pure and simple. Security and a fanatical devotion to privacy. Two things! Okay, reliability, security and a fantaical devotion to privacy. Three things!

Monty Python references aside, an economy that values the data of individuals and groups can only arise if that data is secure. If we live in a topsy-turvy world where the Government, MPAA, RIAA, the Russian Mafia, and Big Hosting Company are pirating our data, then we're hosed. However, if our data is secure and contracts are effective, then we will have a world where data is the currency. There are an incredible number of hurdles to overcome in order for this to happen, however.

• The System - we need a system where user data can be tracked, recorded, and analyzed, and there's enough of it to matter
• Storage - we need our own, personal banks for our data (irrefutable ownership rights and complete power over that data)
• Transactions - we need a mechanism for engaging in secure, data transations
• Identity - when making a transation, we need to be able to prove unequivocally that we are who we say we are
• Anonymity - we need to decouple activity in the system and identity, thus requiring organizations to come to us (or our groups) to get the definitive data they need
• Recourse - we need a legal system and effective laws that protect the individuals and groups against the crimes of data-hungry organizations; fortunately, we will have had years of established precedent protecting the sellers from the buyers... oh my, how the tables turn!

And that's just off the top of my head. There's got to be tons of stuff which hasn't even occurred to me.


Closing Thoughts

Information will be as essential for us as water, yet there is a very interesting divergence from the example of a hydrological empire: each individual is the producer of some of that metaphorical water. By virtue of this difference, we hold the keys of the empire. We will be more a part of the economic and political powerbases than we have every been at any time in human history.

Of course, that means that we've got to get ready :-) This is already being done in many different ways. Everything from community housing cooperatives to small, co-op banks; from capabilities-based programming models to secure online transactions. Like the next 20 years of research needed for ULS systems to become a reality, we've got just as much work to do in order to guarantee our place in the economies of the future.

Ultra Large-Scale Systems: An Example

The ULS Series

• Required Reading: Ultra Large-Scale Systems
• Ultra Large-Scale Systems: An Example

Background

My interest in this topic is as old as my love for science fiction. As a child who had not only just started teaching himself to program but had fallen deeply in love with I, Robot, I consumed everything I could by the Master of the Art himself, Isaac Asimov. Inevitably, an endless steam of science fiction began flowing into my brain: the harder the science, the more cherished it was.

Then came the discovery that computers could actually talk to each other. Holy network, Batman, that changed everything! Oh, how I lamented my Kaypro II's inability to dial out. Science fiction novels began touching on this aspect of technology more and more frequently, while the Internet began taking shape in the "real" world around us. Now, look at it. Regardless of the mess and chaos, it's really quite amazing: beowulf clusters, distributed computing, cloud services, and of course the Internet in general. These advances are actually quite mind-blowing when we take the time to examine them from a historical perspectve.

A lot has changed since those early days of the network. The past 10 years or so has seen the beginnings of a trend with regard to large systems. Certainly my views on the future of networks (and services that utilize them) have been pretty consistent:

• In Praise of Evolvable Systems (2003)
• Dinosaurs and Mammals (2004)
• Evolving Networks (2004)
• Everything as a Distributed Resource (2004)
• The King is dead! Long live the Kinglets! (2005)

As I indicated in the more recent ULS blog post, I have been exposed to some excellent resources for ultra large-scale systems. For some of those I recently provided links, and others I will be referencing in future posts.

Due to their nature, ULS systems pose interesting open source collaboration as well as business opportunities. They entail a massive collection of excellent problems to solve that cannot possibly be completely addressed in the next 6-12 months (where so many projects and businesses tend to put their focus, for obvious practical reasons). As such, there are a great number of research and development areas -- plenty for everyone, in fact. In this series of blog posts, my goal is to expose a wider audience to the topics and encourage folks to start thinking about both interim solutions as well as potential long-term ones.


Characteristics of a ULS

Let's start of with some semblance of a definition :-) What constitutes a ULS system? Here are some characteristics given by Scale Changes Everything:

• an unbelievable amount of code (on the order of trillions of lines of code)
• immense storage needs, network connections, processing
• lots of hardware, lots of people, lots of purposes
• decentralized components
• created by aggregation, not design
• unreliable components, reliable whole
• ongoing and real-time upgrades, changes, and deployments
• lots of functionality, likely in a focused area of concern

Here's an illuminating quote from Richard Gabriel's Design Beyond Human Abilities presentation:

The components that make up a ULS system are diverse as well as many, ranging from servers and clusters down to small sensors, perhaps the size of motes of dust. Some of the components will self-organize like swarms, and others will be carefully designed. The components will not only be computationally active but also physically active, including sensors, actuators...

Sounds like pure science fiction, doesn't it? Think about it, though. Is it really? Divmod's friend Raffi Krikorian co-wrote this paper at MIT. Check out the cheap network node that's smaller than a fingertip. At that size, hundreds of them would be innocuous. In a few years, we could have thousands of them in a room without even knowing it. Within a single home we could have the equivalents of what today are campus or regional networks. We probably can't even wrap our heads around how big these systems will be. But there is plenty of precedence for such natural short-sightedness. From Raffi's (et al.) 2004 paper:

The ARPAnet was ambitiously designed to handle up to 64 sites with up to 4 computers per site, far exceeding any perceived future requirement. Today there are more than 200 million registered hosts on the Internet, with still more computers connected to those.

Here are some other choice quotes:

[Internet 0] is not a replacement for the current Internet (call that Internet 1); it is a set of principles for extending the Internet down to individual devices...

An [Internet 0] network cannot be distinguished from the computers that it connects; it really is the computer. Because it allows devices for communications, computation, storage, sensing, and display to exchange information in exactly the same representation, around the corner or around the world, the components of a system can be dynamically assembled based on the needs of a problem, rather than fixed by the boundaries of a box.

We're already building this stuff. It's not science fiction. We may not have swarming, self-replicating nano machines... yet. But we're already heading in a direction where that's not just a possibility; it's a likelihood.

So, we've got lots of code, machines, storage, sensing and people; much of it decentralized. What else do we need? Failure tolerance and maintenance on-the-fly. Check. Finally, a ULS system will have to actually be useful, or it will never get built. Who would want such a thing besides militaries, big governements, and Dr. Evils? Now we start getting to our example: Health Care. But let's not get ahead of ourselves. First, let's examine why the biggest system of networked devices that we know of isn't a ULS system.


Why the Internet is not a ULS System

Most obvious of the criteria listed above, the Internet is not focused on a single or related set of goals; it's used for everything. However, it does meet many of the criteria. From the Carnegie Mellon report:

The Web foreshadows the characteristics of ULS systems. Its scale is much larger than that of any of today’s systems of systems. Its development, oversight, and operational control are decentralized. Its stakeholders have diverse, conflicting, complex, and changing requirements. The services it provides undergo continuous evolution. The actions of the people making use of the Web influence what services are provided, and the services provided influence the actions of people. It has been designed to avoid the worst problems deriving from the heterogeneity of its elements and to be insensitive to connection failures.

But ... Security was not given much attention in its original design, and its use for purposes for which it was not initially intended ... has revealed exploitable vulnerabilities ... And although the Web is an important element of people’s work lives, it is not as critical as a ULS ... system would be.

Now I think we're in a good place to talk about the health care system of the future...


Health Services as a ULS

Let's start this section with a quote from the presentation that inspired it. Richard Gabriel says:

An example of a ULS system (that doesn’t yet exist) would be a healthcare system that integrates not only all medical records, procedures, and institutions, but also gathers information about individual people continuously, monitoring their health and making recommendations about how to improve it or keep it good. Medical researchers would be hooked into the system, so that whenever new knowledge appeared, it could be applied to all who might need it. Imagining this system, though, requires also imagining that it is protected from the adversaries of governmental and commercial spying / abuse.

Modern hospitals are packed with countless computing devices: everything from charting PDAs to physiological monitors for patients; from mainframes and patient record data warehouses, to terminals and desktops. Wireless medical sensors have already been developed by a research project at Harvard. What's more, despite the concerns over associated health risks, implant research at Johns Hopkins and the University of Maryland is on-going and may produce results that are one day standard practice in hospitals.

As versions of theses decives are developed that produce no ill effects for humans, they will make their way into out-patient clinics, assited living facilities, and ulitmately HMO's, private practices, and our homes. The devices will grow in numbers, shrink in size, and provide more functionality at greater efficiency than their predecessors.

The volumes of information that will be exhanged between devices, analyzed and correlated by other devices, and consumed by end-users, doctors, and researchers will be mind-boggling. It will bring new insights on everything from personal health to epidemiology.

With this, though, will come the obvious need for security and privacy, for defense against information attack and denial of service. These devices will all have to dedicate compuational and storage resources for use by the whole system. Part of the system will have to monitor itself, properly escalate problems, observe and anticipate trends. Protection and defense capabilities will have to exist the likes of which barely exist in our every day lives at the marcoscopic level.

All of this will take time. They will truly be modern wonders of the world. Given that such systems are anticipated to exist sometime in the next 20 years, and will have accreted the component systems over time, where might such a thing start?


Google and a Health Care ULS

If you read my last post (which I think was posted to blogger before the official announcement by Google), you already knew what I was going to say :-) Google Health. Though obviously nowhere near a ULS system in and of itself, why might we suggest Google is moving in this direction?

Here are some interesting bullet points from google.org:

• InSTEDD: $5,000,000 multi-year grant to establish this nonprofit organization focused on improving early detection, preparedness, and response capabilities for global health threats and humanitarian crises
• Global Health and Security Initiative: $2,500,000 multi-year grant to strengthen national and sub-regional disease surveillance systems in the Mekong Basin area (Thailand, Vietnam, Cambodia, Lao PDR, Myanmar, and China-Yunnan province)
• Clark University for Clark Labs: $617,457 to Clark University, with equal funding from the Gordon and Betty Moore Foundation, to support the development of a system to improve monitoring, analysis and prediction of the impacts of climate variability and change on ecosystems, food, and health in Africa and the Amazon
• HealthMap: $450,000 multi-year grant to conduct in-depth research into the use of online data sources for disease surveillance

Does that sound familiar to anyone besides me? All paranoia-induced sinister thoughts and Google Ads jokes aside, it makes sense that this is where we're going with health. In fact, it makes sense this is where we're going with all of our lives. If data privacy, personal ownership of that data, and security concerns can all be addressed, our lives' information will be better served by moving through systems specially designed to provide maximal use of that information with the least work. It won't just be nice to have, it will be essential.

The balance of power, from individuals all the way to the top of whatever organizations exist in the future will rest in information. Not like it is today, however. The "information economy" of the today (+/- 10 years) will look like kids' games and playgrounds. The information economies this will evolve into will be so completely integrated into human existence that they will resemble the basic necessities like water and food.

If you could find yourself a corner of that market, 20 years before everyone else got there, wouldn't that be a smart business move?


Summary

Our world is changing much more than we realize. We're too tied up in our jobs and gas prices to see the larger picture... to see that our future is already being made, that even in our unconscious actions we are propagating it no less than the cells in our bodies conspire to propagate what will become our children.

In the same way that hominid nomadic/migratory patterns begat the distribution of villages and tribal communities, which in turn gave birth to civilization, our silly little Internet will one day have descendants that dwarf it in size, utility, complexity, and computational power. The amazing thing is that we are the ones that actually get to build them!

There is a lot to research, and just as much to prototype. There is a project for everyone, and by starting now, we can make sure that feudal lords of tomorrow don't have absolute control over our food and water. If you have ideas for collaboration, start talking! Get involved! If you have money, fund some research, sponsor some conferences. In simply writing this blog post, I have uncovered gobs of new research I didn't know was out there. We should all be reading more, catching up, and coding. The projects near and dear to our hearts can get a whole new life within the context of ULS systems.

Required Reading: Ultra Large-Scale Systems

At Divmod, we're always talking about the future of computing, software, and the network. This usually focuses on our work with Twisted or the Divmod platform. But we have also spent considerable time assessing research in the area of what is called "ultra large-scale systems." Our primary business interest with this revolves around development, deployment, and management. However, there is a great deal of work that needs to be done to make ULS systems a reality.

I have a series of blog posts planned to discuss ULS systems in the areas where I have a vested interest. Despite the fact that a popular study on the matter was funded by the U.S. Department of Defense (conducted by the Carnegie Mellon Software Engineering Institute), I will not be discussing this technology in the context of war efforts nor national defense. Instead, I will be engaging in this discussion within the context of the medical/health services field, per the example given by Richard Gabriel in his presentation (PDF) to the Chinese University of Hong Kong.

This is especially pertinent today, as Google prepares for its Google Health announcement. Even though there are by some estimates (optimistic ones, in my view) 25 years of research ahead of us, it is generally agreed that ULS systems will consist aggregate sub-systems, built incrementally over time. I don't believe anyone is under the illusion that Google does not want to produce the first ULS system in history, and they are making rapid progress towards this goal. Google Health brings this point home very clearly, in the context of the research that has been done in this area.

The world is rapidly changing. The most important issues in technology and business are not who is going to create the next catchy "Web 2.0" application, or what mega corp games are being played in the Great Silicon Valley Soap Opera. The important issues are how the systems of the next 100-200 years are being built now, who they are being built by, and who as access to that technology.

We are a powerful and creative community. We are concerned about distributing power to the people, education, privacy, and freedom -- open source is built upon these principles. If we want those who are building ULS systems to build them fairly and with our concerns in mind, then we must get involved now. We must start building the necessary tools.

As for the reading, a couple of these were linked to in this blog post. Here is a list that should get your brain revved up and ready to roll (these are all PDF files):

• Design Beyond Human Abilities
• Ultra-Large-Scale Systems: The Software Challenge of the Future
• Scale Changes Everything
• ULS Systems Research Roadmap

I will have more specifics about the sorts of things we're exploring in future blog posts, but until then -- don't get left behind!

Update: Allen Short just pointed out some additional material that predates the Carnegie Mellon research: the Agoric Papers, co-authored by Mark Miller who is a co-creator of the E programming language.

Update 2: Glyph Lefkowitz just sent me a link to the Big Ball of Mud that discusses similar concepts.

Technorati Tags: technology, business, community , open source, uls

The King is dead! Long live the Kinglets!

So what the hell does that mean? Well, there's sort of a back story there (but you can skip to the last paragraph of this post for the punchline).

Earlier this year, I blogged about the language patterns of twisted: Thinking in Twisted. The use of and relationships between such things as protocols, clients, servers, and various factories in Twisted was really starting to make its way into my bones. These and other core aspects of Twisted best practices form the basis of a pattern language... one that is of immense value when writing network code and applications. Here are a couple quotes from that blog entry:

The tricky thing about Twisted is that its patterns are an integral part of its "language" (perhaps it would be better to stop using quotes, and just call it a meta-language...). This is not true of most other languages -- though I would venture to say it is true of good frameworks in general. You don't need to know the patterns to use the grammar/syntax/control structures of most languages. Patterns in a language are usually optional. [However, w]ith Twisted, if you aren't using the patterns, you aren't really using Twisted.

When we learn to think in Twisted and write applications in Twisted, our solutions will be elegant and compact. Before that starts to happen, we try to use our "old" ways of python programming mixed in with the new one we are learning... and that, of course, defeats the purpose of the new "language."

I've recently hit another level of awareness in pattern abstraction, and it has its roots in Twisted. In fact, I never would have been working on what I am now working on if it weren't for Twisted and all the work done by Divmod and its crew. While at PyCon 2004, I met with Amir and Glyph and we discussed twisted's use in the industry at large, and in particular, opportunities for which the Divmod product line could be applicable.

What I liked so much (and immediately) about what Divmod was doing was that they were integrating useful components into a single application (Quotient). It stuck a positive chord with my instincts about where software should be going, and it kept me thinking for months while I worked on other projects. These thoughts culminated in a series of blog entries and software projects. First, a post called Dinosaurs and Mammals explored this. In the intervening time, the projects I have worked on have helped me evolve these ideas usefully. There is now direction, code, and product (more about that later in this post).

Then, in Everything as a Distributed Resource, I took a more sci-fi, radical view... but one that is born out by current trends and tendencies, particularly in the cell phone market. There's a great deal we haven't thought about in regards to the cell phone market. Not so much about cell phones and mobile technology, but rather the usage patterns and demands that are being made and (sometimes) met in this sector.

In an effort to follow through on the "mammal" paradigm, I did some exploration with URLdb and PyKon which I blogged briefly about in New Projects. The two projects mentioned there have since been merged into a project that we are calling Adytum's Twisted Collection. The latest addition to which is ImageDB.

ImageDB is a repository for static graphic media. The images that people upload here can be viewed via it's web interface, but it is more likely the ImageDB would be used as a part of something else. For example, when I get around to finishing PyKon, it will make use of
ImageDB. PyKon will not store any images itself. It's simply going to be an nicely designed, usable image gallery interface. Users won't know that, though. They will use PyKon and be able to create galleries and albums in that gallery by uploading images (which will be stored --
transparently to the user -- in an ImageDB instance), by selecting individual images (JavaScript ImageDB selector/browser), or by selecting "tags" a la del.icio.us (and therefore whole groups of
images).

Analogous to ImageDB is MessageDB, a repository for static text. It's next on the list to be developed. This micro-app will hardly ever be used directly, although it too will provide a basic web interface for viewing and editing stored content. MessageDB will be a storage app for RFC 2822-compliant "messages". For starters, various public mail list emails will get tagged and stored there.

Non-dynamic text for web sites and blogs will be stored as RFC 2822 messages. So the forth-coming tools "SimpleSite" and "WeBlog" (also part of the collection) will actually store and pull their content from MessageDB. But even more, when you want to use a photo, clipart, or logo in your site or on your blog, you will click a "select image" link and an image thumbnail browser will pop up. You will select one of the thumbnails, the popup will go away, and a link to that resource in your ImageDB instance will be inserted for you, in your blog or site text.

The idea is that by abstracting out general functionality into little micro-apps, you not only decrease the amount of work in maintenance and feature addition, you not only are reducing the amount of redundant work you have to do when building content-related applications, but you are providing an infinite amount of flexibility for using these various media in completely unplanned and unexpected ways.

Your media, it's storage, it's access (XML-RPC), now live in self-contained little units. They can "survive" on their own, without a heavy dependence upon some huge CMS that was written for a particular project. The apps are project-agnostic. They can be set up once and then reused repeatedly in different contexts by building new apps that obtain their data from the component micro-apps via simple links and/or XML-RPC calls.

And, again, the ways in which users and designers decide to combine these pieces and how they decide to use that is 100% flexible and thus fertile ground for the unexpected and creative :-)

I say "To Hell!" with large apps, and banish them forever! Let's build tiny, little, single-purpose apps once! And then for larger needs, simply glue together various micro-apps as needed per the project requirements.

As far as I'm concerned, the old King of the Bloated Application is dead. It's been replaced with tons of tiny, decentralized micro-apps. Long live the Kinglets!

Everything as a Distributed Resource

In my blog entry Dinosaurs and Mamals, I mentioned small application servers that were capable of being highly distrubuted. In conversations on this topic with friends, we discussed various scenarios where such things could have an impact on our daily lives. For instance, I'm sure that many people have heard about the potato powered web server joke and then the real thing that it inspired. This demonstrates the possibility of low-power servers; conjoin that with wareable computing, and mammal application servers, and you have a whole new world...

There was this great game I used to play called "Alpha Centuari" -- the best part was choosing the research your culture would engage in. I find a similar "rush" from the communities one can join in the Orkut online friends network. I know lots of people that love that kind of thing, too... possibly pointing to something inherent in human nature about social groups, hobbies, aquisition, or just bright, shiney objects.

Even now, people have taken this one step closer to "reality": from simple association, to actual contribution. We may not all be able to get PhDs in astrophysics or molecular biology and do research that actively contributes to the human body of knowledge, but we can all run client applications on our workstations that borrow idle CPU cycles to work on problems in which we are interested.

So what happens when web servers can fit in the colar of a shirt, when shoes pumps generate electricity as you walk, when fibres woven throughout your clothes act as wireless antenae? What happens when processing power is literally everywhere and highly moble? Universities, companies, governments, special interest groups, neighborhoods, etc., can all lobby you for your computing resources. They don't have to ask you for your time or your money; you don't have to change the way you do things or when you do them. All you have to do is choose to associate with a group and then give them permission to use your cycles. They get to use CPUs that they don't have to power, insure, air condition, troubleshoot, network, maintain, etc. There is a potentially HUGE amount of money that can be saved for each organization.

But with increased mobility and with large numbers of projects to which you can contribute your unused cycles, comes such ubiquitous evils as marketing, advertising, lobbying, etc. If a company can influence you to change your mind and go with them instead of their competitor, then they can do it with thousands of other potential contributors as well. As quickly as you see a "give-us-your-spare-cycles" advertisement or have a conversation with a friend, you can change your mind and give to someone else: the CPU isn't waiting at home, it's there with you.

The consumer becomes a commodity broker; organizations, the consumer. Decisions are immeidate;
impact on a company's R &D is instantaneous. Profits are now. All of a sudden, the entire technology-enabled population is a rich philanthropist and every organization out there is hoping to get your "funding," hoping you'll let them add one more CPU to the thousands already volunteered. This could give birth to a whole new "processing"economy...

Evolving Networks

Note: This is from an email I had with Itamar Shtull-Trauring, who was kind enough to let me blather for a bit...

There was a very interesting article in this year's April issue of Scientific American. It basically discussed a non-neural network in the brain, its own complete entity, but one that is connected to the neural network we know and love. It's a slower network that seems to provide feedback and additional processing/learning power. It uses mechanics and chemistry to communicate along its own network and respond/react to the neural one.It's a great article -- I recommend it (I have severely abused it with that synopsis above).

It got me to thinking... first: what if the original pre-brains were cells that communicated electrically and chemically, but with no neural network capabilities? What if the original multicellular ancestor of the brain was just a simple highly-connected, bi-directional network? What if the chemical and neurological networks grew in tandem and developed together, with the chemical network acting as a "meta-network," providing feedback? And, to really stretch things, what if this feedback is what caused the electrical network to develop into a neural network?Now I'm sure you see where I am going with this and how fruity it is...

Like primitive primordial cellular organisms, our society also has a primitive electrical network: the Internet. How can we develop a functional analog to the biological system (the biological system being proven to work and being insanely efficient... for a neural network...)?

Okay, so that's the visionary's question; no need for reasons or justification -- can we do it? Then let's make it! But why would we want to? Do we really want to turn the insanity of the Internet into something resembling a neural network? Perhaps, but that's not the goal; the ultimate goals might be something like these:

• We know the Internet is slow; how can we make it fast? efficient? (fastest paths, memory/learning, ...)
• We know that the Internet is filled with hostiles (spam, viruses, worms, etc.) - how can we safeguard this?
• There will always be something new that the Internet hasn't seen before - how can we integrate anticipation?

Could we create a virtual separate network inside the internet? Could it use a completely different protocol than anything else in existence? Could it be very simple, very specialized, and very secure? Could it provide protection services? Or perhaps act as some kind of feedback mechanism for Internet devices? A subscription service with intelligence?
This leads to other questions I'm sure people a lot smarter than me have already examined: can we use nature as an analog for developing network communications into something more mature?

• How do complicated organic networks regulate themselves?
• How do they protect themselves from violence?
• How do they self-organize? adapt?
• Are there ways in which we could model these behaviors with special designs (protocols, applications, services)?

So that's it, nothing with much technical content... just some ideas floating around...

Dinosaurs and Mammals

Large software corporations like Microsoft and Oracle in addition to groups of software developers like elements in the Open Source community are giving the world new versions of the same thing: monolithic software for a world that no longer needs or wants such products.

Take for example Microsoft Exchange Server: this is a heavy-duty, if often ridiculed, system of software. This software was meant to handle hundreds to thousands of users and their information. What about the company that wants a Microsoft solution but only has 50 users? They have to pay for Exchange. Yeah, you can get a great discount, etc., but I'm not talking about price -- I'm talking about need. And designing software for need.

Oracle is more guilty of this sin. Let's say I need to have my data highly distributed on remote machines, but that I don't have tons of data and I don't need the horsepower Oracle can provide. Yet, if I use their product or any other large RDBMS like PostgreSQL, SQL Server, or even MySQL, I am using the wrong tool for the job.

Another example: Apache is the world's most popular Web server, but let me commit heresy and ask "Why?" How many people need it's speed? Or its capacity to handle millions of requests? Think of the hundreds of thousands of web sites out there that are visited once a day or once a month. Using something like apache for these is engineering overkill.

Microsoft and Oracle as well as organizations like the Apache Foundation, MySQL, and PostgreSQL are producing software dinosaurs. The have lived, they have thrived, they have helped define our information universe. That universe is changing. With the introduction of programming languages and development processes that foster rapid development, for the first time in the history of modern technology, we are in a place to design enterprise software around the needs of the end user (individulas and organizations), not around the needs of licensing structures and coporate philosophies. Imagine that the motivation behind software development wasn't market domination, but rather market diversity. The right tool for the right job.

I haven't counted the lines of code in the apache web server, but let's just say it's 100,000 lines. What if I only need to serve a couple pages per second and a max of 100 concurrent sessions? What if I could use Web server software that could provide what I need in only 200 lines of code? 100? 6? Why should I use Apache for projects that don't require Apache's strengths? The same thing for my email server and calendar application. What if I could do the same thing for all my networked applications? What if in addition to being small, this software was highly distributable as well? What if I could build office applications perfectly suited for a firm of 40 employees, and they didn't have to pay for licensing of the cliched software products? In addition, as the firm grew, this software could support 100, 1,000, or 100,000 users just as easily simply by adding the required light-weight components.

Software like this is easier to setup and maintain. It's easier to get involved with and understand. It's easier to modify, add to, customize, and enhance. It's easier to provide your firm with what they need.
Enter the new software and application engineering model: the mammal. This is where we are heading. Imgine application servers so light-weight they could run on a 9-volt battery for a week. Imagine them embedded in hundreds of devices all over the country or even the globe... so cheap and distributed, it didn't matter if one went down or even if twenty of them went down. They are quick, nimble mammals, not lumbering dinosaurs.

There are already software developers working on projects with views and goals along these lines. Watch for them: they are small and furry. They move quickly and quietly. Pay attention to them; they will one day rule the world.