Physical Beings with Digital Lives

There's a lot that one could say about that title. In fact, it could be the title of a high-volume collaborative blog... That aside, here's the context for this post: books. Books and Reality. And data.
This post got so long that I now need to add a list of sections here, just to make it more accessible. My apologies :-/

Mini Table of Contents

• Books
• Books in the Sky
• Yeah, I Know: Go Social
• Human Data History
• Reality Merges
• Conclusion

Books

I have tons of books. Actual, physical books. Walls of them. Some I use all the time (reference). Some I read once a year (good books that support multiple reads). Others I've only read once, perhaps as far back as high school (when I started collecting). My bookshelves are like a random associative memory array: reading each title or the act of pulling one from the shelf brings back a flood of memories, relived experiences, sometimes actual sense perceptions. It's a powerfully visceral activity.

But that's just my books. When I'm at friends' homes or offices, I cannot keep my eyes off their bookshelves. It's an irresistible compulsion. I linger and browse, often past any semblance of socially acceptable time limits. My conversational replies experience a rapid exponential die-off -- in duration, gaps, and semantic value -- culminating in grunts and finally silence. (My favorite offices to visit so far? friend mathematicians/maths professors!)


Books in the Sky

Oddly, I love books in digital format. I never really got hung up on the bit about not having the paper entity in my hands (though I have turned a digital book reader over, expecting the next page... though that was deep in the plot of a Greg Egan novel!). Traveling as much as I do, I'm in heaven with ebooks. I feel like Superman, carrying around a library with me everywhere I go.

But when I passed a bookshelf the other day on my way out the door and fell under the spell of a book-memory flashback, I realized what was going away as I transitioned to virtual books. And the painful question arose: How am I going to nurture future layers of book-mulch and text-humus with this new æthereal, cloud-bound library I'm building?

How can I share with others, my stacks of books? How will I browse friends' books in their offices, asking about author X and title Y? How will we borrow from each other? What can be done to add this and related missing richness back into our lives once we adopt the virtual versions?

Light-emitting walls that can display titles from your Amazon account? Virtual over-lays visible with wearable/immersive computing accessories? Whatever we end up with, a gimmick isn't going to cut it. It will need to reflect the same depth of history that stacks of physical books have come to represent to us and the collective human psyche since we first started gathers works of the written word.


Yeah, I Know: Go Social

I'm hung up on books here, I admit it. But the same goes equally well for much of what we experience in online social media as well. Everyone's trying to make a buck on people chatting, playing games, reading, etc. Business as usual.

But the problem is that everyone coming up with their own little solution, one piece at a time. Google, Facebook, LinkedIn, Last.fm, etc. "We socialize X." Wow. Good for you. Now, for every activity or group I'm interested in, I've got some tiny little corner of the internet that I need to pay attention to.
Right.

Maybe I'm just an online social idiot, but this isn't working for me. Too many places and pieces. The physical analogy would be me spending all day on the road, hitting all the social hot spots in the Colorado Front Range. Ain't gonna happen. Ever.


Human Data History

The social data scene is a big stick up my butt. I really don't like it there and I'd love to get rid of it. It's poorly engineered, primitive state makes me grumpy. I don't own a thousand hammers [1]; I don't want a thousand of anything that all do basically the same thing [2]. Most of us probably don't own hundreds of houses, either (for ourselves, that is). We keep most of our stuff in the same location or two.

Speaking of houses, let's talk about settlement. How did we choose where to set up camp, towns, etc.? Trade routes, availability of resources (direct physical presence or presence by virtue of trade routes). Are we doing that now on the internet? Are we looking at the analog to fertile valleys, productive rivers, and protected harbors? Whose priorities do we have in mind? As we set up virtual presences, are we in locations that benefit businesses? Or ourselves?

If we choose the latter, the businesses will come because the people are there. If we do it the other way around, we'll be looking for new virtual homes if the businesses close shop or change the rules too much.

Coming back to data (but on the same anthropological note), historically we've had distinct divisions of our data:

• the secure location of our huts/houses/castles
• what we presented about ourselves in adornment/fashion (public data)
• what we could carry with us in bags/crates/vehicles

Because of their prevalence in our history, any attempt at realizing personal data in a virtual environment would do well to reflect on these. We're naturally already predisposed to such approaches; such divisions are things that anyone can intuitively grasp.

The problem is that we're all used to a single platform: our mutually agreed-upon reality. There's no such thing online yet. And if there were, who would own/run it? Monopolies are eventually overthrown. We hate them. So how do we get around this?


Reality Merges

This very naturally led to thoughts on digital lives in general. And this is more than just a question of usability or human-computer interaction. Rather, this is a question that borders on the metaphysical: how do we solve the problem of syncing divergent realities? Reality-reality interaction.

The problem shouldn't be minimalized by analogy: this isn't a "simple" matter of ensuring that the data in my address book on Google is the same as what's on my iPhone. My self-perception, many reminders of self-reflection, etc., take place as a result of various interactions I have with my surroundings: both real[3] and virtual. No problem. Except that the things that remind me are also things that others can see and interact with as well. Often, they will have associations that spark a neural cascade for them too.

I've had many conversations take place around objects in a shared environment where the name of the object was never mentioned, it was implicitly understood. When there's no shared object (or concept), we have to name the object, define it, share some basic associations, make sure that we're talking about the same thing, etc. Thats all prelude. Only with that done, can we have genuine communication take place around the given concept.

Now rinse and repeat for everything you want to talk about that revolves around or is at least related to something that exists virtually for you and isn't part of your shared, physical environment.
I can't imagine many useful general solutions to this. In fact, I can only imagine one (given our biological wiring): use what we know (in our bones) and overlay or augment our visual reality with another.

With augmented shared realities, there's no platform. You just need hardware that runs it and senses that can perceive it. Just like reality. At that point, we can start sharing what we want, allowing access to data about ourselves and what we like by dumping it into a shared perceptual space, regardless of the original data source. Merged.

Obviously, we're not there yet. We're going to need crazy improvements in mobile technology, storage, computer vision, etc. But once the technology catches up, I think we'll see some powerful needs being filled. And we might start coming out of the internet dark ages...


Conclusion

None of this is new; Pick up any number of books by Charlie Stross[4], and there's all sorts of fun to be had by exploring his ideas. But the point of this post wasn't to be new. While we're all busy enjoying the latest fad in social media, I think it's important we think about where the progressive succession of fads is taking us. At each point, there's a natural next step (more accurately, set of possible next steps). Let's look more than just one in front of us and let's not forget what our biology has made us. We may not be able to engineer truly wise decisions about our future, or even make our lives better/more efficient. But it would be nice if we could at least not make things worse :-)


Footnotes

[1] I think I have three, actually.
[2] This is one of the reasons I'm a big fan of http://ping.fm.
[3] Here I mean "real" in the "conventional" (shared) reality sense of Mahdyamikas. Ultimate reality... well, that's a topic for an entirely different sort of post...
[4] Check out his Amazon page: http://www.amazon.com/Charles-Stross/e/B001H6IW0Q/. Accelerando is probably his most praised book (and likely my favorite), but the ideas touched on in this blog post are explored in other works of his, most notably Glasshouse and Halting State.

The Future of Personal Data: A Followup

The Next Step

A few years ago, I wrote a post about the Future of Personal Data as a result of all the ultra large-scale systems reading and exploration I was doing. Google was foremost in my mind when writing that, but Apple has since come into the spotlight here as well.

Recently, Matt Zimmerman has decided to leave Canonical and join forces with Singly, an exciting startup company focused on secure user data storage and the socialization of (and development around) that data. In this blog post, the following core values were given about the software underlying Singly:

• I own my personal data
• I want my data to be useful to me
• I make the decisions to protect or share my

I would like to see the following added:

• I make the decisions on how my data is used
• If my data is sold, I should get a return for this

This is not petulance speaking :-) This comes from a historical perspective on social and economic fairness. Witness the changes in individual rights and personal finance since the industrial revolution...

Fair Market Value

There are probably many solvent entities out there who would claim users are already getting a return for the use of their data: free email and office docs from Google, free cloud services from Apple, etc. But I would imagine that there are massive margins being made on user data, and services (as valuable as they may be) are a paltry return for such a gold mine. I cannot help but be reminded of the selling price of Manhattan Island or Alaska (in 2011 values, the Lenape Indians got around $29.61/square mile; the Russians got about $150.77/square mile).

Money makes a good point, but personal data (and this post) isn't about the almighty coin. This is about clearly defining who owns what and ensuring that those who don't want to be taken advantage of, aren't. This is about identifying exploitation, and building something better and longer-lasting in its place.

Who's Going to Pay for What?

In David Pakman's "Disruption" blog post about Singly, he makes the following comment:

"I cannot see consumers getting into the business of selling their data to marketers so they can see personalized advertising. Instead, I believe marketers will be encouraged to offer value to us in exchange for access to our data."

I have to agree with him... but I can only offer a qualified agreement. True, I find it hard to believe that users will be selling their data directly to marketers. From the user-side, the pain of inconvenience would not likely be worth the payoff; at the marketer end, individual data is useless, and munging an in-house-built collection would incur a lot of overhead not part of their core business.

However, users' data stored in lockers, updated regularly, pre-processed, has enormous value in the market. Right now, Apple and Google are making eye-crossing amounts of money from data just like this. Again, I would imagine that this data is only really valuable in large quantities and for interesting, identifiable demographics

If users provide their data, but in exchange only get a "nicer app" or a "useful utility" I'm going to cry "foul!" (unless someone can show me the actual numbers involved and unequivocally prove that fair exchange is occurring).

You Say Disruption, I Say Revolution

Instead, if a service such as Singly, were to offer a co-op style dividend payment system to all of its users, that would seem to be much more fair. Not only that, it would be the beginning of a market revolution. This is not to say that co-ops are some perfect economic model, but rather that the data we, as users, generate is of immense value. The more that Singly has, the greater potential for revenue. The more buzz that builds around Singly users generating revenue from their data, the more users they get. With mass-adoption, a new sub-economy is born.

Perhaps a better model than co-op is that of a mutual fund investment firm. Each Singly user has a portfolio of data. Depending upon each user's preferences, any or all of that data could be used by Singly to generate revenue. Some groups of users will generate more than others, and users in these groups would get greater returns.

Whichever analogy you prefer, with a little exploration it seems fairly clear that opportunities for a large payout are present. For instance, I like to imagine a world where entities like Apple and Google can't harvest user data, but must go through brokers whom users have given their permission to sell their data for the most profit. I also imagine there's a lot of lawmaking that would have to take place... and even more lobbying.

Even with that, Google and Apple would still make money hand over fist (or they'd become brokers themselves) -- enough to continue providing free services. Yet at the same time, users would be in control of their data; they'd be financing (or financially augmenting) their data-consumption lives with said data.

With the right press coverage, Singly could find themselves not only swamped with a massively growing user base, but at the very center of a new economy. With the right level of negotiation and coordination, businesses could buy into this new paradigm without losing their shirts in the disruption.

A Plea

In summary, I applaud the goals and vision of Singly. I, for one, would deeply appreciate writing applications against their data locker (to any Facebook or any of its dubious applications), where a user's rights are clear and protected. That being said, Sinlgy would have my eternal allegiance if they also took up the cause of rights for user data in the market place; if they helped transform the current nascent data economy into a world economy capable of achieving as-yet unimagined financial heights.

And if not Singly, my loyalty would be given to whomever did do this.

txSpore: Twisted Spore

I just had a delightful weekend of coding :-) I spent the past two days porting the Spore Python API to Twisted. You can now incorporate Spore data (from static XML as well as REST requests) into your non-blocking Python applications/games!

This was a pretty easy task, really. The API just makes HTTP requests with twisted.web.client.getPage. There was a little bit of work involved in creating object models for the XML, and some head-scratching for the error-catching deferToThread unit test I tried to write (it's still buggy... need to figure that one out). Everything else was pretty much cake.

In fact, it was so much fun to kick back and write some playful code that I might overhaul the sync Python code as well and incorporate both into txSpore.

Do be aware, however, that the code still has some big improvements coming. The first thing I want to hit is actually create a client object. Right now, the client module contains a series of functions (since state's not currently needed). However, I want to start doing some basic object caching in order to limit the number of requests made to spore.com and increase the response time. That's the big item for 0.0.2. Update: 0.0.2 is now released!

Next I'd like to create some more demo apps that show off the API usage better. Right now, there's one demo (a .tac file). All it does is ask for a user name, renders a user page, and then links to a user "Spore assets" page (that's the thumbnail image above).

One thing that might be fun to do is write a script that checks for the latest achievements and publishes them to various microblog/status sites with the Twisted PyngFM client :-)

There's a project page up on Launchpad for txSpore, and I've posted a notice and some updates to the Spore developer forums. It's also been published on PyPI.

Enjoy!

After the Cloud: So Far

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

Systems Engineering in a Box

The recent redefinition of "the cloud" as a service and commodity is a brilliant bit of frugal resource management (making use of idle resources in an expensive data center) coupled with flawless marketing. Yes, from a business perspective, that's an amazing coup. But it's the 30,000 foot technical perspective that really impresses me:

In the same way that software frameworks, their libraries, and best practices have, through the trials of last 40 tears, productized application engineering, the cloud has started to experience something similar. What everyone is now calling the cloud is really the productization of systems engineering.

Systems engineering (and the management of related resources) has proven to be an expensive, time-consuming endeavor best left to the experts. Sadly, those that need it are often in the unenviable position of having to determine who the experts are without having the proper background to do so effectively. When the planning, building, and management of large systems works well, it's a labor of sweat and blood. When it doesn't, it's the same thing, with a nightmare tinge about the whole thing coupled with an odd time-dilation effect.

It seems that in applicable circumstances, some businesses are spared that nightmare by using a cloud service or product.

Bionic CGI

As someone with a long history and interest in application development, I was particularly keen on Google App Engine when it came out. This was a different take on the cloud, one that Mosso also seems to be embracing: upload an application that is capable of having it's data access and views distributed/load balanced across multiple systems (virtual or otherwise).

This is essentially CGI's grandchild. You have an application that needs to be started up by any number of machines in response to demand. A CGI app in Mosso will probably need very few (if any) adjustments required in order to run "in the cloud." Google is a special case, since developers are using custom, black-box infrastructure built by Google (for insights into this, check out these papers), but I'd be willing to bet someone lunch that there is room for a CGI analogy at some level of Google App Engine.I guess with Google, we kind of have both application and systems engineering in a box, in so far as the systems support your application.

At any rate, it's CGI better than it was before. Better, stronger, faster.

The Rub

However fascinating these cloud offerings may be, I find myself not getting what I need. As a developer of Twisted applications, I'm interested in small apps. Hell, I don't even like running databases and full-blown web servers. A while ago, I spent a couple years working on some Twisted-based application components that could be run as independent services (thus load-balanceable) and completely replace the standard web server + database + lots of code routine for application development.

So what about developers out there like me, who want to run tiny apps? We don't need "classic" web hosting, nor CGI in the cloud, nor cloud-virutualized versions of large machines.

As a segment of the population, business consideration for developers such as myself might seem like a waste of time. But before dismissing us, consider this:

1. Exploring small niche's like this one often lead to interesting revelations.
2. Market segments that have proven quite vibrant may be able to expand into even greater territories (e.g., the iPhone apps phenomena).

Next up: Tiny > *

A Champion for Services Architecture Done Right

I've heard a series of discussions lately and have read some articles that I've found rather frustrating, all on the topic of SOA. I've taken a few notes about my objections and had several blog posts planned where I would attempt to articulate the historical failures of software, services, and/or architecture makeovers performed by any number of consulting firms for large IT shops. I was going to discuss the horrible irresponsibility of basing business decisions on buzzword flinging cowboys, and the inevitable price that is paid by companies wooed and seduced by flash.

I was going to start this off by launching a strong criticism of Anne Thomas Manes' views on the "death of SOA", one that I'd read about in a recent article. It seemed that such a view was sensationalist and entirely missed the point of the essence of service oriented architectures.

But before I did that, though, I needed to do some research and present a solid argument, not just take some trade rag's version of the story with a superficial smattering of misquoted details. And holy cow... not only am I glad that I did this, but I am now an ardent fan of Ms. Manes, Vice President and Research Director at the Burton Group.

Her post SOA is Dead; Long Live Services is brilliant. The death of which she speaks is not the mortal (and figurative) soul of SOA itself -- any individual or magazine claiming as much has either completely misunderstood her or has been making its money on the misunderstandings and hype that she would like to see die. She asserts that the very heart of SOA continues to be badly needed, that "the requirement for service-oriented architecture is stronger than ever." She then goes on to say:

The acronym got in the way. People forgot what SOA stands for. They were too wrapped up in silly technology debates (e.g., “what’s the best ESB?” or “WS-* vs. REST”), and they missed the important stuff: architecture and services.

And finishes with this bit of excellence:

The latest shiny new technology will not make things better. Incremental integration projects will not lead to significantly reduced costs and increased agility. If you want spectacular gains, then you need to make a spectacular commitment to change.

And you know what? Her other posts are even better. While reading all of them, I was completely dumbfounded that she said so well and clearly what the industry has been misinterpreting for several years. Why are more people not listening to her? How did I not hear of her until now?! I'll tell you what, I'm now subscribed to Burton Group's Application Platform Strategies blog.

IT shops need to pay close attention to what this woman and her team have to say. She really knows what she's talking about and articulates her views exquisitely. That whole series of blog post I wanted to make? Yeah... she already did that :-) (and way, way better than I could have). Please, I beg you: go read her posts. If you are the director of your IT shop or someone to whom your director listens and respects and you have any interest in distributed services, messaging, and related architectures, go memorize her statements, summaries, arguments and positions. Let her experience and wealth of knowledge guide judicial and wise decisions at your company.

And remember, it is the good, well thought out, and cost-effective architecture that survives. It is this architecture -- with countless peers -- that serve as the basis for the next generation of improvements and advances. Technologies produced in this manner are the ones that propel us into the future and prepare us for a world which we are incapable of imagining or implementing today.

The Future of Personal Data

In a recent post about ULS systems, I said this:

The balance of power, from individuals all the way to the top of whatever organizations exist in the future will rest in information. Not like it is today, however. The "information economy" of today (+/- 10 years) will look like kids' games and playgrounds. The information economies this will evolve into will be so completely integrated into human existence that they will resemble the basic necessities like water and food.

I'm not going to focus on the ULS systems topic in this post, but there is a very deep connection between privacy, personal data and all things ULS. Any thoughts of a ULS system should be coupled with how this will impact the system's users and their data. Any thought of our personal data's future existence should include the anticipated future of computing: ULS systems.


Inside and Out

In a nutshell, here's how things look:

• Yesterday: Paid Services - You want something, you buy it. Demographic research is expensive and mostly outsourced.
• Today: Free Services - You want something, companies give it to you for free... in exchange for your demographic data.
• Tomorrow: Information Economy - You want something, you leverage the value of your information in brokering the the service deals that mean the most to you.

What do we have right now? Companies are fighting for each other over who gets to have our data for free. Yay, free stuff! We used to have to pay for that sort of thing! But paying for people to hold your data was the old, old world. Having them do it for free is the old world. Here's the new world: They pay you.

Why would they do that? Why would things shift from the current status quo? The value of personal information.

There are many ways to assess the value of personal information, but let's look at a few from the perspective of large organizations (entailing everything from government to business). Simplistically, we can assign value to a single individual's data based on the value of a large collection of many individuals' data. The more participants, the greater the value of the whole, and therefore the greater the value for each individual's data. This perspective is limited because it treats data very staticly. The data may change, but in relation to the system it's "acquired" and inside as opposed to "for sale" and outside.


We Are the Markets

But the value of our data is not defined simply by the presense of bits or membership in a valued data conglomerate. Our data is not just our emails, our medical records, our purchasing trends, nor our opinions about local and national politics. Like an organism moving through an ecosystem, our data is dynamic and living; it is the very trace we leave in the world around us, be it digital or otherwise.

Any part of our lives that is ever recorded in "the system" provides data and comprises part of our movements through this system. Our traces through this digital ecosystem impact it, change it, shape its future direction. The collective behaviours (not just collective data) are immensely valuable to organizations. Their value is on-going and growing, with accrued, compounded interest.

Static data bits seem like property to us: you can buy them, you can sell them, you can store them somewhere. But moving, living data... that's a different story. That's not a buy-once commodity; ownership of that might be tantamount to slavery in a future, information-based economy. However, organizations might opt to lease it, or individuals might turn the past back on the future and offer license agreements to organizations.

More likely, though, individuals will form co-ops or communities (we have already seen this happend extensively in today's Internet) with shared mutual interest. Seeing how a group entity with shared values has a larger effect on the system than single individuals, data from such groups would likely be much more interesting and number-crunch-worthy. The greater power a group has to perterb systems' ecomonic or political trends, the more valuable that group's data will be to other groups.

In addition, I'm sure there'd be all sorts of tiered "offerings" from individuals and groups: the juicier/more detailed the data, the higher the premium offered. The changes this will introduce to markets (global and local), legal systems, and politcal organziations are probably barely imaginable right now. But what would it take to get us there? What would it take for my data and your data to be valuable enough to transform the world and make Wall Street look like an old-time, irrelevant boys club?


Privacy

One thing: a fanatical devotion to privacy, pure and simple. Security and a fanatical devotion to privacy. Two things! Okay, reliability, security and a fantaical devotion to privacy. Three things!

Monty Python references aside, an economy that values the data of individuals and groups can only arise if that data is secure. If we live in a topsy-turvy world where the Government, MPAA, RIAA, the Russian Mafia, and Big Hosting Company are pirating our data, then we're hosed. However, if our data is secure and contracts are effective, then we will have a world where data is the currency. There are an incredible number of hurdles to overcome in order for this to happen, however.

• The System - we need a system where user data can be tracked, recorded, and analyzed, and there's enough of it to matter
• Storage - we need our own, personal banks for our data (irrefutable ownership rights and complete power over that data)
• Transactions - we need a mechanism for engaging in secure, data transations
• Identity - when making a transation, we need to be able to prove unequivocally that we are who we say we are
• Anonymity - we need to decouple activity in the system and identity, thus requiring organizations to come to us (or our groups) to get the definitive data they need
• Recourse - we need a legal system and effective laws that protect the individuals and groups against the crimes of data-hungry organizations; fortunately, we will have had years of established precedent protecting the sellers from the buyers... oh my, how the tables turn!

And that's just off the top of my head. There's got to be tons of stuff which hasn't even occurred to me.


Closing Thoughts

Information will be as essential for us as water, yet there is a very interesting divergence from the example of a hydrological empire: each individual is the producer of some of that metaphorical water. By virtue of this difference, we hold the keys of the empire. We will be more a part of the economic and political powerbases than we have every been at any time in human history.

Of course, that means that we've got to get ready :-) This is already being done in many different ways. Everything from community housing cooperatives to small, co-op banks; from capabilities-based programming models to secure online transactions. Like the next 20 years of research needed for ULS systems to become a reality, we've got just as much work to do in order to guarantee our place in the economies of the future.

Ultra Large-Scale Systems: An Example

The ULS Series

• Required Reading: Ultra Large-Scale Systems
• Ultra Large-Scale Systems: An Example

Background

My interest in this topic is as old as my love for science fiction. As a child who had not only just started teaching himself to program but had fallen deeply in love with I, Robot, I consumed everything I could by the Master of the Art himself, Isaac Asimov. Inevitably, an endless steam of science fiction began flowing into my brain: the harder the science, the more cherished it was.

Then came the discovery that computers could actually talk to each other. Holy network, Batman, that changed everything! Oh, how I lamented my Kaypro II's inability to dial out. Science fiction novels began touching on this aspect of technology more and more frequently, while the Internet began taking shape in the "real" world around us. Now, look at it. Regardless of the mess and chaos, it's really quite amazing: beowulf clusters, distributed computing, cloud services, and of course the Internet in general. These advances are actually quite mind-blowing when we take the time to examine them from a historical perspectve.

A lot has changed since those early days of the network. The past 10 years or so has seen the beginnings of a trend with regard to large systems. Certainly my views on the future of networks (and services that utilize them) have been pretty consistent:

• In Praise of Evolvable Systems (2003)
• Dinosaurs and Mammals (2004)
• Evolving Networks (2004)
• Everything as a Distributed Resource (2004)
• The King is dead! Long live the Kinglets! (2005)

As I indicated in the more recent ULS blog post, I have been exposed to some excellent resources for ultra large-scale systems. For some of those I recently provided links, and others I will be referencing in future posts.

Due to their nature, ULS systems pose interesting open source collaboration as well as business opportunities. They entail a massive collection of excellent problems to solve that cannot possibly be completely addressed in the next 6-12 months (where so many projects and businesses tend to put their focus, for obvious practical reasons). As such, there are a great number of research and development areas -- plenty for everyone, in fact. In this series of blog posts, my goal is to expose a wider audience to the topics and encourage folks to start thinking about both interim solutions as well as potential long-term ones.


Characteristics of a ULS

Let's start of with some semblance of a definition :-) What constitutes a ULS system? Here are some characteristics given by Scale Changes Everything:

• an unbelievable amount of code (on the order of trillions of lines of code)
• immense storage needs, network connections, processing
• lots of hardware, lots of people, lots of purposes
• decentralized components
• created by aggregation, not design
• unreliable components, reliable whole
• ongoing and real-time upgrades, changes, and deployments
• lots of functionality, likely in a focused area of concern

Here's an illuminating quote from Richard Gabriel's Design Beyond Human Abilities presentation:

The components that make up a ULS system are diverse as well as many, ranging from servers and clusters down to small sensors, perhaps the size of motes of dust. Some of the components will self-organize like swarms, and others will be carefully designed. The components will not only be computationally active but also physically active, including sensors, actuators...

Sounds like pure science fiction, doesn't it? Think about it, though. Is it really? Divmod's friend Raffi Krikorian co-wrote this paper at MIT. Check out the cheap network node that's smaller than a fingertip. At that size, hundreds of them would be innocuous. In a few years, we could have thousands of them in a room without even knowing it. Within a single home we could have the equivalents of what today are campus or regional networks. We probably can't even wrap our heads around how big these systems will be. But there is plenty of precedence for such natural short-sightedness. From Raffi's (et al.) 2004 paper:

The ARPAnet was ambitiously designed to handle up to 64 sites with up to 4 computers per site, far exceeding any perceived future requirement. Today there are more than 200 million registered hosts on the Internet, with still more computers connected to those.

Here are some other choice quotes:

[Internet 0] is not a replacement for the current Internet (call that Internet 1); it is a set of principles for extending the Internet down to individual devices...

An [Internet 0] network cannot be distinguished from the computers that it connects; it really is the computer. Because it allows devices for communications, computation, storage, sensing, and display to exchange information in exactly the same representation, around the corner or around the world, the components of a system can be dynamically assembled based on the needs of a problem, rather than fixed by the boundaries of a box.

We're already building this stuff. It's not science fiction. We may not have swarming, self-replicating nano machines... yet. But we're already heading in a direction where that's not just a possibility; it's a likelihood.

So, we've got lots of code, machines, storage, sensing and people; much of it decentralized. What else do we need? Failure tolerance and maintenance on-the-fly. Check. Finally, a ULS system will have to actually be useful, or it will never get built. Who would want such a thing besides militaries, big governements, and Dr. Evils? Now we start getting to our example: Health Care. But let's not get ahead of ourselves. First, let's examine why the biggest system of networked devices that we know of isn't a ULS system.


Why the Internet is not a ULS System

Most obvious of the criteria listed above, the Internet is not focused on a single or related set of goals; it's used for everything. However, it does meet many of the criteria. From the Carnegie Mellon report:

The Web foreshadows the characteristics of ULS systems. Its scale is much larger than that of any of today’s systems of systems. Its development, oversight, and operational control are decentralized. Its stakeholders have diverse, conflicting, complex, and changing requirements. The services it provides undergo continuous evolution. The actions of the people making use of the Web influence what services are provided, and the services provided influence the actions of people. It has been designed to avoid the worst problems deriving from the heterogeneity of its elements and to be insensitive to connection failures.

But ... Security was not given much attention in its original design, and its use for purposes for which it was not initially intended ... has revealed exploitable vulnerabilities ... And although the Web is an important element of people’s work lives, it is not as critical as a ULS ... system would be.

Now I think we're in a good place to talk about the health care system of the future...


Health Services as a ULS

Let's start this section with a quote from the presentation that inspired it. Richard Gabriel says:

An example of a ULS system (that doesn’t yet exist) would be a healthcare system that integrates not only all medical records, procedures, and institutions, but also gathers information about individual people continuously, monitoring their health and making recommendations about how to improve it or keep it good. Medical researchers would be hooked into the system, so that whenever new knowledge appeared, it could be applied to all who might need it. Imagining this system, though, requires also imagining that it is protected from the adversaries of governmental and commercial spying / abuse.

Modern hospitals are packed with countless computing devices: everything from charting PDAs to physiological monitors for patients; from mainframes and patient record data warehouses, to terminals and desktops. Wireless medical sensors have already been developed by a research project at Harvard. What's more, despite the concerns over associated health risks, implant research at Johns Hopkins and the University of Maryland is on-going and may produce results that are one day standard practice in hospitals.

As versions of theses decives are developed that produce no ill effects for humans, they will make their way into out-patient clinics, assited living facilities, and ulitmately HMO's, private practices, and our homes. The devices will grow in numbers, shrink in size, and provide more functionality at greater efficiency than their predecessors.

The volumes of information that will be exhanged between devices, analyzed and correlated by other devices, and consumed by end-users, doctors, and researchers will be mind-boggling. It will bring new insights on everything from personal health to epidemiology.

With this, though, will come the obvious need for security and privacy, for defense against information attack and denial of service. These devices will all have to dedicate compuational and storage resources for use by the whole system. Part of the system will have to monitor itself, properly escalate problems, observe and anticipate trends. Protection and defense capabilities will have to exist the likes of which barely exist in our every day lives at the marcoscopic level.

All of this will take time. They will truly be modern wonders of the world. Given that such systems are anticipated to exist sometime in the next 20 years, and will have accreted the component systems over time, where might such a thing start?


Google and a Health Care ULS

If you read my last post (which I think was posted to blogger before the official announcement by Google), you already knew what I was going to say :-) Google Health. Though obviously nowhere near a ULS system in and of itself, why might we suggest Google is moving in this direction?

Here are some interesting bullet points from google.org:

• InSTEDD: $5,000,000 multi-year grant to establish this nonprofit organization focused on improving early detection, preparedness, and response capabilities for global health threats and humanitarian crises
• Global Health and Security Initiative: $2,500,000 multi-year grant to strengthen national and sub-regional disease surveillance systems in the Mekong Basin area (Thailand, Vietnam, Cambodia, Lao PDR, Myanmar, and China-Yunnan province)
• Clark University for Clark Labs: $617,457 to Clark University, with equal funding from the Gordon and Betty Moore Foundation, to support the development of a system to improve monitoring, analysis and prediction of the impacts of climate variability and change on ecosystems, food, and health in Africa and the Amazon
• HealthMap: $450,000 multi-year grant to conduct in-depth research into the use of online data sources for disease surveillance

Does that sound familiar to anyone besides me? All paranoia-induced sinister thoughts and Google Ads jokes aside, it makes sense that this is where we're going with health. In fact, it makes sense this is where we're going with all of our lives. If data privacy, personal ownership of that data, and security concerns can all be addressed, our lives' information will be better served by moving through systems specially designed to provide maximal use of that information with the least work. It won't just be nice to have, it will be essential.

The balance of power, from individuals all the way to the top of whatever organizations exist in the future will rest in information. Not like it is today, however. The "information economy" of the today (+/- 10 years) will look like kids' games and playgrounds. The information economies this will evolve into will be so completely integrated into human existence that they will resemble the basic necessities like water and food.

If you could find yourself a corner of that market, 20 years before everyone else got there, wouldn't that be a smart business move?


Summary

Our world is changing much more than we realize. We're too tied up in our jobs and gas prices to see the larger picture... to see that our future is already being made, that even in our unconscious actions we are propagating it no less than the cells in our bodies conspire to propagate what will become our children.

In the same way that hominid nomadic/migratory patterns begat the distribution of villages and tribal communities, which in turn gave birth to civilization, our silly little Internet will one day have descendants that dwarf it in size, utility, complexity, and computational power. The amazing thing is that we are the ones that actually get to build them!

There is a lot to research, and just as much to prototype. There is a project for everyone, and by starting now, we can make sure that feudal lords of tomorrow don't have absolute control over our food and water. If you have ideas for collaboration, start talking! Get involved! If you have money, fund some research, sponsor some conferences. In simply writing this blog post, I have uncovered gobs of new research I didn't know was out there. We should all be reading more, catching up, and coding. The projects near and dear to our hearts can get a whole new life within the context of ULS systems.

The Problem with and Solution to Google's App Engine

I know everyone is all aglow with the new web development offering from Google, but let me do the unpopular thing and put some things into perspective: there are limitations.

In fact, the limitations that exist will prevent me from using App Engine with all of my projects, save one (that one being a very simple web site). First, the limitations that prevent me from using App Engine (from one of their FAQs):

• Sockets are disabled with Google App Engine
• The system does not allow you to invoke subprocesses, as a result some os module methods are disabled
• Threading is not available

This means that I can't write a deferred wrapper for their data layer, I can't use Twisted for such things as XML-RPC or AMP-based communications, and I can't use an async templating system (like Nevow). I'm stuck with CGI and blocking code. And for all but the simplest projects, that's a big "No Thank You" from me.

This doesn't mean that I won't use it -- I will. I have one project that this will be perfect for... but it's for someone else, not me.

However, these limitations are actually good news :-) Here's the silver lining:

As Glyph as alluded to in his recent blog post (and in our tweets), we've recently completed a massive week-long BizDev Divmod sprint in Boston. One of the results of this is based on community feedback we've had over the last year, and which culminated at PyCon 2008 in Chicago with multiple requests for particular services from The Twisted Company. That result is a set of tools, features, and management options folks will be able to use with our software (app server, smart object db, network services, etc.). People really want to start using our stuff in cloud/grid computing environments. They need support for multiple and diverse network services, inter-store communications, massive deployments, etc. Two months before PyCon, we started working on tickets to support this, and we're making excellent progress toward providing the requested features.

We're still unclear as to which parts of this will be open source, as that will be driven by a combination of business and community demand. Regardless, Google's lack of support for this stuff has (for now) left the field wide open for us. And that, folks, is a big "Thank You Google!" :-)

accountdb

This week I made some great progress on the accountdb twisted/nevow micro app. The prototype is finished, complete with screenshots. One of the nice things that happened while writing it was a personal issue resolution :-)

And, yes, as a programmer, most of my personal issues are with software and software implementation or design.

Here's the issue: I think Nevow's stan is fantastic and ingenious.  However, I tend to adhere pretty strictly to the principle that HTML doesn't belong in code, and should remain in templates. For that  reason, I even try to steer away from programmatically generated HTML.

Two things let me relax this position: 1) stan is *very* clean, and 2) when you have lots of "mini-templates", having them all in separate files *really* starts to affect performance negatively, especially if
you're creating objects to provide an interface for them.

This work has given me some nice insight on how I can optimize imagedb as well. I'm looking forward to that, since I've been using imagedb a lot more, lately.

Oh, I almost forgot: sales pitch for accountdb: this micro app is, as you can guess, responsible for storing users and providing web authentication as a service. Don't worry, there will be many more blog entries on this in the future, and I will describe how this is done and why this is so useful :-)

But for now, back to coding!

The King is dead! Long live the Kinglets!

So what the hell does that mean? Well, there's sort of a back story there (but you can skip to the last paragraph of this post for the punchline).

Earlier this year, I blogged about the language patterns of twisted: Thinking in Twisted. The use of and relationships between such things as protocols, clients, servers, and various factories in Twisted was really starting to make its way into my bones. These and other core aspects of Twisted best practices form the basis of a pattern language... one that is of immense value when writing network code and applications. Here are a couple quotes from that blog entry:

The tricky thing about Twisted is that its patterns are an integral part of its "language" (perhaps it would be better to stop using quotes, and just call it a meta-language...). This is not true of most other languages -- though I would venture to say it is true of good frameworks in general. You don't need to know the patterns to use the grammar/syntax/control structures of most languages. Patterns in a language are usually optional. [However, w]ith Twisted, if you aren't using the patterns, you aren't really using Twisted.

When we learn to think in Twisted and write applications in Twisted, our solutions will be elegant and compact. Before that starts to happen, we try to use our "old" ways of python programming mixed in with the new one we are learning... and that, of course, defeats the purpose of the new "language."

I've recently hit another level of awareness in pattern abstraction, and it has its roots in Twisted. In fact, I never would have been working on what I am now working on if it weren't for Twisted and all the work done by Divmod and its crew. While at PyCon 2004, I met with Amir and Glyph and we discussed twisted's use in the industry at large, and in particular, opportunities for which the Divmod product line could be applicable.

What I liked so much (and immediately) about what Divmod was doing was that they were integrating useful components into a single application (Quotient). It stuck a positive chord with my instincts about where software should be going, and it kept me thinking for months while I worked on other projects. These thoughts culminated in a series of blog entries and software projects. First, a post called Dinosaurs and Mammals explored this. In the intervening time, the projects I have worked on have helped me evolve these ideas usefully. There is now direction, code, and product (more about that later in this post).

Then, in Everything as a Distributed Resource, I took a more sci-fi, radical view... but one that is born out by current trends and tendencies, particularly in the cell phone market. There's a great deal we haven't thought about in regards to the cell phone market. Not so much about cell phones and mobile technology, but rather the usage patterns and demands that are being made and (sometimes) met in this sector.

In an effort to follow through on the "mammal" paradigm, I did some exploration with URLdb and PyKon which I blogged briefly about in New Projects. The two projects mentioned there have since been merged into a project that we are calling Adytum's Twisted Collection. The latest addition to which is ImageDB.

ImageDB is a repository for static graphic media. The images that people upload here can be viewed via it's web interface, but it is more likely the ImageDB would be used as a part of something else. For example, when I get around to finishing PyKon, it will make use of
ImageDB. PyKon will not store any images itself. It's simply going to be an nicely designed, usable image gallery interface. Users won't know that, though. They will use PyKon and be able to create galleries and albums in that gallery by uploading images (which will be stored --
transparently to the user -- in an ImageDB instance), by selecting individual images (JavaScript ImageDB selector/browser), or by selecting "tags" a la del.icio.us (and therefore whole groups of
images).

Analogous to ImageDB is MessageDB, a repository for static text. It's next on the list to be developed. This micro-app will hardly ever be used directly, although it too will provide a basic web interface for viewing and editing stored content. MessageDB will be a storage app for RFC 2822-compliant "messages". For starters, various public mail list emails will get tagged and stored there.

Non-dynamic text for web sites and blogs will be stored as RFC 2822 messages. So the forth-coming tools "SimpleSite" and "WeBlog" (also part of the collection) will actually store and pull their content from MessageDB. But even more, when you want to use a photo, clipart, or logo in your site or on your blog, you will click a "select image" link and an image thumbnail browser will pop up. You will select one of the thumbnails, the popup will go away, and a link to that resource in your ImageDB instance will be inserted for you, in your blog or site text.

The idea is that by abstracting out general functionality into little micro-apps, you not only decrease the amount of work in maintenance and feature addition, you not only are reducing the amount of redundant work you have to do when building content-related applications, but you are providing an infinite amount of flexibility for using these various media in completely unplanned and unexpected ways.

Your media, it's storage, it's access (XML-RPC), now live in self-contained little units. They can "survive" on their own, without a heavy dependence upon some huge CMS that was written for a particular project. The apps are project-agnostic. They can be set up once and then reused repeatedly in different contexts by building new apps that obtain their data from the component micro-apps via simple links and/or XML-RPC calls.

And, again, the ways in which users and designers decide to combine these pieces and how they decide to use that is 100% flexible and thus fertile ground for the unexpected and creative :-)

I say "To Hell!" with large apps, and banish them forever! Let's build tiny, little, single-purpose apps once! And then for larger needs, simply glue together various micro-apps as needed per the project requirements.

As far as I'm concerned, the old King of the Bloated Application is dead. It's been replaced with tons of tiny, decentralized micro-apps. Long live the Kinglets!

PyKon

I got some more work done on PyKon last night after finishing a deadline for a client. I'm excited about having a modern, functional, python-based image gallery application. The first version will allow both:

• manual image uploads (ftp/scp, i.e., changes you would make on the file system with a shell account) and the daemon that runs the web user interface will also regularly check for gallery/album/image changes and process added/changed files), as well as
• through-the-web uploads.

For the second release, I want to enable some work we've been doing on an object registry. This would essentially make PyKon like iPhoto for Mac OS X: a global repository for images, and the capability to create albums with multiple overlapping images (duplicates) with no extra space taken up by the images (as they would only be references/links to the image tagged in repository).

We're hoping to make this better than IDS (written in perl), Gallery (in PHP), and JAlbum (in java). We really want this thing to kick ass and be very easy to use. We're a ways off, but progress so far is exciting.

Oh, and Nevow (for Twisted) rocks.

Nevow: URLdb

Well, I am really, really impressed with Nevow. It really grows on you as you code. I've been posting little tidbits that I learn on the Nevow wiki, the kinds of things that beginners should have easy access to.

Today, I dug really deep into the nevow libs and followed the path of form posts (configurables, validation, etc.). Very interesting stuff. Not only do I like the way Nevow behaves and handles for developers, I like its guts, too. That's actually fairly uncommon when it comes to web app frameworks.

Anyway, today was very successful: my TinyURL knockoff (URLdb) is "fully" functional. It has a form interface for posting/creating shorter URLs as well as a URL you can post to via javascript, etc. The reason that I put "fully" in quotes, is that there is no persistent backend storage yet: it's all in memory. This was for simplifying the development/learning process only, and I have already started on writing the Atop classes.

I look forward to using Nevow in the future -- immediately, on the PyKon project; but in general, for any web app that doesn't require the pre-built functionality of a full-blown CMS.

Everything as a Distributed Resource

In my blog entry Dinosaurs and Mamals, I mentioned small application servers that were capable of being highly distrubuted. In conversations on this topic with friends, we discussed various scenarios where such things could have an impact on our daily lives. For instance, I'm sure that many people have heard about the potato powered web server joke and then the real thing that it inspired. This demonstrates the possibility of low-power servers; conjoin that with wareable computing, and mammal application servers, and you have a whole new world...

There was this great game I used to play called "Alpha Centuari" -- the best part was choosing the research your culture would engage in. I find a similar "rush" from the communities one can join in the Orkut online friends network. I know lots of people that love that kind of thing, too... possibly pointing to something inherent in human nature about social groups, hobbies, aquisition, or just bright, shiney objects.

Even now, people have taken this one step closer to "reality": from simple association, to actual contribution. We may not all be able to get PhDs in astrophysics or molecular biology and do research that actively contributes to the human body of knowledge, but we can all run client applications on our workstations that borrow idle CPU cycles to work on problems in which we are interested.

So what happens when web servers can fit in the colar of a shirt, when shoes pumps generate electricity as you walk, when fibres woven throughout your clothes act as wireless antenae? What happens when processing power is literally everywhere and highly moble? Universities, companies, governments, special interest groups, neighborhoods, etc., can all lobby you for your computing resources. They don't have to ask you for your time or your money; you don't have to change the way you do things or when you do them. All you have to do is choose to associate with a group and then give them permission to use your cycles. They get to use CPUs that they don't have to power, insure, air condition, troubleshoot, network, maintain, etc. There is a potentially HUGE amount of money that can be saved for each organization.

But with increased mobility and with large numbers of projects to which you can contribute your unused cycles, comes such ubiquitous evils as marketing, advertising, lobbying, etc. If a company can influence you to change your mind and go with them instead of their competitor, then they can do it with thousands of other potential contributors as well. As quickly as you see a "give-us-your-spare-cycles" advertisement or have a conversation with a friend, you can change your mind and give to someone else: the CPU isn't waiting at home, it's there with you.

The consumer becomes a commodity broker; organizations, the consumer. Decisions are immeidate;
impact on a company's R &D is instantaneous. Profits are now. All of a sudden, the entire technology-enabled population is a rich philanthropist and every organization out there is hoping to get your "funding," hoping you'll let them add one more CPU to the thousands already volunteered. This could give birth to a whole new "processing"economy...

Dinosaurs and Mammals

Large software corporations like Microsoft and Oracle in addition to groups of software developers like elements in the Open Source community are giving the world new versions of the same thing: monolithic software for a world that no longer needs or wants such products.

Take for example Microsoft Exchange Server: this is a heavy-duty, if often ridiculed, system of software. This software was meant to handle hundreds to thousands of users and their information. What about the company that wants a Microsoft solution but only has 50 users? They have to pay for Exchange. Yeah, you can get a great discount, etc., but I'm not talking about price -- I'm talking about need. And designing software for need.

Oracle is more guilty of this sin. Let's say I need to have my data highly distributed on remote machines, but that I don't have tons of data and I don't need the horsepower Oracle can provide. Yet, if I use their product or any other large RDBMS like PostgreSQL, SQL Server, or even MySQL, I am using the wrong tool for the job.

Another example: Apache is the world's most popular Web server, but let me commit heresy and ask "Why?" How many people need it's speed? Or its capacity to handle millions of requests? Think of the hundreds of thousands of web sites out there that are visited once a day or once a month. Using something like apache for these is engineering overkill.

Microsoft and Oracle as well as organizations like the Apache Foundation, MySQL, and PostgreSQL are producing software dinosaurs. The have lived, they have thrived, they have helped define our information universe. That universe is changing. With the introduction of programming languages and development processes that foster rapid development, for the first time in the history of modern technology, we are in a place to design enterprise software around the needs of the end user (individulas and organizations), not around the needs of licensing structures and coporate philosophies. Imagine that the motivation behind software development wasn't market domination, but rather market diversity. The right tool for the right job.

I haven't counted the lines of code in the apache web server, but let's just say it's 100,000 lines. What if I only need to serve a couple pages per second and a max of 100 concurrent sessions? What if I could use Web server software that could provide what I need in only 200 lines of code? 100? 6? Why should I use Apache for projects that don't require Apache's strengths? The same thing for my email server and calendar application. What if I could do the same thing for all my networked applications? What if in addition to being small, this software was highly distributable as well? What if I could build office applications perfectly suited for a firm of 40 employees, and they didn't have to pay for licensing of the cliched software products? In addition, as the firm grew, this software could support 100, 1,000, or 100,000 users just as easily simply by adding the required light-weight components.

Software like this is easier to setup and maintain. It's easier to get involved with and understand. It's easier to modify, add to, customize, and enhance. It's easier to provide your firm with what they need.
Enter the new software and application engineering model: the mammal. This is where we are heading. Imgine application servers so light-weight they could run on a 9-volt battery for a week. Imagine them embedded in hundreds of devices all over the country or even the globe... so cheap and distributed, it didn't matter if one went down or even if twenty of them went down. They are quick, nimble mammals, not lumbering dinosaurs.

There are already software developers working on projects with views and goals along these lines. Watch for them: they are small and furry. They move quickly and quietly. Pay attention to them; they will one day rule the world.

In Praise of Evolvable Systems

Shirky: In Praise of Evolvable Systems first appeared in the ACM's net_worker, 1996, and there are some particularly thought-provoking comments that Clay Shirky makes:

...the Web's poorer engineering qualities seem not merely desirable but essential.

The effects of this ease of implementation, as opposed to the difficulties of launching a Gopher index or making a CD-ROM, are twofold: a huge increase in truly pointless and stupid content soaking up bandwidth; and, as a direct result, a rush to find ways to compete with all the noise through the creation of interesting work. The quality of the best work on the Web today has not happened in spite of the mass of garbage out there, but in part because of it.

Finally, Orgel's Rule, named for the evolutionary biologist Leslie Orgel -- "Evolution is cleverer than you are". As with the list of the Web's obvious deficiencies above, it is easy to point out what is wrong with any evolvable system at any point in its life... However, the ability to understand what is missing at any given moment does not mean that one person or a small central group can design a better system in the long haul.

Centrally designed protocols start out strong and improve logarithmically. Evolvable protocols start out weak and improve exponentially. It's dinosaurs vs. mammals, and the mammals win every time. The Web is not the perfect hypertext protocol, just the best one that's also currently practical. Infrastructure built on evolvable protocols will always be partially incomplete, partially wrong and ultimately better designed than its competition.

As with the early Web, the 'glue' protocol subsumes the other protocols and produces a kind of weak integration, but weak integration is better than no integration at all, and it is far easier to move from weak integration to strong integration than from none to some. In 5 years, DVD, HDTV, voice-over-IP, and Java will all be able to interoperate because of some new set of protocols which, like HTTP and HTML, is going to be weak, relatively uncoordinated, imperfectly implemented and, in the end, invincible.