Monday, May 19, 2008

Required Reading: Ultra Large-Scale Systems

At Divmod, we're always talking about the future of computing, software, and the network. This usually focuses on our work with Twisted or the Divmod platform. But we have also spent considerable time assessing research in the area of what is called "ultra large-scale systems." Our primary business interest with this revolves around development, deployment, and management. However, there is a great deal of work that needs to be done to make ULS systems a reality.

I have a series of blog posts planned to discuss ULS systems in the areas where I have a vested interest. Despite the fact that a popular study on the matter was funded by the U.S. Department of Defense (conducted by the Carnegie Mellon Software Engineering Institute), I will not be discussing this technology in the context of war efforts nor national defense. Instead, I will be engaging in this discussion within the context of the medical/health services field, per the example given by Richard Gabriel in his presentation (PDF) to the Chinese University of Hong Kong.

This is especially pertinent today, as Google prepares for its Google Health announcement. Even though there are by some estimates (optimistic ones, in my view) 25 years of research ahead of us, it is generally agreed that ULS systems will consist aggregate sub-systems, built incrementally over time. I don't believe anyone is under the illusion that Google does not want to produce the first ULS system in history, and they are making rapid progress towards this goal. Google Health brings this point home very clearly, in the context of the research that has been done in this area.

The world is rapidly changing. The most important issues in technology and business are not who is going to create the next catchy "Web 2.0" application, or what mega corp games are being played in the Great Silicon Valley Soap Opera. The important issues are how the systems of the next 100-200 years are being built now, who they are being built by, and who as access to that technology.

We are a powerful and creative community. We are concerned about distributing power to the people, education, privacy, and freedom -- open source is built upon these principles. If we want those who are building ULS systems to build them fairly and with our concerns in mind, then we must get involved now. We must start building the necessary tools.

As for the reading, a couple of these were linked to in this blog post. Here is a list that should get your brain revved up and ready to roll (these are all PDF files):
I will have more specifics about the sorts of things we're exploring in future blog posts, but until then -- don't get left behind!

Update: Allen Short just pointed out some additional material that predates the Carnegie Mellon research: the Agoric Papers, co-authored by Mark Miller who is a co-creator of the E programming language.

Update 2: Glyph Lefkowitz just sent me a link to the Big Ball of Mud that discusses similar concepts.


  1. thanks for the great overview post.

    the e language was one of the first languages i got really into in college. i thought it was pretty stunningly well composed. i've taken a cursory glance at the agoric papers, and its refreshing to hear many of the same ideas rooted in an earnest attempt at gaining perspective on why we program what goals it accomplishes and how ultra large ultra heterogenous systems factor in.

    i'd kind of forgotten about Nevow for the past year or so. i'm glad to hear you guys are using comet, i'm fiercly interested in seeing what you've done. i'll have to sit down with a debugger and watch your wire protocol for a while, see what i think now. but in general, it was great being reminded about nevow. i found this link from reddit, and am really glad i did. subscribed. :)

  2. rektide,

    Thanks for your comment! It's always refreshing to hear feedback like yours :-) Do stay tuned, because there's lots more I want to talk about wrt ULS... there's a post coming that will focus on exploring protocol research that you might be particularly interested in :-)

    As for the E stuff, we love it (some more than others!) and we've even got a super-secret related project that's been in the works for about 16 months now... I'm trying to talk Allen into saying more about it publicly :-)

  3. In case folks are interested, here's a reddit search that turns up the entry rektide mentioned (I had a hard time finding it, just paging through the programming results):