Ultra Large-Scale Systems: An Example

The ULS Series

• Required Reading: Ultra Large-Scale Systems
• Ultra Large-Scale Systems: An Example

Background

My interest in this topic is as old as my love for science fiction. As a child who had not only just started teaching himself to program but had fallen deeply in love with I, Robot, I consumed everything I could by the Master of the Art himself, Isaac Asimov. Inevitably, an endless steam of science fiction began flowing into my brain: the harder the science, the more cherished it was.

Then came the discovery that computers could actually talk to each other. Holy network, Batman, that changed everything! Oh, how I lamented my Kaypro II's inability to dial out. Science fiction novels began touching on this aspect of technology more and more frequently, while the Internet began taking shape in the "real" world around us. Now, look at it. Regardless of the mess and chaos, it's really quite amazing: beowulf clusters, distributed computing, cloud services, and of course the Internet in general. These advances are actually quite mind-blowing when we take the time to examine them from a historical perspectve.

A lot has changed since those early days of the network. The past 10 years or so has seen the beginnings of a trend with regard to large systems. Certainly my views on the future of networks (and services that utilize them) have been pretty consistent:

• In Praise of Evolvable Systems (2003)
• Dinosaurs and Mammals (2004)
• Evolving Networks (2004)
• Everything as a Distributed Resource (2004)
• The King is dead! Long live the Kinglets! (2005)

As I indicated in the more recent ULS blog post, I have been exposed to some excellent resources for ultra large-scale systems. For some of those I recently provided links, and others I will be referencing in future posts.

Due to their nature, ULS systems pose interesting open source collaboration as well as business opportunities. They entail a massive collection of excellent problems to solve that cannot possibly be completely addressed in the next 6-12 months (where so many projects and businesses tend to put their focus, for obvious practical reasons). As such, there are a great number of research and development areas -- plenty for everyone, in fact. In this series of blog posts, my goal is to expose a wider audience to the topics and encourage folks to start thinking about both interim solutions as well as potential long-term ones.


Characteristics of a ULS

Let's start of with some semblance of a definition :-) What constitutes a ULS system? Here are some characteristics given by Scale Changes Everything:

• an unbelievable amount of code (on the order of trillions of lines of code)
• immense storage needs, network connections, processing
• lots of hardware, lots of people, lots of purposes
• decentralized components
• created by aggregation, not design
• unreliable components, reliable whole
• ongoing and real-time upgrades, changes, and deployments
• lots of functionality, likely in a focused area of concern

Here's an illuminating quote from Richard Gabriel's Design Beyond Human Abilities presentation:

The components that make up a ULS system are diverse as well as many, ranging from servers and clusters down to small sensors, perhaps the size of motes of dust. Some of the components will self-organize like swarms, and others will be carefully designed. The components will not only be computationally active but also physically active, including sensors, actuators...

Sounds like pure science fiction, doesn't it? Think about it, though. Is it really? Divmod's friend Raffi Krikorian co-wrote this paper at MIT. Check out the cheap network node that's smaller than a fingertip. At that size, hundreds of them would be innocuous. In a few years, we could have thousands of them in a room without even knowing it. Within a single home we could have the equivalents of what today are campus or regional networks. We probably can't even wrap our heads around how big these systems will be. But there is plenty of precedence for such natural short-sightedness. From Raffi's (et al.) 2004 paper:

The ARPAnet was ambitiously designed to handle up to 64 sites with up to 4 computers per site, far exceeding any perceived future requirement. Today there are more than 200 million registered hosts on the Internet, with still more computers connected to those.

Here are some other choice quotes:

[Internet 0] is not a replacement for the current Internet (call that Internet 1); it is a set of principles for extending the Internet down to individual devices...

An [Internet 0] network cannot be distinguished from the computers that it connects; it really is the computer. Because it allows devices for communications, computation, storage, sensing, and display to exchange information in exactly the same representation, around the corner or around the world, the components of a system can be dynamically assembled based on the needs of a problem, rather than fixed by the boundaries of a box.

We're already building this stuff. It's not science fiction. We may not have swarming, self-replicating nano machines... yet. But we're already heading in a direction where that's not just a possibility; it's a likelihood.

So, we've got lots of code, machines, storage, sensing and people; much of it decentralized. What else do we need? Failure tolerance and maintenance on-the-fly. Check. Finally, a ULS system will have to actually be useful, or it will never get built. Who would want such a thing besides militaries, big governements, and Dr. Evils? Now we start getting to our example: Health Care. But let's not get ahead of ourselves. First, let's examine why the biggest system of networked devices that we know of isn't a ULS system.


Why the Internet is not a ULS System

Most obvious of the criteria listed above, the Internet is not focused on a single or related set of goals; it's used for everything. However, it does meet many of the criteria. From the Carnegie Mellon report:

The Web foreshadows the characteristics of ULS systems. Its scale is much larger than that of any of today’s systems of systems. Its development, oversight, and operational control are decentralized. Its stakeholders have diverse, conflicting, complex, and changing requirements. The services it provides undergo continuous evolution. The actions of the people making use of the Web influence what services are provided, and the services provided influence the actions of people. It has been designed to avoid the worst problems deriving from the heterogeneity of its elements and to be insensitive to connection failures.

But ... Security was not given much attention in its original design, and its use for purposes for which it was not initially intended ... has revealed exploitable vulnerabilities ... And although the Web is an important element of people’s work lives, it is not as critical as a ULS ... system would be.

Now I think we're in a good place to talk about the health care system of the future...


Health Services as a ULS

Let's start this section with a quote from the presentation that inspired it. Richard Gabriel says:

An example of a ULS system (that doesn’t yet exist) would be a healthcare system that integrates not only all medical records, procedures, and institutions, but also gathers information about individual people continuously, monitoring their health and making recommendations about how to improve it or keep it good. Medical researchers would be hooked into the system, so that whenever new knowledge appeared, it could be applied to all who might need it. Imagining this system, though, requires also imagining that it is protected from the adversaries of governmental and commercial spying / abuse.

Modern hospitals are packed with countless computing devices: everything from charting PDAs to physiological monitors for patients; from mainframes and patient record data warehouses, to terminals and desktops. Wireless medical sensors have already been developed by a research project at Harvard. What's more, despite the concerns over associated health risks, implant research at Johns Hopkins and the University of Maryland is on-going and may produce results that are one day standard practice in hospitals.

As versions of theses decives are developed that produce no ill effects for humans, they will make their way into out-patient clinics, assited living facilities, and ulitmately HMO's, private practices, and our homes. The devices will grow in numbers, shrink in size, and provide more functionality at greater efficiency than their predecessors.

The volumes of information that will be exhanged between devices, analyzed and correlated by other devices, and consumed by end-users, doctors, and researchers will be mind-boggling. It will bring new insights on everything from personal health to epidemiology.

With this, though, will come the obvious need for security and privacy, for defense against information attack and denial of service. These devices will all have to dedicate compuational and storage resources for use by the whole system. Part of the system will have to monitor itself, properly escalate problems, observe and anticipate trends. Protection and defense capabilities will have to exist the likes of which barely exist in our every day lives at the marcoscopic level.

All of this will take time. They will truly be modern wonders of the world. Given that such systems are anticipated to exist sometime in the next 20 years, and will have accreted the component systems over time, where might such a thing start?


Google and a Health Care ULS

If you read my last post (which I think was posted to blogger before the official announcement by Google), you already knew what I was going to say :-) Google Health. Though obviously nowhere near a ULS system in and of itself, why might we suggest Google is moving in this direction?

Here are some interesting bullet points from google.org:

• InSTEDD: $5,000,000 multi-year grant to establish this nonprofit organization focused on improving early detection, preparedness, and response capabilities for global health threats and humanitarian crises
• Global Health and Security Initiative: $2,500,000 multi-year grant to strengthen national and sub-regional disease surveillance systems in the Mekong Basin area (Thailand, Vietnam, Cambodia, Lao PDR, Myanmar, and China-Yunnan province)
• Clark University for Clark Labs: $617,457 to Clark University, with equal funding from the Gordon and Betty Moore Foundation, to support the development of a system to improve monitoring, analysis and prediction of the impacts of climate variability and change on ecosystems, food, and health in Africa and the Amazon
• HealthMap: $450,000 multi-year grant to conduct in-depth research into the use of online data sources for disease surveillance

Does that sound familiar to anyone besides me? All paranoia-induced sinister thoughts and Google Ads jokes aside, it makes sense that this is where we're going with health. In fact, it makes sense this is where we're going with all of our lives. If data privacy, personal ownership of that data, and security concerns can all be addressed, our lives' information will be better served by moving through systems specially designed to provide maximal use of that information with the least work. It won't just be nice to have, it will be essential.

The balance of power, from individuals all the way to the top of whatever organizations exist in the future will rest in information. Not like it is today, however. The "information economy" of the today (+/- 10 years) will look like kids' games and playgrounds. The information economies this will evolve into will be so completely integrated into human existence that they will resemble the basic necessities like water and food.

If you could find yourself a corner of that market, 20 years before everyone else got there, wouldn't that be a smart business move?


Summary

Our world is changing much more than we realize. We're too tied up in our jobs and gas prices to see the larger picture... to see that our future is already being made, that even in our unconscious actions we are propagating it no less than the cells in our bodies conspire to propagate what will become our children.

In the same way that hominid nomadic/migratory patterns begat the distribution of villages and tribal communities, which in turn gave birth to civilization, our silly little Internet will one day have descendants that dwarf it in size, utility, complexity, and computational power. The amazing thing is that we are the ones that actually get to build them!

There is a lot to research, and just as much to prototype. There is a project for everyone, and by starting now, we can make sure that feudal lords of tomorrow don't have absolute control over our food and water. If you have ideas for collaboration, start talking! Get involved! If you have money, fund some research, sponsor some conferences. In simply writing this blog post, I have uncovered gobs of new research I didn't know was out there. We should all be reading more, catching up, and coding. The projects near and dear to our hearts can get a whole new life within the context of ULS systems.

Everything as a Distributed Resource

In my blog entry Dinosaurs and Mamals, I mentioned small application servers that were capable of being highly distrubuted. In conversations on this topic with friends, we discussed various scenarios where such things could have an impact on our daily lives. For instance, I'm sure that many people have heard about the potato powered web server joke and then the real thing that it inspired. This demonstrates the possibility of low-power servers; conjoin that with wareable computing, and mammal application servers, and you have a whole new world...

There was this great game I used to play called "Alpha Centuari" -- the best part was choosing the research your culture would engage in. I find a similar "rush" from the communities one can join in the Orkut online friends network. I know lots of people that love that kind of thing, too... possibly pointing to something inherent in human nature about social groups, hobbies, aquisition, or just bright, shiney objects.

Even now, people have taken this one step closer to "reality": from simple association, to actual contribution. We may not all be able to get PhDs in astrophysics or molecular biology and do research that actively contributes to the human body of knowledge, but we can all run client applications on our workstations that borrow idle CPU cycles to work on problems in which we are interested.

So what happens when web servers can fit in the colar of a shirt, when shoes pumps generate electricity as you walk, when fibres woven throughout your clothes act as wireless antenae? What happens when processing power is literally everywhere and highly moble? Universities, companies, governments, special interest groups, neighborhoods, etc., can all lobby you for your computing resources. They don't have to ask you for your time or your money; you don't have to change the way you do things or when you do them. All you have to do is choose to associate with a group and then give them permission to use your cycles. They get to use CPUs that they don't have to power, insure, air condition, troubleshoot, network, maintain, etc. There is a potentially HUGE amount of money that can be saved for each organization.

But with increased mobility and with large numbers of projects to which you can contribute your unused cycles, comes such ubiquitous evils as marketing, advertising, lobbying, etc. If a company can influence you to change your mind and go with them instead of their competitor, then they can do it with thousands of other potential contributors as well. As quickly as you see a "give-us-your-spare-cycles" advertisement or have a conversation with a friend, you can change your mind and give to someone else: the CPU isn't waiting at home, it's there with you.

The consumer becomes a commodity broker; organizations, the consumer. Decisions are immeidate;
impact on a company's R &D is instantaneous. Profits are now. All of a sudden, the entire technology-enabled population is a rich philanthropist and every organization out there is hoping to get your "funding," hoping you'll let them add one more CPU to the thousands already volunteered. This could give birth to a whole new "processing"economy...

Evolving Networks

Note: This is from an email I had with Itamar Shtull-Trauring, who was kind enough to let me blather for a bit...

There was a very interesting article in this year's April issue of Scientific American. It basically discussed a non-neural network in the brain, its own complete entity, but one that is connected to the neural network we know and love. It's a slower network that seems to provide feedback and additional processing/learning power. It uses mechanics and chemistry to communicate along its own network and respond/react to the neural one.It's a great article -- I recommend it (I have severely abused it with that synopsis above).

It got me to thinking... first: what if the original pre-brains were cells that communicated electrically and chemically, but with no neural network capabilities? What if the original multicellular ancestor of the brain was just a simple highly-connected, bi-directional network? What if the chemical and neurological networks grew in tandem and developed together, with the chemical network acting as a "meta-network," providing feedback? And, to really stretch things, what if this feedback is what caused the electrical network to develop into a neural network?Now I'm sure you see where I am going with this and how fruity it is...

Like primitive primordial cellular organisms, our society also has a primitive electrical network: the Internet. How can we develop a functional analog to the biological system (the biological system being proven to work and being insanely efficient... for a neural network...)?

Okay, so that's the visionary's question; no need for reasons or justification -- can we do it? Then let's make it! But why would we want to? Do we really want to turn the insanity of the Internet into something resembling a neural network? Perhaps, but that's not the goal; the ultimate goals might be something like these:

• We know the Internet is slow; how can we make it fast? efficient? (fastest paths, memory/learning, ...)
• We know that the Internet is filled with hostiles (spam, viruses, worms, etc.) - how can we safeguard this?
• There will always be something new that the Internet hasn't seen before - how can we integrate anticipation?

Could we create a virtual separate network inside the internet? Could it use a completely different protocol than anything else in existence? Could it be very simple, very specialized, and very secure? Could it provide protection services? Or perhaps act as some kind of feedback mechanism for Internet devices? A subscription service with intelligence?
This leads to other questions I'm sure people a lot smarter than me have already examined: can we use nature as an analog for developing network communications into something more mature?

• How do complicated organic networks regulate themselves?
• How do they protect themselves from violence?
• How do they self-organize? adapt?
• Are there ways in which we could model these behaviors with special designs (protocols, applications, services)?

So that's it, nothing with much technical content... just some ideas floating around...

Dinosaurs and Mammals

Large software corporations like Microsoft and Oracle in addition to groups of software developers like elements in the Open Source community are giving the world new versions of the same thing: monolithic software for a world that no longer needs or wants such products.

Take for example Microsoft Exchange Server: this is a heavy-duty, if often ridiculed, system of software. This software was meant to handle hundreds to thousands of users and their information. What about the company that wants a Microsoft solution but only has 50 users? They have to pay for Exchange. Yeah, you can get a great discount, etc., but I'm not talking about price -- I'm talking about need. And designing software for need.

Oracle is more guilty of this sin. Let's say I need to have my data highly distributed on remote machines, but that I don't have tons of data and I don't need the horsepower Oracle can provide. Yet, if I use their product or any other large RDBMS like PostgreSQL, SQL Server, or even MySQL, I am using the wrong tool for the job.

Another example: Apache is the world's most popular Web server, but let me commit heresy and ask "Why?" How many people need it's speed? Or its capacity to handle millions of requests? Think of the hundreds of thousands of web sites out there that are visited once a day or once a month. Using something like apache for these is engineering overkill.

Microsoft and Oracle as well as organizations like the Apache Foundation, MySQL, and PostgreSQL are producing software dinosaurs. The have lived, they have thrived, they have helped define our information universe. That universe is changing. With the introduction of programming languages and development processes that foster rapid development, for the first time in the history of modern technology, we are in a place to design enterprise software around the needs of the end user (individulas and organizations), not around the needs of licensing structures and coporate philosophies. Imagine that the motivation behind software development wasn't market domination, but rather market diversity. The right tool for the right job.

I haven't counted the lines of code in the apache web server, but let's just say it's 100,000 lines. What if I only need to serve a couple pages per second and a max of 100 concurrent sessions? What if I could use Web server software that could provide what I need in only 200 lines of code? 100? 6? Why should I use Apache for projects that don't require Apache's strengths? The same thing for my email server and calendar application. What if I could do the same thing for all my networked applications? What if in addition to being small, this software was highly distributable as well? What if I could build office applications perfectly suited for a firm of 40 employees, and they didn't have to pay for licensing of the cliched software products? In addition, as the firm grew, this software could support 100, 1,000, or 100,000 users just as easily simply by adding the required light-weight components.

Software like this is easier to setup and maintain. It's easier to get involved with and understand. It's easier to modify, add to, customize, and enhance. It's easier to provide your firm with what they need.
Enter the new software and application engineering model: the mammal. This is where we are heading. Imgine application servers so light-weight they could run on a 9-volt battery for a week. Imagine them embedded in hundreds of devices all over the country or even the globe... so cheap and distributed, it didn't matter if one went down or even if twenty of them went down. They are quick, nimble mammals, not lumbering dinosaurs.

There are already software developers working on projects with views and goals along these lines. Watch for them: they are small and furry. They move quickly and quietly. Pay attention to them; they will one day rule the world.

In Praise of Evolvable Systems

Shirky: In Praise of Evolvable Systems first appeared in the ACM's net_worker, 1996, and there are some particularly thought-provoking comments that Clay Shirky makes:

...the Web's poorer engineering qualities seem not merely desirable but essential.

The effects of this ease of implementation, as opposed to the difficulties of launching a Gopher index or making a CD-ROM, are twofold: a huge increase in truly pointless and stupid content soaking up bandwidth; and, as a direct result, a rush to find ways to compete with all the noise through the creation of interesting work. The quality of the best work on the Web today has not happened in spite of the mass of garbage out there, but in part because of it.

Finally, Orgel's Rule, named for the evolutionary biologist Leslie Orgel -- "Evolution is cleverer than you are". As with the list of the Web's obvious deficiencies above, it is easy to point out what is wrong with any evolvable system at any point in its life... However, the ability to understand what is missing at any given moment does not mean that one person or a small central group can design a better system in the long haul.

Centrally designed protocols start out strong and improve logarithmically. Evolvable protocols start out weak and improve exponentially. It's dinosaurs vs. mammals, and the mammals win every time. The Web is not the perfect hypertext protocol, just the best one that's also currently practical. Infrastructure built on evolvable protocols will always be partially incomplete, partially wrong and ultimately better designed than its competition.

As with the early Web, the 'glue' protocol subsumes the other protocols and produces a kind of weak integration, but weak integration is better than no integration at all, and it is far easier to move from weak integration to strong integration than from none to some. In 5 years, DVD, HDTV, voice-over-IP, and Java will all be able to interoperate because of some new set of protocols which, like HTTP and HTML, is going to be weak, relatively uncoordinated, imperfectly implemented and, in the end, invincible.