netaddr Python Library

I recently got several feature requests for my NetCIDR Python library, and in the course of a conversation with one user in particular, I was made aware of the netaddr project. I took some time to explore the code details and was quite impressed: drkjam did a great job. The manner in which he implemented the many features (especially the IP math) was the kind of thing I wanted to do for NetCIDR ... at some point. After about an hour of digging around, testing out the API, and pondering, I decided to retire NetCIDR and encourage my users to migrate to netaddr.

There are a couple more esoteric features in NetCIDR that netaddr currently doesn't have, but we've started talking about adding support for those in netaddr, at which point there will be no need to use NetCIDR.

To facilitate this, I've added a wiki page on the netaddr Google Code project for helping users make the transition from NetCIDR to the netaddr API.

SOA in Practice: A Handbook for Early-Stage ULS Systems (Part 1)

The ULS Series

• Required Reading: Ultra Large Systems
• Ultra Large Systems: An Example
• SOA in Practice: A Handbook for Early-Stage ULS Systems (Part 1)

A Book Review

First off, this is an O'Reilly publication. What's more, if O'Reilly had something like a "criterion collection," this work would be in it. This title is what it says it is, "SOA in Pactice: The Art of Distributed System Design." Authored by Nicolai M. Josuttis, this is one of the best written technical overview works I have ever read, both for writing style and content. For anyone interested in ULS and/or SOA, I have one thing to say: buy this book immediately, with expedited shipping.

I'm not going to write a formal review with pros, cons, deep analysis about message, etc. However, what I will do is spend some time discussing the crossover from SOA to ULS, covering details with quotes from "SOA in Practice." I will not cover the book in detail and reveal all of its precious nuggets, but I will give a taste of what it has to offer and how it applies to ultra large-scale systems.

Divergence

Since most of what I want to discuss is about what we can gain by taking lessons learned from SOA and applying them to efforts in exploring or prototyping ULS systems, I want to initially outline the stark differences between those systems and SOA.

The most obvious difference is scale. To put things in perspective, imagine implementing a large SOA for a large organization. Imagine the requirements, the project planning, the logistics, the code, the bugs, the setbacks, the short-term failures, and finally, the successful delivery. Now multiply that: two related but semi-autonomous SOA projects. And again, with four. How about a third time for eight?

Any reader with experience in working with large projects is probably having heart palpitations right now (and for that, I apologize). You have first hand experience of the difficulties and the pain: with a linear increase in the size of a project, there is an exponential increase in the difficulty of managing that project (people, code, timelines, etc.), asymptotically approaching 100% unmanageability, regardless of the amount of resources you throw at the project.

The point just past the asymptote is where ULS systems and SOA meet. In other words, a ULS system as a whole -- by definition -- cannot be built. Such a system can accrete over time, but is simply too large to be designed, built and managed. Rather, it is emergent. Efforts being made in ULS systems research right now are focused on how we can best facilitate that emergence.

Convergence

One of the profound problem solving skills that maths like analytic geometry teach us is understanding potentially intractable problems by examining discrete and meaningful chunks. It's easy to chop something up; it's quite a different matter to chop such that the pieces are useful and provide further insight.

If working with ULS systems is like integrating over the volume of a complex solid in 11-space, then SOAs provide us with the tools of breaking part of that work up into a manageable chunk, one that we can wrap our heads around. Many of the same problems that technicians are anticipated to encounter when working with ULS systems exist at a smaller scale and are well understood within the context of SOA.

And this is where our friend Nicolai's book comes in.

ULS Systems Review

Before we continue, let's take a quick look back at some of the ULS basics laid out by the report of the Software Engineering Institute (SEI) of Carnegie Mellon:

• ULS systems are systems of systems at internet scale.
• ULS systems will be interdependent webs of software-intensive systems, people, policies, cultures, and economics.

In order to become a functional reality, these systems will require extensive research in the following areas:

• Human Interaction
• Computational Emergence
• Design
• Computational Engineering
• Adaptive System Infrastructure
• Adaptable and Predictable System Quality
• Policy, Acquisition, and Management

This means exploring for use in ultra-large scale systems such things as potential mechanisms for user interfaces, genetic algorithms/programming, new patterns in systems design, behavioral simulations of systems components in varying compositions, decentralized infrastructure, ultra-high availability, and integration with countless third-party support systems. Just to name a very bare minimum.

Intersection

Of those research areas, lessons learned from SOA can be applied to ULS systems research most predominantly in the following areas:

• Human Interaction
• Design
• Adaptive System Infrastructure
• Adaptable and Predictable System Quality

In Part 2, it is with an eye towards these that I will comment on Nicolai Josuttis' excellent work.

MySQL, Storm, and Relationships

I rarely work seriously with databases, but I've been building an API for a contract with PBS.org, and though we have DBAs tasked for the project, everyone's pretty busy. So I dusted off my decade-old DB (formerly known as) skills, and did the work myself.

I've worked with the Storm ORM a fair amount since it was released, but only on small projects. Any time I've needed to use relationships with Storm, I've been using SQLite and so it was all faked. Due to the impact of the PBS gig (which is almost done now!), I really needed to sit down and map everything out. The first thing I needed to do was get a quick refresher on MySQL's dialect with regard to foreign keys. The next thing I needed to clarify was exactly how to ensure that what I've been doing with Storm relationships in SQLite was valid for MySQL and suitable for production use at PBS. It was :-)

Given how infrequently I use this stuff, I thought that my notes would be good to document, for future quick-reference. Given that there are likely users out there who would also benefit from this, a blog post seemed a nice way to do this :-)

The SQL below is modified from an example in the MySQL documentation, slightly tweaked to be a smidge more interesting. The two CREATE TABLE statements define the schemas for a one-to-many table relationship:

Next, to be able to play with this in Storm, we need to define some classes and set up some references:

The parent attribute on the Child class is a Storm reference to whatever parent object is associated with the child object that gets created; the parent_id attribute is what is actually mapped to the MySQL field parent_id (which, in turn, MySQL references to the parent table). I hope I just didn't make that more of a confusing mess than it needed to be :-)

The children attribute that gets added to the Parent class is a reference to all Child instances that are associated with a particular Parent instance. I've got some usage below, if that's not clear.

Let's create a parent:

Note that if you add an __init__ method to your Storm classes, you can save a step or two of typing in these usage examples (see the Storm tutorial for more information).

Next, we'll create and associate a child:

There's more than one way to do this, though, given the way in which Storm has encoded relationships. Above, we created the child and then set the child's parent attribute. Below, we create the child and then use the chilren's add method to associate it with a parent:

We're doing all that flushing so that the created objects refresh with their new ids.

Lastly, let's take a look at what's we've just added to the database:
And that should just about do it :-)