7 Years of Blogasmic Action

Earlier today, I was looking through old blog posts, reminding myself of issues and topics that I want to continue reflecting on actively. When I got to the end of the posts, I looked at the earliest date and realized that my seven-year blogiversary was this past Wednesday. At my vocalized comment to this effect, Marjorie asked what got me started on blogging. Interestingly, it ties in with my professional life over the past twleve years...

As a physics undergrad and employee of the University of Maryland, I was hired away by a large internet startup for my systems and programming background. Already a fan of Office Space, I got to experience the reality of cubicle life as the not-so-funny, real complement of that classic movie. However, it wasn't my personal experience of cubihell that had the biggest impact upon my psyche; it was the corporate atrocities suffered by various customers and partners of dot-boom era, high-rolling companies.

Already a passionate open-source user and contributor, it become one of my personal goals to make sure a greater number of people (especially those making decisions in companies) were informed about the software freedoms that were within their reach. Within two years of this epiphany, I had my own company and was open-source consulting for everything from small businesses to large international corporations and the Federal government. Everything I did was open source; all of it saved my customers from spending millions of dollars unnecessarily and the enslavement of proprietary lock-in.

It was at this time that I started blogging. My primary interest was to be a voice in the wilderness of consultants, finding other like-minded programmers, project managers, sales engineers, etc., who believed in "open business", transparency, and accountability. As time progressed, my blogging habits adapted to fill other needs in various communities. However, all of my professional decisions have remained true to that initial desire to work on technology that was not only free, but actively prevented the bondage of its users.

For the past year and a half, I've been working at Canonical, the company that funds and actively supports Ubuntu and the Ubuntu Community. Every time I work with a different engineer or a new team, I am blown away by two things: the incredible talent embodied in each interaction, and the dedication that each person has to improving how free technology is delivered to the world.

My blog posts have definitely changed in character over the past seven years. But I'm delighted that, where I once felt alone with my ideals, I can count true giants in the field among my fellow community members, cooworkers, and mentors.

Possible Directions in Human Resource Management

This afternoon, while having a nice soak in the tub, I got lost in a Strossian reverie. Or perhaps it was more along the lines of one of his characters, rather than the author himself. Regardless, I was thinking about management styles in modern corporations... and ways in which one might improvise.

I was only half paying attention, until the stream of thought actually started getting interesting. At which point I sat upright in the tub and started taking mental notes. Quickly showering off, I couldn't stop the flood of ideas, hoping to get to a written medium as soon as possible, lest they be forgotten.

I forget how the day-dreaming started... perhaps my usual: pick a random topic where I have some experience, and start playing with it. Run simulations and tweak parameters until there's nothing more to look at or some interesting permutation has popped up. 

Ah! Now I remember. I was thinking about the increasing importance of QA in software as more and more marketshare is gained. Not from the usual perspective ("we gotta make sure users don't see bugs, or our shit's gonna flop"), but rather as part of the original problem domain.  It's very common for open source projects to suffer from a lack of sex appeal when considering QA. QA is essential for success; just as important as the code itself. As such, there should be ways of making QA as interesting, engaging, understood, and respected as the act of programming.

That train of thought spawned a couple more pathways for exploration, but the one that ended up being the most interesting was pondering the QA-interest problem at a human resource management level. What would it take to get talented and skilled engineers who would normally gravitate towards some other field of expertise interested in QA instead? How flexible would a company have to be to start attracting for this and other positions as new needs arose due to new pressures? How could it do so by growing and adapting from the inside?

Let's take these rather arbitrary initial conditions as part of a thought experiment following up on those questions:

• a company with a sufficient number of employees to support multiple departments
  
• a highly intelligent, motivated workforce
  
• flexible employees and teams, probably working in a distributed, remote environment

Imagine with this sort of company that employees are not locked into teams, departments, divisions, etc. Let's say there is some mechanism which allows for an easy re-shuffling of talent throughout the company. For budgeting and reporting purposes, corporate structures would remain, but the individuals that performed specific tasks were granted much more expansive, organic freedoms when it came to projects and tasks.

From the individual's perspective, an employee could choose to work for and closely associate with whatever team they wanted to spend some time with, given of course, that this team could make use of the newly reallocated employee's particular skills and abilities. This is somewhat analogous to the geographical freedoms one has when working remotely: you can move anywhere in the world, to whichever time zone you prefer, whatever culture you want to enjoy, given that you can continue to work effectively. 

Of course, checks and balances would have to be introduced. Otherwise, what's to prevent an employee from team-hopping, and never really getting any work done? Just playing social butterfly? Karma/experience points could be earned on per task, project, team, department, and division bases. Moving at any particular level to a different group would reset an employee's karma for that group and all other organizational units below it. For instance, changing to another department would zero out any team, project, and unfinished task karma that had been accumulated. Perhaps some of the patterns from role playing games could come in handy as sources of inspiration.

The ability to opt for a move could also be governed by an appropriate accumulation of karma points for a given organizational unit, to be determined at the discretion of that organizational unit. For example, hitting a certain karma "level" would allow one new movement opportunities. Obviously, in addition to having enough points, the target group would have to have a need and be willing to bring the requesting person on board.

Intuitively (i.e., without doing any research or checking sources), this feels like something modeled after biological systems, as opposed to the fairly static forms of systems organizations we currently see in many corporations. A company that was able to successfully implement such a human resource management approach could likely see additional, unexpected benefits. Biological systems tend to be well-suited to surviving shock, drastic changes, massive failures. We might see companies with longer-reaching vision, more innovation, less frequent employee turnover, and greater financial stability.

That aside, what appeals to me at a personal level is this: employees would be participating in their work at a new level. The management processes behind professional development would be opened up to them. Not only that, part of one's work would become a game with known rules and clearly defined markers for cumulative achievements. Rewards, though, would not be power-climbing, but rather lateral expansion and exploration. Deeper involvement in other areas of the company.

I'd love to hear from folks who can recommend some good reading/research materials on this topic...

A Sinfonia on Messaging with txAMQP, Part II

A Sinfonia on Messaging:

1. The Voice of Business
   
2. The Voice of Architecture
3. A RabbitMQ and txAMQP Interlude

After writing the last blog post, I found a fantastic site that focuses on messaging in the enterprise. I have really enjoyed the big-picture overview I get from some of the Martin Fowler signature books in this series, so I ordered a copy of this one too.

On the web site, the authors give a nice example for messaging integration in Chapter 1. They provide a more detailed, supplier-version of the kilt store (we're doing "manufacturing" as opposed to distribution) with "Widget-Gadget Corp", but the basic principles are the same. I highly recommend reading that entire page. I used it as the basis for much of this post.


Business Process Overview

At a top-level, we have the following business process for MacGrudder's kilt store:



These are represented by a sales guy or a web store, a third-party billing service, MacGrudder, and a third-party shipping service.

Up until now, the sale process could be either a user deciding to buy something in the online store or the sales guy engaging with a customer. Both generated orders; neither shared resources. The web app interfaced with the payment gateway operating by billing/shipping guy. The sales guy had to call in his orders to the billing/shipping guy. Once orders were charged/approved, a printout was handed to MacGrudder, who then created the ordered kilt. Once completed, he'd set it aside for shipping guy to come box it up and slap a label on it.


The Voice of Architecture

We're now ready to weave in the second voice of our three-part invention. MacGrudder's original infrastructure consists of silos of applications, functionality, data, and process. We want to interconnect these separated areas in order to reduce long-term overhead incurred by redundant components and data. Practically, we want to see the following changes:


Unified orders: at the end of the sales process, there should be one abstraction of the "order", regardless if the source was the web store, a phone call, or the sales guy. The order abstraction will be a message (or series of messages, for orders with multiple items; we'll be addressing only the simple case of a single item).


Unified validation and billing: at the end of the order creation process, there should be a validated order or an invalid order (e.g., if there were insufficient funds). At the end of the billing process, there should be an approved order that has be paid for via a single means (e.g., a single payment gateway, without bothering billing guy for manual entry). Additionally, once an order has been validated, messages should be sent to other components in the system.


Unified status: at the end of the manufacturing process, both the shipping guy and customers should be aware that the product has been completed and is ready to be sent: the shipping guy can connect to our messaging system (probably via a service) and the customer can be notified by email or by checking the order status in the web kilt store.


In the next installment, we will finally start looking at the code. We'll look at the "unified orders" messaging solution after covering some basics with RabbitMQ and Twisted integration, and then see how far we get with implementation details and descriptions. Unified validation, billing, and status might have to be pushed to additional posts.


References

• Enterprise Integration Patterns, Chapter 1
• SOA in Practice

A Sinfonia on Messaging with txAMQP, Part I

A Sinfonia on Messaging:

1. The Voice of Business
   
2. The Voice of Architecture
3. A RabbitMQ and txAMQP Interlude

Prelude

A complete messaging solution is a three-part invention of business need, architecture, and implementation. In its final form, these three voices blend in harmony, with each one taking a dominant role depending upon which part of the solution one examines.

Neither have I the ability nor skill to seamlessly weave three concepts together while clearly explaining their roles. Therefore, I will separate out the voices from each other and leave it as an exercise for the reader to construct an application and practice the principles involved, thus experiencing well-earned contrapuntal pleasures first-hand.

Introduction

As computing and data exchange systems increased in complexity over the past 30 years, so has the need for improvements -- and where possible, simplifications. Some of these efforts have been focused on decentralization of communications (shared, distributed load) and decoupling of messaging from applications (removing redundancy and increasing delivery speed/throughput). The first steps towards this were made in the 1990s with explorations in "middleware" application universe.

Messaging, as we now refer to it in the industry, arouse from those middleware adventures: out of the business drive to refactor old software as new services to wider, more sophisticated audiences. With many new services replacing a single, monolithic application, formal and well-architected solutions were needed for creating, editing, and deleting shared data.

From Wikipedia:

Message-oriented middleware (MOM) is infrastructure focused on message sending that increases the interoperability, portability, and flexibility of an application by allowing the application to be distributed over multiple heterogeneous platforms. It reduces the complexity of developing applications that span multiple operating systems and network protocols by insulating the application developer from the details of the various operating system and network interfaces.

AMQP (Advanced Message Queuing Protocol) is one of these protcols.

In a recent blog post about Ubuntu as a business server, Vaughan-Nichols provides evidence for Ubuntu's and Canonical's commitment to enterprise, saying "... the new [version of] Ubuntu also includes AMQP [...] support. AMQP is an important set of middleware and SOA [...] protocols."

AMQP has demonstrated itself as a compelling protocol for messaging solutions, even to the point of being included in two Linux distributions. The code included in this blog series is txAMQP, an asynchronous Python AMQP library built with Twisted.

Note that last year I had planned to write a blog series on messaging with Twisted and XMPP, but was unable to as a result of time constraints. These days, I'm working with AMQP instead of XMPP, but I still hold some hope that I'll be able to write an anolog for this series from the perspective of XMPP.


The Voice of Business

Dipping into the future, one of the goals for AMQP as developed by a special interest group is the following:

Decentralized, Locally Governed Federated Mesh of AMQP Brokers with standardized Global Addressing. The killer application for AMQP is transacted secure business messages between corporations - e.g. send a banking confirmation message to confirms@bank.com [...]

I find this rather exciting due to my interest in ultra large-scale systems; scenarios like the one described above are the seeds for tomorrow's ULS systems :-)

For now, though, let's look at a more immediate use for AMQP: a messaging protocol for shared services between departments in a small store. In this exercise, the voice of business is the primary melody; everything else (architecture and implementation) is done in support of this theme.


An Example

Fhionnlaidh MacGrudder creates hand-made kilts to order. He's got a sales guy who works with movie costume design shops and the like. He's got a web girl who wrote and maintains a custom store front app. He's got a friend who does shipping and billing for him (as well as some other local Glen Orchy artisans). Until now, these three business "groups" associated with the kilt shop have been maintaining their own records, sometimes copying and updating them manually from each others' various export files.

Fhionnlaidh's niece Fíona is programmer, business student, and is dating shipping guy's son. Horrified by the inefficiencies in her uncle's business processes (and tired of her boyfriend's father's complaints), she has proposed the following:

• sales guy will maintain customer contact info offer this as a service to the store and billing
• the web store will dynamically update displayed data when sales guy changes it
• the CRM will dynamically update displayed data when a web store customer updates their info
• MacGrudder will have a new web page he goes to where all pending orders are presented with their full details; changes can be updated by a customer in real-time until MacGrudder has started working on the order
  
• billing/shipping guy will be notified instantly as soon as MacGrudder marks a kilt order as completed
  

This setup has the following benefits:

• Contacts will be maintained in a single data store
• There is zero latency between customer-driven updates and sales guy-driven updates
• Customers have increased post-purchase flexibility with their orders
• Shipping guy can plug into MacGudder's messaging and be notified when packages are ready for pickup
• Everyone has more time for buttered scones and tea (especially shipping guy, who will no longer be making unneeded trips down the glen)
  

The following changes will be made to the current software:

• Contacts will need to be merged into the CRM
• A read/write data service for the contacts will need to be created
• The CRM front-end will need to be upgraded to an AJAX-enabled version
• The web store app will need to be updated to support AJAX
• A new page will be created which displays the status of all orders and allows MacGrudder to change an order from "pending" to "in-progress" to "completed"
• The current "new order" email notification code in the web app will need to be changed so that it uses the same messaging as MacGrudder's status page
  
• A new service needs to be created for shipping guy so that he can choose to be notified about pending pickups by email or he can check a web page or even make a query directly to the service, thus preventing unnecessary trips to MacGrudder's isolated little shop
  
• After all the work is done, somone's going to need to order more scones

This example is not meant to fully justify messaging for businesses, but rather to provide a simple use case for which we can write some simple (and less than robust) code. It is a toy, but a conceptually useful one with a solid, concrete foundation.

In the next installment, we'll review the business process (with diagrams!) and the explore the architecture of the system, before and after. Another post will take that architecture and combine it with MacGrudder's already extant infrastructure, reusing as much as possible. With that in place, we will have the opportunity to look at some RabbitMQ basics and some actual txAMQP code.


References

• SOA in Practice
  
• Message-oriented middleware on Wikipedia
  
• AMQP on Wikipedia
  
• The AMQP Spec
• A synchronous Python library for AMQP
• The async Python library for AMQP
• A txAMQP example
• An intro to messaging and py-amqplib
  

After the Cloud: Epilogue

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

Though wildly exciting to imagine a future of computing where ubiquitous devices are more deeply integrated into the infrastructure we use to power our applications, the real purpose of these posts has been to explore possibilities.

Let's have crazy thoughts. Let's build upon them, imagining ways in which they could become a reality. Let's not only munch on the regular diet of the technical "now"; let's plant seeds in many experiments for the future.

This post is about thinking of small, mobile devices and cloud computing. But it's also a rough template. Let's do the same thing for a the desktop: how might it evolve? Where will users be spending their time? Let's do it for the OS and the kernel: what radical changes can we envision there? The technology behind health care. Education. Our new patterns of behaviour in a constantly changing world. All of these and more deserve our attention.

The more we discuss such topics in a public forum, the more thought will be given to them. Such increased awareness and attention might spark the light of innovation years ahead of time, and do so in the context of an open exchange of ideas. Let's have Moore's law for the improved quality of life with regard to technology; let's take it out of the chip and into our lives.

Canonical's Vision

Canonical's most recent AllHands meeting finished last night (this morning, really... I can't believe I got up at 7:30am), and I'm somewhat at a loss for words. In a good way.

But I'll try anyway :-)

As someone who was highly skeptical of the validity of Canonical's business model prior to working here, I can say that not only do I not doubt our ability to be a hugely successful company, but I am deeply committed to that success. Before AllHands, Canonical had earned my respect and loyalty through the consistent support and care of its employees. After AllHands, I have a much greater practical, hands-on understanding of Canonical's strategies and the various projects involved in creating a reality of success.

What's more, though, is the completeness of my belief in the people and the vision. This is thanks to the massive exposure we've had during AllHands to the collective vision; the team projects; all the individuals with amazing histories, skills, unbelievable talent and ability to deliver; and most of all, the dedication that each employee of Canonical has to truly making the world a better place for anyone who depends upon technology.

Ubuntu is free, and that's great. But Canonical needs to be a huge commercial success if its free OS distribution is going to have the power to transform the market and thus people's lives. This AllHands has given me a complete picture of how that will happen: we're all working on a different part of this puzzle, and we're all making it happen.

Success in the marketplace is crucial. Not because of greed or the lust for power, but because we live in a world where value is exchanged. As part of that ecosystem, we want to bring the greatest value to the people. This is not "selling out"; it's selling. This does not give away a user's freedom; it helps guarantee its continued safety in a competitive, capitalist society.

If we want anyone to embrace Ubuntu instead of a non-free OS -- without asking our users to sacrifice anything -- we're going to need to make very serious changes in design, usability, integration, and stability. To do this in a clean, unified manner really only comes with a significant investment of time, direction, and capital. Due to the seriousness of Canonical's altruistic vision, as we generate this capital, we're making the dream come true for the world.

At AllHands, I've seen designs that will seriously challenge Apple. I've seen a usability team's plans for true computing goodness. I've seen revenue models that have made my jaw drop. I've seen glimpses of the bright future.

And baby, it's exciting as hell.

After the Cloud: Heaps of Cash

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

A One-Two Punch

Let me give you the punchline first, this time. Imagine a service that:

1. Combines the cloud management features of EC2 for any system (in this case, mobile devices), the monitoring and update management of Canonical's Landscape, the buying/selling power of an e-commerce application, the auctioning capabilities of eBay, and the data of marketing campaigns.
2. Seamlessly integrates all this into a cloud (Open Heap!) provisioning/acquisition system or as part of your mobile provider's billing and information web pages.

So where does the cash come in?


Power to the People

As usual, revenue would depend upon market adoption. That would depend upon appeal (addressed with marketing), usefulness (addressed by software engineering and usability), and viability. That last one's particularly interesting, as it's where people and the cash intersect.

A product suite and service, all built around open heaps, could have a long and fruitful life if implemented with the end user in mind. Users would have the opportunity to become partners in an extraordinary way: they would be consuming a service, while at the same time, being given the opportunity to resell a portion of that service for use in the cloud-like architectures of open heaps.

The first company that does this really well would have a continuously growing following of users. This company would be helping consumers earn immediate cash back on their property. This is something I believe deeply in; it's a positive manifestation of the continuing evolution of the consumer's role in the market. I'm convinced that the more symbiotic a relationship between consumer and producer, the healthier an economy will be.


Providers

In an open heap scenario, there are two providers: the mobile phone provider and the heap provider. Phone companies get to make money from the deal passively: through a partnership that provides them with a certain percentage of the revenue or indirectly through heap-related network use.

The heap provider (e.g., someone like Amazon or RightScale), would stand to make the most of everyone. Even though they wouldn't own the devices themselves (in contrast to current cloud providers), they would be able to assess fees on various transactions and for related services.

Imagine application developers "renting" potential CPU, memory, storage and bandwitdth from a heap that included 100s of 1000s of mobile users. The heap provider would be the trusted third party between the device owner and the application developer. In this way, the provider acts like an escrow service and can asses feeds accordingly.

Imagine a dynamic sub-market that arises out of this sort of provisioning: with millions of devices to choose from, a user is going to want to make theirs more appealing than 1000s of others. Enter auctions. Look at how much money eBay makes. Look at the fractional fees that they asses... fees which have earned them billions of dollars.

Throw in value-adds like monitoring and specialized management features, and you've got additional sources of revenue. There's a lot of potential in something like this...


Review
Obviously, all of this is little more than creative musing. The technology isn't quite there yet for a lot of what is required to make this a reality. Regardless, given the shere numbers of small, networkable devices in our society, we need to explore how to best exploit untapped resources in mobile computing, providing additional, cheaper environments for small applications, decreasing the dependency we have upon large data centers, and hopefully reducing the draw on power grids. We need decentralized, secure storage and processing. We need smarter, more fair, consumer-as-beneficiary, economies.

Next, we develope a new segment of the market, where any user or company with one or more networked devices would be able to log in to an open heap provider's software and offer their machine as another member in that cloud. There's a lot of work involved in making that happen, much of it focused on the design and implementation of really good software.

If we can accomplish all that, we will have reinvented the cloud as something far greater and more flexible than it is today.

After the Cloud: Open Heaps

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

Refresher

Up to now, we've considered technical explorations and possible related future directions for the technology surrounding the support of distributed applications and infrastructure. This post takes a break and returns to thoughts of provisioning resources on small devices such as mobile phones. As stated in To Atomic Computation and Beyond:

This could be just the platform for running small processes in a distributed environment. And making it a reality could prove to be quite lucrative. A forthcoming blog post will explore more about the possibilities involved with phone clouds...

But first, I'm so tired of the term "cloud," so I did some free-association... from cloud to clouds to "tons of little clouds" to "close to the ground" to cumulus to heap (Latin for "cumulus"). Heap! It's irresistible :-)

"Open" is such a terribly abused word these days (more so than cloud), but using it as an adjective for a wild collection of ad-hoc, virtualized process spaces satisfies some subtle sense of humor. Open Heaps it is.


Starting Points

Let's think about the medium in our example: cellular telephony. Is there a potential market here? Here are some raw numbers from Wikipedia:

By November 2007, the total number of mobile phone subscriptions in the world had reached 3.3 billion, or half of the human population (although some users have multiple subscriptions, or inactive subscriptions), which also makes the mobile phone the most widely spread technology and the most common electronic device in the world.

I think we can count that as a tentative "yes."

Can we do this the easy way and just use TCP/IP? In other words, what about using WiFi phones or dual-mode mobile phones as the communication medium for devices in our open heaps? Well, that would certainly make many things much easier, since everything would stay in the TCP/IP universe. However, the market penetration of standard mobile phones is so much greater in comparison.

That being said, how many currently operating phones are capable of serving content on the internet, running background processes, etc.? Maybe only a small fraction, perhaps even enough to justify supporting only devices such as handhelds, smartphones, MIDs, UMPCs, and Netbooks.

Two possibilities for ventures here might be:

1. A startup that developed an Open Heap offering for any Internet-connected device.
2. A company that formed a partnership with one or more mobile carriers, acting as a bridge between the carrier-controlled network/device-management capabilities and the Internet.

The Business Problem

So, let's say we've got the technology ready to go that will allow users to upload a process hypervisor to their phones, and that this technology provides the ability for users to allot process resources (e.g., RAM, CPU, storage). There are still a couple basic principles to address to justify a business in this area:

1. How will people be better off with this than without it?
2. How will this technology generate revenue?

In general, I believe that consumers are always better off with more choices. I also believe that balanced systems run better than those that are rigged to benefit just one group. As such, I have an idealist's interest in things like Open Heaps, as they will empower interested consumers to earn revenue (however small) on their own property (mobile phones and other devices with marketable resources). What's more, if there are billions of devices available as nodes in Open Heaps, and there is a computing demand for those resources, then there will inevitably be competitors aiming to capitalize on them. Generally speaking, I also believe that increased competition provides a better chance for improved quality of service.

Conversely, imagine that Open Heaps don't happen, that the idle resources of mobile devices (or any other eligible equipment) either remain untapped of their potential or, worse, are put to use by corporations that only desire the end consumer to have limited power over their own property and how it's used. Dire scenarios aren't difficult to imagine, thanks to various examples of anti-consumer behaviour we've seen from large corporations and special interest organizations in the recent past.

So, yes -- I think we can make a case for this being of benefit to consumers (and thus a marketer's dream!). The more prevalent mobile devices become, the more they will integrate into our daily lives... and the more important it will be that these are managed as the rightful property of the consumer, people that have the right to rent or lease and profit from their property as they see fit.

But, how could this generate revenue?

Next up: Gimme da cash!

After the Cloud: Prelude

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

These days, it seems that no matter where we go, we hear something about "the cloud." It's not really buzz anymore... it has become far more accepted and widely discussed to be that. For some organizations it's actually part of their current, every-day infrastructure. For others, it soon will be. As far as I'm concerned, now's the perfect time to start discussing what's next :-)

If you've spent any time reading some of the blog content I've managed to post over the past several years, you've probably noted that I like to explore the long view (if rather informally). Well, that's what I've got in store for you now: a series of blog posts that explore the long view of a post-cloud industry. Hopefully, with some new twists and turns along the way.

First off, I want to cover some basic ground, so the first couple posts might be a little less interesting that those that follow. Fortunately, I've been pondering these particular ideas since my month sabbatical last August -- this means I've already got most of the material written and ready to go!

These posts are going to take a peek at practical, hands-on ideas regarding the ways in which one might make use of current nascent tech to build prototypes for tomorrow's infrastructure, what that infrastructure might be, business ideas about what do do with that tech, and even future possibilities for information-based markets.

Hope you enjoy it as much as I've enjoyed thinking about it :-)

Ubuntu Developer Summit

For the past two weeks, I've been listening, learning, discussing, and hacking various Landscape and Ubuntu initiatives with members of the Ubuntu community and fellow Canonical employees. It was an amazing experience, and we've got the next 6 months crammed full of plans... with the next 3 months already spec'ed out.

Canonical has surprised me. It's an extraordinary company... both in the modern business-sense of the word as well as the original sense: a fellowship of companions with a common goal. While so far I have only had a chance to hear some personal histories, it's evident that every member of this company is an extraordinary individual with a rich background and a great deal to offer to the whole. Everyone works with an unprecedented amount of motivation towards the company vision, one that is well and tightly integrated into the corporate culture.

There is a bright future ahead for this amazing group...

New Directions

Yesterday I submitted my resignation as COO to the Divmod officers, and today I forwarded it to the rest of the team. Divmod is headed in a new and wonderful direction, and I'm happy to have contributed to raising public awareness about our team, community, and the tech we use, thus increasing its value in the market. I am taking this opportunity to rest and then pursue interests of my own.

Even more than that contribution, I'm delighted to have worked with these guys for the past year. I've long been a supporter and fan of Divmod (since shortly after its inception, in fact). I was a community contributor before I was an employee and I will remain so for the foreseeable future. But I gotta say, that team in incomparable. The combination of technical excellence, creativity, pragmatic problem solving, quality engineering, humor and insight has made my time there a rich experience. They have made my time at Divmod one for the personal record books.

I'm ready for a break, though; the past year has been a long, hard pull...

I was originally courted by them for management, due to my community work. I deferred, and worked as a coder instead. This ended up being invaluable, as far as the insight it provided. After some early successes with a product release, I was offered the position of CEO, but deferred there too, with the mutual agreement that COO might be a better match for my skills. After a couple months as COO, I was put in charge of managing the direction of the company and raising funds, so I ended up being acting CEO anyway. I poured my heart and soul into Divmod, and it looks like that has payed off: the team is happy and they're headed for some good success. What's more, that leaves me in the enviable position of finally being able to surrender a massive workload :-)

For the next week or two, I'll be camping in the Rocky Mountains catching up on some rest and enjoying nature at her best :-) I've also got some fun sci-fi reading to catch up on (Stross and MacLeod).

When I get back, I'll be exploring 6-12 month consulting contracts... so if anyone hears anything interesting, do let me know!

Divmod/Holden Web Partnership

As mentioned before in a few tweets and blog posts, Divmod's been working with Holden Web a lot lately. After lots of brainstorming, sweet jam sessions, and planning, we're finally ready to talk about it: our latest news item says it all :-)

We've got information up on the site about the topics covered in our joint training courses and workshops. You can contact either Divmod or Holden Web with any questions.

The Twisted Think Tank

I'm pleased to announce that we've got a new Divmod site up! We're still making tweaks, but it's ready for public viewing, and open for business.

This change currently doesn't affect our subscriber services... but it will, very shortly :-) JP's working on that now.

Anyone who knows us, knows that we know Twisted. We really know it. And how could we not, with Twisted superheroes like Glyph and JP? We've been solving very interesting problems for the past couple years, and other companies have availed themselves of this expertise. We're no longer trying to hide from our destiny as "the Twisted company."

We've found that providing specialized consulting services has not detracted from our core competency as software developers, but has rather done quite the reverse: provided a great deal of insight and clarity. The two activities have established a complementary feedback mechanism for growth and invention.

The Problem with and Solution to Google's App Engine

I know everyone is all aglow with the new web development offering from Google, but let me do the unpopular thing and put some things into perspective: there are limitations.

In fact, the limitations that exist will prevent me from using App Engine with all of my projects, save one (that one being a very simple web site). First, the limitations that prevent me from using App Engine (from one of their FAQs):

• Sockets are disabled with Google App Engine
• The system does not allow you to invoke subprocesses, as a result some os module methods are disabled
• Threading is not available

This means that I can't write a deferred wrapper for their data layer, I can't use Twisted for such things as XML-RPC or AMP-based communications, and I can't use an async templating system (like Nevow). I'm stuck with CGI and blocking code. And for all but the simplest projects, that's a big "No Thank You" from me.

This doesn't mean that I won't use it -- I will. I have one project that this will be perfect for... but it's for someone else, not me.

However, these limitations are actually good news :-) Here's the silver lining:

As Glyph as alluded to in his recent blog post (and in our tweets), we've recently completed a massive week-long BizDev Divmod sprint in Boston. One of the results of this is based on community feedback we've had over the last year, and which culminated at PyCon 2008 in Chicago with multiple requests for particular services from The Twisted Company. That result is a set of tools, features, and management options folks will be able to use with our software (app server, smart object db, network services, etc.). People really want to start using our stuff in cloud/grid computing environments. They need support for multiple and diverse network services, inter-store communications, massive deployments, etc. Two months before PyCon, we started working on tickets to support this, and we're making excellent progress toward providing the requested features.

We're still unclear as to which parts of this will be open source, as that will be driven by a combination of business and community demand. Regardless, Google's lack of support for this stuff has (for now) left the field wide open for us. And that, folks, is a big "Thank You Google!" :-)

Generic Functions and Rule-Based Systems

Well, I've been pretty bummed that business is keeping me too occupied to attend PyCon 2005 this year. Last year's was so great, and I had a chance to sprint with Chris McDonough and Donovan Preston and learned a great deal from them both.

In fact, I was decompressing the data I took in during the python sprints for the following 6 months. I guess it shows what a geek I am too, that I bemoan too much business, as it's preventing me attending this year.

I was checking out the presentations here, and I came across Phillip Eby's presentation here. This really hit home, since one of the projects I am working on is for processing *massive* amounts of data produced while testing equipment at manufacturing plants. There are sets of sets of sets of ... tests. Thousands of the buggers. Many of them are very similar in nature, and they all have associated rules for pass and failure.

I can't wait to tryout Phillip's generic functions in PyProtocol's... in fact, I haven't waited ;-) I'm already writing some prototype code for it...

Python rocks.

Internet Super Heros

Allen Lerner and I were hanging out the other day, jamming about AdytumSolutions and E-Secure Systems... he was talking about working out, and feeling better about himself - more confident, more
comfortable. As he was describing his self image and the introspection that had led him to want to redefine himself, I got the coolest visual image...

I saw something akin to the super hero transformation that occurs in the time of need: from work-a-day cloths, to flying-out-of-phone-booth tights and cape. Not that I saw Allen in tights, or anything... ;-)

But as that image soaked into my brain, it struck me how well it fit what we do - especially what E-Secure Systems does. We are the techno super-heros; all of us that work in this industry; we're like the Hall of Justice, where uptimes, network security, and customer satisfaction are the law that we support. We really started getting into this, too!

Allen called me later that night, and just went off. I was laughing my ass off. He did this whole improve on internet superheros that was just hysterical - and true.

It was just fantastic :-)