A Sinfonia on Messaging:
A complete messaging solution is a three-part invention of business need, architecture, and implementation. In its final form, these three voices blend in harmony, with each one taking a dominant role depending upon which part of the solution one examines.
Neither have I the ability nor skill to seamlessly weave three concepts together while clearly explaining their roles. Therefore, I will separate out the voices from each other and leave it as an exercise for the reader to construct an application and practice the principles involved, thus experiencing well-earned contrapuntal pleasures first-hand.
As computing and data exchange systems increased in complexity over the past 30 years, so has the need for improvements -- and where possible, simplifications. Some of these efforts have been focused on decentralization of communications (shared, distributed load) and decoupling of messaging from applications (removing redundancy and increasing delivery speed/throughput). The first steps towards this were made in the 1990s with explorations in "middleware" application universe.
Messaging, as we now refer to it in the industry, arouse from those middleware adventures: out of the business drive to refactor old software as new services to wider, more sophisticated audiences. With many new services replacing a single, monolithic application, formal and well-architected solutions were needed for creating, editing, and deleting shared data.
Message-oriented middleware (MOM) is infrastructure focused on message sending that increases the interoperability, portability, and flexibility of an application by allowing the application to be distributed over multiple heterogeneous platforms. It reduces the complexity of developing applications that span multiple operating systems and network protocols by insulating the application developer from the details of the various operating system and network interfaces.AMQP (Advanced Message Queuing Protocol) is one of these protcols.
In a recent blog post about Ubuntu as a business server, Vaughan-Nichols provides evidence for Ubuntu's and Canonical's commitment to enterprise, saying "... the new [version of] Ubuntu also includes AMQP [...] support. AMQP is an important set of middleware and SOA [...] protocols."
AMQP has demonstrated itself as a compelling protocol for messaging solutions, even to the point of being included in two Linux distributions. The code included in this blog series is txAMQP, an asynchronous Python AMQP library built with Twisted.
Note that last year I had planned to write a blog series on messaging with Twisted and XMPP, but was unable to as a result of time constraints. These days, I'm working with AMQP instead of XMPP, but I still hold some hope that I'll be able to write an anolog for this series from the perspective of XMPP.
The Voice of Business
Dipping into the future, one of the goals for AMQP as developed by a special interest group is the following:
Decentralized, Locally Governed Federated Mesh of AMQP Brokers with standardized Global Addressing. The killer application for AMQP is transacted secure business messages between corporations - e.g. send a banking confirmation message to firstname.lastname@example.org [...]I find this rather exciting due to my interest in ultra large-scale systems; scenarios like the one described above are the seeds for tomorrow's ULS systems :-)
For now, though, let's look at a more immediate use for AMQP: a messaging protocol for shared services between departments in a small store. In this exercise, the voice of business is the primary melody; everything else (architecture and implementation) is done in support of this theme.
Fhionnlaidh MacGrudder creates hand-made kilts to order. He's got a sales guy who works with movie costume design shops and the like. He's got a web girl who wrote and maintains a custom store front app. He's got a friend who does shipping and billing for him (as well as some other local Glen Orchy artisans). Until now, these three business "groups" associated with the kilt shop have been maintaining their own records, sometimes copying and updating them manually from each others' various export files.
Fhionnlaidh's niece Fíona is programmer, business student, and is dating shipping guy's son. Horrified by the inefficiencies in her uncle's business processes (and tired of her boyfriend's father's complaints), she has proposed the following:
- sales guy will maintain customer contact info offer this as a service to the store and billing
- the web store will dynamically update displayed data when sales guy changes it
- the CRM will dynamically update displayed data when a web store customer updates their info
- MacGrudder will have a new web page he goes to where all pending orders are presented with their full details; changes can be updated by a customer in real-time until MacGrudder has started working on the order
- billing/shipping guy will be notified instantly as soon as MacGrudder marks a kilt order as completed
- Contacts will be maintained in a single data store
- There is zero latency between customer-driven updates and sales guy-driven updates
- Customers have increased post-purchase flexibility with their orders
- Shipping guy can plug into MacGudder's messaging and be notified when packages are ready for pickup
- Everyone has more time for buttered scones and tea (especially shipping guy, who will no longer be making unneeded trips down the glen)
- Contacts will need to be merged into the CRM
- A read/write data service for the contacts will need to be created
- The CRM front-end will need to be upgraded to an AJAX-enabled version
- The web store app will need to be updated to support AJAX
- A new page will be created which displays the status of all orders and allows MacGrudder to change an order from "pending" to "in-progress" to "completed"
- The current "new order" email notification code in the web app will need to be changed so that it uses the same messaging as MacGrudder's status page
- A new service needs to be created for shipping guy so that he can choose to be notified about pending pickups by email or he can check a web page or even make a query directly to the service, thus preventing unnecessary trips to MacGrudder's isolated little shop
- After all the work is done, somone's going to need to order more scones
In the next installment, we'll review the business process (with diagrams!) and the explore the architecture of the system, before and after. Another post will take that architecture and combine it with MacGrudder's already extant infrastructure, reusing as much as possible. With that in place, we will have the opportunity to look at some RabbitMQ basics and some actual txAMQP code.
- SOA in Practice
- Message-oriented middleware on Wikipedia
- AMQP on Wikipedia
- The AMQP Spec
- A synchronous Python library for AMQP
- The async Python library for AMQP
- A txAMQP example
- An intro to messaging and py-amqplib