After the Cloud: Prelude

After the Cloud:

1. Prelude
2. So Far
3. The New Big
4. To Atomic Computation and Beyond
5. Open Heaps
6. Heaps of Cash
7. Epilogue

These days, it seems that no matter where we go, we hear something about "the cloud." It's not really buzz anymore... it has become far more accepted and widely discussed to be that. For some organizations it's actually part of their current, every-day infrastructure. For others, it soon will be. As far as I'm concerned, now's the perfect time to start discussing what's next :-)

If you've spent any time reading some of the blog content I've managed to post over the past several years, you've probably noted that I like to explore the long view (if rather informally). Well, that's what I've got in store for you now: a series of blog posts that explore the long view of a post-cloud industry. Hopefully, with some new twists and turns along the way.

First off, I want to cover some basic ground, so the first couple posts might be a little less interesting that those that follow. Fortunately, I've been pondering these particular ideas since my month sabbatical last August -- this means I've already got most of the material written and ready to go!

These posts are going to take a peek at practical, hands-on ideas regarding the ways in which one might make use of current nascent tech to build prototypes for tomorrow's infrastructure, what that infrastructure might be, business ideas about what do do with that tech, and even future possibilities for information-based markets.

Hope you enjoy it as much as I've enjoyed thinking about it :-)

ULS-SIG: New Python Special Interest Group

After several discussions at the end of last summer and an incubation in Meta-SIG, Steve Holden, Jim Baker and I are pleased to announce the Python special interest group for ultra large-scale systems. For more about ULS system, you might want to read this post or this one. The SIGs page has been updated, so you can find us there with the rest of them (subscribe and archive links, too), and we've even got our own page :-)

The initial group of interested parties (and thus the first members of the list) represent an interesting cross-section of the Python community, including the following:

• Jython hackers
• Twisted hackers
• Stackless hackers
• XMPP experts
• MMPORG developers
• SOA and Business Process consultants
• General technology and software companies

The technological umbrella of ULS systems covers a vast array of topics and interests, but basic principle unifying all of these is their potential contribution to making massive, highly distributed systems a functional reality.

One overview of ULS systems research states that we currently don't have an effective understanding of software (it's nature, development, and management) at the scale anticipated for ULS systems. These "fundamental gaps" will hinder the development of such systems until they can be crossed. Doing so will require breakthroughs in many fields with insights and experience gained over time.

Python programmers represent an extraordinary segment of the population: creative, curious, motivated, communicative, and deeply intelligent individuals. Our community is filled with minds that continuously produce solutions for a vast array of problems across a great many disciplines.
If there's any one group out there that could pull this off, I think it's ours :-)

A future post will provide a sketch of areas of interest in Python that are already sneaking up on the gaps outlined in ULS systems reports. There is a lot of software and supporting libraries that have an obvious connection to ULS systems, but even more fun are the ones that don't... and isn't always the darkest corners that yield the most unlooked for surprises?

While you're waiting for that, though, feel free to join us on the mail list!

Newest Members of the PSF

Since it's now official, I can blog about it: I was delighted to discover that JP, radix, glyph, and I were voted into the Python Software Foundation this year at PyCon :-) Not only is that four for Twisted, but it's two for Divmod and two more for Canonical.

There was another Canonical employee -- Matthias Klose -- voted in as well, and with Barry Warsaw and Gustavo Niemeyer, that brings a tally for Canonical/Ubuntu to at least 5, and maybe more (let me know if I've missed you!).

It gets even better, though: check out the rest of the new members list:

• Jim Baker
• Ben Bangert
• James Bennett
• Graham Dumpleton
• Martijn Faassen
• Michael Fletcher
• Michael Foord
• Doug Hellmann
• Adrian Holovaty
• Jacob Kaplan-Moss
• Jesse Noller
• Benjamin Peterson
• Ted Pollari
• Mark Ramm
• Malcolm Tredinnick
• Kirby Urner
• Robert Dino Viehland
• Thomas Waldmann
• Frank Wierzbicki

I am thrilled to be a PSF member and look forward to deepening my support of Python through this new level of involvement. Even more, though, I'm honored and delighted to be working with both the esteemed veteran members as well as these amazing new additions.

txLoadBalancer Lightening Talk?

As I mentioned in this post, I wasn't able to make it to PyCon this year. I'm super bummed now, 'cause I just got an email about a potential lighting talk for txLoadBalancer.

I'm not sure when tomorrow it has been tentatively scheduled for, but go! And then tell me how it was :-)

As I told one of the presenters (Jehiah Czebotar), this is really motivating me to get the next release out :-)

New PyRRD Release: 0.0.7

Version 0.1 is nearing as the 7th release made it out the door tonight. The latest features include RRD info/fetch methods, a simple RRD object mapper, and the ability to dump files programmatically. Various bug fixes have been applied, thanks to feedback and patches from the community. In particular,

Type the characters you see in the picture below.

Letters are not case-sensitive

and

Aaron Westendorf of Agora Games, Leem Smit, and nasvos.

You can download from PyPI (or use setuptools to install it) at the expected location:
http://pypi.python.org/pypi/PyRRD/

Note that the the PyPI page has a quick run-down on the basic features and how to use them.

The next feature to be implemented has already been planned out (it's just a matter of sitting down and writing the code now): adding support for the Python bindings. If this is important to you, be sure to star the issue in the bug tracker:
http://code.google.com/p/pyrrd/issues/detail?id=5&can=1

Hot Talks at PyCon 2009

This year's PyCon talks look absolutely fabulous, and should be a real treat for attendees. I'm probably not going to be able to make it (working for a startup last year put me in a financial position I'm only slowly recovering from). Since it looks like the Landscape team won't be sprinting there this year, Canonical won't be footing the bill.

Were I to attend, though, the talks I have my eyes on are the following (listed in visual scanning order of the schedule):

• Twisted, AMQP and Thrift: Bridging messaging and RPC for building scalable distributed applications (#40)
• Panel: Python VMs (#15)
• Building tests for large, untested codebases (#112)
• Introduction to Multiprocessing in Python (#6)
• The State of the Python Community: Leading the Python tribe (#118)
• PyPy status talk. (#72)
• Concurrency and Distributed Computing with Python Today (#49)
• Batteries Included! Python on Low Cost Tiny Embedded Wireless Devices (#28)
• Easy AI with Python (#115)
• Abstraction as Leverage (#110)
• Metaprogramming with Decorators and Metaclasses (#120)

There are many different wonderful topics being covered at PyCon this year -- this just happens to be the list that converges most closely with my own interests :-) No offense if I've left you out!

I will be sorry to have to miss these, but I will greatly look forward to their audio or video recordings as well as the presenters' published slides and papers.

A Champion for Services Architecture Done Right

I've heard a series of discussions lately and have read some articles that I've found rather frustrating, all on the topic of SOA. I've taken a few notes about my objections and had several blog posts planned where I would attempt to articulate the historical failures of software, services, and/or architecture makeovers performed by any number of consulting firms for large IT shops. I was going to discuss the horrible irresponsibility of basing business decisions on buzzword flinging cowboys, and the inevitable price that is paid by companies wooed and seduced by flash.

I was going to start this off by launching a strong criticism of Anne Thomas Manes' views on the "death of SOA", one that I'd read about in a recent article. It seemed that such a view was sensationalist and entirely missed the point of the essence of service oriented architectures.

But before I did that, though, I needed to do some research and present a solid argument, not just take some trade rag's version of the story with a superficial smattering of misquoted details. And holy cow... not only am I glad that I did this, but I am now an ardent fan of Ms. Manes, Vice President and Research Director at the Burton Group.

Her post SOA is Dead; Long Live Services is brilliant. The death of which she speaks is not the mortal (and figurative) soul of SOA itself -- any individual or magazine claiming as much has either completely misunderstood her or has been making its money on the misunderstandings and hype that she would like to see die. She asserts that the very heart of SOA continues to be badly needed, that "the requirement for service-oriented architecture is stronger than ever." She then goes on to say:

The acronym got in the way. People forgot what SOA stands for. They were too wrapped up in silly technology debates (e.g., “what’s the best ESB?” or “WS-* vs. REST”), and they missed the important stuff: architecture and services.

And finishes with this bit of excellence:

The latest shiny new technology will not make things better. Incremental integration projects will not lead to significantly reduced costs and increased agility. If you want spectacular gains, then you need to make a spectacular commitment to change.

And you know what? Her other posts are even better. While reading all of them, I was completely dumbfounded that she said so well and clearly what the industry has been misinterpreting for several years. Why are more people not listening to her? How did I not hear of her until now?! I'll tell you what, I'm now subscribed to Burton Group's Application Platform Strategies blog.

IT shops need to pay close attention to what this woman and her team have to say. She really knows what she's talking about and articulates her views exquisitely. That whole series of blog post I wanted to make? Yeah... she already did that :-) (and way, way better than I could have). Please, I beg you: go read her posts. If you are the director of your IT shop or someone to whom your director listens and respects and you have any interest in distributed services, messaging, and related architectures, go memorize her statements, summaries, arguments and positions. Let her experience and wealth of knowledge guide judicial and wise decisions at your company.

And remember, it is the good, well thought out, and cost-effective architecture that survives. It is this architecture -- with countless peers -- that serve as the basis for the next generation of improvements and advances. Technologies produced in this manner are the ones that propel us into the future and prepare us for a world which we are incapable of imagining or implementing today.

Expert Python Programming, a Book Review

A few months back, I received an email from someone at Packt publishing asking whether I'd like a complementary copy of their Expert Python Programming book, and if I would blog about it. I accepted and read the book immediately. However, their request caused me to deeply consider my personal position on grassroots marketing for corporations.

Ultimately, I decided that it wasn't the publisher nor even the author that really mattered or entered into my assessment equation at all. Instead, it's the potential benefit that can be brought to members of the community. So, at long last, I am writing the review :-)

The author of this work is Tarek Ziadé, is a Python contributor, Plone contributor, CTO of Ingeniweb, author of a French Python book, spoke at OSCON last year, and will be talking at PyCon this year. I don't know him personally, but given his credentials, he is a well-qualified and experienced programmer.

Expert Python Programming is actually a shorter book than I expected, coming in at 352 numbered pages. It's also a fairly different book than I expected as well. If these were times when books were given outrageously long titles, I might have suggested something along the lines of:

Expert Python Programming

Wherein the Tools and Methodologies of the Experts

are Described and Discussed

The amount of material that is covered between its covers, though, is impressive. Check out the table of contents to confirm this. This is essentially a book of best practices and tools for Python developers wanting to move from an intermediate (or advanced beginner) stage towards the levels at which experienced programmers operate. In addition to discussing tools and methodologies, Tarek takes the reader on a quick tour of the entire development process -- an invaluable guide for programmers wanting to hit their stride in larger open source projects or on critical company software initiatives.

In Tarek's introductory blog post, he quoted Shannon -jj Behrens as saying the following:

"If you’re looking to progress from knowing Python to mastering Python, this is the book for you. In fact, this is exactly the type of book I wish I had had five years ago. What took me years to discover by steadfastly attending talks at PyCon and my local Python users’ group is now available in a succinct book form."

This is well said... though I would add that this book in conjunction with Python Cookbook (especially the last several chapters) can get you this progress that Shannon mentions :-)

One tiny little quibble ... Given that this book might be read by developers who have no experience with distributed version control systems, I would have liked to have seen Bazaar and Darcs mentioned in addition to Mercurial and Git. The reason being that a developer reading this book as her introduction to working with others in a distributed environment may be given the impression that Mercurial is "the Python way", and that's not necessarily the case ;-)

I've already personally recommended this book to curious and motivated beginning Python programmer friends (as well as an intermediate programmer who felt like he was stuck in a skill rut). If you're one of these, consider this my recommendation to you too :-)

Window Maker and Ubuntu

Pictured right is the new setup that I've recently started using for development. I've found it to be much, much faster and more memory efficient than Gnome. When I'm developing, sometimes I want to run several server and/or client instances, and I need as many resources as possible, ready for crazy, unexpected usages. Window Maker fits the bill.

As a high school student in the 80s, I lusted after NeXT boxes and the look of NeXTSTEP. I had a collection of glossy pamphlets from the company that were kept out on my desk for regular ogling. The best I could do though, was incessantly use the Macs at the University of Maine at Orono (where I spent as much time as possible). When, as a new Linux user in '96 or '97, I discovered AfterStep, I switched from the Motif-alike FVWM. Immediately after Window Maker was released, I choose it as my primary window manager.

Since then, the Linux distributions and operating systems I've used have been distractingly varied. However, after several excellent years with Mac OS X, I'm actually quite happy to be back in Linux. Though I have generally enjoyed Gnome, I was really unhappy with how it seemed run rather slow, with delayed transitions between applications and other similar operations. With a new machine having 4GB of RAM, this seemed rather unnecessary. (However, I do run some heavy services on my machine... databases, web servers, etc.)

When I switched to Window Maker, I was stunned. Number one, it still works on a modern distro! Secondly, it's super fast. Finally, I *love* the clean desktop (I'm not a fan of having the desktop background display the contents of a folder). Not only that, but a quick test drive of some Python dockapp-writing software left me with some fun ideas for side-projects I could write to even better augment my working environment. (For example, dock apps for NetworkManager, update notifications, and Rhythmbox.)

Now I'm ready to start playing with architecture emulation for some exploratory networking projects...

Python and Rhythmbox

I've got a fairly large collection of digital music on a networked drive, and I access it from multiple machines on the network. I consolidated it 5 years ago when I started using iTunes, and over the past few years picked up about 400 songs from the iTunes store. This is something I avoid now, since Amazon offers songs at higher bitrates and without the crippling, non-Fair Use of DRM. (Apple seems to have recently changed it's policy, though the pain their DRM crap has caused me doesn't make me a very loyal customer.)

Since I started working at Canonical, I've been using Ubuntu for more than just development -- it's my main-use machine. I still use iTunes every once in a while, but my primary media player is Rhythmbox. There are a couple of issues with the old library, though.

Obviously, Rhythmbox can't play Apple's encrypted .m4p files or a couple audio book files I have. What's more, there are about 200 files that iTunes is able to locate but which Rhythmbox cannot (this may be due to the differing case sensitivities of the respective OSs). In order to track all these issues down conveniently, I wanted to export the import errors and missing files as a text file. Sadly, Rhythmbox doesn't have this functionality.

Fortunately, it comes with a Python console :-)

The missing files export was fairly easy, after some digging around and poking at the Python objects:

Try as I might, I was completely unable to obtain similar data for the import errors. After looking at the C code, I was able to determine that though the import errors were treated generally as a media source, due to their nature (not being able to provide the actual media itself), the related meta data was handled differently. Yet I wasn't able to decipher how, exactly.

So, I hopped on their mail list and asked for help :-) After a few quick exchanges, I was pointed in the right direction by one of the developers, who said that I needed to make use of the db object and some constants. After a quick test, this advice resulted in the following:


Note that the shell and rhythmdb objects are exposed by Rhythmbox in Python console sessions.

I now have a complete list of files that either need some file name updates or need to be burned to CD in iTunes and ripped to OGG.

So far, I've been pretty pleased with Rhythmbox. Thanks to their use of Python, I find I'm now becoming somewhat of a fan :-)