txLoadBalancer Lightening Talk?

As I mentioned in this post, I wasn't able to make it to PyCon this year. I'm super bummed now, 'cause I just got an email about a potential lighting talk for txLoadBalancer.

I'm not sure when tomorrow it has been tentatively scheduled for, but go! And then tell me how it was :-)

As I told one of the presenters (Jehiah Czebotar), this is really motivating me to get the next release out :-)

New PyRRD Release: 0.0.7

Version 0.1 is nearing as the 7th release made it out the door tonight. The latest features include RRD info/fetch methods, a simple RRD object mapper, and the ability to dump files programmatically. Various bug fixes have been applied, thanks to feedback and patches from the community. In particular,

Type the characters you see in the picture below.

Letters are not case-sensitive

and

Aaron Westendorf of Agora Games, Leem Smit, and nasvos.

You can download from PyPI (or use setuptools to install it) at the expected location:
http://pypi.python.org/pypi/PyRRD/

Note that the the PyPI page has a quick run-down on the basic features and how to use them.

The next feature to be implemented has already been planned out (it's just a matter of sitting down and writing the code now): adding support for the Python bindings. If this is important to you, be sure to star the issue in the bug tracker:
http://code.google.com/p/pyrrd/issues/detail?id=5&can=1

Hot Talks at PyCon 2009

This year's PyCon talks look absolutely fabulous, and should be a real treat for attendees. I'm probably not going to be able to make it (working for a startup last year put me in a financial position I'm only slowly recovering from). Since it looks like the Landscape team won't be sprinting there this year, Canonical won't be footing the bill.

Were I to attend, though, the talks I have my eyes on are the following (listed in visual scanning order of the schedule):

• Twisted, AMQP and Thrift: Bridging messaging and RPC for building scalable distributed applications (#40)
• Panel: Python VMs (#15)
• Building tests for large, untested codebases (#112)
• Introduction to Multiprocessing in Python (#6)
• The State of the Python Community: Leading the Python tribe (#118)
• PyPy status talk. (#72)
• Concurrency and Distributed Computing with Python Today (#49)
• Batteries Included! Python on Low Cost Tiny Embedded Wireless Devices (#28)
• Easy AI with Python (#115)
• Abstraction as Leverage (#110)
• Metaprogramming with Decorators and Metaclasses (#120)

There are many different wonderful topics being covered at PyCon this year -- this just happens to be the list that converges most closely with my own interests :-) No offense if I've left you out!

I will be sorry to have to miss these, but I will greatly look forward to their audio or video recordings as well as the presenters' published slides and papers.

A Champion for Services Architecture Done Right

I've heard a series of discussions lately and have read some articles that I've found rather frustrating, all on the topic of SOA. I've taken a few notes about my objections and had several blog posts planned where I would attempt to articulate the historical failures of software, services, and/or architecture makeovers performed by any number of consulting firms for large IT shops. I was going to discuss the horrible irresponsibility of basing business decisions on buzzword flinging cowboys, and the inevitable price that is paid by companies wooed and seduced by flash.

I was going to start this off by launching a strong criticism of Anne Thomas Manes' views on the "death of SOA", one that I'd read about in a recent article. It seemed that such a view was sensationalist and entirely missed the point of the essence of service oriented architectures.

But before I did that, though, I needed to do some research and present a solid argument, not just take some trade rag's version of the story with a superficial smattering of misquoted details. And holy cow... not only am I glad that I did this, but I am now an ardent fan of Ms. Manes, Vice President and Research Director at the Burton Group.

Her post SOA is Dead; Long Live Services is brilliant. The death of which she speaks is not the mortal (and figurative) soul of SOA itself -- any individual or magazine claiming as much has either completely misunderstood her or has been making its money on the misunderstandings and hype that she would like to see die. She asserts that the very heart of SOA continues to be badly needed, that "the requirement for service-oriented architecture is stronger than ever." She then goes on to say:

The acronym got in the way. People forgot what SOA stands for. They were too wrapped up in silly technology debates (e.g., “what’s the best ESB?” or “WS-* vs. REST”), and they missed the important stuff: architecture and services.

And finishes with this bit of excellence:

The latest shiny new technology will not make things better. Incremental integration projects will not lead to significantly reduced costs and increased agility. If you want spectacular gains, then you need to make a spectacular commitment to change.

And you know what? Her other posts are even better. While reading all of them, I was completely dumbfounded that she said so well and clearly what the industry has been misinterpreting for several years. Why are more people not listening to her? How did I not hear of her until now?! I'll tell you what, I'm now subscribed to Burton Group's Application Platform Strategies blog.

IT shops need to pay close attention to what this woman and her team have to say. She really knows what she's talking about and articulates her views exquisitely. That whole series of blog post I wanted to make? Yeah... she already did that :-) (and way, way better than I could have). Please, I beg you: go read her posts. If you are the director of your IT shop or someone to whom your director listens and respects and you have any interest in distributed services, messaging, and related architectures, go memorize her statements, summaries, arguments and positions. Let her experience and wealth of knowledge guide judicial and wise decisions at your company.

And remember, it is the good, well thought out, and cost-effective architecture that survives. It is this architecture -- with countless peers -- that serve as the basis for the next generation of improvements and advances. Technologies produced in this manner are the ones that propel us into the future and prepare us for a world which we are incapable of imagining or implementing today.

Expert Python Programming, a Book Review

A few months back, I received an email from someone at Packt publishing asking whether I'd like a complementary copy of their Expert Python Programming book, and if I would blog about it. I accepted and read the book immediately. However, their request caused me to deeply consider my personal position on grassroots marketing for corporations.

Ultimately, I decided that it wasn't the publisher nor even the author that really mattered or entered into my assessment equation at all. Instead, it's the potential benefit that can be brought to members of the community. So, at long last, I am writing the review :-)

The author of this work is Tarek Ziadé, is a Python contributor, Plone contributor, CTO of Ingeniweb, author of a French Python book, spoke at OSCON last year, and will be talking at PyCon this year. I don't know him personally, but given his credentials, he is a well-qualified and experienced programmer.

Expert Python Programming is actually a shorter book than I expected, coming in at 352 numbered pages. It's also a fairly different book than I expected as well. If these were times when books were given outrageously long titles, I might have suggested something along the lines of:

Expert Python Programming

Wherein the Tools and Methodologies of the Experts

are Described and Discussed

The amount of material that is covered between its covers, though, is impressive. Check out the table of contents to confirm this. This is essentially a book of best practices and tools for Python developers wanting to move from an intermediate (or advanced beginner) stage towards the levels at which experienced programmers operate. In addition to discussing tools and methodologies, Tarek takes the reader on a quick tour of the entire development process -- an invaluable guide for programmers wanting to hit their stride in larger open source projects or on critical company software initiatives.

In Tarek's introductory blog post, he quoted Shannon -jj Behrens as saying the following:

"If you’re looking to progress from knowing Python to mastering Python, this is the book for you. In fact, this is exactly the type of book I wish I had had five years ago. What took me years to discover by steadfastly attending talks at PyCon and my local Python users’ group is now available in a succinct book form."

This is well said... though I would add that this book in conjunction with Python Cookbook (especially the last several chapters) can get you this progress that Shannon mentions :-)

One tiny little quibble ... Given that this book might be read by developers who have no experience with distributed version control systems, I would have liked to have seen Bazaar and Darcs mentioned in addition to Mercurial and Git. The reason being that a developer reading this book as her introduction to working with others in a distributed environment may be given the impression that Mercurial is "the Python way", and that's not necessarily the case ;-)

I've already personally recommended this book to curious and motivated beginning Python programmer friends (as well as an intermediate programmer who felt like he was stuck in a skill rut). If you're one of these, consider this my recommendation to you too :-)

Window Maker and Ubuntu

Pictured right is the new setup that I've recently started using for development. I've found it to be much, much faster and more memory efficient than Gnome. When I'm developing, sometimes I want to run several server and/or client instances, and I need as many resources as possible, ready for crazy, unexpected usages. Window Maker fits the bill.

As a high school student in the 80s, I lusted after NeXT boxes and the look of NeXTSTEP. I had a collection of glossy pamphlets from the company that were kept out on my desk for regular ogling. The best I could do though, was incessantly use the Macs at the University of Maine at Orono (where I spent as much time as possible). When, as a new Linux user in '96 or '97, I discovered AfterStep, I switched from the Motif-alike FVWM. Immediately after Window Maker was released, I choose it as my primary window manager.

Since then, the Linux distributions and operating systems I've used have been distractingly varied. However, after several excellent years with Mac OS X, I'm actually quite happy to be back in Linux. Though I have generally enjoyed Gnome, I was really unhappy with how it seemed run rather slow, with delayed transitions between applications and other similar operations. With a new machine having 4GB of RAM, this seemed rather unnecessary. (However, I do run some heavy services on my machine... databases, web servers, etc.)

When I switched to Window Maker, I was stunned. Number one, it still works on a modern distro! Secondly, it's super fast. Finally, I *love* the clean desktop (I'm not a fan of having the desktop background display the contents of a folder). Not only that, but a quick test drive of some Python dockapp-writing software left me with some fun ideas for side-projects I could write to even better augment my working environment. (For example, dock apps for NetworkManager, update notifications, and Rhythmbox.)

Now I'm ready to start playing with architecture emulation for some exploratory networking projects...

Python and Rhythmbox

I've got a fairly large collection of digital music on a networked drive, and I access it from multiple machines on the network. I consolidated it 5 years ago when I started using iTunes, and over the past few years picked up about 400 songs from the iTunes store. This is something I avoid now, since Amazon offers songs at higher bitrates and without the crippling, non-Fair Use of DRM. (Apple seems to have recently changed it's policy, though the pain their DRM crap has caused me doesn't make me a very loyal customer.)

Since I started working at Canonical, I've been using Ubuntu for more than just development -- it's my main-use machine. I still use iTunes every once in a while, but my primary media player is Rhythmbox. There are a couple of issues with the old library, though.

Obviously, Rhythmbox can't play Apple's encrypted .m4p files or a couple audio book files I have. What's more, there are about 200 files that iTunes is able to locate but which Rhythmbox cannot (this may be due to the differing case sensitivities of the respective OSs). In order to track all these issues down conveniently, I wanted to export the import errors and missing files as a text file. Sadly, Rhythmbox doesn't have this functionality.

Fortunately, it comes with a Python console :-)

The missing files export was fairly easy, after some digging around and poking at the Python objects:

Try as I might, I was completely unable to obtain similar data for the import errors. After looking at the C code, I was able to determine that though the import errors were treated generally as a media source, due to their nature (not being able to provide the actual media itself), the related meta data was handled differently. Yet I wasn't able to decipher how, exactly.

So, I hopped on their mail list and asked for help :-) After a few quick exchanges, I was pointed in the right direction by one of the developers, who said that I needed to make use of the db object and some constants. After a quick test, this advice resulted in the following:


Note that the shell and rhythmdb objects are exposed by Rhythmbox in Python console sessions.

I now have a complete list of files that either need some file name updates or need to be burned to CD in iTunes and ripped to OGG.

So far, I've been pretty pleased with Rhythmbox. Thanks to their use of Python, I find I'm now becoming somewhat of a fan :-)

Twisted Mail Server: The Conclusion

Holy old code, Batman -- it's been about 2.5 years since I first blogged about the Twisted mail server I cobbled together. In the intervening time, I've received tons of emails and instant messages requesting that the code be put up somewhere. Well, that time has come...

I recently decommissioned a box I had at tummy.com which was my primary mail server for several years. After many years of qmail and a few months with Postfix, I wanted a solution that I had more immediate control over. At that time, Gmail for Domains was just announced and I became one of the beta testers for it. However, it was missing a few critical features, so I sat down and put together a mail server for myself. Abe Fetting had some nice examples in his O'Reilly book, so I started there. Then I jumped into the Twisted source code and discovered the rest that I needed.

A year or so later, Gmail for Domains (now Google Apps for business) was rockin' out, so I started moving my mail there. I eventually aliased the remainder of my domains and then stopped using the Twisted mail server. I kept the tummy.com box around for a while for DNS, but due to a tightening of the budget at home, I later had to pull the plug. When I reviewed the backup files from the server, I saw the old mail code and put it at the top of the list for my down-time projects.

I've sort of done that, but the process is incomplete. In the spirit of r0ml's OSCON 2008 talk, I'm placing this code in a public space for people to play with, find bugs in, report them, add patches, and branch from. The project page on Launchpad.net is here:

https://launchpad.net/tx/txmailserver

There are a couple non-standard parts of the code (in particular, the account, alias, and mail list management), but it can be improved very easily. The Dspam stuff was nascent at best (I didn't end up using it in production). Also, the relaying really needs to be looked into to make sure that it's done safely. A quick task someone could jump on right away is to convert the copious print statments to twisted.python.log calls.

Having the code in the state that it's in could be a lot of fun, really. The code base is tiny and there's nothing too tricky going on. For those that haven't had a chance to play with Bazaar, you can easily branch it to your own lp home dir, and if you make some cool improvements, I can easily merge those back in to the main branch. Looking at the list of branches, potential users and developers will be able to easily see the efforts that others have made. txMailServer could end up being a nice little piece of utilitarian code for folks...

Enjoy! And sorry it took so long :-(

Intellectual Property and Open Source

A few months ago, I received a complementary copy of Van Lindberg's new O'Reilly book Intellectual Property and Open Source: A Practical Guide to Protecting Code and the first thing that happened at home when the book was unwrapped was three of us began arguing over who got to read it first.

This may seem like an odd thing to happen for what one could easily assume was a dry and less than interesting topic. However, at the time I was strongly considering the possibility of beginning a non-tech-industry startup built with both open source and proprietary code. The discussions with the potential founders of the startup had been very vigorous and exciting, but the big questions that remained revolved around patents, protecting IP, and providing protection against big business while still offering powerful, free code for use by individuals/private consumers. If you've read the book or even seen the table of contents, you can see why everyone wanted to be the first to read it and learn from the insights provided between its covers.

Instead of jumping into another startup, I ended up joining Canonical; this has kept me both very busy and exceptionally happy. The holiday break has provided an opportunity to finish reading the book, and it has been a delight. I have friends working on startups that depend upon exciting code to power some or all of the business models for their visions, and reading this book should be on their shelves, close at hand. Even if you're not involved directly with open source and intellectual property, this book is an excellent read.

Intellectual Property and Open Source accomplishes a difficult goal of sharing dense information while making the subject matter engaging. This is done through examples, thought experiments, and well developed analogies. Van does an excellent job of igniting a powerful curiosity on the part of the reader while providing rewards for this in the lucid explanations of related laws and perspectives. I am resisting the urge to turn this post into a long series of quotes, but at the very least I want to mention a few little "spoilers" ;-)

The book starts off with an excellent foundation, giving an overview of the origins of intellectual property from an economic and legal perspective. This was particularly useful for me, as I have no background in this field. Van Lindberg does a really great job of expressing some of the widely held (and diverse) views of IP in the open source community.

The book then launches the reader into an array of well organized chapters on patents, the patent system, trademarks, copyright, trade secrets and licenses. Every open source developer should read chapter 10 on choosing an open source license (the opening dialog had me laughing out loud, a hilarious parody of news groups and IRC arguments as well as a nod to Princess Bride). There's also a chapter dedicated to patches and their relationships to copyright; another on reverse engineering; and the final one provides information and advice on establishing non-profits for open source projects -- the author even gives mention to our friends at the Software Freedom Conservancy (the umbrella non-profit for the Twisted Software Foundation).

In all honesty, I can't rave enough about this book. I've re-read parts of it just because I enjoyed the clarity of the explanations so much. Law is a twisty maze of easily confused subtleties to those who have not been trained in its dark arts. Through explicit language and examples, the author guides us past pitfalls of misunderstanding and brings us directly to all the major points.

If you are an Amazon shopper, you may want to act quickly: last I checked, there were only two copies left.

Enjoy!

Ubuntu Developer Summit

For the past two weeks, I've been listening, learning, discussing, and hacking various Landscape and Ubuntu initiatives with members of the Ubuntu community and fellow Canonical employees. It was an amazing experience, and we've got the next 6 months crammed full of plans... with the next 3 months already spec'ed out.

Canonical has surprised me. It's an extraordinary company... both in the modern business-sense of the word as well as the original sense: a fellowship of companions with a common goal. While so far I have only had a chance to hear some personal histories, it's evident that every member of this company is an extraordinary individual with a rich background and a great deal to offer to the whole. Everyone works with an unprecedented amount of motivation towards the company vision, one that is well and tightly integrated into the corporate culture.

There is a bright future ahead for this amazing group...