NASA/EOSDIS Earthdata

Update

It's been a few years since I posted on this blog -- most of the technical content I've been contributing to in the past couple years has been in the following:

• The LFE Blog
• The Clojang Blog

But since the publication of the Mastering matplotlib book, I've gotten more and more into satellite data. The book, it goes without saying, focused on Python for the analysis and interpretation of satellite data (in one of the many topics covered). After that I spent some time working with satellite and GIS data in general using Erlang and LFE. Ultimately though, I found that more and more projects were using the JVM for this sort of work, and in particular, I noted that Clojure had begun to show up in a surprising number of Github projects.

EOSDIS

Enter NASA's Earth Observing System Data and Information System (see also earthdata.nasa.gov and EOSDIS on Wikipedia), a key part of the agency's Earth Science Data Systems Program. It's essentially a concerted effort to bring together the mind-blowing amounts of earth-related data being collected throughout, around, and above the world so that scientists may easily access and correlate earth science data for their research.

Related NASA projects include the following:

• EOS
• ESDIS
• ESMO

The acronym menagerie can be bewildering, but digging into the various NASA projects is ultimately quite rewarding (greater insights, previously unknown resources, amazing research, etc.).

Clojure

Back to the Clojure reference I made above:  I've been contributing to the nasa/Common-Metadata-Repository open source project (hosted on Github) for a few months now, and it's been amazing to see how all this data from so many different sources gets added, indexed, updated, and generally made so much more available to any who want to work with it. The private sector always seems to be so far ahead of large projects in terms of tech and continuously improving updates to existing software, so its been pretty cool to see a large open source project in the NASA Github org make so many changes that find ways to keep helping their users do better research. More so that users are regularly delivered new features in a large, complex collection of libraries and services thanks in part to the benefits that come from using a functional programming language.

It may seem like nothing to you, but the fact that there are now directory pages for various data providers (e.g., GES_DISC, i.e., Goddard Earth Sciences Data and Information Services Center) makes a big difference for users of this data. The data provider pages now also offer easy access to collection links such as UARS Solar Ultraviolet Spectral Irradiance Monitor. Admittedly, the directory pages still take a while to load, but there are improvements on the way for page load times and other related tasks. If you're reading this a month after this post was written, there's a good chance it's already been fixed by now.

Summary

In summary, it's been a fun personal journey from looking at Landsat data for writing a book to working with open source projects that really help scientists to do their jobs better :-) And while I have enjoyed using the other programming languages to explore this problem space, Clojure in particular has been a delightfully powerful tool for delivering new features to the science community.

OpenStack Developer Summit: Heat Followup

Folks are finally starting to recover from the OpenStack Developer Summit that was held in Portland, Oregon recently. All reports indicate that it was a truly phenomenal experience, record-breaking in many ways, and something that has inspired incredible enthusiasm within the community. And that's great news, since there's an enormous amount of work to be done this release ;-)

Of particular importance to many in the community is the work around maturing the autoscaling feature in OpenStack Heat. There was a fantastic session at the summit, facilitated by the bow-tied and most dapper Ken Wronkiewicz (his notes from the Summit were published on the Rackspace blog).

In preparation for the session, the following resources were created:

• https://blueprints.launchpad.net/heat/+spec/heat-autoscaling
• https://etherpad.openstack.org/heat-autoscaling
• http://summit.openstack.org/cfp/details/172

That one in the middle is important, as it is also where notes were taken during the actual session itself (see the section entitled "ODS Session Notes"). Devs at Rackspace have started going through the notes from the session and started planning work around this -- all of which will be carried on in the open, on the OpenStack mail list (tagged with "[Heat]"), on Freenode, and on github/gerrit.

The discussion at the Summit indicated strong interest in building a REST API for the existing autoscaling feature. Needless to say, there is a lot involved in this, touching upon significant OpenStack components like Quantum, LBaaS, and Ceilometer. Once the appropriate code is in place, a REST API will need to be created, features will need to be expanded/added, etc., and we'll be off and running :-)

Lots to do, and lots of great energy and excitement around this to keep us all chugging through this cycle.

On that note, we'd like to send out a special "thanks" to all the countless folks who worked so hard to make ODS happen. This event anchors us in a most excellent way, providing the insight and fuel that supports future development work so well!

Autoscale and Orchestration: the Heat of OpenStack

Several months before I joined Rackspace last year, there were efforts under way to provide an Autoscaling solution for Rackspace customers. Features that we needed in OpenStack and Heat hadn't been released yet, and there were no OpenStack experts on the Autoscaling team. As such, the engineers began developing a product that met Rackspace customer needs, integrated with the
existing monitoring and load-balancing infrastructure, and made calls to OpenStack Nova APIs as part of the scaling up and down process.

At PyCon this year, Monty Taylor, Robert Collins, Clint Byrum, Devananda van der Veen, and I caught up and chatted about what their views were of the current status of autoscaling support in OpenStack Heat. It seems that the two pieces we need the most -- LBaas and support for external monitoring systems (perhaps via webhooks) -- are nascent and not ready for prime-time yet. Regardless, Monty and his team encouraged us to dive into Heat, contribute patches, and in general, release our work for consumption by other Stackers.

Deeply encouraged by these interactions, we took this information to Rackspace management and, to quote Monty Python, there was much rejoicing. Obviously OpenStack is huge for Rackspace. Even more, there is a lot of excitement about Heat, the existing autoscaling features in OpenStack, and getting our engineers involved and contributing to these efforts.

In the course of these conversations, we discovered that Heat was getting lots of attention internally. It turns out that another internal Rackspace project had been doing something pretty cool: they were experimenting with the development of a portable syntax for application description and deployment orchestration. Their work had started to converge on some of the functionality provided by Heat, and they had a similar experience as the Autoscaling team. The timing was right to contribute what they have learned and align all of their continued efforts with adding value to Heat.

Along these lines, we are building two new teams that will focus on Heat development: one

contributing to features related to autoscaling (not necessarily limited to Heat) and the other contributing to the ongoing conversations regarding the separation of concerns between orchestration and configuration management. Everyone -- from engineers to management -- is very excited about this new direction in which our teams are moving. Not only will it bring new developers to OpenStack, but it is aligning our teams with Rackspace's OpenStack roots and the company's vision for supporting the growing cloud community.

Simply put: we're pretty damned pumped and looking forward to more good times with OpenStack :-)

CERN, OpenStack Keep Resonance Cascades at Bay

Tim Bell preparing to get his
OpenStack on

As previously mentioned, there's a growing momentum around ops-oriented participation in the OpenStack community. DreamHost is deeply invested in DevOps, seeing how that's where we're going to be living in a few months! As Simon Anderson, CEO of DreamHost, recently said:

"When we're running a complex fabric of apps on over 5,000 servers across three data centers, we need a lean and nimble approach to software development and operational implementation. Without a DevOps approach, we wouldn't be able to push code into production as fast or as efficiently as we do, and our customers would not be happy! Today's developers demand up-to-the-hour security and performance updates to Internet infrastructure, so we aim to deliver just that with DevOps."

Though expressed in the context of our work, the import of DevOps that Simon's comment generally highlights is going to be increasingly important for nearly anyone running cloud services. 

In particular, I've been following the work of the intrepid folks at CERN. As such, this post is not about DreamHost; rather, it's a mad tale of OpenStack, DevOps, and averting alien invasion.

After countless long-distance phone conversations, a flight to Switzerland, and spending several days buying pints for a security guard in the know (referred to from now on as "Barney"), I've uncovered some profound truths -- Mulder-style -- and have confirmed that the impact of OpenStack at CERN is huge. 

Superficial examinations turn up the usual: CERN's planning slides, nice quotes, discussions of features and savings in time and money. For instance, in a recent email conversation with Tim "Gordon Freeman" Bell at CERN, I learned that 

"The CERN Agile Infrastructure project aims to develop CERN's computing resources and processes to support the expanding needs of LHC physicists and the CERN organisation."

I think these guys have been hanging out with Simon! But once you slip behind the scenes, peek at some of the whiteboards in unattended rooms, or rifle through notes lying about, you see that things are not what they appear. I've included a shot of Mr. OpenStack-at-CERN himself; this was my first clue.

Publicly, he's been working with other teams at CERN to:

• modernise the data centre configuration tools and automating operations procedures
• exploit wide scale use of virtualisation, improving flexibility and efficiency
• enhance monitoring such that the usage of the infrastructure can be fully understood and tuned to maximise the resources available

But privately, it seems that he and his team have been doing much, much more. This was alluded to in a statement made by team member Jan van Eldik: "We expect the number of requests to insert non-standard specimens into the scanning beam of the Anti-Mass Spectrometer to significantly decrease, once automation is in place and everyone is using the standard infrastructure we are setting up."

That isn't to say there haven't been incidents...

Innocuously enough, the current toolchains are based around:

• OpenStack as a single Infrastructure-as-a-Service providing physics experiment services, developer boxes, applications servers as well as the large batch farm
• Puppet for configuration management
• Scientific Linux CERN as the dominant operating system with sizeable chunk of Windows installs

But that second bullet caught my eye, and one of Barney's pub mates confirmed a rumor that we'd heard: the Puppet instances are actually trained headcrabs. The primary training tool? You guessed it, a crowbar. Barney said that the folks from Dell took inspiration from this and developed it further for their OpenStack deployment framework after an extended visit to CERN.

Although Barney hadn't seen any evidence of resonance cascades, there have been minor cross-dimensional disturbances as a result of some "cowboy" activity and folks not following DevOps best practices. This has been kept quiet for obvious reasons, but has led to a small pest problem in some of CERN's older tunnel complexes. As rouge elements are discovered, CERN has been educating transgressors aggressively. (Sometimes they go as far as sending employees to Xen training... or was it Xen training?)

One artist's conception of what success will
look like for OpenStack at CERN

Despite the minor hiccoughs along the way, CERN is aiming for success. (Given the lack of Combine and forced relocation programs, they're already doing better than Black Mesa's Anomalous Materials team.) Plans are in place for an initial pre-production service, OpenStack deployment this year. Following that, they will be moving towards 300,000 virtual machines on 15,000 hosts spread across two data centres by 2015.

The OpenStack community is supporting them in their efforts with fantastic new features, high-quality discussions on the mail lists, and real-time interaction on the IRC channels. In an act of reciprocity and community spirit, operators at CERN have volunteered to contribute back to the OpenStack community with regard to operations best practices, reference architecture documentation, and support on the operators' mail list.

To see how other institutions were taking this news, I spent several days waiting on hold. In particular, Aperture Science could not be reached for comment. However, Ops team member Belmiro Rodrigues Moreira did say that there's an audio file being circulated at CERN of Cave Johnson threatening to "burn down OpenStack" ... with lemons. Given Aperture Science's failure record with time machine development, it's generally assumed to be a prank audio reconstruction. CloudStack developers are considered to be the prime suspects, seeing how much time they have on their hands while waiting for ant to finish compiling the latest Java contributions.

When asked what advice he could give to shops deploying OpenStack, Tim said simply: "Remember, the cake is a lie. Don't get distracted and don't stop. Just keep hacking."

Alyx, explaining to her dad why she loves DreamHost

Couldn't have said it better myself.

In closing, and interestingly enough, one of DreamHost's employees has an uncle who works at the Black Mesa Research Facility. Though his teleportation research team was too busy for an extended interview, his daughter did mention that she is a DreamHost customer and can't wait to use OpenStack while interning at CERN next summer. After all, that's what she uses to auto-scale her WordPress blog (she's in our private beta program).

It's a small world.

And, thanks to Tim and the rest at CERN, a safer one, too.

Successful Hack-In, 01 Mar 2012!

DreamHost has a new core set of cloud developers now based in Atlanta, and a new Meetup group to go with that :-) Today there was a global OpenStack Hack-In, and I just posted a summary to our Meetup discussion page, but since I desperately need to do some blogging, I'm republishing here :-) (with some minor tweaks...)

We had fun in person, there was good chatter on IRC with the Colorado and San Francisco teams, and we had a great time digging into OpenStack some more.

Technical highlights include:

• testing out development deployments of OpenStack using Vagrant (some successes, some blockers)
• testing out dev deployments of OpenStack using VirtualBox directly
• filed some bugs for issues in horizon regarding error feedback to users and how the documentation is generated
• dug into issues with logging and inconsistencies in datestamps
• uncovered some weirdness with the usage of gnu screen and hanging services/partial devstack installs due to sudo assumptions (devstack assumes a passwordless sudo, and will label an install as failed if it gets hung up on the apache log tail, waiting for a password, even if the install was successful and all the services started correctly)
• Doug Hellmann made his first commit upstream to OpenStack

On the non-technical, fun side:

• Thanks for Zenoss for the fun swag today (the Zebras are still staring at me... I think they're going to be making an appearance in ToyStory 5)
• Even more thanks to Zenoss for the offer to become an OpenStack Atlanta Sponsor (food, drinks, and swag)!
• Thanks to DreamHost for the AMAZING coffee and danishes from The Village Corner Restaurant/Basket Bakery. Seriously. That was the best coffee I have ever had. In. My. Life.
• Also, thanks DreamHost for the pizza and the sweet potato pies!

We took a couple snapshots of the event, and I'll be posting those soon on the Meetup page, but for the super-impatient, they're up on Flickr right now

http://www.flickr.com...

Ubuntu Foundations and Maverick Meerkat 10.10

Introduction

For those that don't know, The Ubuntu Foundations Team is responsible for delivering the core Ubuntu system, which is common to the whole Ubuntu family of products and services. For the past couple months, I had the pleasure and honour to work with the Foundations team, assisting in preparation for the Foundations Track at UDS and planning for the 10.10 cycle.

Below I will give a brief summary of the major topics covered at UDS which, in turn, generated scheduled work items. They are listed in alphabetical order.

Boot Work

Several boot-related areas were identified for work during Maverick. These include the following:

• cd boot - by converting CD boot to use grub2 with its new graphical goodness, we will only need to maintain use of a single bootloader
• continued performance improvements
• grub2 framebuffer - the end goal being a near flicker-free graphical boot splash experience
• UEFI - support booting on systems that use UEFI firmware

Related blueprints:

• foundations-m-cd-boot
• foundations-m-boot-performance
• foundations-m-grub2-boot-framebuffer
• foundations-m-uefi-support

btrfs

In Maverick, we will be adding support for btrfs. Our tasks include such work as making ureadahead work with btrfs, adding btrfs support to grub2, integration work, and features support.

Related blueprint:

• foundations-m-btrfs-support

Cleanup

Just after an LTS release is a perfect time to clean house. We will be taking this opportunity to do so, with such work as dropping unused/unneeded packages from the base system, double-checking package dependencies, and investigating space-saving measures.

Related blueprints:

• foundations-m-spring-cleaning
• foundations-m-package-culling

Installer Redesign

The installer is getting a serious make-over. Foundations and the Design team are working very closely together, improving the workflow, minimizing user clicks, improving the look-and-feel, and providing utility with increased ease of use.

Related blueprint:

• foundations-m-installer-redesign

Software Center

We want to get new applications into the Software Center, ideally providing developers with a means of generating revenue with the applications. For the former, we need to define some good social and technical processes to ensure ongoing quality and excellent producer/consumer experience. In conjunction with that, we need to work on getting a billing system in place.

Related blueprint:

• foundations-m-software-center-roadmap

Upstart

Upstart is getting major work this cycle. New and improved features include the following:

• Manual mode
• Resource limits
• Dependencies
• Better support for UIs that want to use Upstart
• Simple skeleton to make life easier for sysadmins
• Provide an API for services and tasks so that folks don't have to think about the event-based model if they don't need to
• Explore the conversion of conf files into jobs
• Possibly extend the debug capabilities into an interactive mode
• Improve job disabling

Foundations will also be working closely with the server team to get their init scripts converted to Upstart. Conversely, the Kernel team will be providing new features that will allow Foundations to fully develop the planned Upstart features.

Related blueprint:

• foundations-m-finish-upstart

Miscellaneous

There is lots of other work we'll be doing, some of which are highlighted in the following:

• i686 Default Compile
• Stop building the ia64 and sparc community ports
• Multiarch Support for gcc, binutils, dpkg, and apt
• Foundations Python improvements
• Upgrade and install testing

Related blueprints:

• foundations-m-686-compile
• foundations-m-drop-ia64sparc
• foundations-m-multiarch-support
• foundations-m-python-continuous-integration
• foundations-m-python-versions
• foundations-m-robust-python-packaging
• foundations-m-upgrade-and-install-testing

References

• Foundations Work Items for Maverick
• Maverick Release Schedule
• btrfs
• Upstart

7 Years of Blogasmic Action

Earlier today, I was looking through old blog posts, reminding myself of issues and topics that I want to continue reflecting on actively. When I got to the end of the posts, I looked at the earliest date and realized that my seven-year blogiversary was this past Wednesday. At my vocalized comment to this effect, Marjorie asked what got me started on blogging. Interestingly, it ties in with my professional life over the past twleve years...

As a physics undergrad and employee of the University of Maryland, I was hired away by a large internet startup for my systems and programming background. Already a fan of Office Space, I got to experience the reality of cubicle life as the not-so-funny, real complement of that classic movie. However, it wasn't my personal experience of cubihell that had the biggest impact upon my psyche; it was the corporate atrocities suffered by various customers and partners of dot-boom era, high-rolling companies.

Already a passionate open-source user and contributor, it become one of my personal goals to make sure a greater number of people (especially those making decisions in companies) were informed about the software freedoms that were within their reach. Within two years of this epiphany, I had my own company and was open-source consulting for everything from small businesses to large international corporations and the Federal government. Everything I did was open source; all of it saved my customers from spending millions of dollars unnecessarily and the enslavement of proprietary lock-in.

It was at this time that I started blogging. My primary interest was to be a voice in the wilderness of consultants, finding other like-minded programmers, project managers, sales engineers, etc., who believed in "open business", transparency, and accountability. As time progressed, my blogging habits adapted to fill other needs in various communities. However, all of my professional decisions have remained true to that initial desire to work on technology that was not only free, but actively prevented the bondage of its users.

For the past year and a half, I've been working at Canonical, the company that funds and actively supports Ubuntu and the Ubuntu Community. Every time I work with a different engineer or a new team, I am blown away by two things: the incredible talent embodied in each interaction, and the dedication that each person has to improving how free technology is delivered to the world.

My blog posts have definitely changed in character over the past seven years. But I'm delighted that, where I once felt alone with my ideals, I can count true giants in the field among my fellow community members, cooworkers, and mentors.

Recent Work on Various Open Source Projects

Over the last few months, I've been doing lots of work on various open source projects. I've been so burried in them that I haven't blogged (or microblogged) much about them. So much has been happening, though, that I needed to take a break from the coding and communicate some of this :-)

txAWS

Over the last few months, Robert Collins, Thomas Herve, Jamshed Kakar and I have been putting lots of effort into improving cloud support in the async (Twisted) Python Amazon EC2 library. It's been a lot of fun to see that part of the library take shape and start getting production use from Canonical. We have implemented the following functionality groups and their associated API methods:

• Instances
• Key Pairs
• Security Groups
• Elastic IPs
• Availability Zones
• EBS

There is a ticket for the following two groups of functionality, and branches in progress for both:

• Images (ticket #461136 on Launchpad)
  
• Image Attributes (ticket #461139 on Launchpad)
  

Once those are merged, EC2 support in txAWS will be complete.

As a bonus, we've added support for arbitrary endpoints and with that in place, have successfully tested txaws.ec2 against Eucalyptus clouds :-)

txJSON-RPC

There's a new release of txJSON-RPC out now, downloadable from PyPI. Work on the next version has been a great deal of fun. What started out as a conversation on IRC with Terry Jones, ended up as spec-driven doctest work on trunk for implementing support for multiple versions of the JSON-RPC spec (pre-1.0, 1.0, and 2.0).

With these changes in spec support, txJSON-RPC has really started to mature, and that's been fantastic. Even more, the jsonrpclib module that's included in txJSON-RPC (and can be used with non-Twisted projects) is getting spec version support as well.

SOOM in txULS

As some may know, one of my computing passions is ultra large-scale systems. After a phone conversation with Jamshed Kakar and some nice exchanges on the Python ULS-SIG mail list with Alex Drahon, I started working on a set of coding experiments in self-organizing objects. The Google Doc informally outlines the various stages and goals.

For now, the code is living in a txULS series on Launchpad. The reason for its inclusion in txULS is that ultimate goal of the SOOM (self-organizing object meshes) code is to produce an async API for building Twisted services that provide behaviours as outlined in the Google Doc (linked above).

I would to emphasize the networking-library-agnostic nature of the ULS-SIG: Twisted comes up since I spend a lot of time with Twisted, but ever networking library is welcome. I'm personally interested in exploring (or watching other developers explore) various Stackless Python experiments in the ULS systems space.

txSpore

This project was a spontaneous effort resulting from an evening of code review when I first discovered the official Python API from EA/Maxis for the game Spore. It's been a blast and something that Chris Armstrong and I have been working on together. The API is currently feature-complete, but Chris has some excellent ideas about improving usage as well as some additional API augmentations that will make life easier for game developers.

Already more featureful and usable than the official Spore Python API, there are great things in store for this library. Chris has come up with several very cool demo ideas that take advantage of the new API and will push it to the limits. We're both pretty excited :-)

Isomyr

I love isometric games. I'm a freak for the classic look. One night about a month ago, Chris and I discovered Isotope, an isometric Python game engine by Simon Gillespie. It was last updated in 2005 at version 0.9, so I started working on a branch that could be released as 1.0. I never heard back from Simon after an inquiry for his permission to release as 1.0, so I forked the code to a new project: Isomyr. I released the code rewrite work I had done to that point (plus some changes such as replacing some old code with Numpy) as 0.1. At which point things just went nuts...

Isomyr now has support for multiple worlds, customizable (per world) in-game time and calendars, and basic interactive fiction development. The latest chunk of code (that hasn't been pushed up to Launchpad yet) is adding support for general planetary simulation (e.g., axial title, varying daylight hours, seasons, and weather). As you might imagine, this has been a great deal of fun to work on!

PyRRD

PyRRD has gotten some recent community love, with requests for a mail list, new developer-oritnted features, etc. Currently at version 0.0.7, the 0.1.0 release isn't to far away. Folks have been using trunk for a while, which added support for the RRDTool Python bindings back in March of this year. (PyRRD's focus has primarily been for users/developers who didn't have the RRDTool Python bindings installed). In the next couple weeks or months, I expect that we'll be adding a few more features, and then preparing the new release.

PyRTF

Another fixer-upper project, PyRTF (mirroed on Google code as pyrtf-ng) has been on hiatus for a while, due to my diminished need to manipulate and interact with RTF files. However, a new developer has joined the project and the code-cleanup and unit test development now continues. Thanks Christian Simms!

A while ago, Simon Cusack (the original author of PyRTF) and I had some great discussions about the future development of PyRTF and his interest in merging the recent changes into trunk on SourceForge. I deferred on that action, wanting to wait until the code cleanup, unit tests, and API changes had been completed. With Christian's help, we may get there now :-)

Wrap-up

It's been about a year since I've been so active in open source development, and it feels really good to be at it again :-) Being back in Colorado seems to have helped in subtle ways, but mostly it's been the increased interaction and interest from developers in the community that I can thank for my increased activity (and thus enjoyment). You guys are awesome. You're the reason for any code I produce.

Intellectual Property and Open Source

A few months ago, I received a complementary copy of Van Lindberg's new O'Reilly book Intellectual Property and Open Source: A Practical Guide to Protecting Code and the first thing that happened at home when the book was unwrapped was three of us began arguing over who got to read it first.

This may seem like an odd thing to happen for what one could easily assume was a dry and less than interesting topic. However, at the time I was strongly considering the possibility of beginning a non-tech-industry startup built with both open source and proprietary code. The discussions with the potential founders of the startup had been very vigorous and exciting, but the big questions that remained revolved around patents, protecting IP, and providing protection against big business while still offering powerful, free code for use by individuals/private consumers. If you've read the book or even seen the table of contents, you can see why everyone wanted to be the first to read it and learn from the insights provided between its covers.

Instead of jumping into another startup, I ended up joining Canonical; this has kept me both very busy and exceptionally happy. The holiday break has provided an opportunity to finish reading the book, and it has been a delight. I have friends working on startups that depend upon exciting code to power some or all of the business models for their visions, and reading this book should be on their shelves, close at hand. Even if you're not involved directly with open source and intellectual property, this book is an excellent read.

Intellectual Property and Open Source accomplishes a difficult goal of sharing dense information while making the subject matter engaging. This is done through examples, thought experiments, and well developed analogies. Van does an excellent job of igniting a powerful curiosity on the part of the reader while providing rewards for this in the lucid explanations of related laws and perspectives. I am resisting the urge to turn this post into a long series of quotes, but at the very least I want to mention a few little "spoilers" ;-)

The book starts off with an excellent foundation, giving an overview of the origins of intellectual property from an economic and legal perspective. This was particularly useful for me, as I have no background in this field. Van Lindberg does a really great job of expressing some of the widely held (and diverse) views of IP in the open source community.

The book then launches the reader into an array of well organized chapters on patents, the patent system, trademarks, copyright, trade secrets and licenses. Every open source developer should read chapter 10 on choosing an open source license (the opening dialog had me laughing out loud, a hilarious parody of news groups and IRC arguments as well as a nod to Princess Bride). There's also a chapter dedicated to patches and their relationships to copyright; another on reverse engineering; and the final one provides information and advice on establishing non-profits for open source projects -- the author even gives mention to our friends at the Software Freedom Conservancy (the umbrella non-profit for the Twisted Software Foundation).

In all honesty, I can't rave enough about this book. I've re-read parts of it just because I enjoyed the clarity of the explanations so much. Law is a twisty maze of easily confused subtleties to those who have not been trained in its dark arts. Through explicit language and examples, the author guides us past pitfalls of misunderstanding and brings us directly to all the major points.

If you are an Amazon shopper, you may want to act quickly: last I checked, there were only two copies left.

Enjoy!

Twisted and Divmod: A Cheater's Setup Guide

I've been helping a few folks out on IRC lately. They've wanted to know how to setup Twisted and Divmod without doing any installs, running directly from SVN. They've been in luck, because that's actually how we develop at Divmod :-)

Here are the Cliff Notes (this stuff is available on the wikis, but it's spread out):

Install the dependencies:

pycrypto 2.0
SQLite 3.2.1
PySQLite 2.0
PyTZ 2005m-1
PIL 1.1.6

Get the Divmod code first (we'll get Twisted next):

mkdir ~/lab
cd ~/lab
svn co http://divmod.org/svn/Divmod/trunk Divmod/trunk

Set the Combinator env vars (if you want to persist this, then you'll need to put it in your .profile or shell .rc file):

eval `python ~/lab/Divmod/trunk/Combinator/environment.py`

Have Combinator start "tracking" Divmod and Twisted, thus managing PYTHONPATH for them (note that chbranch will detect that Twisted has not been checked out and will do so automatically):

chbranch Divmod trunk
chbranch Twisted trunk svn://svn.twistedmatrix.com/svn/Twisted/trunk

Get the new project dirs into the env:

eval `python ~/lab/Divmod/trunk/Combinator/environment.py`

Executing the whbranch command should give you the following:

Divmod: trunk
Twisted: trunk

If you start up a Python interpreter, you'll be able to import from twisted, mantissa, axiom, etc.

Update: the instructions have been edited and shortened, thanks to insight from Glyph.

Mantissa: An Alternative to LAMP

I first started drafting this post a few months ago, out of excitement for the work that JP and Glyph have been doing in the Divmod open source stack codebase. I was planning on entering the acronym fray with a title like "*MAP: An Alterantive to LAMP" where *MAP (pronounced "starmap") would be "Any OS, Mantissa, Axiom, and Python." A good friend of mine whose opinion I value said that *MAP was a terrible name, and after chatting about it with Glyph, he commented "Why not keep it really simple? Just say 'Mantissa.'"

And so it is :-)

For those that don't know, Mantissa is the Twisted application server and Axiom is a Twisted-based object database. By virtue of what are called "deferreds," Twisted allows you to write highly concurrent applications. Mantissa -- the Divmod stack (Mantissa entails Python, Twisted, and Axiom because it requires them) -- provides developers a means of scaling their Twisted-based, asynchronous applications. This means that you can go from experiments or prototypes to multi-node production deployments with the same set of tools and code.

As such, this is a direct competitor for LAMP. Here are some questions about that: What is the value of a full stack? Why is an alternative to LAMP good or needed? What is a good alternative?

Stacked Development Value

What does a full stack give us, as developers? From a practical perspective, it:

• eliminates the overheard involved in setting up a system in preparation for development
• provides a development toolset
• provides a context within which design patterns have been established and utilized

In other words, we can do things like pop in a CD, install an OS, have it meet all the software dependencies for our development tasks (since we're talking about LAMP, we mean development for the web), and either know how to build what we need or who to ask that can point us in the right direction. LAMP gives us this and, thanks to OS distributions like Ubuntu, gives it to us cheaply through simple button-pushing.

Do notice, however, that I said nothing about "going live" or "pushing to production"...

An Engineer's Perspective

In a recent conversation, Sean Reifschneider of tummy.com had this to say about LAMP:

"The problem with the LAMP stack is that it's not a solution for the worst case scenarios. It's great for development: you throw it all together and start writing code. It's fairly okay for low-volume production use. But you need to plan for DoS attacks, search engine bot crawls, and spammer email address harvesting. Default LAMP installs fall over under such conditions."

This is a point that bears repeated belaboring: the network is violent and unpredictable. Connectivity can go away at any moment due to causes at pretty much all layers of the OSI model. The best practices for deploying applications in a production environment that keep this in mind are vast and varried. This is the domain of systems experts.

Sean made further comments concerning Google, that App Engine is so great because you write your code and then just throw the whole thing in their grid, and bam! instant scalability, protected by the (hopefully) same mechanisms that protect all of Google's publicly-facing web assets.

LAMP distributions productized and made freely available the otherwise painstaking process of compiling and installing a Linux kernel, Apache, a database, and your preferred programming language. The painstaking process was one that developers engaged in for software development. But what about the ones that systems engineers engage in for production deployments?

Google has addressed this in a "small way": massive in infrastructure support, but small in features. Knowning Google's penchant for incremental and steady service improvements, they've got plans for additional features. But I think everyone can agree that they're not going to try to meet everyone's needs all the time. Regardless, they are moving in the right direction: innovating a new platform.

Something for Both

On just this topic (innovating or finding a new platform), Albert Wenger of Union Square Ventures said the following:

"What then is needed? A platform that is created from the ground up ... What would such a platform look like? It would be hosted and (nearly) infinitely scaleable. It would provide object storage that’s as simple as saying 'here’s an object, store it' ... user authentication, authorization and access control. Flexible processing of pretty URLs. Easy creation and maintenance of page templates. Ability to send emails and process bounces. Handling of RSS feeds (inbound and outbound). Support for mobile access and possibly even voice capabilities."

Anyone that knows the Divmod software will know why this tickled us so: we have an object database (Axiom) with built-in user authentication, we've got object publishing (even with pretty URLs) and templating with Nevow, we've got mail services, feed support, mobile access and SIP. However! This isn't an advertisement; it's an illustration. The platform is part of the network, and in a sense, it is the network. Considerations for rapid application development need to be regarded very highly; I think it's fairly uncontested common knowledge that LAMP has proved this. Just as highly, though, we need to consider the needs of systems and of the engineers that are integrating them.

Google is making parts of its infrastructure available to developers now. With the dual ease of development and deployment, they are innovating engineering for us. They are only one of many, however. We need to be asking ourselves what our applications are, what the network is, what services are, and what our dev teams and engineers need.

Epilogue

This brings me to what I want for my birhtday :-) Hey IBM! Sun! I want access to a Blue Gene (a la Project Kittyhawk) or a Sun Grid. I want to prove the efficacy of LAMP alternatives in the changing internet, of Python's continued pertinence, Twisted's developmental power and Mantissa's deployment capabilities.

darkstat, For the Win

This is a quick self-response to my tweet to the lazyweb (is it still a tweet when it's Pownce and not Twitter?) today. I couldn't remember the name of a really handy network monitoring tool I used to use. It was similar to ntop but used a fraction of the resources and had a very limited yet perfectly satisfactory feature set. I've been having some crazy network utilization weirdness at the office lately, and I've wanted to peek at some trends without setting up NetFlow for my router or messing with ntop.

The answer was darkstat. It was my own memory that eventually came to the rescue, not Keyword Roulette on Google. Version 3.x is out and available for Mac OS X 10.5 via the latest MacPorts version (1.6).

This is all I needed to get it running:

sudo port selfupdate
sudo port install darkstat
sudo /opt/local/sbin/darkstat --debug -i en0 -l 192.168.4.0/255.255.255.0

Then, I just had to hit localhost:667...

I don't know what's up with the Google Juice for this guy's page, but it took me forever to find! I was searching for all the keywords like "ntop" (which was mentioned on his site at one point, I think), "network", "dark", "lightweight", "monitoring", etc. You get the picture. Hopefully this blog post will help when others are looking for it, too.