Twisted and Divmod: A Cheater's Setup Guide

I've been helping a few folks out on IRC lately. They've wanted to know how to setup Twisted and Divmod without doing any installs, running directly from SVN. They've been in luck, because that's actually how we develop at Divmod :-)

Here are the Cliff Notes (this stuff is available on the wikis, but it's spread out):

Install the dependencies:

pycrypto 2.0
SQLite 3.2.1
PySQLite 2.0
PyTZ 2005m-1
PIL 1.1.6

Get the Divmod code first (we'll get Twisted next):

mkdir ~/lab
cd ~/lab
svn co http://divmod.org/svn/Divmod/trunk Divmod/trunk

Set the Combinator env vars (if you want to persist this, then you'll need to put it in your .profile or shell .rc file):

eval `python ~/lab/Divmod/trunk/Combinator/environment.py`

Have Combinator start "tracking" Divmod and Twisted, thus managing PYTHONPATH for them (note that chbranch will detect that Twisted has not been checked out and will do so automatically):

chbranch Divmod trunk
chbranch Twisted trunk svn://svn.twistedmatrix.com/svn/Twisted/trunk

Get the new project dirs into the env:

eval `python ~/lab/Divmod/trunk/Combinator/environment.py`

Executing the whbranch command should give you the following:

Divmod: trunk
Twisted: trunk

If you start up a Python interpreter, you'll be able to import from twisted, mantissa, axiom, etc.

Update: the instructions have been edited and shortened, thanks to insight from Glyph.

Mantissa: An Alternative to LAMP

I first started drafting this post a few months ago, out of excitement for the work that JP and Glyph have been doing in the Divmod open source stack codebase. I was planning on entering the acronym fray with a title like "*MAP: An Alterantive to LAMP" where *MAP (pronounced "starmap") would be "Any OS, Mantissa, Axiom, and Python." A good friend of mine whose opinion I value said that *MAP was a terrible name, and after chatting about it with Glyph, he commented "Why not keep it really simple? Just say 'Mantissa.'"

And so it is :-)

For those that don't know, Mantissa is the Twisted application server and Axiom is a Twisted-based object database. By virtue of what are called "deferreds," Twisted allows you to write highly concurrent applications. Mantissa -- the Divmod stack (Mantissa entails Python, Twisted, and Axiom because it requires them) -- provides developers a means of scaling their Twisted-based, asynchronous applications. This means that you can go from experiments or prototypes to multi-node production deployments with the same set of tools and code.

As such, this is a direct competitor for LAMP. Here are some questions about that: What is the value of a full stack? Why is an alternative to LAMP good or needed? What is a good alternative?

Stacked Development Value

What does a full stack give us, as developers? From a practical perspective, it:

• eliminates the overheard involved in setting up a system in preparation for development
• provides a development toolset
• provides a context within which design patterns have been established and utilized

In other words, we can do things like pop in a CD, install an OS, have it meet all the software dependencies for our development tasks (since we're talking about LAMP, we mean development for the web), and either know how to build what we need or who to ask that can point us in the right direction. LAMP gives us this and, thanks to OS distributions like Ubuntu, gives it to us cheaply through simple button-pushing.

Do notice, however, that I said nothing about "going live" or "pushing to production"...

An Engineer's Perspective

In a recent conversation, Sean Reifschneider of tummy.com had this to say about LAMP:

"The problem with the LAMP stack is that it's not a solution for the worst case scenarios. It's great for development: you throw it all together and start writing code. It's fairly okay for low-volume production use. But you need to plan for DoS attacks, search engine bot crawls, and spammer email address harvesting. Default LAMP installs fall over under such conditions."

This is a point that bears repeated belaboring: the network is violent and unpredictable. Connectivity can go away at any moment due to causes at pretty much all layers of the OSI model. The best practices for deploying applications in a production environment that keep this in mind are vast and varried. This is the domain of systems experts.

Sean made further comments concerning Google, that App Engine is so great because you write your code and then just throw the whole thing in their grid, and bam! instant scalability, protected by the (hopefully) same mechanisms that protect all of Google's publicly-facing web assets.

LAMP distributions productized and made freely available the otherwise painstaking process of compiling and installing a Linux kernel, Apache, a database, and your preferred programming language. The painstaking process was one that developers engaged in for software development. But what about the ones that systems engineers engage in for production deployments?

Google has addressed this in a "small way": massive in infrastructure support, but small in features. Knowning Google's penchant for incremental and steady service improvements, they've got plans for additional features. But I think everyone can agree that they're not going to try to meet everyone's needs all the time. Regardless, they are moving in the right direction: innovating a new platform.

Something for Both

On just this topic (innovating or finding a new platform), Albert Wenger of Union Square Ventures said the following:

"What then is needed? A platform that is created from the ground up ... What would such a platform look like? It would be hosted and (nearly) infinitely scaleable. It would provide object storage that’s as simple as saying 'here’s an object, store it' ... user authentication, authorization and access control. Flexible processing of pretty URLs. Easy creation and maintenance of page templates. Ability to send emails and process bounces. Handling of RSS feeds (inbound and outbound). Support for mobile access and possibly even voice capabilities."

Anyone that knows the Divmod software will know why this tickled us so: we have an object database (Axiom) with built-in user authentication, we've got object publishing (even with pretty URLs) and templating with Nevow, we've got mail services, feed support, mobile access and SIP. However! This isn't an advertisement; it's an illustration. The platform is part of the network, and in a sense, it is the network. Considerations for rapid application development need to be regarded very highly; I think it's fairly uncontested common knowledge that LAMP has proved this. Just as highly, though, we need to consider the needs of systems and of the engineers that are integrating them.

Google is making parts of its infrastructure available to developers now. With the dual ease of development and deployment, they are innovating engineering for us. They are only one of many, however. We need to be asking ourselves what our applications are, what the network is, what services are, and what our dev teams and engineers need.

Epilogue

This brings me to what I want for my birhtday :-) Hey IBM! Sun! I want access to a Blue Gene (a la Project Kittyhawk) or a Sun Grid. I want to prove the efficacy of LAMP alternatives in the changing internet, of Python's continued pertinence, Twisted's developmental power and Mantissa's deployment capabilities.

darkstat, For the Win

This is a quick self-response to my tweet to the lazyweb (is it still a tweet when it's Pownce and not Twitter?) today. I couldn't remember the name of a really handy network monitoring tool I used to use. It was similar to ntop but used a fraction of the resources and had a very limited yet perfectly satisfactory feature set. I've been having some crazy network utilization weirdness at the office lately, and I've wanted to peek at some trends without setting up NetFlow for my router or messing with ntop.

The answer was darkstat. It was my own memory that eventually came to the rescue, not Keyword Roulette on Google. Version 3.x is out and available for Mac OS X 10.5 via the latest MacPorts version (1.6).

This is all I needed to get it running:

sudo port selfupdate
sudo port install darkstat
sudo /opt/local/sbin/darkstat --debug -i en0 -l 192.168.4.0/255.255.255.0

Then, I just had to hit localhost:667...

I don't know what's up with the Google Juice for this guy's page, but it took me forever to find! I was searching for all the keywords like "ntop" (which was mentioned on his site at one point, I think), "network", "dark", "lightweight", "monitoring", etc. You get the picture. Hopefully this blog post will help when others are looking for it, too.

Required Reading: Ultra Large-Scale Systems

At Divmod, we're always talking about the future of computing, software, and the network. This usually focuses on our work with Twisted or the Divmod platform. But we have also spent considerable time assessing research in the area of what is called "ultra large-scale systems." Our primary business interest with this revolves around development, deployment, and management. However, there is a great deal of work that needs to be done to make ULS systems a reality.

I have a series of blog posts planned to discuss ULS systems in the areas where I have a vested interest. Despite the fact that a popular study on the matter was funded by the U.S. Department of Defense (conducted by the Carnegie Mellon Software Engineering Institute), I will not be discussing this technology in the context of war efforts nor national defense. Instead, I will be engaging in this discussion within the context of the medical/health services field, per the example given by Richard Gabriel in his presentation (PDF) to the Chinese University of Hong Kong.

This is especially pertinent today, as Google prepares for its Google Health announcement. Even though there are by some estimates (optimistic ones, in my view) 25 years of research ahead of us, it is generally agreed that ULS systems will consist aggregate sub-systems, built incrementally over time. I don't believe anyone is under the illusion that Google does not want to produce the first ULS system in history, and they are making rapid progress towards this goal. Google Health brings this point home very clearly, in the context of the research that has been done in this area.

The world is rapidly changing. The most important issues in technology and business are not who is going to create the next catchy "Web 2.0" application, or what mega corp games are being played in the Great Silicon Valley Soap Opera. The important issues are how the systems of the next 100-200 years are being built now, who they are being built by, and who as access to that technology.

We are a powerful and creative community. We are concerned about distributing power to the people, education, privacy, and freedom -- open source is built upon these principles. If we want those who are building ULS systems to build them fairly and with our concerns in mind, then we must get involved now. We must start building the necessary tools.

As for the reading, a couple of these were linked to in this blog post. Here is a list that should get your brain revved up and ready to roll (these are all PDF files):

• Design Beyond Human Abilities
• Ultra-Large-Scale Systems: The Software Challenge of the Future
• Scale Changes Everything
• ULS Systems Research Roadmap

I will have more specifics about the sorts of things we're exploring in future blog posts, but until then -- don't get left behind!

Update: Allen Short just pointed out some additional material that predates the Carnegie Mellon research: the Agoric Papers, co-authored by Mark Miller who is a co-creator of the E programming language.

Update 2: Glyph Lefkowitz just sent me a link to the Big Ball of Mud that discusses similar concepts.

Technorati Tags: technology, business, community , open source, uls

The Twisted Think Tank

I'm pleased to announce that we've got a new Divmod site up! We're still making tweaks, but it's ready for public viewing, and open for business.

This change currently doesn't affect our subscriber services... but it will, very shortly :-) JP's working on that now.

Anyone who knows us, knows that we know Twisted. We really know it. And how could we not, with Twisted superheroes like Glyph and JP? We've been solving very interesting problems for the past couple years, and other companies have availed themselves of this expertise. We're no longer trying to hide from our destiny as "the Twisted company."

We've found that providing specialized consulting services has not detracted from our core competency as software developers, but has rather done quite the reverse: provided a great deal of insight and clarity. The two activities have established a complementary feedback mechanism for growth and invention.

App Engine Haiku

I've been playing with Google App Engine a bit more tonight, and I've come to some additional conclusions: 

1. For the best results, you gotta drink the whole pitcher of koolaide; and
2. It's really quite tasty koolaide.

One of the things I really wanted to test was imaging. I use Flickr far more than Picasa, but Flickr's authentication API is kind of a pain in the ass (especially when used with App Engine). Google auth is (for obvious reasons) easier to work with. This, of course, led me to explore Picasa. To my surprise, it was delightful to use. I chopped up gdata for only the parts needed to support Picasa and was uploading images within mere minutes. And not only was the effort minimal, but the performance was outstanding.

One of the things that makes application development a more efficient process is having a unified platform on which to work. This is why working with the Divmod platform is so nice when I do extensive Twisted application development: all the infrastructure is already built and ready for me to use. Today, I got to really taste that same experience with the Google platform. And I liked it.

I did notice one interesting psychological side effect of working with the file count limitation: I inadvertently treated it like a game. Not unlike the rules in poetry, it provided a structure and bounds within which I was forced to operate, adjust for, and test my creativity against. I rather enjoyed this microgame, and that was a surprise :-)

Founding Sponsor Opportunity Closing

Last month we announced that there were only 30 days left to become a Founding Sponsor of the Twisted Software Foundation. There are only two days left, so get your donate on!

We've had some really amazing sponsor conversations during this process, and we're excited about all the new Twisted stories that can be told. We've got enough material for 3 years of Twisted Shows, so I expect you'll be hearing more about how folks are using Twisted in the coming months.

In particular, I want to give a shout out to two different organizations.

Zenoss recently published a press release about their sponsorship, exposing Twisted to an "enterprise" audience that may not have heard of it yet. From the release:

"Twisted provides the foundation for Zenoss' scalable, agent-less collectors. It allows Zenoss daemons to talk to hundreds of enterprise systems simultaneously, with low overheads, using standard protocols like SNMP. In addition, Twisted provides the internal communications between distributed collectors."

United Business Media (formerly known as CMP) has also donated to Twisted, becoming a recent Founding Sponsor.

This was a rather unexpected bit of good fortune... but it gets better: they hadn't heard of Twisted before, and after reading up on it, they were so impressed that they immediately agreed to have three of their research and publication organizations become founding sponsors. They were amazed at the sophistication and power that the Twisted framework provides to a community of developers who are creating the future trends in software. I expect we'll be hearing more from them in the future ;-)

Update: The three UBM companies which sponsored the TSF are Contentinople, Internet Evolution, and Light Reading.

Twisting the Planet

As Steve blogged the other day, we've been jamming on some Twisted lately. But it's not the kind of thing you usually hear from us. We're not doing something esoteric and mind-blowing. We're doing something much harder: working out how to bring Twisted to the masses.

The motivation for this is philanthropic: we believe in Twisted's goodness :-) As Allen Short paraphrased on IRC the other day after listening to MIT entrepreneur Raffi Krikorian "it sounds like he's saying Twisted makes you smarter." Humor aside, Allen is right. Twisted does make you smarter: with increased familiarity and experience, you start thinking outside the box. Twisted helps you become a more creative problem solver.

In particular, we're reviewing the "Teach Me Twisted" open space session we had at PyCon. A bunch of you showed up for it, and the energy in that room was just phenomenal. 30 minutes after the session, people were still talking excitedly about what they were learning or how they were using Twisted or just sharing their love for the code :-)

For those of you that missed it, Steve Holden was the headliner while Alex Martelli played impromptu co-star. The humor and enthusiasm from these two was just incredible. Glyph, Itamar, and Chris played educators while JP, Zooko and I handled one-on-one questions in the audience. There were more players, but you get the point: it was a highly dynamic, lively and fun experience. Folks were so jazzed that conversations that night lasted long into the wee hours of the morning.

After almost two months' worth of post-PyCon follow-up, we're finally getting around to comparing notes. My biggest concern is for the absolute new-comer and the lack of intuitive and useful metaphors that would help aspiring Twisted users grasp the event-driven concepts of our code quickly. Steve and I are both interested in establishing a Proper and Good motivation for using Twisted. My girlfriend, who has a Masters in anthropology, was also there. Thanks to her insight and background, she has a completely different perspective of the community (and the new-comer dynamic at the session that night) and has some completely unique ideas for crafting a new generation of tutorial materials.

We're just getting started, but it's quite exciting. We expect to have more thoughts to share on the matter... in the form of materials as well as potential news items.

One last parting thought: despite the rumors and well-earned reputation to the contrary, Twisted coders are not exclusionists: everyone's invited to the party. We're just trying to make it easier to get there :-)