Twisted on Nexenta/OpenSolaris

A few hours ago, I had the chance to install Nexenta/OpenSolaris in Parallels. The install was pretty straight-forward and quick. Gnome isn't on the .iso, but this was easily addressed with a/etc/apt/sources.list update, and an apt-get (a ~450MB dependency download and install). With Gnome was up and running, I was amazed at its responsiveness: Gnome on Nexenta seems to be much snappier than Ubuntu 7.10. This is the first time I've seen something I could use instead of Ubuntu, and that's saying a lot.

Next up was getting the Twisted and Divmod code installed. This required the following additional package installs:

sudo apt-get install python2.4-zopeinterface
sudo apt-get install python2.4-profiler
sudo apt-get install python2.4-pyopenssl
sudo apt-get install python2.4-crypto

With that done, I ran trial twisted, and watched the tests zip by. The end result? Only 1 failure and 2 errors; that's a pretty significant improvement over Twisted in Solaris 10. The failure was actually a little bit weird: the test can't find /dev/tty, however the device does exist (and I can open it from the python prompt). The two errors came from the UDP "multi listen" test, and were are result of the test timing out.

If I can get those two tests sorted out, I'll start testing the Divmod code. If all goes well, this could very well end up being my new development platform.

The Problem with and Solution to Google's App Engine

I know everyone is all aglow with the new web development offering from Google, but let me do the unpopular thing and put some things into perspective: there are limitations.

In fact, the limitations that exist will prevent me from using App Engine with all of my projects, save one (that one being a very simple web site). First, the limitations that prevent me from using App Engine (from one of their FAQs):

• Sockets are disabled with Google App Engine
• The system does not allow you to invoke subprocesses, as a result some os module methods are disabled
• Threading is not available

This means that I can't write a deferred wrapper for their data layer, I can't use Twisted for such things as XML-RPC or AMP-based communications, and I can't use an async templating system (like Nevow). I'm stuck with CGI and blocking code. And for all but the simplest projects, that's a big "No Thank You" from me.

This doesn't mean that I won't use it -- I will. I have one project that this will be perfect for... but it's for someone else, not me.

However, these limitations are actually good news :-) Here's the silver lining:

As Glyph as alluded to in his recent blog post (and in our tweets), we've recently completed a massive week-long BizDev Divmod sprint in Boston. One of the results of this is based on community feedback we've had over the last year, and which culminated at PyCon 2008 in Chicago with multiple requests for particular services from The Twisted Company. That result is a set of tools, features, and management options folks will be able to use with our software (app server, smart object db, network services, etc.). People really want to start using our stuff in cloud/grid computing environments. They need support for multiple and diverse network services, inter-store communications, massive deployments, etc. Two months before PyCon, we started working on tickets to support this, and we're making excellent progress toward providing the requested features.

We're still unclear as to which parts of this will be open source, as that will be driven by a combination of business and community demand. Regardless, Google's lack of support for this stuff has (for now) left the field wide open for us. And that, folks, is a big "Thank You Google!" :-)

We're in the Kitchen, Cookin Ur Mealz...

As Glyph just said, our community/development site is "reloaded" with a fresh look and the beginnings of some new structure. This comes as a result of many interdependent activities in the offing here at Divmod, most of which are still in the oven (and we wouldn't want to ruin the surprise now, would we?).

At PyCon, many of you approached us about more than the Twisted stuff we've been working on, and we had some good conversations. We've listened to all of you and have been making Herculean strides to provide a clearer view into what we do and how it can help you. We've got a long way to go regarding site content improvements and enhancements to documentation, both of which are genuinely at the top of our list right now. Yes, we've got a vision; but that is truly nothing without the continued interest and support of curious and creative folks like you, who want to architect extraordinary software.

There's been an extraordinary amount of energetic development, conversation, brainstorming and contribution that's been happening on IRC, at our offices, and in the code base, by employees and community members, at work and at play -- it's terrific to be a part of something so genuine and organic (the rumors of inorganic overlords have been greatly exaggerated).

Come take a look! ... And stay for a while :-)

P.S. If there are any CSS-on-IE Super Freaks in the House, will you please stand up? We're trying to work on some styling oddities that exist on the site and we could use some help!

Update: Paul Hummer has been amazingly helpful and got us up and running on IE when he had spare moments today. Thanks Paul!

Technorati Tags: business, community, design, divmod, mantissa, nevow, python, web, work

Genshi in Nevow, Revisited

About a year ago, I blogged about using Genshi in Nevow. It was purely an exercise in curiosity on my part: I wanted to know how flexible Nevow's template parsing was. Could I just drop something else in there that was completely different? The answer was yes and no :-)

The "no" part of that was discovered by a kind reader, Karl Bartel, who found that the Genshi templates in Nevow were one-time wonders: after the first render, data was not refreshed. I didn't have time to take a look at why, since Zenoss was occupying so much of my time then. Now that I'm rocking out at Divmod, this sort of thing is more or less my business now :-) What's more, a friend recently made a deeply heart-felt plea to get Genshi working on Nevow, so I took another look tonight.

Here's the "yes" part of that answer :-) This only represents about an hour or so of work, so this hasn't been tested very thoroughly, but there's a trivial way to get last year's old, static code working such that Genshi templates are dynamically rendered in Nevow. Here's what I did to make it work:

• preserve the original template data in a temp variable
• reassign to self.template after Nevow's load() method has been called
• clean Nevow's template cache

That's it :-)

If you want a single template to be able to be processed by both Nevow's templating machinery as well as Genshi's, you're going to have to make some sacrifices in efficiency. However, if you want a pure-Genshi solution for Nevow, you should be able to use this code to get something up in no time.

In addition to the bullet points above, I also did the following:

• added some "method filters" so that Nevow stuff didn't get called by the Genshi loader
• checked to see if the python attributes available to the Genshi templates were callable and if so, called them

As usual, leave a comment if you run across any problems.

Technorati Tags: community, genshi, howto, python, software, templating, twisted, web

Twisted on Solaris: Fail

No, this isn't the fail blog, but right now it ain't the supper happy dev blog, either. Let's do the Good first:

Good

To mis^H^Habuse Wind In His Hair's quote from Dances with Wolves, here's what I say to Sun: Do you see that I am your fanboy? Can you see that I will always be your fanboy?

As I said to Glyph last night while Solaris was installing on my iMac, I have the fondest memories of Sun, SunOS and Solaris. It ranks right up there with the time I spent on the Kaypro II CP/M machine as a kid in the early 80s, and far more special than the considerable time I spent on VAX/VMS machines at APSU and UMASS in the mid 90s. I remember doing physics projects in Mathematica and creating the first site for my web design business in '96 on SunOS 5 machines at UMCP. It felt real and solid... and it was exciting. After using SunOS, I rarely touched a PC. Even my early Linux experiments were just that: hacking ethernet drivers so that Slackware was usable for me (it was a couple years before I was running Mathematica on Linux :-)); it was the Sun machines I used for the real work.

So yesterday, once the install finished and I started up a CDE session, you can imagine my nostalgia :-) In case you can't, I took some pictures to remember it by... I'm gonna miss the CDE when it goes :-( It was great being back in these Motif stomping grounds, iconizing windows to the desktop, tweaking the environment in the same old ways. I look forward to the possibility of more of this in the near future.

As for add-on software, those Sunfreeware guys are both still at it and still insanely awesome. What would I ever do without them? I"ve been using the packages at this site almost as long as I've been using Solaris. This may be a question for the Lazy Web, but can we get an apt-get for sunfreeware.com? Post-OS-install installs are so slow, due to the manual pick-and-choose-and-download-and-unzip-and-pkgadd... honestly. Oh, wow -- after some digging, I just discovered that this already exists. (doesn't seem to work with Sunfreeware; does work with the old SunSite/ibiblio hosts, though).

Thanks to Sunfreeware, I had Python 2.5 and Subversion up and running in no time, co'ed Twisted and the Divmod code base, got Combinator installed, and started running Twisted test suites. Quick Combinator plug: with just four commands (2 per repository), I was on branch-management easy street. If you're using svn, Combinator should be your best friend...

Bad

You already know the big-bad, since you can see the title of this post :-) Twisted unit tests had 3 failures and 66 errors. Given that this represents about 0.07% and 1.59% of the tests, respectively, that's not too bad. The errors were higher at first, but my pyOpenSSL build was pointing at the wrong libs; there were obvious improvements once I pointed to the right libs.

Solaris x86 doesn't seem to perform as well. This matches the stories I heard a couple years ago, so I'm not that surprised. Solaris has a proven track record on Sparc; high performance on x86 may be a ways off. I ran the Twisted test suite on a VM install of Solaris and Ubuntu in Parallels (and thus both using identical hardware). During each test run, I left my machine alone, so there were no resource drains (or differences in utilization) either from the VM or Mac OS X.

Ubuntu 7.10
real 3m25.151s
user 0m54.655s
sys 1m22.189s

Solaris 10
real 5m32.329s
user 1m17.935s
sys 0m37.875s

As you can see, Solaris x86 runs a little more slowly. Once the test errors are addressed, it will be interesting to see if there is any change.

Ugly

It seems that one or more of the partitions started filling up towards the end of the tests, so several of the errors were exceptions.OSError: [Errno 12] Not enough space. During the install, I just accepted the default partitioning that Solaris offered for the 32GB Parallels virtual drive, so this is going to need some tweaking. For starters, /tmp is way too tiny, especially since that is what is used for package installs (unzipped temp files). Likewise / is way over-used; I should probably put /usr on its own partition.

However, the point is not to blow the dust off my old sysadmin skills, but rather that a default install isn't actually a proper environment for developers. If Sun wants devs to be able to jump right in and get going, then they need to either provide a better default or, if there is one, make it more obvious how to get to that default.

What's more (less?), when attempting to run the unit tests on Solaris, I kept getting GUI popup error messages saying that there were too many processes open and I should close some applications. I didn't dig at all to see which resources were limited; the purpose was to just get a complete test suite run, not do log file forensics and discover the source of the issues (which I hope to do in the future). I had to close all windows, save the terminal where I was running the tests.

Summary

So, what have I taken away from all this? Possibly the following:

1. Sun needs to put some resources into integrating the work that has been done with pkg-get; user communities thrive on easily available software, and developers count on it.
2. If I'm going to seriously consider using Solaris as a development platform, I need to start hanging out on the forums, get back up to speed; I'm sure there's lots of good stuff out there that simply hasn't gotten any exposure.
3. Similarly, I'm going to have to do some Solaris brush-up and then uncover some best practices -- using/adapting those that are extant, or creating new ones where there is a need.

The last two would be an investment on my part, but if Sun is serious about the Python community and supporting Python on Solaris, I'm willing to give it a shot.

So what's next? If that mythical entity "free time" truly does exist, I'm going to take a deeper look at Twisted on Solaris and start identifying the problems. I'll probably explore some system setup best practices too, and see what it would take to get pkg-get to support Sunfreeware.com.

Technorati Tags: community, open source, python, solaris, sun, testing, twisted

High Dynamic Range Imaging

So David Reid has a killer desktop background that looks like a game screenshot from Half Life 2. Turns out, it's a real photo and it was done using multiple exposure/composite techniques, something known as "high dynamic range" imaging. There's more info about that here:

http://en.wikipedia.org/wiki/High_dynamic_range_imaging

Here are some nice ones that I have found on flickr and elsewhere:

• http://farm3.static.flickr.com/2071/2271947578_5b09f10150_o.jpg
  
• http://farm3.static.flickr.com/2002/2236003621_ac35fb1a3e_o.jpg
  
• http://upload.wikimedia.org/wikipedia/commons/d/d5/Trencin_hdr_001.jpg
  
• http://upload.wikimedia.org/wikipedia/commons/2/22/New_York_City_at_night_HDR.jpg
  
• http://farm3.static.flickr.com/2157/2186079657_a970c3e4ea_o.jpg
• http://farm3.static.flickr.com/2004/2079908663_85832e9461_o.jpg
  
• http://farm3.static.flickr.com/2030/1524399896_a93560592b_o.jpg
  
• http://farm3.static.flickr.com/2056/1552881168_2cf7411b84_o.jpg
  
• http://farm1.static.flickr.com/87/232598586_bf083fbe68_o.jpg
  
• http://farm1.static.flickr.com/55/108078035_eabc7da078_o.jpg
  

I still haven't found the one that is on his desktop background, but when I do, I'll post a link :-)

Technorati Tags: photography , pycon

Twisted Announces TSF

Hey everyone, in case you weren't at PyCon (or were, but slept-in on Saturday morning... or aren't subscribed to any of the Twisted feeds), we made an announcement about the Twisted's new membership in the Software Freedom Conservancy. There's a news post about it here where you can get the latest details.

Update: Thanks to feedback from Grig Gheorghiu, we now have two domains that direct to the TSF page:

• twistedsoftwarefoundation.org
• tsf.name

Technorati Tags: community, tsf, pycon, twisted

Twisted News Resources and Divmod Sprint at PyCon

These days, there's a lot going on right now in the Twisted community as well as here at Divmod. It's really quite exciting :-) But, in an effort to not spam the world, I'll try to group things together when possible.

First off is Twisted news: in case you haven't noticed yet, Twisted Labs now has a group blog here. We're making an effort to share more of our conversations and vision with the wider community. We'll also be posting news items there, which means (at long last!) we now have an RSS feed for the latest Twisty Happenings. To just read the news items (if that's your preference), you can visit this page (it's a "news" tag filter). What's more, if you want an RSS feed that just covers the news, you can use this one, which is also being maintained. As always, our News Archives page continues its non-feed-having, wiki-ish existence.

Next up is the Divmod Sprint at PyCon 2008: we're having one! I posted an entry on our company blog, so please check it out and follow the links to join our sprint, if you are so inclined.

More soon!

Technorati Tags: community, divmod, pycon, twisted

zope.testbrowser: Automating the Web

Speaking of little known python modules, if you haven't used zope.testbrowser, please raise your hand. Okay, all of you that raised your hand and have ever wanted to automate web forms, check a web site's functionality, or perform screen-scraping related tasks, pull up a chair. This won't take long, because zope.testbrowser's API rocks so hard and is so easy to use.

I first started using zope.testbrowser sometime before consulting for Zenoss. However, when I implemented Zenoss' "Synthetic Transactions" (I believe this was originally part of their open source offering, but has since been moved into the Enterprise Edition), I used it again and profusely. It provided me with a wonderful Python API (a mechanize wrapper and then some), one so intuitive that I could use not only without thinking, but for which I didn't even have to read the docs. This solution proved to be friendly to programmers, but we were really targeting non-programming network administrators and IT managers with the Zenoss plugin, so I completed the plugin using Twill and TestGen4Web (whose python runner I rewrote and to which I added support for zope.testbrowser).

I used it again at Zenoss later when building the community software, enabling users to set their Mailman preferences from their Plone settings. Most recently, I've been using it to publish news articles to various sites from a single file used by a script.

Here are a couple of standard use cases with sample code:

Visit a page and follow some links


That gives us the actual link object... but what if we want to follow that link?

Simple enough :-)

Check for content

Let's continue on the Google link trail, and check for content:


Sign into a site

Signing into a site requires filling in form information and submitting that data.


A couple things to note here:

1. you need to look at the HTML so that you know what the form elements are named;
2. there are two form elements on the page that are named loginpage_email, and we want the first one; and
3. I omitted the part of the code where I pulled my credentials from the file system.

Submit form data

Now that we're logged in, let's change some out-dated information in my profile (my old blog's link has been there for too long):


Now that we saved it, let's check our results:

Excellent; just what we expected to see.

These are all really simple examples, but they should be helpful in providing some insight and inspiring you to use it :-) Working with radio buttons is a little more complex in that you have to get the containing object first and use getControl on that object in order to get the selection you want. However, if you've worked with HTML more than any sane person should have to (I think that exact amount is "any") , this will all be quite expected.

So, who's up for making an async version to work with Twisted? ;-)

A tangential caveat: I made a comment above about the zope.testbrowser API rocking, and that needs some clarification. As far as I am concerned, there are two major ways in which API's can be graded:

1. how easy is the API to use, how well is it documented, how much of the problem domain does it cover? (developer as user)
2. how elegantly/efficiently was the solution implemented (developer as contributor)

zope.testbrowser rocks from the perspective of a coder using the API. I've never looked at how it is implemented, because I haven't needed to. It does everything I require without me having to dig around in its guts. As such, I can make no assertions as to the quality of its implementation. However, it's packages, modules, classes and methods are all named just as I would expect them to be named and do what I would expect them to do. That may not sound like a big deal, but it really is.

Technorati Tags: howto, programming, python, web, zope

PyCon 2008 Registration Discounts Expire Tomorrow!

Hey all, in case you're like me and wait for the last minute to do everything, you need to hop on the good foot and do the bad thing: pay now. Or do the worse thing: pay later, and pay more.

Currently, the rates are as follows:

• hobbyist registration: $220
• corporate registration: $400
• single room: $99 (maybe all rooms are the same price?)

After tomorrow, the rates jump up to the following (until March 7):

• hobbyist registration: $300
• corporate registration: $500
• single room: dunno... I've heard rumors that you might be able to get something for $160/night

Don't wait any longer... :-)

Technorati Tags: community, pycon