Thursday, March 27, 2008

We're in the Kitchen, Cookin Ur Mealz...

As Glyph just said, our community/development site is "reloaded" with a fresh look and the beginnings of some new structure. This comes as a result of many interdependent activities in the offing here at Divmod, most of which are still in the oven (and we wouldn't want to ruin the surprise now, would we?).

At PyCon, many of you approached us about more than the Twisted stuff we've been working on, and we had some good conversations. We've listened to all of you and have been making Herculean strides to provide a clearer view into what we do and how it can help you. We've got a long way to go regarding site content improvements and enhancements to documentation, both of which are genuinely at the top of our list right now. Yes, we've got a vision; but that is truly nothing without the continued interest and support of curious and creative folks like you, who want to architect extraordinary software.

There's been an extraordinary amount of energetic development, conversation, brainstorming and contribution that's been happening on IRC, at our offices, and in the code base, by employees and community members, at work and at play -- it's terrific to be a part of something so genuine and organic (the rumors of inorganic overlords have been greatly exaggerated).

Come take a look! ... And stay for a while :-)

P.S. If there are any CSS-on-IE Super Freaks in the House, will you please stand up? We're trying to work on some styling oddities that exist on the site and we could use some help!

Update: Paul Hummer has been amazingly helpful and got us up and running on IE when he had spare moments today. Thanks Paul!

Technorati Tags: , , , , , , , ,

Genshi in Nevow, Revisited

About a year ago, I blogged about using Genshi in Nevow. It was purely an exercise in curiosity on my part: I wanted to know how flexible Nevow's template parsing was. Could I just drop something else in there that was completely different? The answer was yes and no :-)

The "no" part of that was discovered by a kind reader, Karl Bartel, who found that the Genshi templates in Nevow were one-time wonders: after the first render, data was not refreshed. I didn't have time to take a look at why, since Zenoss was occupying so much of my time then. Now that I'm rocking out at Divmod, this sort of thing is more or less my business now :-) What's more, a friend recently made a deeply heart-felt plea to get Genshi working on Nevow, so I took another look tonight.

Here's the "yes" part of that answer :-) This only represents about an hour or so of work, so this hasn't been tested very thoroughly, but there's a trivial way to get last year's old, static code working such that Genshi templates are dynamically rendered in Nevow. Here's what I did to make it work:
  • preserve the original template data in a temp variable
  • reassign to self.template after Nevow's load() method has been called
  • clean Nevow's template cache
That's it :-)

If you want a single template to be able to be processed by both Nevow's templating machinery as well as Genshi's, you're going to have to make some sacrifices in efficiency. However, if you want a pure-Genshi solution for Nevow, you should be able to use this code to get something up in no time.

In addition to the bullet points above, I also did the following:
  • added some "method filters" so that Nevow stuff didn't get called by the Genshi loader
  • checked to see if the python attributes available to the Genshi templates were callable and if so, called them
As usual, leave a comment if you run across any problems.

Technorati Tags: , , , , , , ,

Sunday, March 23, 2008

Twisted on Solaris: Fail

No, this isn't the fail blog, but right now it ain't the supper happy dev blog, either. Let's do the Good first:


To mis^H^Habuse Wind In His Hair's quote from Dances with Wolves, here's what I say to Sun: Do you see that I am your fanboy? Can you see that I will always be your fanboy?

As I said to Glyph last night while Solaris was installing on my iMac, I have the fondest memories of Sun, SunOS and Solaris. It ranks right up there with the time I spent on the Kaypro II CP/M machine as a kid in the early 80s, and far more special than the considerable time I spent on VAX/VMS machines at APSU and UMASS in the mid 90s. I remember doing physics projects in Mathematica and creating the first site for my web design business in '96 on SunOS 5 machines at UMCP. It felt real and solid... and it was exciting. After using SunOS, I rarely touched a PC. Even my early Linux experiments were just that: hacking ethernet drivers so that Slackware was usable for me (it was a couple years before I was running Mathematica on Linux :-)); it was the Sun machines I used for the real work.

So yesterday, once the install finished and I started up a CDE session, you can imagine my nostalgia :-) In case you can't, I took some pictures to remember it by... I'm gonna miss the CDE when it goes :-( It was great being back in these Motif stomping grounds, iconizing windows to the desktop, tweaking the environment in the same old ways. I look forward to the possibility of more of this in the near future.

As for add-on software, those Sunfreeware guys are both still at it and still insanely awesome. What would I ever do without them? I"ve been using the packages at this site almost as long as I've been using Solaris. This may be a question for the Lazy Web, but can we get an apt-get for Post-OS-install installs are so slow, due to the manual pick-and-choose-and-download-and-unzip-and-pkgadd... honestly. Oh, wow -- after some digging, I just discovered that this already exists. (doesn't seem to work with Sunfreeware; does work with the old SunSite/ibiblio hosts, though).

Thanks to Sunfreeware, I had Python 2.5 and Subversion up and running in no time, co'ed Twisted and the Divmod code base, got Combinator installed, and started running Twisted test suites. Quick Combinator plug: with just four commands (2 per repository), I was on branch-management easy street. If you're using svn, Combinator should be your best friend...


You already know the big-bad, since you can see the title of this post :-) Twisted unit tests had 3 failures and 66 errors. Given that this represents about 0.07% and 1.59% of the tests, respectively, that's not too bad. The errors were higher at first, but my pyOpenSSL build was pointing at the wrong libs; there were obvious improvements once I pointed to the right libs.

Solaris x86 doesn't seem to perform as well. This matches the stories I heard a couple years ago, so I'm not that surprised. Solaris has a proven track record on Sparc; high performance on x86 may be a ways off. I ran the Twisted test suite on a VM install of Solaris and Ubuntu in Parallels (and thus both using identical hardware). During each test run, I left my machine alone, so there were no resource drains (or differences in utilization) either from the VM or Mac OS X.

Ubuntu 7.10
real 3m25.151s
user 0m54.655s
sys 1m22.189s

Solaris 10
real 5m32.329s
user 1m17.935s
sys 0m37.875s

As you can see, Solaris x86 runs a little more slowly. Once the test errors are addressed, it will be interesting to see if there is any change.


It seems that one or more of the partitions started filling up towards the end of the tests, so several of the errors were exceptions.OSError: [Errno 12] Not enough space. During the install, I just accepted the default partitioning that Solaris offered for the 32GB Parallels virtual drive, so this is going to need some tweaking. For starters, /tmp is way too tiny, especially since that is what is used for package installs (unzipped temp files). Likewise / is way over-used; I should probably put /usr on its own partition.

However, the point is not to blow the dust off my old sysadmin skills, but rather that a default install isn't actually a proper environment for developers. If Sun wants devs to be able to jump right in and get going, then they need to either provide a better default or, if there is one, make it more obvious how to get to that default.

What's more (less?), when attempting to run the unit tests on Solaris, I kept getting GUI popup error messages saying that there were too many processes open and I should close some applications. I didn't dig at all to see which resources were limited; the purpose was to just get a complete test suite run, not do log file forensics and discover the source of the issues (which I hope to do in the future). I had to close all windows, save the terminal where I was running the tests.


So, what have I taken away from all this? Possibly the following:

  1. Sun needs to put some resources into integrating the work that has been done with pkg-get; user communities thrive on easily available software, and developers count on it.
  2. If I'm going to seriously consider using Solaris as a development platform, I need to start hanging out on the forums, get back up to speed; I'm sure there's lots of good stuff out there that simply hasn't gotten any exposure.
  3. Similarly, I'm going to have to do some Solaris brush-up and then uncover some best practices -- using/adapting those that are extant, or creating new ones where there is a need.
The last two would be an investment on my part, but if Sun is serious about the Python community and supporting Python on Solaris, I'm willing to give it a shot.

So what's next? If that mythical entity "free time" truly does exist, I'm going to take a deeper look at Twisted on Solaris and start identifying the problems. I'll probably explore some system setup best practices too, and see what it would take to get pkg-get to support

Technorati Tags: , , , , , ,

Sunday, March 16, 2008

Twisted Announces TSF

Hey everyone, in case you weren't at PyCon (or were, but slept-in on Saturday morning... or aren't subscribed to any of the Twisted feeds), we made an announcement about the Twisted's new membership in the Software Freedom Conservancy. There's a news post about it here where you can get the latest details.

Update: Thanks to feedback from Grig Gheorghiu, we now have two domains that direct to the TSF page:

Technorati Tags: , , ,