Mac OS X --> Solaris via Serial

My google foo was pretty low tonight, and I had a hell of a time connecting to a Netra 240. After lots of searching, trial, and error, I'm now in like Flynn. Here's what I ended up using:

• Sun Netra 240
• Mac OS X 10.4, PowerBook G4
• Keyspan USB/Serial adapter (P/N: USA-19HS)
• Keyspan drivers
• Sun-provided RJ-45 serial cable (I think the part number is 530-2093-01)
• minicom (installed via Fink)

Instructions for getting connected with minicom:

• Start minicom from the command line
• Hit "ctl-a" an then "z"
• Hit "o" to adjust the configuration
• Down-arrow to the "Serial port setup" menu and select
• Make sure that you're pointed at the right serial device and that your speed/parity setting is "9600 8N1"
• Hit "enter" to save and go back to the menu
• Select "Modem and dialing" and change the "Init string" to "~^M~"
• Hit "enter" and select "Save setup as dfl", exit, and restart minicom

Here are the gotchas I ran into:

• be sure to reboot OS X after installing the adapter drivers
• be sure to use the right cable (I was using a Cisco RJ-45 serial cable, and it seems to be pinned differently)
• be sure to use minicom (I tried with ZTerm, but I just don't grok GUI apps)

Once all these were in place, it was a piece of cake. With a BREAK (no keyboard for the Netra, so no "Stop-A") during boot-up, I'm ready to wipe this puppy and install Solaris 10.

Damn, Sun makes solid hardware. It's been years since I've played with their stuff. Even if Python apps don't run so well on Solaris, it's still going to be good to use these machines again.

Update: Added step-by-step instructions for connecting with minicom.

Nevow for Rapid-Deployments

I think I remember Guido making the comment that Twisted and/or Nevow scared him. I also think that (ignoring the causal effects of his statement) many people feel the same way, and if those people saw this blog post, they'd say I was deluded.

I've been using Nevow (on and off) since fzZzy first presented it at PyCon2004. I've been blogging about it since a year after that.

My first "web framework" was shell-script-generated HTML in the mid-90s. Then I found PHP. Zope. Perl (Interchange and Mason). Plone. Then a whole string of custom solutions.

I tried Django when it first came out, and hated it. I tried it again recently, and was most pleased with the progress they've made. I'm using it on a project with some non-programmer folk who wanted to convert their application from PHP to Python (I couldn't sell them on twisted). After digging around in the Django code base, I can safely say I will probably enjoy myself while working on the project.

I've also been a fan of the z3 CA for a couple years, and disagree with much of Ian Bicking's recent assessment of it (and agree very strongly with Martijn's comments). I continue to do work with z3 and enjoy almost all of it (being able to pick and choose helps greatly).

But when it comes down to it, when I need to roll something out fast, and I can choose the framework without concern for such things as technical support, community popularity, buzz, or political considerations, I choose Nevow. Case in point, the project I mentioned in this post. 30 minutes to casually convert a project from z3 to Nevow.

Though important, development speed is not the only consideration to make. Everyone talks about how fast you can roll a project out in web framework X, etc. But there's something that has a much higher priority for me: how insane will it let me be? Can I do anything I want? Once it's built, can I plug "stuff" in and out? Can I make unexpected changes easily and quickly, without compromising the integrity of the architecture? And even more: can I build my own system(s) with it?

And that's where Nevow cinches it for me. I am the most comfortable with it's design, templating, internals, and programmer freedom. I feel I have a little more freedom and flexibility with it than I do with z3. Nevow provides me with the tools and comfort level to build my own systems easily, quickly, and extensibly.

As an example, take this work in progress.

I've built a couple game-world oriented sites in the past. The first one used PHP and the second two used Plone. In both cases, it was very difficult to easily manage what I wanted to manage. I learned a lot of lessons about how information management works for me. When I started working on Myðgarður, I put these lessons into effect. Nevow let me do that.

I have a highly customized brain that needs things done in a certain way in order to be maximally productive. I think lots of brains are like that ;-) I want a framework that reflects my brain's needs, not only to deliver a result quickly, but to deliver it in a way the fits me best... and can adapt to the future best.

As a side note, Nevow is going through an overhaul (more) right now that will not only improve its general efficiency, but will actually fit my brain even better. When the context-less Nevow is released, I plan on producing at least one screencast on how to go form 0 to 60 with it in less than 20 minutes.

So stay tuned...

Technorati Tags: internet, nevow, programming, python, twisted, web

A New Experience

As I told Bill Karpovich today, I don't think I've ever had this happen before: add a feature to software and then have someone on a major news site blog about it.

To be fair, the Zenoss objects-to-XML (and vice versa) code was written quite a while ago by Erik Dahl. What I finished this week was providing a means of exporting Zenoss templates (RRD and Nagios) and then importing them from the file system or a URL (as well as some initial UI plumbing).

The code may sound mundane, but when you have a highly active, intense, and enthusiastic user base, writing code that makes their lives easier is a profoundly satisfying activity. Because, inevitably, you get emails and IRC comments like "you guys are awesome!" and "we love you!"

The next level of deployment will likely include a dedicated mail list for users to share the systems management/monitoring templates (and download them directly into Zenoss). This will be a transitional feature while we continue working on the portal where users will have the ability to upload, edit, and publish (share) their templates, and then download these templates through the Zenoss management UI.

I'm having a great time with these guys -- an awesome team with an incredible product and an amazing community :-)

Now playing:

Muse - Darkshines

Technorati Tags: media, monitoring, programming, software, work

Work Break

Tired of coding twisted? Need a break to relax? Play a game...

Now playing:

Muse - Butterflies and hurricanes

Technorati Tags: games, programming, twisted

Python and Monitoring

One of the first things I did when I got back home on Sunday was check the laptop I left at home... and sort the 1800+ emails, spam a-taunting. Of the odd hundred or so that were valid communications, one was from a friend with a heads-up/contact about RedHat's new effort to break into the burgeoning monitoring market. Apperantly, they are looking for python talent with coders having a proven background in providing systems management and monitoring solutions.

I find this interesting not only because of my long involvement in the monitoring arena (including the pymon and CoyMon projects), but also because of my recent work with Zenoss and the contact I have had with competitors in the field over the past year. My interest in architecting efficient and fast monitoring systems led me to develop pymon in twisted. This was, in part, a reaction to my not-so-happy experience with NetSaint/Nagios. On the other hand, CoyMon was positively inspired by Cacti, with an aim to give similar usability to NetFlow data. I have found it instructive to discover the various impetuses that drove me to develop monitoring solutions and to find what inspired others to do the same. I wonder what is driving RedHat. Business need? Business opportunity? Chronic issues with their current system?

On a side note, I did some more work on pymon this weekend, and I am really happy with the way the internals of it are progressing, primarily

1. how damned fast it runs, and
2. how easy it is to setup and configure.

Most of all, though, I find the design increasingly flexible and elegant. There's still some horrible crud tucked in the older crevices, but that stuff's on its way out.pymon is not explicitly systems management software, but rather a service-checking application. A part of me is curious, though, about what could be built using pymon's design principles. There could be some interesting possibilities to explore there.

Anyway, back to the topic: Systems monitoring and management a huge market and it is getting a lot of development effort and investment attention. When the dust settles in a year or two, it will be most interesting to see where all of this leads and what code bases/companies are left standing.

Now playing:

Muse - Ruled by secrely

Technorati Tags: monitoring, pymon, python, software

Roadtrip

Sunday I returned from a two-week road trip through Wyoming, Montana, Idaho, Washington, Utah and then back to Colorado. Though the scenery was amazing, the best part was the actual purpose of the trip: seeing friends, some of whom it has been 15 years since we last hung out.

Of the many very cool things that happened on the trip, one of the things I can share in a public forum is the RPG experience I had in Pullman, WA visiting my dear friend Jim Roach. He and his good friend Jacob DMed/GMed two sessions each and it was just phenomenal. Jacob's was a Ravenloft campaign I got to join with a 54-year old character named Li Po (monk) on a journey to Romania in the late 1800s. The other was a GURPS character -- a kilt-clad goblin samurai. I haven't gamed with sessions like that in over 10 years, and it was so much fun I will probably joining them again via web cam.

Upon my return to Colorado, I had a new Sector 9 bamboo longboard waiting for me. Two days and no spills. I'm totally not into tricks, but damn, I love to cruise and slalom. I've had a delightful time getting reacquainted with my skating muscles, though my body currently disagrees, as I can barely walk up the stairs right now.

Now playing:

Muse - Plug In Baby

Technorati Tags: friends, games, skating, travel

Knights of Cydonia

This is the BEST VIDEO EVAR.

How can you possible beat it? It's got kung fu, tai chi, spaghetti western, gun fighting, post apocalyptic America, lasers guns, holograms, and rock and roll. Take the best parts of the 70s (pop-culturally and ideologically) and pack as much of that as you can into a 6-minute video, and this is what you get. It's the music video equivalent of this poster.

And it's now responsible for making me a Muse fan.

Technorati Tags: entertainment, music, videos

Python and kcachegrind

I've recently needed to profile some very subtle issues that cropped up in a customer's python application. However, when I tried to use hotshot, I consistently got tracebacks. After some digging around on the net, I saw folks saying that profiling is basically busted in python2.4 (and then I remembered Itamar saying basically the same thing at PyCon 2006 when we were looking at web2 slowness).

To get around this, I built python2.5 from svn and copied its cProfile, _lsprof and pstats files to my python2.4 libs. This was a complete desperation move and I totally didn't expect it to work -- but it did (with only a warning about a version mismatch).

Earlier this year, JP and Itamar updated an lsprof patch to work as a standalone. However, I've never done any profiling in python, so it took a few minutes to get up to speed. Looking at the patch source and the python2.5 cProfile docs and then doing the usual dir() and help() on cProfile.Profile in the python interpreter is what helped the most.

To give others new to profiling a jumpstart, I'm including a quick little toy howto below.

Import the junk:

>>> import os
>>> import cProfile
>>> import lsprofcalltree

Define a silly test function:

>>> def myFunc():
...   myPath = os.path.expanduser('~/kcrw_s')
...   print "Hello, world! This is my home:"
...   print myPath
...

Define a profile object and run it:

>>> p = cProfile.Profile()
>>> p.run('myFunc()')
Hello, world! This is my home:
/home/kcrw_s/kcrw_s
<cProfile.Profile object at 0xb7c87304>

Get the stats in a form kcachegrind can use and save it:

>>> k = lsprofcalltree.KCacheGrind(p)
>>> data = open('prof.kgrind', 'w+')
>>> k.output(data)
>>> data.close()

You can now open up the prof.kgrind file in kcachegrind and view the (in this case, very uninteresting) results to your heart's content.

Technorati Tags: profiling, python

Twisted Mail Server

Update: The saga now has a conclusion! See this blog post for details.

Well, these last two months have been hell. Working on sucky projects just plain sucks. It ate up so much of my time that I've got an unhealthy backlog of blog posts waiting for release into the wild.

The first post I am compelled to write is on my new mail server. In the past, I've used sendmail, qmail and postfix. I have not been happy with any of those (though I did really like qmail, until it got too cumbersome to keep it updated). I didn't have the level of control I wanted -- and this was solely because I couldn't fit my brain into those applications (though, again, qmail came the closest).

I decided to chuck it all, and write my own mail server in twisted, using the pre-built lego code that twisted offers for this sort of thing. I've been running the server for about a month now, and all I can say is "wow". Just WOW.

The level of control I have over the operation of my mail server is insane. I can get this thing to do exactly what I want, when I want. I've got a bazillion domain names for which I (or others) receive email. I was able to write the code that lets me handle that the way that makes sense for ME (and *not* the author(s) of Postfix, etc.).

Today, I needed to add support for aliases that were actually lists of recipients. One "if" statement and an additional implementation of smtp.IMessage later, it was operational. I don't know how I ever ran a mail server any other way.

I've been testing my mail server all month, and it's running beautifully. It has continued to be free of relay issues and spammer attacks. I couldn't be happier with the results.

Now that I am feeling more secure in the custom code, I'm ready to start adding additional features I need:

• white listing: automatically updated with the address of people to whom I send email
• black listing: I am starting to maintain a list of the most heinous offenders in my junk mail box; these will be regularly pushed to the mail server
• greylisting: I have begun planning an implementation of greylisting, but this will take some time to get right
• spammer databases: I am considering using one or more of these. My only problem with them is that I don't trust them. I don't want to block someone inadvertantly just because they were unlucky enough to have one of their boxes raped into becoming an open relay 3 years ago.

Having a mail server that runs on twisted seemed a little daunting at first. I feared maintenance and security burdens, however, I have already begun reaping the benefits and my fears have been shown to be baseless. I spend 1/10th the maintenance time on the twisted version. I have *fun* while updating the server or configurations. I can plug my own code into it instead of using third party applications I don't like or patches I don't understand.

My first exposure to twisted was late 2002 as I was writing my first "real" python script (a networking script, naturally). Since that time, twisted has integrated itself into my life such that I can't imagine living without it. I literally use it for all of my coding activities: my professional life depends upon it nearly 100%, and 50% of my entertainment is derived from programming activities, all of which incorporate some aspect of twisted.

I, for one, welcome our twisted overlords.

Now playing:

Bagpipes - Flow Gently Sweet Afton

Trac Spam

It seems I was the last to find out about the "trac spam" phenomenon. I've got a bunch of projects that use trac here, and as a result, I've been cleaning up the huge mess these bastards left on the wiki. Gotta love page history, though...

The pages that are most viewed are the ones with code examples on them, and the spammers were kind enough to delete that code, making the pages useless. The code is now restored and can be referenced again. In particular, the following projects were impacted most heavily:

• AmazonDBDriver


• NetCIDR


• PyRRD

I've now got the wiki set for auth users only. If you want to contribute to the wikis, just send me an email.

Technorati Tags: internet, spam, trac