Thursday, April 23, 2009

Generators and Coroutines

This came up in the blog comments yesterday, but it really deserves a post of its own. A few days back, I was googling for code and articles that have been written regarding the use of Python 2.5 generators to build coroutines. All I got were many dead ends. Nothing really seemed to have any substance nor did materials dive into the depths in the manner I wanted and was hoping to find. I was at a loss... until I came across a blog post by Jeremy Hylton.

This was heaven-sent. Until then, I'd never looked at David Beazley's instruction materials, but immediately was dumbstruck at the beautiful directness, clarity, and simplicity of his style. He was both lucid in the topics while conveying great enthusiasm for them. With Python 2.5 generator-based coroutines, this is something that few have been able to do a fraction as well as David has. I cannot recommend the content of these two presentations highly enough:
Even though I've never been to one of his classes, after reading his fascinating background, I'd love the chance to pick his brain... for about a year or two (class or no!).

I've done a little bit of prototyping using greenlets before, and will soon need to do much more than that with Python 2.5 generators. My constant companion in this work will be David's Curious Course. Also, don't give the other slides a pass, simply because you already understand generators. David's not just about conveying information: he's gifted at sharing and shifting perspective.


Simon Wittber said...

Hi Duncan,

You may want to take a look at the Fibra package, which I maintain.

Fibra does most of the things David B. wrote about, and has been in production use for a few years now.

Anonymous said...

Hi Duncan!

As you know, the widely used definition of "coroutine" involves a construct which is capable of switching through multiple levels of call stack frames. As generators in Python are limited to switched through only a single level, they are not really coroutines. Enough people already make this mistake. Please don't encourage them. :)

Duncan McGreggor said...

Hey JP!

I've always wondered about that, though. Isn't a coroutine just something that allows for suspending and resuming execution in a function/method? Coroutines built with Python 2.5 generators seem to fit the bill, if that's the case.

It seems like the distinction you're making is between two different types of coroutine.

Regardless, I've always wanted to pin down the objections that people have had about using the term "coroutine" calling generator-based Python 2.5 coroutines. Maybe we could do another interview ;-)

Kirby Urner said...

Thanks for hyping David Beazley as a miracle man, I feel suitably awed having studied those slides. I also noticed his 'death to Fibonacci' refrain, shorthand for dumbing 'em down (generators) with trivial not-useful examples (in contrast to his sysadmin piping stuff, with yield generators a "pull train" and send coroutines a push train -- helpful to a noob like me (in this area, I admit my beginnger status)).

Kirby Urner said...

So in reference to foolish Fibonacci numbers Sir David (= knight in Python Nation) complains about, I've published a more
useful intro to coroutines, based on his excellent slides.

Duncan McGreggor said...

Very cool, Kirby :-) Thanks for sharing the links!