The Future of Programming - Adopting The Functional Paradigm?

Series Links

• An Overview
• Themes at OSCON 2014
• Adopting the Functional Paradigm?
• Retrospective on Paradigms
• The Rise of Polyglotism
• Preparing for the Future

Survivors' Breakfast

The previous post covered some thoughts on the future-looking programming themes present at OSCON 2014.

Following that wonderful conference, long-time Open Source advocate, Pythonista, and instructor Steve Holden, was kind enough to host his third annual "OSCON Survivors' Breakfast" with tens of esteemed attendees, speakers, and organizers enjoying great company and conversation, relaxing together after the flurry of conference activity, planning a leisurely day in Portland, and -- most immediately -- having some much-needed breakfast.

The view from the 23rd floor was quite an eyeful, and the conversation ranged across equally panoramic topics. Sitting with Alex Martelli, Anna Ravenscroft, and Katie Miller, the conversation inevitably turned to thoughts programmatical. One thread of the discussion was so compelling that it helped crystallize this series of blog posts. That was kicked off with Katie's question:

Why [have some large companies] not embraced functional programming to the extent that other large ones have?

Multiple points of discussion spawned from this, some of which still continue. The rest of this post explores these. 

Large Companies?

What constitutes a large company? We settled on discussing Fortune 500 companies, which, by definition are:

• U.S. Companies
• Ranked by gross revenue (after adjustments for excise taxes).

Afterwards, I looked up the 2013 top 25 tech companies in the Fortune 500. I've listed them below; in parentheses is the Fortune 500 ranking. After the dash are the functional programming languages used on various company projects -- these are listed only if I have talked to someone who has worked on a project (or interviewed for a job that used the language), or if I have read an article by an employee who has stated that they use the listed language(s) [1].

1. Apple (6) - Swift, Clojure, Scala
2. AT&T (11) - Haskell
3. HP (15) - F#, Scala
4. Verizon Communications (16) - Scala
5. IBM (20) - Scala
6. Microsoft (35) - F#, F*
7. Comcast (46) - Scala
8. Amazon (49) - Haskell, Scala, Erlang
9. Dell (51) - Erlang, Scala
10. Intel (54) - Haskell, SML, PLT Scheme
11. Google (55) - Haskell [2]
12. Cisco (60) - Scala
13. Ingram Micro (76) - ?
14. Oracle (80) - Scala
15. Avnet (117) - ?
16. Tech Data (119) - ?
17. Emerson Electric (123) - ?
18. Xerox (131) - Scala
19. EMC (133) - Scala
20. Arrow Electronics (141) - ?
21. Century Link (150) - ?
22. Computer Sciences Corp. (176) - ?
23. eBay (196) - Scala 
24. TI (218) - ?
25. Western Digital (222) - ?

The companies which have committed to projects guessed to be of significant business value written in FP languages include: Apple, HP, and eBay. Possibly also Oracle and Intel. So, a rough estimate of between 3 to 5 of the top 25 U.S. tech companies have made a significant investment in FP.

Why not Google?

The next two sections offer summaries of some views on this.

Ideal Use Case?

Is an FP language suitable for large organisations? Are smaller companies better served by them? During breakfast, It was postulated that dealing with such things as immutable data, handling I/O in pure FP languages, and creating/using higher order functions is easier for small startups due to the shorter amount of time required to hire or train a critical mass of skilled programmers.

It is certainly true that it will take larger organisations longer to train its personnel simply due to sheer numbers and, even with enough trainers, logistics. But this argument can be made for any corporate level of instruction; in my book, this cancels out on both sides and is not an argument unique to hard topics, even less, specifically pertinent to FP adoption.

Brain Fit?

I've heard this one a bit: "Some people just don't think in FP terms." They need loops and iteration, not higher order functions and recursion. Joel Spolsky makes reference to this in his article The Guerrilla Guide to Interviewing. In particular, he says that "For some reason most people seem to be born without the part of the brain that understands pointers." This has been applied to topics in FP as well as C.

To be fair, Joel's comment was probably made with a bit of lightness and not meant to be a statement on the nature of mind or a theory of cognition. The context of the article is a very practical one: hiring. When trying to identify whether a programmer would be an asset for your team, you're not living in the space of cognitive theory, rather you inhabit the realm of quick approximations, gut instincts, and fast, economical decisions.

Regardless, I find this perspective -- Type Physicalism [3] -- fairly objectionable. This is because I see it as a kind of intellectual "racism." Early social sciences utilized this form of reasoning to justify all sorts of discriminatory thinking in the name of "science", reinforcing a rigid mentality of "us" vs. "them." In my own experience, I've seen this sort of approach used to shutdown exploration, to enforce elitism, and dismiss ideas that threaten the authority of the status quo.

Rather than seeing the problem of comprehending FP as a physical limitation of the individual, I see instructional failure as the obstacle to overcome. If we start with the proposition that certain brains are deficient, we are essentially abandoning education. It is the responsibility of the instructor to engage creatively with each student's learning style. When adhering to the idea that certain brains are limited, one discards creative engagement; one doesn't even consider working with the students and their learning styles. This is a view that, however implicitly, can be used to shun diversity and dismiss potential.

I believe the essence of what Joel was shooting for can be approached in a much kinder fashion (adapted for an FP discussion):

None of us was born knowing GOTO statements, global state, mutable data, or for loops. There are many programmers alive, though, whose first contact with programming involved one or more of these. That's their "home town", as it were; their programmatic birth place. Having utilized -- as well as taught -- imperative, OOP, and functional styles of programming, I do not feel that one is intrinsically any harder than another. However, they are sometimes so vastly different from each other in style or syntax or semantics that once a student has solidified around the concepts of a particular paradigm, it can be a challenge retraining to work easily in another.

Why the Objections?

If both "ideal use case" and "brain fit" are given as arguments against adopting FP (or any other new paradigm) in large organisations, and neither are considered logically or philosophically valid, what's at the root of the resistance?

It is not uncommon for changes in an industry or field of study to be met with resistance. The bigger or more different the change from the status quo, very often is proportional to the amount of resistance. I suspect that this is really what we're seeing when companies take a stance against FP. There are very often valid business concerns: "we've made an investment in OOP" or "it will cost too much to train/hire/migrate to FP." 

I would remind those company leaders, though, that new sources of revenue, that product innovation and changes in market adoption do not often come from maintaining or enforcing the current state. Instead, that is an identifying characteristic of companies whose relevance is fading.

Even if your company has market dominance or is a monopoly, there is still a good incentive for exploring alternative paradigms. At the very least, one can uncover inefficiencies and apply new knowledge to remove duplication of efforts, increase margins, etc.

Careers

As a manager, I have found that about half of the senior engineers up for promotion have very little to no interest in taking on different (new to them) programmatic paradigms. They consider current burdens sufficient (or too much) and would rather spend what little free time they have available to them in improving existing systems.

Senior engineers who have a more academic or research bent (or are easily bored) are much more likely to embrace this sort of change. Interestingly, senior engineers who have little to no competitive drive will more readily pick up something new if the need arises. This may be due to such things as not perceiving accumulated knowledge as territory to defend, for example.

Younger engineers with less experience (and less of an investment made in a particular school of thought) are much more willing to take on new challenges. I believe there are many reasons for this, one of which may include an interest in becoming more professionally competitive with their peers.

Junior or senior, I have found that programmers who are currently looking to find new employment are nearly invariably not only willing to take on the challenge of learning different paradigms, but are usually going about that proactively and engaging in self-study.

I want to work with programmers who can take on any problem space in any paradigm and find creative solutions, contributing as valued members of a team. This is certainly an ideal set of characteristics, but one that I have seen in the wilds of the workplace on multiple occasions. It has nothing to do with FP or OOP paradigms, but rather with the people themselves.

Even if a company is locked into well-established processes and views on programming, they may find it in their best interests to provide a more open-minded approach with their employees who would enjoy that. Their retention rates could very well increase dramatically.

Do We Need To?

Philosophy and hiring strategies aside, do we -- as programmers, software projects, or organizations that support programming -- need to take on the burden of learning or adopting functional programming? Quite possibly not.

If Google's plans around Go involve building a new operating system (in the spirit of 1970s C and UNIX), the systems programmers may find pure functions too cumbersome to work with. FP may be too burdensome a fit for that type of work.

If one is not tied to a historical analogy with UNIX, as Mozilla is not with Rust, doing something like creating a new browser engine (or running a remote services company) may be a good fit for FP, especially if one has data showing reduced error counts when using type systems.

As we shall see illustrated in the next post, the usual advice continues to apply: the decision of which paradigm to employ for any given project should be dictated by the best fit and not ideological inflexibility. The bearing this has on programming is innovation: it is the early adopters who have the best chance of leading us into the future.

Up next: Retrospective on Programming Paradigms
Previously: Themes at OSCON 2014

Footnotes

[1] If anyone has additional information as to which FP languages are used by these top 25 companies, please let me know, and I will include that information. Bonus points for knowing of business-critical applications.

[2] Google Switzerland are using Haskell.

[3] Type Physicality is a form of reductive materialism, also known as the Mind-Brain Identity Theory that does not allow for mental states to be realized in organisms or computational systems that do not have a brain. See "Criticisms of Type Physicality" at http://en.wikipedia.org/wiki/Identity_theory_of_mind#Multiple_realizability.

The Future of Programming - Themes at OSCON 2014

Series Links

• An Overview
• Themes at OSCON 2014
• Adopting the Functional Paradigm?
• Retrospective on Paradigms
• The Rise of Polyglotism
• Preparing for the Future

A Qualitative OSCON Debrief

As you might have noticed from the OSCON Twitter-storm this year, the conference was a blast. Even if you weren't physically present, given the 17 tracks, you can imagine that the presentations -- and subsequent conversations -- were deeply varied.

This was the second OSCON I'd attended; the first was was in 2008 as a guest of Michael Bernstein, a friend who was speaking there. OSCON 2008 was a zoo - I'm not sure of the actual body count, but I've heard that attendees + vendors + miscellaneous topped 12,000 people over the course of the week (I would love to hear if someone has hard data on that -- googling didn't reveal much). OSCON 2008 was dominated by Big Data, Hadoop, endless buzzword bingo, and business posturing by all sorts. The most interesting bits of that conference were the outlines that formed around the conversations people weren't having. In fact, over the following 6 months, that's what I spent my spare time pondering: what people didn't say at OSCON.

This year's conference seemed like a completely different animal. It felt like easily 1/2 to 1/3rd the number of attendees in 2008. Where that one had all the anonymizing feel of rush-hour in a major metropolitan hub, OSCON 2014 had a distinctly small-town vibe to it -- I was completely charmed. Conversations (overheard as well as participated in) were not littered with examples from the latest bizspeak, but rather focused on essence. The interactions were not continually distracted, but rather steadily focused, allowing people to form, express, and dispute complete thoughts with their peers.


Conversations

So what were people talking about this year? Here are some of the topics I heard covered during lunches, in hallways, and at podiums; at pubs, in restaurants and at parks [1]:

• What communities are thriving?
• Which [projects, organisations, companies, etc.] are treating their people right?
• What successful processes are being followed at [project, organisation, etc.]?
• Who is hiring and why should someone want to work there?
• Where can I go to learn X? Who is teaching X? Who shares the most about X?
• Which [projects, organisations] support X?
• Why don't more [people, projects, organisations] care about [possible future X]?
• Why don't more [people, projects, organisations] spend more time investigating the history of X for "lessons learned"?
• There was so much more X in computing during the 60s and 70s -- what happened? [2]
• Why are we reinventing X?
• When is X going to be invented, and who's going to do it?
• Everything is changing! I can't keep up anymore.
• I want to keep up, but how?
• Why can't we stop making so many X?
• Nobody cares about Y anymore; we're all doing X now.
• Full stack developers!
• Haskell!
• Fault-tolerant systems!

After lots of reflection, here's how I classified most of the conversations I heard:

• Developing communities,
• Developing careers and/or personal/professional qualities, and
• Developing software, 

along lines such as:

• Effective maintenance, maturity, and health,
• Focusing on the "art",  eventual mastery, and investments of time,
• Tempering bare pragmatism with something resembling science or academic excellence,
• Learning the new to bolster the old,
• Inspiring innovation from a place of contemplation and analysis,
• Mining the past for great ideas, and
• Figuring out how to better share and spread the adoption of good ideas.

Themes

Generalized to such a degree, this could have been pretty much any congregation of interested, engaged minds since the dawn of civilization. So what does it look like if we don't normalize quite so much? Weighing these with what may well be my own bias (and the bias of like-minded peers), I submit to your review these themes:

• A very strong interest in programming (thinking and creating) vs. integration (assessing and consuming).
• An express desire to become better at abstraction (higher-order functions, composition, and types) to better deal with growing systems complexities.
• An interest in building even more complicated systems.
• A fear of reimplementing past mistakes or of letting dust gather on past intellectual achievements.

As you might have guessed, these number very highly among the reasons why the conference was such an unexpected pleasure for me. But it should also not come as a surprise that these themes are present:

• We have had several years of companies such as Google and Amazon (AWS) building and deploying some of the most sophisticated examples of logic-made-manifest in human history. This has created perceived value in our industry and many wish to emulate it. Similarly, we have single purpose distributed systems being purchased for nearly 20 billion USD -- a different kind of complexity, with a different kind of perceived reward.
• In the 70s and 80s, OOP adoption brought with it the ability to create large software systems in ways that people had not dared dream or were impractical to realize. Today's growing adoption of the Functional paradigm is giving early signs of allowing us to better integrate complex systems with more predictability and fewer errors.
• Case studies of improvements in productivity or the capacity to handle highly complex or previously intractable problems with better abstractions, has ignited the passions of many. Not wanting to limit their scope of knowledge or sources of inspiration, people are not simply limiting themselves to the exploration of such things as Category Theory -- they are opening the vaults of computer science with such projects as Papers We Love.

There's a brave new world in the making. It's a world for programmers and thinkers, for philosophers and makers. There's a lot to learn, but it's really not so different from older worlds: the same passions drive us, the same idealism burns brightly. And it's nice to see that these themes arise not only in small, highly specialized venues such as university doctoral programs and StrangeLoop (or LambdaJam), but also in larger intersections of the industry like OSCON (or more general-audience ones like Meetups).

Up next: Adopting the Functional Paradigm?
Previously: An Overview


Footnotes

[1] It goes without saying that any one attendee couldn't possibly be exposed to enough conversations to form a perfectly accurate sense of the total distribution of conversation topics. No claim to the contrary is being made here :-)

[2] I strongly adhere to the multifaceted hypothesis proposed by Bret Victor

here in the section titled "Why did all these ideas happen during this particular time period?"

The Future of Programming - An Overview

Art by Philip Straub

There's a new series of blog posts coming, inspired by on-going conversations with peers, continuous inspection of the development landscape, habitual navel-gazing, and participation at the catalytic OSCON 2014. As you might have inferred, these will be on the topic of "The Future of Programming."

Not to be confused with Bret Victor's excellent talk last year at DBX, these posts will be less about individual technologies or developer user experience, and more about historic trends and viewing the present (and near future) through such a lense.

In this mini-series, the aim is to present posts on following topics:

• An Overview
• Themes at OSCON 2014
• Adopting the Functional Paradigm?
• Retrospective on Paradigms
• The Rise of Polyglotism
• Preparing for the Future

I did a similar set of posts, conceived in late 2008 and published in 2009 on the future of cloud computing entitled After the Cloud. It was a very successful series and the cloud industry seems to be heading towards some of the predictions made in it -- ZeroVM and Docker are an incremental step towards the future of distributed processes/functions outlined in To Atomic Computation and Beyond. 

In that post, though, are two quotes from industry greats. These provide an excellent context for this series as well, hinting at an overriding theme:

• Alan Kay, 1998: A crucial key to growing large systems is effective communications between components.
• Joe Armstrong, 2004: To effectively model and solve problems in a distributed manner, we need concurrency... this is made easier when we isolate processes and do not share data.

In the decade since these statements were made, we have seen individuals, projects, and companies take that vision to heart -- and succeeding as a result. But as an industry, we continue to struggle with the definition of our art; we still are tormented by change -- both from within and externally -- and do not seem to adapt to it well.

These posts will peer into such places ... in the hope that such inspection might guide us better through the tangled forest of our present into the unimagined forest of our future.

Up next: Themes at OSCON 2014 ...

Prefix Operators in Haskell

I wanted to name this post something a little more catchy, like "McCarthy's Dream: Haskell as Lisp's M-Expressions" but I couldn't quite bring myself to do it. If s-expressions had greater support in Haskell, I would totally have gone for it, but alas, they don't.

However, there is still reason to celebrate: many Haskell operators do support prefix notation! This was a mind-blower to me, since I hadn't  heard about this until last night...

At the Data Day Texas conference this past Saturday, O'Reilly/StrataConf had a fantastic booth. Among the many cool give-aways they were doing, I obtained a coupon for a free ebook and another for 50% off. Upon returning home and having made my free book decision, I was vacillating between an OCaml book and the famous Haskell one. I've done a little Haskell in the past but have never touched OCaml, so I was pretty tempted.

However, something amazing happened next. I stumbled upon a page that was comparing OCaml and Haskell, which led to another page... where Haskell prefix notation was mentioned. I know many Haskellers who might read this would shrug, or say "yeah, we know", but this was quite a delightful little discovery for me :-)

I don't remember the first page I found, but since then, I've come across a couple more resources:

• http://www.haskell.org/haskellwiki/Infix_operator
• http://learninghaskell.wordpress.com/2009/07/23/haskell-prefix-and-infix-notations-for-functions/

That's pretty much it, though. (But please let me know if you know of or find any other resources!)

As such, I needed to do a lot more exploration. Initially, I was really excited and thought I'd be able to convert all Haskell forms to s-expressions (imports, lets, etc.), but I quickly found this was not the case. But the stuff that did work is pretty cool, and I saved it in a series of gists for your viewing pleasure :-)

Addition

The first test was pretty simple. Finding success, I thought I'd try something I do when using a Lisp/Scheme interpreter as a calculator. As you can see below, that didn't work (the full traceback is elided). Searching on Hoogλe got me to the answer I was looking for, though. Off to a good start:

More Operators

I checked some basic operators next, and a function. Everything's good so far:

Lists

Here are some basic list operations, including the ever-so-cool list difference operator, (\\). Also, I enjoyed the cons (:) so much that I made a t-shirt of it :-) (See the postscript below for more info.)

List Comprehensions

I'm not sure if one can do any more prefixing than this for list comprehensions:

Functions

Same thing for functions; nothing really exciting to see. (Btw, these examples were lifted from LYHGG.)

Lambdas

Things get a little more interesting with lambda expressions, especially when a closure is added for spice:

Function Composition

Using the compose operator in prefix notation is rather... bizarre :-) It looks much more natural as a series of compositions in a lambda. I also added a mostly-standard view of the compose operator for comparison:

Monads

I've saved the best for last, an example of the sort of thing I need when doing matrix operations in game code, graphics, etc. The first one is standard Haskell...

Wow. Such abstract. So brains. 

Note that to make the prefix version anywhere close to legible, I added whitepsace. (If you paste this in ghci, you'll see a bunch of prompts, and then the result. If you're running in the Sublime REPL, be sure to scroll to the right.)

And that pretty much wraps up what I did on Sunday afternoon :-)

Other Resources

• Travis Hartwell, Twisted hacker from back in the day, let me know about a great Haskell tutorial for Lispers.
• Here's a fun Lisper-oriented monad tutorial. Also, here's the best monad tutorial I've ever read.
• Liskell (ILC 2007 presentation; paper; 2009 update; source code)
• Lisk (Chris' lisk github repo seems to have disappeared, with the possible exception of an old fork)
• Husk Scheme (main page; github; Hackage)

Post Script

If you're in the San Antonio / Austin area (or SF; I'll be there in March) and want to go in on a t-shirt order, let me know :-) We need at least six for the smallest order (tri-blend Next Level; these are a bit pricey, but they feel GREAT). I'll collect payment before I make an order.

Cruising HTTP with LFE

In the last post, you learned how to get LFE running on Ubuntu. This one will give you some insight into how LFE can be used in something approaching real-world problems. In the next post, we're going to jump back into the lambda calculus, and we'll see some more LFE shortly after that.

Because Lisp Flavored Erlang is 100% compatible with Erlang Core, it has access to all the Erlang libraries, OTP, and many third-party modules, etc.  Naturally, this includes the Erlang HTTP client, httpc. Today we're going to be taking a look at how to use httpc from LFE. Do note, however that this post is only going to provide a taste, just enough to give you a sense of the flavor, as it were.

If you would like more details, be sure to not only give the official docs a thorough reading, but to take a look at the HTTP Client section of the inets Reference Manual.

Note that for the returned values below, I elide large data structures. If you run them in the LFE REPL yourself, you can view them in all of the line-consuming glory.

Synchronous GET

Let's get started with a simple example. The first thing we need to do is start the inets application. With that done, we'll then be able to make client requests:
Now we can perform an HTTP GET: This just makes a straight-forward HTTP request (defaults to GET) and returns a bunch of associated data:

• HTTP version
• status code
• reason phrase
• headers
• body

All of that data is dumped into our result variable. Here's the same GET but with pattern matching set up so that we can easily access all that data:
For those not familiar with Erlang patterns, we've just told LFE the following:

• the return value of the function we're going to call is going to be a tuple composed of an atom ('ok) and another tuple
• the nested tuple is going to be composed of a tuple, some headers, and a body
• the next nested tuple is going to be composed of the HTTP version, status code, and status code phrase

If you'd like to learn more about using patterns in LFE, be sure to view the patterns page of the LFE User Guide.

Once the request returns, we can check out the variables we set in the pattern:

That's great if everything goes as expected and we get a response from the server. What happens if we don't?

Well, errors don't have the same nested data structure that the non-error results have, so we're going to have to make some changes to our pattern if we want to extract parts of the error reason. Pattern matching for just the 'error atom and the error reason, we can get a sense of what that data structure looks like:

Looking at just the data stored in the reason variable, we see:
If you check out the docs for httpc request and look under "Types", you will see that the error returned can be one of three things:

• a tuple of connect_failed and additional data
• a tuple of send_failed and additional data
• or just unspecified additional data

In our example our additional data is a tuple of the address we were trying to connect to and the specific error type that for our failed connection.


Async GET

Now that we've taken a quick look at the synchronous example, let's make a foray into async. We'll still be using httpc's request function, but we'll need to use one of the longer forms were extra options need to be passed, since that's how you tell the request function to perform the request asynchronously and not synchronously.

For clarity of introducing the additional options, we're going to define some variables first: You can read more about the options in the httpc docs.

With the variables defined, let's make our async call: The sender receives the results, and since we sent from the LFE REPL, that's the process that will receive the data. Let's keep our pattern simple at first -- just the request id and the result data:
Needless to say, parsing the returned data is a waste of Erlang's pattern matching, so let's go back and do that again, this time with a nice pattern to capture the results. We'll need to do another request, though, so that something gets sent to the shell:
Now we can set up a pattern that will allow us to extract and print just the bits that we're looking for. The thing to keep in mind here is that the scope for the variables is within the receive call, so we'll need to display the values within that scope:
This should demonstrate the slight differences in usage and result patterns between the sync and async modes.

Well, that about sums it up for an intro to the HTTP client in LFE! But one last thing, for the sake of completeness. Once we're done, we can shut down inets:

Getting Started with LFE on Ubuntu

For those that don't know, there is a fully Erlang Core-compatible Lisp-2 that runs on the Erlang VM and produces .beam files that can be used in any Erlang application. This manna from heaven is LFE, or Lisp Flavored Erlang. It was started about 5 years ago by Robert Virding, one of the co-creators of the Erlang programming language, and has inspired other similar efforts: Elixir (a Ruby-alike) and Joxa (a Lisp-1). (Incidentally, Robert has also created Prolog and Lua implementations that run on top of the Erlang VM!)

The new LFE docs site (a continuous work in progress) has some good introductory materials for the curious reader:

• A Quick Start guide
• An as yet incomplete User Guide (most of the introductory chapters are finished)
• A tutorial on Erlang's Lightweight Processes in LFE.

This blog post aims to bring some of those hidden materials into the consciousness of Ubuntu users. If you are averse to Erlang syntax, LFE opens up a whole new world to you :-)

The examples below assume Ubuntu 12.10.


Getting Erlang

Erlang R15B01 comes with Ubuntu 12.10. If that's all you need, then this will suite you just fine:

$ sudo apt-get install erlang

If you are wanting to test against multiple versions of Erlang, you should check out the kerl project, which lets you install a wide variety of Erlang versions (including the latest releases) side-by-side.

You'll also need git, if you don't yet have it installed:

$ sudo apt-get install git

Currently, rebar is required to build all the LFE files. If you're going to be building LFE projects, you'll want this anyway ;-) Rebar will be in Ubuntu 13.04, but it's not in 12.10, so you'll need to get it:

$ wget https://github.com/rebar/rebar/wiki/rebar
$ chmod 755 rebar
$ sudo mv rebar /usr/local/bin

Getting and Building LFE

Here's what you need to do to build LFE:

$ mkdir -p ~/lab/erlang && cd ~/lab/erlang
$ git clone https://github.com/rvirding/lfe.git
$ cd lfe
$ make compile

If you looked at your ./ebin directory when you cloned the repo, you would have seen that there were no .beam files in it. After compiling, it is full of .beams ;-)

Sidebar: A common pattern in Erlang applications is the use of a deps directory under one's project dir where dependencies can be installed without conflicting with any system-wide installs, providing versioning independence, etc. Managing these with rebar has been very effective for projects, where simply calling rebar compile puts everything your app needs in ./deps. Projects that depend upon LFE are doing this, but we'll cover that in a future blog post.


Using LFE

With everything compiled, we can jump right in! Let's fire up the REPL, and do some arithmetic as a sanity check:

How about a message to stdout?

Any form starting with : is interpreted as a call to a module. The full form is (: <module name> <function name> <arguments>). As such, you can see that we're calling the format function in the (built-in) io module.

Also, it's good to know that there are certain things that you can't do in the REPL, e.g., defining modules, macros, functions, and records. Erlang expects that these sorts of activities take place in modules. However, we can explore a little more before we write our first module. Let's use the REPL's set form and lambda to define a function anyway (albeit, in a somewhat awkward fashion):

That wasn't too bad ;-) We're seeing the external module call, again -- this time to the math library. Now let's use a module of our own devising...


Creating Modules

In another terminal (but same working directory), let's create a module in a file called my-module.lfe, with the following content:


Note that the module name in the code needs to match the file name (minus the extension) that you used for the module.

Back in the REPL terminal window, let's compile this module and run the defined function:

Let's add another function to the module that demonstrates the benefits of Erlang's multiple-arity support:

Re-compiling and running the functions, we are greeted with success:

Lastly, let's convert the power function we defined in the previous section using our REPL-workaround to a "real" function, defined in our new module:

And then let's try it out:

Perfect.

(Of course, it's rather absurd to redefine pow to exp, when there is basically nothing to gain by it ;-) It's just a quick demo...)


Conclusion

There's lots more to learn; this has been just a small sip from a hot mug o' LFE.

However, it's definitely enough to get you started and, should you be interested in following along in future LFE blog posts, you'll have everything you need to get the most out of those.

The Lambda Calculus: A Quick Primer

The λ-Calculus Series

1. A Brief History
2. A Quick Primer for λ-Calculus
3. Reduction Explained
4. Church Numerals
5. Arithmetic
6. Logic
7. Pairs and Lists
8. Combinators

To the untrained eye, the notation used in λ-calculus can be a bit confusing. And by "untrained", I mean your average programmer. This is a travesty: reading the notation of λ-calculus should be as easy to do as recognizing that the following phrase demonstrates variable assignation:

x = 123

So how do we arrive at a state of familiarity and clarity from a starting state of confusion? Let's dive in with some examples, and take it step at a time :-) Once we've got our heads wrapped around Alonzo Church's notation, we'll be able to easily read it -- and thus convert it into code! (We will have lots of practice in the coming posts to do just that.)

A Quick Primer for λ-Calculus

Here's one of the simplest definitions in λ-calculus that you're going to see: the identity function:

λx.x

This reads as "Here is a function that takes x as an argument and returns x." Let's do some more:

λxy.x

"Here is a function that takes x and y as arguments and returns only x."

λx.λy.xy

"An outer function takes x as an argument and an inner function takes y as an argument, returning the x and the y." Note that this is exactly equivalent to the following (by convention):

λxy.xy

Let's up the ante with a function application:

λf.λx.f x

"Here is a function that takes a function f as its argument; the inner function takes x as its argument; return the result of the function f when given the argument x." For example, if we pass a function f which returns its input multiplied by 2, and we supplied a value for x as 6, then we would see an output of 12.

Let's take that a little further:

λf.λx.f (f (f x))

"Here is a function that takes a function f as its argument; the inner function takes x as its argument. Apply the function f to the argument x; take that result and apply f to it. Then do it a third time, returning that result." If we had the same function as the example above and passed the same value, our result this time would be 48 (i.e., 6 * 2 * 2 * 2).

That's most of what you need to read λ-calculus expressions. Next we'll take a peek into the murky waters of  λ-calculus reduction and find that it's quite drinkable, that we were just being fooled by the shadows.

The Lambda Calculus: A Brief History

Over this past weekend I took a lovely journey into the heart of the lambda calculus, and it was quite amazing. My explorations were made within the context of LFE. Needless to say, this was a romp of pure delight. In fact, it was so much fun and helped to clarify for me so many nooks and crannies of something that I had simply not explored very thoroughly in the past, that I had to share :-)

The work done over the past few days is on its way to becoming part of the documentation for LFE. However, this is also an excellent opportunity to share some clarity with a wider audience. As such, I will be writing a series of blog posts on λ-calculus from a very hands-on (almost practical!) perspective. There will be some overlap with the LFE documentation, but the medium is different and as such, the delivery will vary (sometimes considerably).

This series of posts will cover the following topics:

1. A Brief History
2. A Quick Primer for λ-Calculus
3. Reduction Explained
4. Church Numerals
5. Arithmetic
6. Logic
7. Pairs and Lists
8. Combinators

The point of these posts is not to expound upon that which has already been written about endlessly. Rather, the hope is to give a very clear demonstration of what the lambda calculus really is, and to do so with clear examples and concise prose. When the gentle reader is able see the lambda calculus in action, with lines of code that clearly show what is occurring, the mystery will disappear and an intuition for the subject matter will quite naturally begin to arise. This post is the first in the series; I hope you enjoy them as much as I did rediscovering λ-calculus :-)

Let us start at the beginning...

A Brief History

The roots of functional programming languages such as Lisp, ML, Erlang, Haskell and others, can be traced to the concept of recursion in general and λ-calculus in particular. In previous posts, I touched upon how we ended up with the lambda as a symbol for the anonymous function as well as how recursion came to be a going concern in modern mathematics and then computer science.

In both of those posts we saw Alonzo Church play a major role, but we didn't really spend time on what is quite probably considered his greatest contribution to computer science, if not mathematics itself: λ-calculus. Keep in mind that the Peano axioms made use of recursion, that Giuseppe Peano played a key role in Bertrand Russell’s development of the Principia, that Alonzo Church sought to make improvements on the Principia, and λ-calculus eventually arose from these efforts.

Invented in 1928, Alonzo didn't publish λ-calculus until 1932. When an inconsistency was discovered, he revised it in 1933 and republished. Furthermore, in this second paper, Church introduced a means of representing positive integers using lambda notation, now known as Church numerals. With Church and Turing both publishing papers on computability in 1936 (based respectively upon λ-calculus and the concept of Turing machines), they proposed solutions to the Entscheidungsproblem. Though Gödel preferred Turing's approach, Rosser suggested that they were equivalent definitions in 1939. A few years later, Kleene proposed the Church Thesis (1943) and then later formally demonstrated the equivalence between his teacher's and Turing's approaches giving the combination the name of the Church-Turing Thesis (1952, in his Introduction to Metamathematics). Within eight years, John McCarthy published his now-famous paper describing the work that he had started in 1958: "Recursive Functions of Symbolic Expressions and Their Computation by Machine". In this paper, McCarthy outlined his new programming language Lisp, citing Church's 77-page book  (1941, Calculi of Lambda Conversion), sending the world off in a whole new direction.

Since that time, there has been on-going research into λ-calculus. Indisputably, λ-calculus has had a tremendous impact on research into computability as well as the practical applications of programming languages. As programmers and software engineers, we feel its impact -- directly and indirectly -- on a regular, almost daily basis.

Maths and Programming: Whence Recursion?

As a manager in the software engineering industry, one of the things that I see on a regular basis is a general lack of knowledge from less experienced developers (not always "younger"!) with regard to the foundations of computing and the several related fields of mathematics. There is often a great deal of focus on what the hottest new thing is, or how the industry can be changed, or how we can innovate on the decades of profound research that has been done. All noble goals.

Notably, another trend I've recognized is that in a large group of devs, there are often a committed few who really know their field and its history. That is always so amazing to me and I have a great deal of admiration for the commitment and passion they have for their art. Let's have more of that :-)

As for myself, these days I have many fewer hours a week which I can dedicate to programming compared to what I had 10 years ago. This is not surprising, given my career path. However, what it has meant is that I have to be much more focused when I do get those precious few hours a night (and sometimes just a few per week!). I've managed this in an ad hoc manner by taking quick notes about fields of study that pique my curiosity. Over time, these get filtered and a few pop to the top that I really want to give more time.

One of the driving forces of this filtering process is my never-ending curiosity: "Why is it that way?" "How did this come to be?" "What is the history behind that convention?" I tend to keep these musings to myself, exploring them at my leisure, finding some answers, and then moving on to the next question (usually this takes several weeks!).

However, given the observations of the recent years, I thought it might be constructive to ponder aloud, as it were. To explore in a more public forum, to set an example that the vulnerability of curiosity and "not knowing" is quite okay, that even those of us with lots of time in the industry are constantly learning, constantly asking.

My latest curiosity has been around recursion: who first came up with it? How did it make it's way from abstract maths to programming languages? How did it enter the consciousness of so many software engineers (especially those who are at ease in functional programming)? It turns out that an answer to this is actually quite closely related to a previous post I wrote on the secret history of lambda. A short version goes something like this:

Giuseppe Peano wanted to establish a firm foundation for logic and maths in general. As part of this, he ended up creating consistent axioms around the hard-to-define natural numbers, counting, and arithmetic operations (which utilized recursion).  While visiting a conference in Europe, Bertrand Russell was deeply impressed by the dialectic talent of Peano and his unfailing clarity; he queried Peano as to his secret for success (Peano told him) and them asked for all of his published works. Russell proceeded to study these quite deeply. With this in his background, he eventually co-wrote the Principia Mathematica. Later, Alonzo Church (along with his grad students) sought to improve upon this, and in the process Alonzo Church ended up developing the lambda calculus. His student, John McCarthy, later created the first functional programming language, Lisp, utilizing concepts from the lambda calculus (recursion and function composition).

In the course of reading between 40-50 mathematics papers (including various histories) over the last week, I have learned far more than I had originally intended. So much so, in fact, that I'm currently working on a very fun recursion tutorial that not only covers the usual practical stuff, but steps the reader through programming implementations of the Peano axioms, arithmetic definitions, the Ackermann function, and parts of the lambda calculus.

I've got a few more blog post ideas cooking that dive into functions, their history and evolution. We'll see how those pan out. Even more exciting, though, was having found interesting papers discussing the evolution of functions and the birth of category theory from algebraic topology. This, needless to say, spawned a whole new trail of research, papers, and books... and I've got some great ideas for future blog posts/tutorials around this topic as well. (I've encountered category theory before, but watching it appear unsearched and unbidden in the midst of the other reading was quite delightful).

In closing, I enjoy reading not only the original papers (and correspondence between great thinkers of a previous era), but also the meanderings and rediscoveries of my peers. I've run across blog posts like this in the past, and they were quite enchanting. I hope that we continue to foster that in our industry, and that we see more examples of it in the future.

Keep on questing ;-)

Erlang Meetup at Rackspace, San Francisco

I'm very pleased to announce that Rackspace's San Francisco office will be hosting an Erlang Meetup immediately after the closing session of last day of SF Erlang Factory on 22 March.

We are honored to announce that both Robert Virding and Francesco Cesarini will be present for the event. The night will open up at 6:30pm with Mediterranean food, refreshments (and I hear there will be a keg). Around 7:15pm we will gather for a presentation by the very talented folks at Boundary which will start at 7:30pm.

The Boundary presentation will be followed by one from Robert Virding. We'll finish the night with a door prize give-away of two copies of Francesco and Simon's book Erlang Programming: A Concurrent Approach to Software Development. Perhaps the winners might be able to talk Francesco into signing their copies? ;-)

Last but not least, some of Rackspace's own Erlangers will be on hand for conversations about how we're using Erlang/OTP to deliver Fanatical Support in in our Data Center and Cloud Ops groups.

We're going to start with a cap of 60 people for the event, but if there's high demand, we have room to adjust this. Due to some communications hiccups, we're going to do the planning for this event the old-fashioned way: send an email to duncan.mcgreggor@rackspace.com, and let me know that you'd like to attend. Your name will be added to the list in the order I receive emails. The list of attendees and the waiting list are published here and will be refreshed with each new addition.

Update! The event is now published on meetup.com -- Thanks for all your help, Andra!

As special thanks go out to Robert Virding, Phil Toland, and Brian Troutwine -- conversations with them were the inspiration for putting this event together. Thanks guys! (Note that I've put that background info into its own blog post, so as not to confuse the event announcement ... to much ;-) ).

See you at Erlang Factory SF and then on Friday at the Rackspace office!

Lisp Flavored Erlang

I've flirted with Lisp since the 90s, really started getting into it around 2008 when I started playing with genetic programming, and more recently investigated Common Lisp, Scheme (Gambit and Chicken), and Clojure with the intent of discovering the best way to write distributed programs in Lisp. (I even seriously explored implementing chunks of Twisted in Common Lisp. If only I had more time...)

Needless to say, I kept coming back to Erlang as it is a natural choice for the sort of concurrent programming I was interested in doing. On several occasions, I'd run across Robert Virding's Lisp-2 that he had written on top of the Erlang VM. At first blush, this appeared quite promising. Yet faced with the perceived burden of learning Erlang, I consistently set it aside for a future date.

"Excuse me, could I have some Erlang, please? Yes, just a cup. Oh, and can I get that Lisp-flavored? Thanks so much."

After bouncing between Clojure and CL, after running into difficulties with Termite on Chicken Scheme, and finally, after being quite impressed with the efforts made by the Elixir folks (who I believe took inspiration from LFE!), I decided to give LFE a chance. Within minutes of that decision, I came to two conclusions:

1. LFE is brilliant.
2. LFE needs docs like Elixir... and tutorials... and exposure! Why haven't I been using LFE all along?!

At which point, I started hacking on some docs to see if I could stick with it. When, after a few days, I proved to myself that I could, I contacted Robert and let him know not only how much I adored his masterpiece, but that I really wanted to write tons and tons of docs for it so that anyone could pick it up and start using it right away. I wanted to write docs for an audience like me, that didn't know Erlang, who weren't Lisp gurus.

This seemed like a doable goal, since I had about 5 minutes' worth of Erlang experience at the time I was having these conversations with Robert. I was learning Erlang at a rapid pace simply by virtue of the Lisp hook that LFE provided.

Our interactions led to the publicizing of the new docs site for Lisp Flavored Erlang on the LFE google groups list. We also created a Twitter account (we both have full access to it, but I tend to maintain it) whose sole purpose is to bring LFE to more people, keep the news around LFE fresh, etc.

"I could have sworn you just said 'Lisp'..."

A side note about Lisp: S-expressions are concise and elegant. Code and data using the same form is quite powerful. I do believe that the technology industry has the capacity to realize that old biases against Lisp are just that: old and outdated. The many dialects of Lisp are anything but. Clojure and (I believe) LFE are perfect examples of this. Whole new generations of programmers are delighting in the expressive power of a language whose roots can be traced back to actual manipulations of memory registers.

To resume the narrative: in the course of various efforts focused on documenting LFE, asking questions on the mail list, and having various other discussions, Robert pointed out that some of my coworkers at Rackspace had been working on Erlang projects. I subsequently reached out to Phil Toland. Then, within minutes of this (and entirely coincidentally), Kai Janson emailed a group of us about Erlang Factory SF and his desire to provide Erlang workshops for engineers at Rackspace.

This led to further conversations with Robert, then with Francesco, with several Rackers signing up for Erlang Factory this year, and finally, with me volunteering to put a Meetup together afterwards, hosted at Rackspace's SF office (more on that in a few hours).

For the curious, I do continue to work in Python and Twisted; I am excited about the new async support that Guido is spearheading for Python 3 and which has electrified so many hard-core Python hackers. Similarly, I continue to hack on Common Lisp projects. However, I am quite delighted that I have found a way to interface with Erlang which matches how I think, matches my aesthics. And finally, I look forward to many fruitful years of LFE in my life :-)

Thanks Joe! Thanks Mike! Thanks Robert!

The Secret History of Lambda

Being a bit of an origins nut (I always want to know how something came to be or why it is a certain way), one of the things that always bothered me with regard to Lisp was that no one seemed to talking about the origin of lambda in the lambda calculus. I suppose if I wasn't lazy, I'd have gone to a library and spent some time looking it up. But since I was lazy, I used Wikipedia. Sadly, I never got what I wanted: no history of lambda. [1] Well, certainly some information about the history of the lambda calculus, but not the actual character or term in that context.

Why lambda? Why not gamma or delta? Or Siddham ṇḍha?

To my great relief, this question was finally answered when I was reading one of the best Lisp books I've ever read: Peter Norvig's Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp. I'll save my discussion of that book for later; right now I'm going to focus on the paragraph at location 821 of my Kindle edition of the book. [2]

The story goes something like this:

• Between 1910 and 1913, Alfred Whitehead and Bertrand Russell published three volumes of their Principia Mathematica, a work whose purpose was to derive all of mathematics from basic principles in logic. In these tomes, they cover two types of functions: the familiar descriptive functions (defined using relations), and then propositional functions. [3]
• Within the context of propositional functions, the authors make a typographical distinction between free variables and bound variables or functions that have an actual name: bound variables use circumflex notation, e.g. x̂(x+x). [4]
• Around 1928, Church (and then later, with his grad students Stephen Kleene and J. B. Rosser) started attempting to improve upon Russell and Whitehead regarding a foundation for logic. [5]
• Reportedly, Church stated that the use of x̂ in the Principia was for class abstractions, and he needed to distinguish that from function abstractions, so he used ⋀x [6] or ^x [7] for the latter.
• However, these proved to be awkward for different reasons, and an uppercase lambda was used: Λx. [8].
• More awkwardness followed, as this was too easily confused with other symbols (perhaps uppercase delta? logical and?). Therefore, he substituted the lowercase λ. [9]
• John McCarthy was a student of Alonzo Church and, as such, had inherited Church's notation for functions. When McCarthy invented Lisp in the late 1950s, he used the lambda notation for creating functions, though unlike Church, he spelled it out. [10] 

It seems that our beloved lambda [11], then, is an accident in typography more than anything else.

Somehow, this endears lambda to me even more ;-)

---

[1] As you can see from the rest of the footnotes, I've done some research since then and have found other references to this history of the lambda notation.

[2] Norvig, Peter (1991-10-15). Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp (Kindle Locations 821-829). Elsevier Science - A. Kindle Edition. The paragraph in question is quoted here:

The name lambda comes from the mathematician Alonzo Church’s notation for functions (Church 1941). Lisp usually prefers expressive names over terse Greek letters, but lambda is an exception. Abetter name would be make-function. Lambda derives from the notation in Russell and Whitehead’s Principia Mathematica, which used a caret over bound variables: x̂(x + x). Church wanted a one-dimensional string, so he moved the caret in front: ^x(x + x). The caret looked funny with nothing below it, so Church switched to the closest thing, an uppercase lambda, Λx(x + x). The Λ was easily confused with other symbols, so eventually the lowercase lambda was substituted: λx(x + x). John McCarthy was a student of Church’s at Princeton, so when McCarthy invented Lisp in 1958, he adopted the lambda notation. There were no Greek letters on the keypunches of that era, so McCarthy used (lambda (x) (+ x x)), and it has survived to this day.

[3] http://plato.stanford.edu/entries/pm-notation/#4

[4] Norvig, 1991, Location 821.

[5] History of Lambda-calculus and Combinatory Logic, page 7.

[6] Ibid.

[7] Norvig, 1991, Location 821.

[8] Ibid.

[9] Looking at Church's works online, he uses lambda notation in his 1932 paper A Set of Postulates for the Foundation of Logic. His preceding papers upon which the seminal 1932 is based On the Law of Excluded Middle (1928) and Alternatives to Zermelo's Assumption (1927), make no reference to lambda notation. As such, A Set of Postulates for the Foundation of Logic seems to be his first paper that references lambda.

[10] Norvig indicates that this is simply due to the limitations of the keypunches in the 1950s that did not have keys for Greek letters.

[11] Alex Martelli is not a fan of lambda in the context of Python, and though a good friend of Peter Norvig, I've heard Alex refer to lambda as an abomination :-) So, perhaps not beloved for everyone. In fact, Peter Norvig himself wrote (see above) that a better name would have been make-function.

Python for iOS

I do a lot of traveling, and I don't always like to lug my laptop around with me. Even when I do, I'd rather leave it in the bag unless I absolutely need to get it out (or if I'm setting up my mobile workspace). As such, I tend to use my iPhone for just about everything: reading, emails, calendar, etc.

So, imagine my delight, when I found out (just after PyCon this year) that I can now run Python 2.7.2 on my iPhone (and, when I get it, my iPad 3 ;-) ). This is just too cool for words... and given what pictures are worth, I'll use those instead :-)

I've put together a small Flickr set that highlights some of the functionality offered in this app, and each image in the set describes a nifty feature. For the image-challenged, here's a quick list:

• an interactive Python prompt for entering code directly using the iPhone keyboard
• a secondary, linear "keyboard" that one can use in conjunction with the main keyboard, extending one's ability to type faster
• multiple options for working with/preserving one's code (email, saving to a file, viewing command history)

I can't even begin to count the number of times such an awesome Python scratchpad would have come in handy. And now we have it :-) At $2.99, this is a total steal.

Thanks Jonathan Hosmer!

(And thanks to David Mertz for pointing it out to folks on a Python mail list.)

Canonical and Codethink at Bostom GNOME Summit

Today is the second day of the Boston GNOME Summit, and the second day of Canonical providing morning sustenance for the hackers here. Codethink and Canonical coordinated these efforts, with Codethink sponsoring food later today. It warms my heart that we can do this sort of thing.

Yesterday Cody Russell and I held a session about getting a gesture API into GTK 3.x. There were a great many questions about the uTouch framework, how we're handling multi-touch in the absence of MT support in X (coming in XInput 2.1), and what sort of dependencies would be needed (none! if GEIS is present on the system, gesture support will be added at build-time). At the end of the session, there was a consensus for Cody to present his plans to the GTK developers list and then to start getting branches reviewed for merge. We're hoping to make it for GTK 3.2.

In this vein, Cody and I have been hacking on libgrope for GTK compatibility, and this is serving as the sandbox for the GTK 3 gesture API development. My efforts have been focused on creating the GTK 2 Python C extension for grope. Given that the last time I coded C was in 1989 (and then a bit later in the mid-90s, when I had to hack a slackware driver to get ethernet working), this has been quite an effort. However, after a night and morning of hacking, I've got a handle on C extensions and am using the example code I wrote as the basis for pygrope. I've even managed to rope Barry Warsaw into reviewing the C extension code for us, to be sure we're not doing anything too crazy :-)

The Python C extension will be of immediate use to us in our test harness for gestures and exercising the stack. We will be creating a GTK app for recording user gestures for later playback and inclusion in test suites.

The State of Graphs in Python

There is a sad need for standardization of graphs in Python. The topic has come up numerous times on various mail lists, news groups, forums, etc. There is even a wiki page dedicated to the discussion of the topic on python.org. Ach, when will the madness end?

As far as I can tell, Guido van Rossum essentially solved this issue 10 years ago when he published his paper on Python Patterns - Implementing Graphs. The graph representation is a simple dict and he provided a few functions for demonstration purposes. In 2004, UC Irvine professor David Eppstein started making public his Python graph-theoretic efforts (with a functional programming approach). Both of these represent a direct approach that appeals to my aesthetic sense.

Now, after years of tracking the lack of progress made in standardizing graph representations in Python, I've recently had strong need of them. I did some checking around, and found projects that potentially met my needs. Sadly, none of them had the simplicity of Guido's original implementation (and therefore, anticipated speed benefits).

I was looking for graph implementations with no cruft, no external dependencies, no afterthoughts. I need something that balances runtime performance with a usable API, preferably created using PEP-8 (or similar) coding style.

Here's what I found, with some notes that I used to make a decision for my own needs:

• PADS - David Eppstein's work; functional programming style; very strong math; leaves the implementation of the graph up to the developer-user
  
• altgraph - too many utility and special-purpose methods for my taste; uses a custom graph object
• python-graph - a new implementation; uses its own objects; seems to take the "framework" approach to graph implementation
• graph - requires the use of custom vertex and edge objects
• NetworkX - fairly complete; lots of redundant code; covers more than just a graph implementation (I only include it here because it seems to be fairly highly used)
  

If you know me, then you've guessed what's coming next. Yes, I'm going to contribute to the general chaos and announce yet another graph library. What I hope to accomplish with this is provide a very simple implementation based on Guido van Rossum's approach (dictionary-based) that doesn't consume much memory, can be operated on quickly, and can be used anywhere.

In keeping with this motivation, I've started a new project on Launchpad and named it simple-graph. My initial efforts will be aimed at implementing a dict-based graph per Guido's paper, with the possible inclusion of some of David's functions (updated to operate on a dict object). I will then spend some time taking inspiration from the best of what the other graph libraries have to offer while keeping things simple.

As I stated on the web panel at PyCon 2007, diversity is a good thing; it gives us a rich gene pool from which a full and healthy process of natural selection may occur. Let's hope that the efforts of so many Python programmers eventually lead to the inclusion of a graph object in the Python standard library.

Synthesis Studios Twisted Interview

This is a re-blog, in case some of you aren't subscribed to http://labs.twistedmatrix.com/ -- we've got a new episode of the Twisted Show up, and it was a excellent interview with Synthesis Studios. Go check it out!

Thinking in Twisted

I've been emailing with a fellow developer and friend about learning to write applications and parts of applications using the Twisted networking framework. Even though Twisted is written in Python, conceptually -- and from an abstract point of view -- it is like a language of its own, with a foundation, syntax, and grammar that is Python.

My friend was trying to make use of basic Twisted stuff like deferreds, combine that with other parts of Twisted, but traverse the problem-solving process from an essentially ground-up, Python approach. I had tried to provide the shift in perspective needed in order to write Twisted apps with much less effort than he was exerting, but I fear that I failed.

Then, this morning, I thought of the perfect analogy. Actually, it's more than an ananlog -- it is the thing itself. Since really starting to learn Twisted, my programming has changed. Well, that and since absorbing such books as Patterns of Enterprise Application Architecture and Refactoring to Patterns. The deal with writing applications in Twisted is that you're no longer programming in the same language (Python); you're learning a new language. A more abstract one.

When we learn to think in Twisted and write applications in Twisted, our solutions will be elegant and compact. Before that starts to happen, we try to use our "old" ways of Python programming mixed in with the new one we are learning... and that, of course, defeats the purpose of the new "language."

The tricky thing about Twisted is that its patterns are an integral part of its "language" (perhaps it would be better to stop using quotes, and just call it a meta-language...). This is not true of most other languages -- though I would venture to say it is true of good frameworks in general. You don't need to know the patterns to use the grammar/syntax/control structures of most languages. Patterns in a language are usually optional.

With Twisted, if you aren't using the patterns, you aren't really using Twisted.

I am midway through this learning process. Therefore, I can't point to the solutions for many Twisted problems immediately or directly. I have, however, learned to see most of the critical components of a Twisted implementation. I still miss a few, though, and I have to keep reviewing code until I catch them all. Seeing these components clearly is what leads to the solution. As the time between "component vision" and "solution" gets shorter and shorter, we come closer and closer to thinking in Twisted.