The Future of Programming - Adopting The Functional Paradigm?

Series Links

• An Overview
• Themes at OSCON 2014
• Adopting the Functional Paradigm?
• Retrospective on Paradigms
• The Rise of Polyglotism
• Preparing for the Future

Survivors' Breakfast

The previous post covered some thoughts on the future-looking programming themes present at OSCON 2014.

Following that wonderful conference, long-time Open Source advocate, Pythonista, and instructor Steve Holden, was kind enough to host his third annual "OSCON Survivors' Breakfast" with tens of esteemed attendees, speakers, and organizers enjoying great company and conversation, relaxing together after the flurry of conference activity, planning a leisurely day in Portland, and -- most immediately -- having some much-needed breakfast.

The view from the 23rd floor was quite an eyeful, and the conversation ranged across equally panoramic topics. Sitting with Alex Martelli, Anna Ravenscroft, and Katie Miller, the conversation inevitably turned to thoughts programmatical. One thread of the discussion was so compelling that it helped crystallize this series of blog posts. That was kicked off with Katie's question:

Why [have some large companies] not embraced functional programming to the extent that other large ones have?

Multiple points of discussion spawned from this, some of which still continue. The rest of this post explores these. 

Large Companies?

What constitutes a large company? We settled on discussing Fortune 500 companies, which, by definition are:

• U.S. Companies
• Ranked by gross revenue (after adjustments for excise taxes).

Afterwards, I looked up the 2013 top 25 tech companies in the Fortune 500. I've listed them below; in parentheses is the Fortune 500 ranking. After the dash are the functional programming languages used on various company projects -- these are listed only if I have talked to someone who has worked on a project (or interviewed for a job that used the language), or if I have read an article by an employee who has stated that they use the listed language(s) [1].

1. Apple (6) - Swift, Clojure, Scala
2. AT&T (11) - Haskell
3. HP (15) - F#, Scala
4. Verizon Communications (16) - Scala
5. IBM (20) - Scala
6. Microsoft (35) - F#, F*
7. Comcast (46) - Scala
8. Amazon (49) - Haskell, Scala, Erlang
9. Dell (51) - Erlang, Scala
10. Intel (54) - Haskell, SML, PLT Scheme
11. Google (55) - Haskell [2]
12. Cisco (60) - Scala
13. Ingram Micro (76) - ?
14. Oracle (80) - Scala
15. Avnet (117) - ?
16. Tech Data (119) - ?
17. Emerson Electric (123) - ?
18. Xerox (131) - Scala
19. EMC (133) - Scala
20. Arrow Electronics (141) - ?
21. Century Link (150) - ?
22. Computer Sciences Corp. (176) - ?
23. eBay (196) - Scala 
24. TI (218) - ?
25. Western Digital (222) - ?

The companies which have committed to projects guessed to be of significant business value written in FP languages include: Apple, HP, and eBay. Possibly also Oracle and Intel. So, a rough estimate of between 3 to 5 of the top 25 U.S. tech companies have made a significant investment in FP.

Why not Google?

The next two sections offer summaries of some views on this.

Ideal Use Case?

Is an FP language suitable for large organisations? Are smaller companies better served by them? During breakfast, It was postulated that dealing with such things as immutable data, handling I/O in pure FP languages, and creating/using higher order functions is easier for small startups due to the shorter amount of time required to hire or train a critical mass of skilled programmers.

It is certainly true that it will take larger organisations longer to train its personnel simply due to sheer numbers and, even with enough trainers, logistics. But this argument can be made for any corporate level of instruction; in my book, this cancels out on both sides and is not an argument unique to hard topics, even less, specifically pertinent to FP adoption.

Brain Fit?

I've heard this one a bit: "Some people just don't think in FP terms." They need loops and iteration, not higher order functions and recursion. Joel Spolsky makes reference to this in his article The Guerrilla Guide to Interviewing. In particular, he says that "For some reason most people seem to be born without the part of the brain that understands pointers." This has been applied to topics in FP as well as C.

To be fair, Joel's comment was probably made with a bit of lightness and not meant to be a statement on the nature of mind or a theory of cognition. The context of the article is a very practical one: hiring. When trying to identify whether a programmer would be an asset for your team, you're not living in the space of cognitive theory, rather you inhabit the realm of quick approximations, gut instincts, and fast, economical decisions.

Regardless, I find this perspective -- Type Physicalism [3] -- fairly objectionable. This is because I see it as a kind of intellectual "racism." Early social sciences utilized this form of reasoning to justify all sorts of discriminatory thinking in the name of "science", reinforcing a rigid mentality of "us" vs. "them." In my own experience, I've seen this sort of approach used to shutdown exploration, to enforce elitism, and dismiss ideas that threaten the authority of the status quo.

Rather than seeing the problem of comprehending FP as a physical limitation of the individual, I see instructional failure as the obstacle to overcome. If we start with the proposition that certain brains are deficient, we are essentially abandoning education. It is the responsibility of the instructor to engage creatively with each student's learning style. When adhering to the idea that certain brains are limited, one discards creative engagement; one doesn't even consider working with the students and their learning styles. This is a view that, however implicitly, can be used to shun diversity and dismiss potential.

I believe the essence of what Joel was shooting for can be approached in a much kinder fashion (adapted for an FP discussion):

None of us was born knowing GOTO statements, global state, mutable data, or for loops. There are many programmers alive, though, whose first contact with programming involved one or more of these. That's their "home town", as it were; their programmatic birth place. Having utilized -- as well as taught -- imperative, OOP, and functional styles of programming, I do not feel that one is intrinsically any harder than another. However, they are sometimes so vastly different from each other in style or syntax or semantics that once a student has solidified around the concepts of a particular paradigm, it can be a challenge retraining to work easily in another.

Why the Objections?

If both "ideal use case" and "brain fit" are given as arguments against adopting FP (or any other new paradigm) in large organisations, and neither are considered logically or philosophically valid, what's at the root of the resistance?

It is not uncommon for changes in an industry or field of study to be met with resistance. The bigger or more different the change from the status quo, very often is proportional to the amount of resistance. I suspect that this is really what we're seeing when companies take a stance against FP. There are very often valid business concerns: "we've made an investment in OOP" or "it will cost too much to train/hire/migrate to FP." 

I would remind those company leaders, though, that new sources of revenue, that product innovation and changes in market adoption do not often come from maintaining or enforcing the current state. Instead, that is an identifying characteristic of companies whose relevance is fading.

Even if your company has market dominance or is a monopoly, there is still a good incentive for exploring alternative paradigms. At the very least, one can uncover inefficiencies and apply new knowledge to remove duplication of efforts, increase margins, etc.

Careers

As a manager, I have found that about half of the senior engineers up for promotion have very little to no interest in taking on different (new to them) programmatic paradigms. They consider current burdens sufficient (or too much) and would rather spend what little free time they have available to them in improving existing systems.

Senior engineers who have a more academic or research bent (or are easily bored) are much more likely to embrace this sort of change. Interestingly, senior engineers who have little to no competitive drive will more readily pick up something new if the need arises. This may be due to such things as not perceiving accumulated knowledge as territory to defend, for example.

Younger engineers with less experience (and less of an investment made in a particular school of thought) are much more willing to take on new challenges. I believe there are many reasons for this, one of which may include an interest in becoming more professionally competitive with their peers.

Junior or senior, I have found that programmers who are currently looking to find new employment are nearly invariably not only willing to take on the challenge of learning different paradigms, but are usually going about that proactively and engaging in self-study.

I want to work with programmers who can take on any problem space in any paradigm and find creative solutions, contributing as valued members of a team. This is certainly an ideal set of characteristics, but one that I have seen in the wilds of the workplace on multiple occasions. It has nothing to do with FP or OOP paradigms, but rather with the people themselves.

Even if a company is locked into well-established processes and views on programming, they may find it in their best interests to provide a more open-minded approach with their employees who would enjoy that. Their retention rates could very well increase dramatically.

Do We Need To?

Philosophy and hiring strategies aside, do we -- as programmers, software projects, or organizations that support programming -- need to take on the burden of learning or adopting functional programming? Quite possibly not.

If Google's plans around Go involve building a new operating system (in the spirit of 1970s C and UNIX), the systems programmers may find pure functions too cumbersome to work with. FP may be too burdensome a fit for that type of work.

If one is not tied to a historical analogy with UNIX, as Mozilla is not with Rust, doing something like creating a new browser engine (or running a remote services company) may be a good fit for FP, especially if one has data showing reduced error counts when using type systems.

As we shall see illustrated in the next post, the usual advice continues to apply: the decision of which paradigm to employ for any given project should be dictated by the best fit and not ideological inflexibility. The bearing this has on programming is innovation: it is the early adopters who have the best chance of leading us into the future.

Up next: Retrospective on Programming Paradigms
Previously: Themes at OSCON 2014

Footnotes

[1] If anyone has additional information as to which FP languages are used by these top 25 companies, please let me know, and I will include that information. Bonus points for knowing of business-critical applications.

[2] Google Switzerland are using Haskell.

[3] Type Physicality is a form of reductive materialism, also known as the Mind-Brain Identity Theory that does not allow for mental states to be realized in organisms or computational systems that do not have a brain. See "Criticisms of Type Physicality" at http://en.wikipedia.org/wiki/Identity_theory_of_mind#Multiple_realizability.

The Future of Personal Data: A Followup

The Next Step

A few years ago, I wrote a post about the Future of Personal Data as a result of all the ultra large-scale systems reading and exploration I was doing. Google was foremost in my mind when writing that, but Apple has since come into the spotlight here as well.

Recently, Matt Zimmerman has decided to leave Canonical and join forces with Singly, an exciting startup company focused on secure user data storage and the socialization of (and development around) that data. In this blog post, the following core values were given about the software underlying Singly:

• I own my personal data
• I want my data to be useful to me
• I make the decisions to protect or share my

I would like to see the following added:

• I make the decisions on how my data is used
• If my data is sold, I should get a return for this

This is not petulance speaking :-) This comes from a historical perspective on social and economic fairness. Witness the changes in individual rights and personal finance since the industrial revolution...

Fair Market Value

There are probably many solvent entities out there who would claim users are already getting a return for the use of their data: free email and office docs from Google, free cloud services from Apple, etc. But I would imagine that there are massive margins being made on user data, and services (as valuable as they may be) are a paltry return for such a gold mine. I cannot help but be reminded of the selling price of Manhattan Island or Alaska (in 2011 values, the Lenape Indians got around $29.61/square mile; the Russians got about $150.77/square mile).

Money makes a good point, but personal data (and this post) isn't about the almighty coin. This is about clearly defining who owns what and ensuring that those who don't want to be taken advantage of, aren't. This is about identifying exploitation, and building something better and longer-lasting in its place.

Who's Going to Pay for What?

In David Pakman's "Disruption" blog post about Singly, he makes the following comment:

"I cannot see consumers getting into the business of selling their data to marketers so they can see personalized advertising. Instead, I believe marketers will be encouraged to offer value to us in exchange for access to our data."

I have to agree with him... but I can only offer a qualified agreement. True, I find it hard to believe that users will be selling their data directly to marketers. From the user-side, the pain of inconvenience would not likely be worth the payoff; at the marketer end, individual data is useless, and munging an in-house-built collection would incur a lot of overhead not part of their core business.

However, users' data stored in lockers, updated regularly, pre-processed, has enormous value in the market. Right now, Apple and Google are making eye-crossing amounts of money from data just like this. Again, I would imagine that this data is only really valuable in large quantities and for interesting, identifiable demographics

If users provide their data, but in exchange only get a "nicer app" or a "useful utility" I'm going to cry "foul!" (unless someone can show me the actual numbers involved and unequivocally prove that fair exchange is occurring).

You Say Disruption, I Say Revolution

Instead, if a service such as Singly, were to offer a co-op style dividend payment system to all of its users, that would seem to be much more fair. Not only that, it would be the beginning of a market revolution. This is not to say that co-ops are some perfect economic model, but rather that the data we, as users, generate is of immense value. The more that Singly has, the greater potential for revenue. The more buzz that builds around Singly users generating revenue from their data, the more users they get. With mass-adoption, a new sub-economy is born.

Perhaps a better model than co-op is that of a mutual fund investment firm. Each Singly user has a portfolio of data. Depending upon each user's preferences, any or all of that data could be used by Singly to generate revenue. Some groups of users will generate more than others, and users in these groups would get greater returns.

Whichever analogy you prefer, with a little exploration it seems fairly clear that opportunities for a large payout are present. For instance, I like to imagine a world where entities like Apple and Google can't harvest user data, but must go through brokers whom users have given their permission to sell their data for the most profit. I also imagine there's a lot of lawmaking that would have to take place... and even more lobbying.

Even with that, Google and Apple would still make money hand over fist (or they'd become brokers themselves) -- enough to continue providing free services. Yet at the same time, users would be in control of their data; they'd be financing (or financially augmenting) their data-consumption lives with said data.

With the right press coverage, Singly could find themselves not only swamped with a massively growing user base, but at the very center of a new economy. With the right level of negotiation and coordination, businesses could buy into this new paradigm without losing their shirts in the disruption.

A Plea

In summary, I applaud the goals and vision of Singly. I, for one, would deeply appreciate writing applications against their data locker (to any Facebook or any of its dubious applications), where a user's rights are clear and protected. That being said, Sinlgy would have my eternal allegiance if they also took up the cause of rights for user data in the market place; if they helped transform the current nascent data economy into a world economy capable of achieving as-yet unimagined financial heights.

And if not Singly, my loyalty would be given to whomever did do this.