NASA/EOSDIS Earthdata

Update

It's been a few years since I posted on this blog -- most of the technical content I've been contributing to in the past couple years has been in the following:

• The LFE Blog
• The Clojang Blog

But since the publication of the Mastering matplotlib book, I've gotten more and more into satellite data. The book, it goes without saying, focused on Python for the analysis and interpretation of satellite data (in one of the many topics covered). After that I spent some time working with satellite and GIS data in general using Erlang and LFE. Ultimately though, I found that more and more projects were using the JVM for this sort of work, and in particular, I noted that Clojure had begun to show up in a surprising number of Github projects.

EOSDIS

Enter NASA's Earth Observing System Data and Information System (see also earthdata.nasa.gov and EOSDIS on Wikipedia), a key part of the agency's Earth Science Data Systems Program. It's essentially a concerted effort to bring together the mind-blowing amounts of earth-related data being collected throughout, around, and above the world so that scientists may easily access and correlate earth science data for their research.

Related NASA projects include the following:

• EOS
• ESDIS
• ESMO

The acronym menagerie can be bewildering, but digging into the various NASA projects is ultimately quite rewarding (greater insights, previously unknown resources, amazing research, etc.).

Clojure

Back to the Clojure reference I made above:  I've been contributing to the nasa/Common-Metadata-Repository open source project (hosted on Github) for a few months now, and it's been amazing to see how all this data from so many different sources gets added, indexed, updated, and generally made so much more available to any who want to work with it. The private sector always seems to be so far ahead of large projects in terms of tech and continuously improving updates to existing software, so its been pretty cool to see a large open source project in the NASA Github org make so many changes that find ways to keep helping their users do better research. More so that users are regularly delivered new features in a large, complex collection of libraries and services thanks in part to the benefits that come from using a functional programming language.

It may seem like nothing to you, but the fact that there are now directory pages for various data providers (e.g., GES_DISC, i.e., Goddard Earth Sciences Data and Information Services Center) makes a big difference for users of this data. The data provider pages now also offer easy access to collection links such as UARS Solar Ultraviolet Spectral Irradiance Monitor. Admittedly, the directory pages still take a while to load, but there are improvements on the way for page load times and other related tasks. If you're reading this a month after this post was written, there's a good chance it's already been fixed by now.

Summary

In summary, it's been a fun personal journey from looking at Landsat data for writing a book to working with open source projects that really help scientists to do their jobs better :-) And while I have enjoyed using the other programming languages to explore this problem space, Clojure in particular has been a delightfully powerful tool for delivering new features to the science community.

Hash Maps in LFE: Request for Comment

As you may have heard, hash maps are coming to Erlang in R17. We're all pretty excited about this. The LFE community (yes, we have one... hey, being headquartered on Gutland keeps us lean!) has been abuzz with excitement: do we get some new syntax for Erlang maps? Or just record-like macros?

That's still an open question. There's a good chance that if we find an elegant solution, we'll get some new syntax.

In an effort to (re)start this conversation and get us thinking about the possibilities, I've drawn together some examples from various Lisps. At the end of the post, we'll review some related data structures in LFE... as a point of contrast and possible guidance.

Note that I've tried to keep the code grouped in larger gists, not split up with prose wedged between them. This should make it easier to compare and contrast whole examples at a glance.

Before we dive into the Lisps, let's take a look at maps in Erlang:

Erlang Maps

Common Lisp Hash Tables

Racket Hash Tables

Clojure Hash Maps

Shen Property Lists

OpenLisp Hash Tables

LFE Property Lists

LFE orddicts

I summarized some very basic usability and aesthetic thoughts on the LFE mail list, but I'll restate them here:

• Erlang syntax really is quite powerful; I continue to be impressed.
• Clojure was by far the most enjoyable to work with... however, doing something similar in LFE would require quite a bit of additions for language or macro infrastructure. My concern here is that we'd end up with a Clojure clone rather than something distinctly Erlang-Lispy.
• Racket had the fullest and most useful set of hash functions (and best docs).
• Chicken Scheme was probably second.
• Common Lisp was probably (I hate to say it) the most awkward of the bunch). I'm hoping we can avoid pretty much everything the way it was done there :-/

One of the things that makes Clojure such a joy to work with is the unified aspect of core functions and how one uses these to manipulate data structures of different types. Most other implementations have functions/macros that are dedicated to working with just maps. While that's clean and definitely has a strong appeal, Clojure reflects a great deal of elegance.

That being said, I don't think today is the day to propose unifying features for LFE/Erlang data types ;-) (To be honest, though, it's certainly in the back of my mind... this is probably also true for many folks on the mail list.)

Given my positive experience with maps (hash tables) in Racket, and Robert's initial proposed functions like map-new, map-set, I'd encourage us to look to Racket for some inspiration:

• http://docs.racket-lang.org/reference/hashtables.html

Additional thoughts:

• "map" has a specific meaning in FPs (: lists map), and there's a little bit of cognitive dissonance for me when I look at map-*
• In my experience, applications generally don't have too many records; however, I've known apps with 100s and 1000s of instances of hash maps; as such, the idea of creating macros for each hash-map (e.g., my-map-get, my-map-set, ...) terrifies me a little. I don't believe this has been proposed, and I don't know enough about LFE's internals (much less, Erlang's) to be able to discuss this with any certainty.
• The thought did occur that we could put all the map functions in a module e.g., (: maps new ... ), etc. I haven't actually looked at the Erlang source and don't know how maps are implemented in R17 yet (nor how that functionality is presented to the developer). Obviously, once I have, this point will be more clear for me.

With this done, I then did a thought experiment in potential syntax additions for LFE. Below are the series of gists that demonstrate this.

Looking at this Erlang syntax:

My fingers want to do something like this in LFE:

That feels pretty natural, from the LFE perspective. However, it looks like it might require hacking on the tuple-parsing logic (or splitting that into two code paths: one for regular tuple-parsing, and the other for maps...?).

The above syntax also lends itself nicely to these:

The question that arises for me is "how would we do this when calling functions?" Perhaps one of these:

Then, for Joe's other example:

We'd have this for LFE:

Before we pattern match on this, let's look at Erlang pattern matching for tuples:

Compare this with pattern matching elements of a tuple in LFE:

With that in our minds, we turn to Joe's matching example against a specific map element:

And we could do the same in LFE like this:

I'm really uncertain about add-pair and update-pair, both the need for them and the names. Interested to hear from others who know how map is implemented in Erlang and the best way to work with that in LFE...

Physical Beings with Digital Lives

There's a lot that one could say about that title. In fact, it could be the title of a high-volume collaborative blog... That aside, here's the context for this post: books. Books and Reality. And data.
This post got so long that I now need to add a list of sections here, just to make it more accessible. My apologies :-/

Mini Table of Contents

• Books
• Books in the Sky
• Yeah, I Know: Go Social
• Human Data History
• Reality Merges
• Conclusion

Books

I have tons of books. Actual, physical books. Walls of them. Some I use all the time (reference). Some I read once a year (good books that support multiple reads). Others I've only read once, perhaps as far back as high school (when I started collecting). My bookshelves are like a random associative memory array: reading each title or the act of pulling one from the shelf brings back a flood of memories, relived experiences, sometimes actual sense perceptions. It's a powerfully visceral activity.

But that's just my books. When I'm at friends' homes or offices, I cannot keep my eyes off their bookshelves. It's an irresistible compulsion. I linger and browse, often past any semblance of socially acceptable time limits. My conversational replies experience a rapid exponential die-off -- in duration, gaps, and semantic value -- culminating in grunts and finally silence. (My favorite offices to visit so far? friend mathematicians/maths professors!)


Books in the Sky

Oddly, I love books in digital format. I never really got hung up on the bit about not having the paper entity in my hands (though I have turned a digital book reader over, expecting the next page... though that was deep in the plot of a Greg Egan novel!). Traveling as much as I do, I'm in heaven with ebooks. I feel like Superman, carrying around a library with me everywhere I go.

But when I passed a bookshelf the other day on my way out the door and fell under the spell of a book-memory flashback, I realized what was going away as I transitioned to virtual books. And the painful question arose: How am I going to nurture future layers of book-mulch and text-humus with this new æthereal, cloud-bound library I'm building?

How can I share with others, my stacks of books? How will I browse friends' books in their offices, asking about author X and title Y? How will we borrow from each other? What can be done to add this and related missing richness back into our lives once we adopt the virtual versions?

Light-emitting walls that can display titles from your Amazon account? Virtual over-lays visible with wearable/immersive computing accessories? Whatever we end up with, a gimmick isn't going to cut it. It will need to reflect the same depth of history that stacks of physical books have come to represent to us and the collective human psyche since we first started gathers works of the written word.


Yeah, I Know: Go Social

I'm hung up on books here, I admit it. But the same goes equally well for much of what we experience in online social media as well. Everyone's trying to make a buck on people chatting, playing games, reading, etc. Business as usual.

But the problem is that everyone coming up with their own little solution, one piece at a time. Google, Facebook, LinkedIn, Last.fm, etc. "We socialize X." Wow. Good for you. Now, for every activity or group I'm interested in, I've got some tiny little corner of the internet that I need to pay attention to.
Right.

Maybe I'm just an online social idiot, but this isn't working for me. Too many places and pieces. The physical analogy would be me spending all day on the road, hitting all the social hot spots in the Colorado Front Range. Ain't gonna happen. Ever.


Human Data History

The social data scene is a big stick up my butt. I really don't like it there and I'd love to get rid of it. It's poorly engineered, primitive state makes me grumpy. I don't own a thousand hammers [1]; I don't want a thousand of anything that all do basically the same thing [2]. Most of us probably don't own hundreds of houses, either (for ourselves, that is). We keep most of our stuff in the same location or two.

Speaking of houses, let's talk about settlement. How did we choose where to set up camp, towns, etc.? Trade routes, availability of resources (direct physical presence or presence by virtue of trade routes). Are we doing that now on the internet? Are we looking at the analog to fertile valleys, productive rivers, and protected harbors? Whose priorities do we have in mind? As we set up virtual presences, are we in locations that benefit businesses? Or ourselves?

If we choose the latter, the businesses will come because the people are there. If we do it the other way around, we'll be looking for new virtual homes if the businesses close shop or change the rules too much.

Coming back to data (but on the same anthropological note), historically we've had distinct divisions of our data:

• the secure location of our huts/houses/castles
• what we presented about ourselves in adornment/fashion (public data)
• what we could carry with us in bags/crates/vehicles

Because of their prevalence in our history, any attempt at realizing personal data in a virtual environment would do well to reflect on these. We're naturally already predisposed to such approaches; such divisions are things that anyone can intuitively grasp.

The problem is that we're all used to a single platform: our mutually agreed-upon reality. There's no such thing online yet. And if there were, who would own/run it? Monopolies are eventually overthrown. We hate them. So how do we get around this?


Reality Merges

This very naturally led to thoughts on digital lives in general. And this is more than just a question of usability or human-computer interaction. Rather, this is a question that borders on the metaphysical: how do we solve the problem of syncing divergent realities? Reality-reality interaction.

The problem shouldn't be minimalized by analogy: this isn't a "simple" matter of ensuring that the data in my address book on Google is the same as what's on my iPhone. My self-perception, many reminders of self-reflection, etc., take place as a result of various interactions I have with my surroundings: both real[3] and virtual. No problem. Except that the things that remind me are also things that others can see and interact with as well. Often, they will have associations that spark a neural cascade for them too.

I've had many conversations take place around objects in a shared environment where the name of the object was never mentioned, it was implicitly understood. When there's no shared object (or concept), we have to name the object, define it, share some basic associations, make sure that we're talking about the same thing, etc. Thats all prelude. Only with that done, can we have genuine communication take place around the given concept.

Now rinse and repeat for everything you want to talk about that revolves around or is at least related to something that exists virtually for you and isn't part of your shared, physical environment.
I can't imagine many useful general solutions to this. In fact, I can only imagine one (given our biological wiring): use what we know (in our bones) and overlay or augment our visual reality with another.

With augmented shared realities, there's no platform. You just need hardware that runs it and senses that can perceive it. Just like reality. At that point, we can start sharing what we want, allowing access to data about ourselves and what we like by dumping it into a shared perceptual space, regardless of the original data source. Merged.

Obviously, we're not there yet. We're going to need crazy improvements in mobile technology, storage, computer vision, etc. But once the technology catches up, I think we'll see some powerful needs being filled. And we might start coming out of the internet dark ages...


Conclusion

None of this is new; Pick up any number of books by Charlie Stross[4], and there's all sorts of fun to be had by exploring his ideas. But the point of this post wasn't to be new. While we're all busy enjoying the latest fad in social media, I think it's important we think about where the progressive succession of fads is taking us. At each point, there's a natural next step (more accurately, set of possible next steps). Let's look more than just one in front of us and let's not forget what our biology has made us. We may not be able to engineer truly wise decisions about our future, or even make our lives better/more efficient. But it would be nice if we could at least not make things worse :-)


Footnotes

[1] I think I have three, actually.
[2] This is one of the reasons I'm a big fan of http://ping.fm.
[3] Here I mean "real" in the "conventional" (shared) reality sense of Mahdyamikas. Ultimate reality... well, that's a topic for an entirely different sort of post...
[4] Check out his Amazon page: http://www.amazon.com/Charles-Stross/e/B001H6IW0Q/. Accelerando is probably his most praised book (and likely my favorite), but the ideas touched on in this blog post are explored in other works of his, most notably Glasshouse and Halting State.

The Future of Personal Data: A Followup

The Next Step

A few years ago, I wrote a post about the Future of Personal Data as a result of all the ultra large-scale systems reading and exploration I was doing. Google was foremost in my mind when writing that, but Apple has since come into the spotlight here as well.

Recently, Matt Zimmerman has decided to leave Canonical and join forces with Singly, an exciting startup company focused on secure user data storage and the socialization of (and development around) that data. In this blog post, the following core values were given about the software underlying Singly:

• I own my personal data
• I want my data to be useful to me
• I make the decisions to protect or share my

I would like to see the following added:

• I make the decisions on how my data is used
• If my data is sold, I should get a return for this

This is not petulance speaking :-) This comes from a historical perspective on social and economic fairness. Witness the changes in individual rights and personal finance since the industrial revolution...

Fair Market Value

There are probably many solvent entities out there who would claim users are already getting a return for the use of their data: free email and office docs from Google, free cloud services from Apple, etc. But I would imagine that there are massive margins being made on user data, and services (as valuable as they may be) are a paltry return for such a gold mine. I cannot help but be reminded of the selling price of Manhattan Island or Alaska (in 2011 values, the Lenape Indians got around $29.61/square mile; the Russians got about $150.77/square mile).

Money makes a good point, but personal data (and this post) isn't about the almighty coin. This is about clearly defining who owns what and ensuring that those who don't want to be taken advantage of, aren't. This is about identifying exploitation, and building something better and longer-lasting in its place.

Who's Going to Pay for What?

In David Pakman's "Disruption" blog post about Singly, he makes the following comment:

"I cannot see consumers getting into the business of selling their data to marketers so they can see personalized advertising. Instead, I believe marketers will be encouraged to offer value to us in exchange for access to our data."

I have to agree with him... but I can only offer a qualified agreement. True, I find it hard to believe that users will be selling their data directly to marketers. From the user-side, the pain of inconvenience would not likely be worth the payoff; at the marketer end, individual data is useless, and munging an in-house-built collection would incur a lot of overhead not part of their core business.

However, users' data stored in lockers, updated regularly, pre-processed, has enormous value in the market. Right now, Apple and Google are making eye-crossing amounts of money from data just like this. Again, I would imagine that this data is only really valuable in large quantities and for interesting, identifiable demographics

If users provide their data, but in exchange only get a "nicer app" or a "useful utility" I'm going to cry "foul!" (unless someone can show me the actual numbers involved and unequivocally prove that fair exchange is occurring).

You Say Disruption, I Say Revolution

Instead, if a service such as Singly, were to offer a co-op style dividend payment system to all of its users, that would seem to be much more fair. Not only that, it would be the beginning of a market revolution. This is not to say that co-ops are some perfect economic model, but rather that the data we, as users, generate is of immense value. The more that Singly has, the greater potential for revenue. The more buzz that builds around Singly users generating revenue from their data, the more users they get. With mass-adoption, a new sub-economy is born.

Perhaps a better model than co-op is that of a mutual fund investment firm. Each Singly user has a portfolio of data. Depending upon each user's preferences, any or all of that data could be used by Singly to generate revenue. Some groups of users will generate more than others, and users in these groups would get greater returns.

Whichever analogy you prefer, with a little exploration it seems fairly clear that opportunities for a large payout are present. For instance, I like to imagine a world where entities like Apple and Google can't harvest user data, but must go through brokers whom users have given their permission to sell their data for the most profit. I also imagine there's a lot of lawmaking that would have to take place... and even more lobbying.

Even with that, Google and Apple would still make money hand over fist (or they'd become brokers themselves) -- enough to continue providing free services. Yet at the same time, users would be in control of their data; they'd be financing (or financially augmenting) their data-consumption lives with said data.

With the right press coverage, Singly could find themselves not only swamped with a massively growing user base, but at the very center of a new economy. With the right level of negotiation and coordination, businesses could buy into this new paradigm without losing their shirts in the disruption.

A Plea

In summary, I applaud the goals and vision of Singly. I, for one, would deeply appreciate writing applications against their data locker (to any Facebook or any of its dubious applications), where a user's rights are clear and protected. That being said, Sinlgy would have my eternal allegiance if they also took up the cause of rights for user data in the market place; if they helped transform the current nascent data economy into a world economy capable of achieving as-yet unimagined financial heights.

And if not Singly, my loyalty would be given to whomever did do this.

The Future of Personal Data

In a recent post about ULS systems, I said this:

The balance of power, from individuals all the way to the top of whatever organizations exist in the future will rest in information. Not like it is today, however. The "information economy" of today (+/- 10 years) will look like kids' games and playgrounds. The information economies this will evolve into will be so completely integrated into human existence that they will resemble the basic necessities like water and food.

I'm not going to focus on the ULS systems topic in this post, but there is a very deep connection between privacy, personal data and all things ULS. Any thoughts of a ULS system should be coupled with how this will impact the system's users and their data. Any thought of our personal data's future existence should include the anticipated future of computing: ULS systems.


Inside and Out

In a nutshell, here's how things look:

• Yesterday: Paid Services - You want something, you buy it. Demographic research is expensive and mostly outsourced.
• Today: Free Services - You want something, companies give it to you for free... in exchange for your demographic data.
• Tomorrow: Information Economy - You want something, you leverage the value of your information in brokering the the service deals that mean the most to you.

What do we have right now? Companies are fighting for each other over who gets to have our data for free. Yay, free stuff! We used to have to pay for that sort of thing! But paying for people to hold your data was the old, old world. Having them do it for free is the old world. Here's the new world: They pay you.

Why would they do that? Why would things shift from the current status quo? The value of personal information.

There are many ways to assess the value of personal information, but let's look at a few from the perspective of large organizations (entailing everything from government to business). Simplistically, we can assign value to a single individual's data based on the value of a large collection of many individuals' data. The more participants, the greater the value of the whole, and therefore the greater the value for each individual's data. This perspective is limited because it treats data very staticly. The data may change, but in relation to the system it's "acquired" and inside as opposed to "for sale" and outside.


We Are the Markets

But the value of our data is not defined simply by the presense of bits or membership in a valued data conglomerate. Our data is not just our emails, our medical records, our purchasing trends, nor our opinions about local and national politics. Like an organism moving through an ecosystem, our data is dynamic and living; it is the very trace we leave in the world around us, be it digital or otherwise.

Any part of our lives that is ever recorded in "the system" provides data and comprises part of our movements through this system. Our traces through this digital ecosystem impact it, change it, shape its future direction. The collective behaviours (not just collective data) are immensely valuable to organizations. Their value is on-going and growing, with accrued, compounded interest.

Static data bits seem like property to us: you can buy them, you can sell them, you can store them somewhere. But moving, living data... that's a different story. That's not a buy-once commodity; ownership of that might be tantamount to slavery in a future, information-based economy. However, organizations might opt to lease it, or individuals might turn the past back on the future and offer license agreements to organizations.

More likely, though, individuals will form co-ops or communities (we have already seen this happend extensively in today's Internet) with shared mutual interest. Seeing how a group entity with shared values has a larger effect on the system than single individuals, data from such groups would likely be much more interesting and number-crunch-worthy. The greater power a group has to perterb systems' ecomonic or political trends, the more valuable that group's data will be to other groups.

In addition, I'm sure there'd be all sorts of tiered "offerings" from individuals and groups: the juicier/more detailed the data, the higher the premium offered. The changes this will introduce to markets (global and local), legal systems, and politcal organziations are probably barely imaginable right now. But what would it take to get us there? What would it take for my data and your data to be valuable enough to transform the world and make Wall Street look like an old-time, irrelevant boys club?


Privacy

One thing: a fanatical devotion to privacy, pure and simple. Security and a fanatical devotion to privacy. Two things! Okay, reliability, security and a fantaical devotion to privacy. Three things!

Monty Python references aside, an economy that values the data of individuals and groups can only arise if that data is secure. If we live in a topsy-turvy world where the Government, MPAA, RIAA, the Russian Mafia, and Big Hosting Company are pirating our data, then we're hosed. However, if our data is secure and contracts are effective, then we will have a world where data is the currency. There are an incredible number of hurdles to overcome in order for this to happen, however.

• The System - we need a system where user data can be tracked, recorded, and analyzed, and there's enough of it to matter
• Storage - we need our own, personal banks for our data (irrefutable ownership rights and complete power over that data)
• Transactions - we need a mechanism for engaging in secure, data transations
• Identity - when making a transation, we need to be able to prove unequivocally that we are who we say we are
• Anonymity - we need to decouple activity in the system and identity, thus requiring organizations to come to us (or our groups) to get the definitive data they need
• Recourse - we need a legal system and effective laws that protect the individuals and groups against the crimes of data-hungry organizations; fortunately, we will have had years of established precedent protecting the sellers from the buyers... oh my, how the tables turn!

And that's just off the top of my head. There's got to be tons of stuff which hasn't even occurred to me.


Closing Thoughts

Information will be as essential for us as water, yet there is a very interesting divergence from the example of a hydrological empire: each individual is the producer of some of that metaphorical water. By virtue of this difference, we hold the keys of the empire. We will be more a part of the economic and political powerbases than we have every been at any time in human history.

Of course, that means that we've got to get ready :-) This is already being done in many different ways. Everything from community housing cooperatives to small, co-op banks; from capabilities-based programming models to secure online transactions. Like the next 20 years of research needed for ULS systems to become a reality, we've got just as much work to do in order to guarantee our place in the economies of the future.