Blog

We recently had to prototype a data application over a supercomputer tuned for graph analysis. We built a proof-of-concept leveraging multiple APIs, Cray’s Urika-GX and Graph Engine (CGE), and a handful of programming languages in less than a week.

Parallel coordinates is one way to visually compare many variables at once and to see the correlations between them. Each variable is given a vertical axis, and the axes are placed parallel to each other. A line representing a particular sample is drawn between the axes, indicating how the sample compares across the variables.

Often times, when we're looking at a mass of data, we're trying to get a sense for relationships within that data. Who is the leader in this social group? What is a common thread between different groups of people? Such relationships can be represented hundreds of ways graphically, but few are as powerful as the classic network diagram.

Nexflix has collected an impressive amount of data on Hollywood entertainment, made possible by tracking the viewing habits of its more-than 90 million members. In 2013, Netflix took an educated guess based on that data to stream its own original series, risking its reputation and finances in the process. When people were subscribing to Netflix to watch a trove of television series and movies created by well-established networks and studios, why create original content? Now, few would question the move.

There are data visualizations. There are web applications. If they had a baby, you'd get a data application.

Data applications are a big part of where our data-driven world is headed. They're how data science gets operationalized. They are how end-users - whether they're subject matter experts, business decision makers, or consumers - interact with data, big and small. We all use a data application when we book a flight, for instance. 

Bayesian inference is a statistical method used to update a prior belief based on new evidence, an extremely useful technique with innumerable applications. Uncertainty about probabilities that are hard to quantify is one of the challenges of Bayesian inference, but there is a solution that is exciting for its cross-disciplinary origins and the elegant chain of ideas of which it is composed.

Can we predict what the next hub of tech entrepreneurship will be? Could we pinpoint where the next real estate boom will be and invest there? Thanks to advances in machine learning and easier access to public data through Open Data initiatives, we can now explore these types of questions.

Continuing with our recent theme of abstraction in data applications, Dave King gave a talk last month explaining his design principles for "Making Code Sing: Finding the Right Abstractions." Nailing the best abstractions is a quintessential software challenge. We strive for generality, flexibility, and reuse, but we are often forced to compromise in order to get the details right for one particular use case. We end up with projects that we know have amazing potential for use in other applications but are too hardcoded to make repurposing easy. It’s frustrating to see the possibilities locked away, just out of reach.

Six months ago I didn't understand the concept of abstraction. Now it comes up almost daily. It’s foundational to my thinking on everything from software to entrepreneurship. I can’t believe how simple it seems. When I finally grokked abstraction, it felt like my first taste of basic economics. Given a new framework, something that had always been there, intuited but blurry, came into focus.