Data Engineering in 2022: Storage and Access

In the beginning was the word, and the word was an expensive relational datawarehouse that wasn’t flexible or scalable enough for the cool kids in Silicon Valley.

Then came Hadoop and scalability was won, at the vast cost of usability. You literally had to write your own Java code to move data around and transform it. You needed to serialise and deserialise the data yourself, and could store whatever you wanted - it didn’t have to be structured. This was sold as a benefit — "Schema on Read" they said, "It’ll be a good idea", they said. "Oh bugger, where’s my schema", they said when they came to use it.

Through the virtues of open source a fantastic ecosystem grew particularly around the Apache Software Foundation and we got such wonderfully named projects as Sqoop, Oozie, Pig, and Flume emerged. Hive brought with it the familiar and comforting bosom of SQL and table structures but with limited functionality (including no DELETE or UPDATE at first) and performance.

Over the years things improved, with Spark replacing MapReduce and enabling a generation of Python coders to get into the big data lark too, along with SQL.

Amongst all of this pioneering work and technology was the assumption that the resting place for analytical data was HDFS. Other stores like HBase existed for special purposes, but the general we’ve-got-a-bunch-of-data-in-this-org-and-need-to-collect-it destination was HDFS. Because "general dumping ground" wasn’t sexy enough for the marketing folk it became sold as a "Data Lake" with all the associated puns and awful cliches (fishing for data, data swamp, etc etc etc).

The general pitch around the data lake was to collect all the data, structured and unstructured (or structured that you’ve made unstructured by chucking away its schema when you loaded it), and then wait for the data lake fairy to conjure magical value out of the pile of data you’ve dumped there make the raw data available for teams in the company to process and use for their particular purposes. This may have been direct querying of the data in place, or processing it and landing it in another data store for serving (for example, aggregated and structured for optimal query access in an RDBMS or columnar data store).

Accessing the data in HDFS was done with Hive and other tools including Impala, Drill, and Presto. All had their pros and cons particularly in early releases, often with limitations around performance and management of the data.

All of this was built around the closely-intertwined Hadoop platform, whether self-managed on-premises including with deployments from Cloudera, MapR, and Hortonworks, or with a cloud provider such as AWS and its EMR service.

This was pretty much the state of the Big Data world (as it was called then) as I last saw it before getting distracted by the world of stream processing - fragmented, more difficult to use, less functionality than an RDBMS for analytics, and evolving rapidly.

Coming back to this after my attention being elsewhere for a few years means that I have the slightly uninformed but helpfully simplistic view of things. What the relational data warehouses used to do (bar scale, arguably), we are now approaching something roughly like parity again with a stack of tools that have stabilised and matured in large, with a layer on top that’s still being figured out.

Underpinning it all is the core idea of separation of storage and compute. Instead of one box (the traditional RDBMS), we have two. This is important for two vital reasons:

HDFS and Hive gave us this separation, right? Well, it did, but with a long list of problems and limitations for Hive. These include poor perfomance, a lack of support for transactions, point-in-time querying, streaming updates, and more. In addition, HDFS has nowadays been overtaken by S3 as the object store of choice with APIs supported by numerous non-S3 platforms, both Cloud based (e.g. Google’s Cloud Storage (GCS), and Cloudflare’s R2) and on-premises (e.g. Minio).

So if it’s not HDFS and Hive, what’s the current state and future of analytics data storage & access looking like?

Leaving aside table formats and lakehouses for the moment—and coming back to the big picture of how we store and access data nowadays—one idea that’s caught my attention is that of being able to apply git-like semantics to the data itself. Here’s a copy of a recent Twitter thread that I wrote.

Having watched @gwenshap and @ozkatz100 talk about "git for data" I would definitely say is a serious idea. However to the point at the end of the video, RTFM—it took reading page from the docs and some other pages subsequently to really grok the concept in practice.

Where I struggled at first with the git analogy alone was that data changes, and I couldn’t see how branch/merge fitted into that outside of the idea of branching for throwaway testing alone. The 1PB accidental deletion example was useful for illustrating the latter point for sure.

But then reading this page made me realise that I was thinking about the whole thing from a streaming PoV—when actually the idea of running a batch against a branch with a hook to validate and then merge is a freakin awesome idea

(As the roadmap issue notes, doing this for streaming data is conceptually possible but more complex to implement.)

I’m also still trying to think through the implications of merging one branch into another in which there are changes; can data really be treated the same as code in that sense, or could one end up with inconsistent data sets?

Lastly, having been reading up on table formats, I’d be interested to dig into quite how much LakeFS works already with them vs roadmap alone (the docs are not entirely consistent on this point)—but with both in place it sounds like a fantastic place for data eng to be heading.

The "git for data" pitch is a great way to articulate things, but it also shifted my brain off some of the central uses. For code, git is an integral part of the development process but once it hits Production git steps back from an active role. However, in the case of LakeFS some of their most exciting use cases are as part of the Production data process. The docs have several examples which I think are just great:

I kicked the tyres on LakeFS and wrote about it here