Interesting links - May 2026

🔥 Apache Kafka 4.3.0 has been released. Check out the release announcement, as well as a video from Sandon Jacobs covering the new features.

🔥 After a few quiet months on his blog, Jack Vanlightly is back with a bang! He’s written a new tool, Dimster, a performance benchmarking tool for Apache Kafka, and has written several more blog posts off the back of it:

🔥 I had the absolute pleasure to watch Victor Rentea present at Devoxx UK earlier this month. This guy redefines what it means to be an entertaining, energetic, enthusiastic—and educational presenter. Whilst his specific talk, "Event-Driven Architecture Pitfalls" isn’t online yet, you can find the slides here, and a recording from Devoxx last year of a similar talk.

The Parallel Consumer library from Confluent has been marked as no longer maintained, prompting a discussion of alternatives (and the concept itself) on LinkedIn, as well as a fork from one of the original authors, Tony Stubbs.

Mariano Gonzalez - Benchmarking KPipe against the parallel-Kafka libraries you would actually pick.

Michel Tricot - Event-Driven vs. Polling Architectures for Agent Triggers.

An interesting idea from Florent Ramiere and colleagues: what if you specify a set of interesting additions to Kafka’s functionality, with strict rules around the implementation, and then have LLMs take their best shot at it? You can see the ideas and results in the branches of this repository.

Viquar Khan - Architecting Cloud-Native Kafka: From Tiered Storage Towards a Diskless Future.

Elad Eldor - Kafka’s Real Compression Problem Is Batch Depth, and Kafka Compute Is Cheap. Network Is Not.

Kroxylicious version 0.21.0 has been released, and Sam Barker from the Kroxylicious project has been running some benchmarks to look at the impact that the proxy has, both pass-through and when encrypting records.

Aiven’s Juha Mynttinen explores why they think Apache Kafka Deserves Topic Types.

Details of a Coinbase outage involving their Kafka provider, which based on blogs from 2022 and 2023 is MSK.

Andy Muir - Kafka Schema Registry doesn’t guarantee compatibility (and what actually does).

Bruno Cadonna - OpenData Buffer: HA pipelines without Kafka.

Jeffrey J. Jennings - Kafka’s quiet observability superpower - Kafka Interceptors.

Grzegorz Kocur - Do Kafka metrics have to be so difficult?

Flink’s Stateful Functions (StateFun) is not maintained by the project any more, so kzmlabs' Oleksandr Kazimirov forked it to continue developing it.

Olena Vodzianova - How Chandy-Lamport Inspired Apache Flink Checkpointing.

🔥 Two good posts from the team at Grab:

Details of how Smartsheet use Flink for optimising both costs and performance by filtering messages.

flink-state-explorer is, as the name suggests, a tool for exploring Apache Flink 1.20 canonical savepoints interactively.

A hands-on github repo from Patrick Neff showing off Stream processing pipeline using dbt and Flink on Confluent Cloud.

Shuva Jyoti Kar - Designing stateful serverless Agentic Loop with Kafka and Flink.

A couple of security issues for Flink to be aware of if you’re running it:

🔥 Tristan Handy - BI’s Second Unbundling.

A good writeup from Cloudflare’s James Morrison and Christian Endres about tracing performance issues in ClickHouse.

Several posts from StarRocks covering new features in 4.1:

Two BigQuery optimisation/cost saving articles, from Christophe Oudar and Azeem Jalageri.

Daniel Beach - Spark is Dead. Long Live DuckDB.

Alibaba added DuckDB into their fork of MySQL, AliSQL, providing storage and query for OLAP workloads.

Simon Aubury - I don’t need an untrusted LLM to tell me I’m spending too much on coffee.

The DuckDB team announced Quack: The DuckDB Client-Server Protocol.

Ben Fleis explores DuckDB’s support for Delta and Unity Catalog.

🔥 I’ve been a fan of Mark Litwintschik’s no-nonsense blog posts showing current technologies and exploring interesting data sets for many years. In this one he uses DuckDB to analyse details of 10K+ Satellites in Space.

🔥 Nikola Ilic - Data Modeling for Analytics Engineers: The Complete Primer.

AirTable’s Matthew Jin details how they optimised their costs by moving PBs of cold data from MySQL to S3, and wrote a query engine using Data Fusion to serve it.

Brian Brunner and his colleagues at Cloudflare published details of how they built Cloudflare’s data platform and an AI agent on top of it.

Caesario Kisty - A Practical Implementation of Medallion Architecture Using ClickHouse.

Xinran Waibel - Data Engineering Open Forum 2026 Recap.

🔥 After doing a bit of fairly naïve experimentation with Claude and dbt earlier this year, I was very interested to read Jason Ganz’s article What data agent benchmarks do and don’t tell us, and hope to try out the referenced ADE-bench ("a framework for evaluating AI agents on data analyst tasks") soon.

Whilst Thijs Nieuwdorp’s article about Handling Schema Issues in Polars is specific to Polars, it’s a useful reference for the kind of schema changes one will want to make in a data pipeline, and the challenges it can cause depending on how or if your implementation technology of choice supports it.

🔥 Pedram Navid - We need to talk about dbt.

A summary/re-write by Alex Yu (a.k.a. ByteByteGo) looking at How Figma Upgraded Data Pipeline from Multi-Day Latency to Real-Time (based on the original blog post by Yichao Zhao from last year).

Netflix - The Evolution of Cassandra Data Movement at Netflix.

Alexey Makhotkin has a two parter on Data Quality part 1 / part 2.

Chris Hillman - Don’t Go Dark: Visibility Is a Data Engineering Skill.

After the excellent survey and results that Joe Reis published about data engineering earlier this year, he’s now following up with a survey on The Organizational State of Data Engineering (open for submissions until Sunday, June 21).

Mahendran Vasagam - From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines.

Dana Rabba - Building Self-Healing Data Pipelines at Halodoc.

rocky is a dbt alternative that looks quite interesting. It describes itself as "the trust plane for your warehouse", and targeting Databricks users primarily, with Snowflake and BigQuery to follow. There’s a built-in playground feature that’s worth poking around to get a feel for it.

Apache Iceberg 1.11 has been released (I even got some small contributions merged 🎉). There are more details of the release in these blog posts:

🔥 The talks from Iceberg Summit 2026 are now online.

Alex Merced begins an epic 15-part series about Apache Iceberg by looking at What Are Table Formats and Why Were They Needed?

Yelp’s Nick Del Nano looks at How Partition Access Visualizations Reduced Data Lake S3 Cost by 33%.

Honest words from Fresha’s Samuel Valente as he looks at the use of Iceberg with Snowflake in practice: Snowflake with Iceberg: Lakekeeper, dbt, and some Sparks Flying.

Daniel Guzman-Burgos describes bintrail which provides time-travel SQL for MySQL. Renato Losio has a summary on InfoQ.

Teiva Harsanyi - How Linux 7.0 Broke PostgreSQL: The Preemption Regression Explained.

Radim Marek covers the ORDER BY jungle, as well as PostgreSQL’s TOAST.

Markus Winand also looks at ORDER BY and the evolution of support in different RDBMS.

🔥 James Blackwood-Sewell writes up details of the benchmarking platform they built, whilst Ben Dicken muses on benchmarking at PlanetScale too.

An opinionated, and fairly concise, set of recommendations for the use of different Open Standards for Modern Data Architecture.

LinkedIn’s Pratikmohan Srivastav writes about a performance troubleshooting experience - The 58-Million-Key Freeze: What a HashMap Resize Taught Us About Memory Allocation at Scale.

Sem Sinchenko - Same buffers, same instructions, same hardware. Where Is the JVM Tax?

Gergely Orosz shares some excerpts from Martin Kleppmann’s second edition of Designing Data Intensive Applications.

🔥 Ben Evans - AI Eats the World.

🔥 TikTok is my guilty pleasure, but instead of dogs misbehaving in comical ways, here’s an excellent piece to camera from Scott Hanselman reflecting on the impact of AI in our lives as software developers.

Nate Berkopec - Thoughts on LLMs in 2026.

Julien Hurault - Time for AI Coding to Turn Boring?

Kate Holterhoff - AI Slop & the Vulnerability Treadmill.

Paulo Arruda - What I Learned Building Multi-Agent Systems from Scratch at Shopify.

🔥 Loris Cro - Contributor Poker and Zig’s AI Ban.

Lucia Cerchie - Why You Need More Than a SKILL.md.