As 2026 hurtles towards the holiday season and then on into 2027, I took a moment to canvass opinion from some of y’all as to the shape of this newsletter.
For the last year I’ve included a section on AI, which has grown from a passing note through to a significant proportion of the newsletter (a third of it at times). So yesterday I asked on both Substack and LinkedIn whether I should split this out into its own post. I figured that some people might want only AI…and some people no AI.
The vote was relatively evenly split, but came down in favour of keeping it as is. So, that is what I shall do :)
As always, I’m very open to feedback on the newsletter - good and bad. Drop me an email, or use the comments section below.
Next week I’m looking forward to returning to San Francisco for Current 2026. If you’ll be there, come and say hi :) I’ll be at the unofficial run on Wednesday morning, and taking part in the Streamhouse AMA at midday in the meetup hub too.
|
Kafka and Event Streaming 🔗
-
🔥 Apache Kafka 4.4 has been released - Omnia Ibrahim has the release announcement and Sandon Jacobs a nice summary of the changes and new features. Check out the release notes for all the details.
-
Joshua Oluikpe (PolyAI) - Building a Session-Ordered Kafka Pipeline in Go.
-
Pravesh Gupta and Pranav Mutharia from LinkedIn discuss Scaling, Migrating, and Hardening a High-Volume Event Pipeline.
-
Yaroslav Tkachenko - Improving Cost Efficiency of Data Streaming Pipelines.
-
A couple of neat tools from Addison Huddy - a replacement for one of my favourite tools ever,
kafkacatkcat, a lightweight CLI standalone binary called kite, billed as an ultra-lightweight Kafka CLI for agents and shell pipelines. Complementing it is wing, a Schema Registry CLI. -
Geoff Williams - Streaming IoT sensor data from MQTT to Kafka
-
🔥 Confluent have released a new Rust Client for Apache Kafka.
-
Raynor Elgie - Kafgres: Embedding a Kafka Broker into Postgres & 700 MB/s of Kafka throughput, on Postgres.
-
What started out as a diskless Kafka project from Peter Morgan called Tansu, then renamed to Nisshi, has now been acquired by Solace.
-
Mayank (JioHotstar) - The Kafka Optimization That Saved Us $350K a Year.
Stream Processing 🔗
-
Katya Gorshkova - Hands-on with Flink — Part 10: Joining Streams with Reference Data. (Previous parts: 9, 8, 7, 6, 5, 4, 3, 2, 1).
-
David Anderson - Deduplicating Streams with Flink SQL
-
Aravind Patnam (LinkedIn) - Scaling Spark and Flink Shuffle with Apache Celeborn.
-
Dale Lane - How Flink’s Kafka connector commits offsets to Kafka.
-
🔥 Qianrui Zhang and team at Pinterest Engineering - Partition Finalization in Pinterest’s Next-Generation DB Ingestion Framework.
-
George Zefkilis - Query the Stream in Practice: A Local Streamhouse with Fluss, Flink and Iceberg.
-
Yaroslav Tkachenko has a nice writeup from the Data Streaming Summit 2026.
It’s a Kafka Streams Special! 🎉 🔗
-
🔥 Republished: 15 Kafka Streams deep dives from the Responsive archive.
-
Almog Gavra - Don’t Panic: The Definitive Guide to Kafka Streams State.
-
Rohan Desai - A Size for Every Stream: The Expert’s Guide to Sizing Kafka Streams.
-
Sophie Blee-Goldman - Defense Against the Dark Arts of Rebalancing in Kafka Streams.
-
Sophie Blee-Goldman - Kafka Streams and Rebalancing Through the Ages.
-
Apurva Mehta - Kafka Streams Sightings in the Wild.
-
Almog Gavra - You Will Survive Monitoring Kafka Streams.
-
Apurva Mehta, Sophie Blee-Goldman - Enter the Matrix: A practical guide to exactly-once semantics in Kafka Streams.
-
Almog Gavra - So You Want to Write a Stream Processor? Beware of the Duck Syndrome..
-
Sophie Blee-Goldman - Kafka Streams Wrapped: 2024 in review.
-
Almog Gavra, Sophie Blee-Goldman - Kafka Streams 101: Windows & Time.
-
Sophie Blee-Goldman - Kafka Streams 101: Topology Upgrades.
-
Almog Gavra - Kafka Streams 101: The Most Important Configs.
-
Almog Gavra - Kafka Streams 101: Application Lifecycle.
-
Sophie Blee-Goldman - Kafka Streams 101: Optimizing your apps for maximum performance.
-
Rohan Desai - Kafka Streams 101: Memory Management.
-
Analytics 🔗
-
🔥 DuckDB 2.0.0 has been released - check out the release blog post plus follow-up articles covering new features such as Agent Mode in the DuckDB CLI (this one’s cool), and other Gems of DuckDB 2.0.0.
-
Benjamin Wan Rothschild (Meta) - Scaling AI-Native Analytics Patterns at Meta.
-
DuckLabs was acquired by AWS earlier this year, and the fruits of that for AWS are already beginning to show: Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data in your data lake (using embedded DuckDB).
-
Gabriel Musat Mestre (Datadog) - How we extended Apache DataFusion to execute one query across many machines.
Data Platforms & Architectures 🔗
-
🔥 Joe Reis - The Pendulums of Data.
-
Couple of good interviews from Gergely Orosz this month:
-
Rafał Gancarz at InfoQ is curating a series of articles about Agentic AI Architecture.
-
🔥 Julien Millau (Michelin) - From Real-Time Ingestion to Cold Analytics: Designing for Data Usage.
-
Giannis Polyzos & Anton Borisov discuss the Streamhouse from an Apache Fluss perspective.
-
Hou Zhong (JD.com) - How JD.com Built a Lakestream with Apache Fluss and Apache Hudi.
-
🔥 Jon Lee and team at OpenAI - Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part 2). (see also Part 1)
Data Engineering and Pipelines 🔗
-
SeattleDataGuy - Are Data Teams Solving the Right Problems?.
-
🔥 Travis Thompson has some thoughts on how data-related job shapes and titles might evolve as the world shifts towards its adoption of AI.
-
Simon Späti interviews Josh Wills.
-
Pushkar Kurhekar (Whoop) - Scaling an ML Inference Pipeline for Batch Workloads.
-
Pengyu Hou and team at Airbnb - Extending Airbnb’s sequence recommender for real-time with Chronon. (Chronon - Open-Source Feature Platform for Machine Learning).
-
Paarth Chothani, Chirag Agrawal, Amrith M - Taming the ML Firehose: Scaling Feature Consistency.
-
Lukas Sénicourt - Renart - Open source data platform for SQL and Python pipelines, notebooks and dashboards, with type checking, scheduling and definitions in Git..
-
🔥 Chris Gambill has a good strategy if you’re trying to break into Data Engineering: Count Before You Study.
CDC 🔗
-
Vu Trinh has a good introduction to Change Data Capture (CDC).
-
Kodukulla Mohnish Mythreya - Introducing PyDebeziumAI: Real-Time CDC for LangChain and LangGraph.
-
🔥 Apache Flink CDC 3.7.0 has been released.
-
Jack Leitch - CDC at WHOOP: Self-Service Replication for Hundreds of Postgres Tables.
Data Modelling 🔗
I’m broadening this category to include articles about context layers and semantic layers.
-
🔥 Joe Reis - The Five Camps of Data Modeling (and Which One You’re Stuck In).
-
Olek Gorajek, Samuel Redai, and Yian Shang from Netflix write about DataJunction for providing a semantic layer.
-
Tushar Gupta (Zalando) - Automating schema mapping for diverse supplier data using a RAG Flywheel and LLMs.
-
🔥 Jessica Talisman - Ontologies, Context Graphs, and Semantic Layers: What AI Actually Needs in 2026.
-
Sanjeev Mohan - FAQ on Metadata, Semantics, Taxonomy, Ontology, Knowledge Graphs, and Context.
-
Michele Ceccacci and colleagues at Pinterest - Metrics Board: Building an Agent-ready Metrics Layer.
-
Joe Reis - Fan-Out: Why Your Dashboard Counts Customers Twice.
-
As many software companies look at their offering and try and work out how it fits in the hawt new AI landscape, both Neo4j and Elastic are making plausible plays for being a Context Engine.
-
Ilho Ye, Wei Li, and Jianlong Zhong (LinkedIn) - Inside LinkedIn’s data context layer: curating trustworthy context for humans and agents.
Open Table Formats (OTF), Catalogs, Lakehouses etc. 🔗
-
🔥 Joshua Buss has written zpq, a very neat tool for working with Parquet files. Read more about it in this blog post.
-
A Lance explainer from Vu Trinh.
-
Apache Iceberg 1.12 has been released. Alex Merced has a recap of what’s changed.
RDBMS 🔗
-
🔥 Simeon Griggs - What was the turning point for Postgres?
-
Manish R Jain - Egres: A Postgres-compatible database for heavy, multi-tenant workloads.
-
Thomas Neumann - The Benchmarking Crisis.
-
Zuofei Wang (Airbnb) - Beyond synthetic testing: Capturing and replaying real database workloads.
-
🎥 Andy Pavlo interviewed by Matt Turck - What Happens When Billions of AI Agents Hit Your Database.
-
Peter Eisentraut - The reverts will continue until morale improves.
-
David Laban and Lambros Petrou (Cloudflare) - Under the hood, off the hot path: how D1 scales to millions of databases.
-
Petros Angelatos - Building a Linearizable Log on READ COMMITTED Isolation.
-
Sai Srirampur - Postgres Isn’t Slow. Your Storage Is.
General Data Stuff 🔗
-
🔥 Excellent writeup from Murat Demirbas on Day 1 of the High Performance Transaction Systems (HPTS) conference.
-
Syamla Bandla and team at Meta have some impressive numbers and interesting analysis of user behaviour during the 2026 World Cup.
-
In the same vein, Channy Yun at AWS has a roundup of mind-boggling numbers from Amazon’s recent Prime Day, including 2.1T HTTP requests, 100s of billions of RDBMS transactions, and so on.
-
Ben Horowitz of a16z gives a (lengthy) recap of where Databricks came from, and his thoughts on its future: Databricks & the Future of the Firm.
AI 🔗
| For calm and well-reasoned analysis of AI and tech in general, I’d recommend Benedict Evans’s newsletter. It’s free to subscribe. |
The risk I’m worried about isn’t a future machine deciding to wipe us out. It’s today’s AI, being wired into everything, carelessly and fast.
Hands-On 🔗
-
Joshua Buss - How I 'write software' as of Oct 2026.
-
🔥 Cindy Sridharan - How to effectively debug when you no longer understand the code.
-
Addy Osmani and colleagues at Anthropic describe How the Claude Code team does code review. This pairs nicely with another article about how Anthropic built their own source control system, Scion. Gergely Orosz also discusses the latter.
-
Robin Wieruch - Toward a Self-Improving Agentic Code Review Loop.
-
Patrick Debois - Context is the New Code.
-
Jina Yoon - WTF is context engineering? (with real examples).
-
Dimitri Mazmanov (Spotify) describes how he cut Claude Code token usage by 90%. James Governor has a nice summary of the article.
-
Ryan Skidmore (Cloudflare) - Cutting our inference bill by 68% by being obsessive about optimization.
Careers, culture, and slop 🔗
-
🔥 Sean Goedecke with a couple of great posts: Getting good at AI takes time and practice, and Software engineers don’t get to be Luddites about AI.
-
Matheus Lima - The Brain Sandwich.
-
Mark Seemann discusses the notion raised by Robert C. Martin of not actually reading the code that’s written by agents.
-
Ian Duncan - Who Owns the Machines?.
-
Tamas Mezei - More Agents, Same Mythical Month (TLDR).
-
Swizec Teller - When code is cheap, judgement becomes the job.
-
Annie Vella - Supervisory Engineering Work.
-
Chris Kiehl - It’s still OK to enjoy programming.
-
Kirill Bobrov - I Miss Writing Code.
-
Sara A. Metwalli - What Makes AI-Written Stories Feel Off?.
-
🔥 Linus Torvalds (for it is he) makes it very clear what he thinks of some of the absolutism around not using AI:
I don’t understand this whole idiotic concept of "don’t use AI for X".
Here’s a new rule just for you: don’t use compilers for any of your kernel submissions.
Because if you can’t show that you really understand what the machine is actually doing by writing your code in assembler, then I don’t want to see your code contribution. You clearly aren’t competent if you can’t write it that way.
[…]
So no. We’re not going to make stupid rules that say "you can’t use tools for X".
The kind of expectations I have is that people do their best and I expect people to understand what they are doing.
But that understanding has very little to do with what tools they use.
-
Daniel Stenberg (of curl fame) explains why we can’t just “ignore the slop”.
AI in Data 🔗
-
Bonnie Xu (OpenAI) - Diving Through Data at OpenAI: How a Data Agent Navigates 70,000 Datasets.
-
Trang NT - Which LLM should run your agentic analytics? Accuracy vs. cost of Claude, Gemini and GPT.
-
Shreya Shankar and colleagues in the Full Stack Data Lab at CMU are building Quail and write about it here: Building an Ultra-High Throughput AI-SQL Engine.
-
Luke Kranz - Can Your LLM Vibe Code Your Semantic Layer?.
-
Paul Iusztin - Build an LLM Wiki From First Principles.
-
Pinterest Engineering - From Activity to Intent: Generating User Journeys with LLMs.
Fun and games 🔗
-
🔥 Simon Willison is one of my favourite bloggers in the AI space. He has great insights, he avoids a lot of the blather and empty calories of some writers, and he also just writes really well. I like how he doesn’t just dump some vibe-coded project and walk away, but uses it to illustrate where the current frontier models are at. This month he’s been partaking in, and commenting on, the inflection point for vibe game development. He built his own very cool game, and shares the details and prompts of quite how, which is educational to read: Battle Drones—a vibe coded RTS with audio-based multiplayer.
-
Popcar - AI is mass decompiling games. That’s both awesome and scary..
-
Christian Mathiesen - Jev plays Pokémon Red.
-
Sarah Perez - People are using AI to make their smart TVs dumb again.
And finally… 🔗
Nothing to do with data, but stuff that I’ve found interesting or has made me smile.
Misc 🔗
-
Charity Majors - If your team is happy, are you doing a good job?.
-
Veronika Durgin - Step Outside Your Bubble. Stay Inside Your Square..
-
Cory Doctorow - Expedia amazoned hotel rooms - "Everything more expensive for everyone, monopolies everywhere, and we’re all screwed".
Security 🔗
Nerd 🔗
-
🔥 Tiny Engineer is a small desktop robot that brings an AI coding assistant to life, physically acting out when it reads files, thinks, writes code, and finishes work.
-
Wiesław Šoltés has created a very realistic version of Visual Basic 6 Studio in the browser. Pointless, but so cool!
-
Another blast of nostalgia with this mock-up of a Windows desktop and getting online via a 56k modem.
-
🔥 I love these kinds of visualisations. I don’t even know what I’m really looking at, other than it’s friggin cool. Space Now.
-
The last deploy is very cool. Kinda similar to the above VB6 and dial-up simulations, but for the modern day (Slack! iMessages!). I’ve not actually played the game through properly, I’m too chicken ;)
|