Apr 20, 2020
Apr 20, 2020

Using Confluent Cloud when there is no Cloud (or internet)

☁️Confluent Cloud is a great solution for a hosted and managed Apache Kafka service, with the additional benefits of Confluent Platform such as ksqlDB and managed Kafka Connect connectors. But as a developer, you won’t always have a reliable internet connection. Train, planes, and automobiles—not to mention crappy hotel or conference Wi-Fi. Wouldn’t it be useful if you could have a replica of your Cloud data on your local machine?
Apr 20, 2020
Apr 20, 2020

How to install kafkacat on Fedora

kafkacat is one of my go-to tools when working with Kafka. It’s a producer and consumer, but also a swiss-army knife of debugging and troubleshooting capabilities. So when I built a new Fedora server recently, I needed to get it installed. Unfortunately there’s no pre-packed install available on yum, so here’s how to do it manually. Note kafkacat is now known as kcat (ref). When invoking the command you will need to use kcat in place of kafkacat.
Apr 16, 2020
Apr 16, 2020

Converting from AsciiDoc to Google Docs and MS Word

Updated 16 April 2020 to cover formatting tricks & add import to Google Docs info Short and sweet this one. I’ve written in the past how I love Markdown but I’ve actually moved on from that and now firmly throw my hat in the AsciiDoc ring. I’ll write another post another time explaining why in more detail, but in short it’s just more powerful whilst still simple and readable without compilation.
Apr 16, 2020
Apr 16, 2020

A quick and dirty way to monitor data arriving on Kafka

I’ve been poking around recently with capturing Wi-Fi packet data and streaming it into Apache Kafka, from where I’m processing and analysing it. Kafka itself is rock-solid - because I’m using ☁️Confluent Cloud and someone else worries about provisioning it, scaling it, and keeping it running for me. But whilst Kafka works just great, my side of the setup—tshark running on a Raspberry Pi—is less than stable. For whatever reason it sometimes stalls and I have to restart the Raspberry Pi and restart the capture process.

Mar 13, 2020
Mar 13, 2020

Are Tech Conferences Dead?

🦠COVID-19 has well and truly hit the tech scene this week. As well as being full of "WFH tips" for all the tech workers suddenly banished from their offices, my particular Twitter bubble is full of DevRel folk musing and debating about what this interruption means to our profession. For sure, in the short term, the Spring conference season is screwed— all the conferences are cancelled (or postponed). But what about the future?
Mar 11, 2020
Mar 11, 2020

Streaming Wi-Fi trace data from Raspberry Pi to Apache Kafka with Confluent Cloud

Wi-fi is now ubiquitous in most populated areas, and the way the devices communicate leaves a lot of 'digital exhaust'. Usually a computer will have a Wi-Fi device that’s configured to connect to a given network, but often these devices can be configured instead to pick up the background Wi-Fi chatter of surrounding devices. There are good reasons—and bad—for doing this. Just like taking apart equipment to understand how it works teaches us things, so being able to dissect and examine protocol traffic lets us learn about this.
Feb 25, 2020
Feb 25, 2020

Kafka Connect JDBC Sink - setting the key field name

I wanted to get some data from a Kafka topic:

ksql> PRINT PERSON_STATS FROM BEGINNING;
Key format: KAFKA (STRING)
Value format: AVRO
rowtime: 2/25/20 1:12:51 PM UTC, key: robin, value: {"PERSON": "robin",
 "LOCATION_CHANGES":1, "UNIQUE_LOCATIONS": 1}

into Postgres, so did the easy thing and used Kafka Connect with the JDBC Sink connector.

Feb 13, 2020
Feb 13, 2020

Adventures in the Cloud, Part 94: ECS

My name’s Robin, and I’m a Developer Advocate. What that means in part is that I build a ton of demos, and Docker Compose is my jam. I love using Docker Compose for the same reasons that many people do:

  • Spin up and tear down fully-functioning multi-component environments with ease. No bespoke builds, no cloning of VMs to preserve "that magic state where everything works"

  • Repeatability. It’s the same each time.

  • Portability. I can point someone at a docker-compose.yml that I’ve written and they can run the same on their machine with the same results almost guaranteed.

Feb 7, 2020
Feb 7, 2020

Primitive Keys in ksqlDB

ksqlDB 0.7 will add support for message keys as primitive data types beyond just STRING (which is all we’ve had to date). That means that Kafka messages are going to be much easier to work with, and require less wrangling to get into the form in which you need them. Take an example of a database table that you’ve ingested into a Kafka topic, and want to join to a stream of events. Previously you’d have had to take the Kafka topic into which the table had been ingested and run a ksqlDB processor to re-key the messages such that ksqlDB could join on them. Friends, I am here to tell you that this is no longer needed!