Streaming messages from RabbitMQ into Kafka with Kafka Connect
Analysing network behaviour with ksqlDB and MongoDB
Detecting and Analysing SSH Attacks with ksqlDB
Copy MongoDB collections from remote to local instance
Kafka Connect - Request timed out
A short & sweet blog post to help people Googling for this error, and me next time I encounter it.
The scenario: trying to create a connector in Kafka Connect (running in distributed mode, one worker) failed with the curl
response
HTTP/1.1 500 Internal Server Error
Date: Fri, 29 Nov 2019 14:33:53 GMT
Content-Type: application/json
Content-Length: 48
Server: Jetty(9.4.18.v20190429)
{"error_code":500,"message":"Request timed out"}
Using tcpdump With Docker
I was doing some troubleshooting between two services recently and wanting to poke around to see what was happening in the REST calls between them. Normally I’d reach for tcpdump
to do this but imagine my horror when I saw:
root@ksqldb-server:/# tcpdump
bash: tcpdump: command not found
Common mistakes made when configuring multiple Kafka Connect workers
Streaming data from SQL Server to Kafka to Snowflake ❄️ with Kafka Connect
Running Dockerised Kafka Connect worker on GCP
I talk and write about Kafka and Confluent Platform a lot, and more and more of the demos that I’m building are around Confluent Cloud. This means that I don’t have to run or manage my own Kafka brokers, Zookeeper, Schema Registry, KSQL servers, etc which makes things a ton easier. Whilst there are managed connectors on Confluent Cloud (S3 etc), I need to run my own Kafka Connect worker for those connectors not yet provided. An example is the MQTT source connector that I use in this demo. Up until now I’d either run this worker locally, or manually build a cloud VM. Locally is fine, as it’s all Docker, easily spun up in a single docker-compose up -d
command. I wanted something that would keep running whilst my laptop was off, but that was as close to my local build as possible—enter GCP and its functionality to run a container on a VM automagically.
You can see the full script here. The rest of this article just walks through the how and why.