tl;dr: specify the --user root argument:
docker exec --interactive \
--tty \
--user root \
--workdir / \
container-name bashConfluent Cloud is not only a fully-managed Apache Kafka service, but also provides important additional pieces for building applications and pipelines including managed connectors, Schema Registry, and ksqlDB. Managed Connectors are run for you (hence, managed!) within Confluent Cloud - you just specify the technology to which you want to integrate in or out of Kafka and Confluent Cloud does the rest.
Running IBM MQ in a Docker container and the client connecting to it was throwing repeated Channel was blocked errors.
Kafka Connect (which is part of Apache Kafka) supports pluggable connectors, enabling you to stream data between Kafka and numerous types of system, including to mention just a few:
Databases
Message Queues
Flat files
Object stores
The appropriate plugin for the technology which you want to integrate can be found on Confluent Hub.
I was doing some troubleshooting between two services recently and wanting to poke around to see what was happening in the REST calls between them. Normally I’d reach for tcpdump to do this but imagine my horror when I saw:
root@ksqldb-server:/# tcpdump
bash: tcpdump: command not foundI talk and write about Kafka and Confluent Platform a lot, and more and more of the demos that I’m building are around Confluent Cloud. This means that I don’t have to run or manage my own Kafka brokers, Zookeeper, Schema Registry, KSQL servers, etc which makes things a ton easier. Whilst there are managed connectors on Confluent Cloud (S3 etc), I need to run my own Kafka Connect worker for those connectors not yet provided. An example is the MQTT source connector that I use in this demo. Up until now I’d either run this worker locally, or manually build a cloud VM. Locally is fine, as it’s all Docker, easily spun up in a single docker-compose up -d command. I wanted something that would keep running whilst my laptop was off, but that was as close to my local build as possible—enter GCP and its functionality to run a container on a VM automagically.
You can see the full script here. The rest of this article just walks through the how and why.
I use Docker and Docker Compose a lot. Like, every day. It’s a fantastic way to build repeatable demos and examples, that can be torn down and spun up in a repeatable way. But…what happens when the demo that was working is spun up and then tail spins down in a blaze of flames?
Tiny little snippet this one. Given a list of images:
$ docker images|grep confluent
confluentinc/cp-enterprise-kafka 5.0.0 d0c5528d7f99 3 months ago 600MB
confluentinc/cp-kafka 5.0.0 373a4e31e02e 3 months ago 558MB
confluentinc/cp-zookeeper 5.0.0 3cab14034c43 3 months ago 558MB
confluentinc/cp-ksql-server 5.0.0 691bc3c1991f 4 months ago 493MB
confluentinc/cp-ksql-cli 5.0.0 e521f3e787d6 4 months ago 488MB
…Now there’s a new version available, and you want to pull down all the latest ones for it:
docker images|grep "^confluentinc"|awk '{print $1}'|xargs -Ifoo docker pull foo:5.1.0Doing some funky Docker Compose stuff, including:
(This was cross-posted on the Confluent.io blog)
This question comes up on StackOverflow and such places a lot, so here’s something to try and help.
tl;dr : You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. Otherwise they’ll try to connect to the internal host address–and if that’s not reachable then problems ensue.
Put another way, courtesy of Spencer Ruport:
LISTENERSare what interfaces Kafka binds to.ADVERTISED_LISTENERSare how clients can connect.
