Sometimes you might want to access Apache Kafka that’s running on your local machine from another device not on the same network. I’m not sure I can think of a production use-case, but there are a dozen examples for sandbox, demo, and playground environments.

In this post we’ll see how you can use ngrok to, in their words, Put localhost on the internet. And specifically, your local Kafka broker on the internet.

The problem - nested messages in Kafka

Data comes into Kafka in many shapes and sizes. Sometimes it’s from CDC tools, and may be nested like this:

• Check out the jmxterm repository
• Download jmxterm from https://docs.cyclopsgroup.org/jmxterm

See also docs. To help future Googlers… with the Confluent docker images for Kafka, KSQL, Kafka Connect, etc, if you want to access JMX metrics from within, you just need to pass two environment variables: <x>_JMX_HOSTNAME and <x>_JMX_PORT, prefixed by a component name. <x>_JMX_HOSTNAME - the hostname/IP of the JMX host machine, as accessible from the JMX Client. This is used by the JMX client to connect back into JMX, so must be accessible from the host machine running the JMX client.

(SO answer repost) You can use kafkacat to send messages to Kafka that include line breaks. To do this, use its -D operator to specify a custom message delimiter (in this example /): kafkacat -b kafka:29092 \ -t test_topic_01 \ -D/ \ -P <<EOF this is a string message with a line break/this is another message with two line breaks! EOF Note that the delimiter must be a single byte - multi-byte chars will end up getting included in the resulting message See issue #140

(This was cross-posted on the Confluent.io blog)

---

This question comes up on StackOverflow and such places a lot, so here’s something to try and help.

tl;dr : You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. Otherwise they’ll try to connect to the internal host address–and if that’s not reachable then problems ensue.

Put another way, courtesy of Spencer Ruport:

LISTENERS are what interfaces Kafka binds to. ADVERTISED_LISTENERS are how clients can connect.

In this article I demonstrated how to use KSQL to filter streams of network event data. As well as filtering, KSQL can be used to easily enrich streams. In this article we’ll see how this enriched data can be used to drive analysis in Elasticsearch and Kibana—and how KSQL again came into use for building some stream processing as a result of the discovery made. The data came from my home Ubiquiti router, and took two forms: