When you run Kafka Connect in distributed mode it uses a Kafka topic to store the offset information for each connector. Because it’s just a Kafka topic, you can read that information using any consumer.

It may be that you want to query this information. You can use a CLI tool like kafkacat:

$ kafkacat -b localhost:9092 -t docker-kafka-connect-offsets -C -K:
["source-debezium-orders-00",{"server":"asgard"}]:{"file":"mysql-bin.000003","pos":154}
["source-debezium-orders-00",{"server":"asgard"}]:{"ts_sec":1556791017,"file":"mysql-bin.000003","pos":9041,"row":1,"server_id":223344,"event":2}

You may want to access this information over HTTP though. Here’s how, using the Confluent REST Proxy. For more cheat-sheet of the REST proxy see here.

  • Create a consumer

    echo '{"name": "rmoff_consumer_instance", "format": "json", "auto.offset.reset": "earliest"}' | \
  http POST http://localhost:8082/consumers/rmoff_consumer_group \
  Content-Type:application/vnd.kafka.v2+json

  • Create a subscription to the offset topic for the consumer. Make sure you put the correct topic name here

    echo '{"topics":["docker-kafka-connect-offsets"]}' | \
  http POST http://localhost:8082/consumers/rmoff_consumer_group/instances/rmoff_consumer_instance/subscription \
  Content-Type:application/vnd.kafka.v2+json

  • Read any new messages from the topic

    http http://localhost:8082/consumers/rmoff_consumer_group/instances/rmoff_consumer_instance/records \
  Accept:application/vnd.kafka.json.v2+json
    HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 154
Content-Type: application/vnd.kafka.json.v2+json
Date: Thu, 02 May 2019 09:50:27 GMT
Server: Jetty(9.4.14.v20181114)
Vary: Accept-Encoding, User-Agent

[
    {
        "key": [
            "source-debezium-orders-00",
            {
                "server": "asgard"
            }
        ],
        "offset": 0,
        "partition": 20,
        "topic": "docker-kafka-connect-offsets",
        "value": {
            "file": "mysql-bin.000003",
            "pos": 154
        }
    }
]