Jan 23, 2020

Notes on getting data into InfluxDB from Kafka with Kafka Connect

You can download the InfluxDB connector for Kafka Connect here. Documentation for it is here.

When a message from your source Kafka topic is written to InfluxDB the InfluxDB values are set thus:

Timestamp is taken from the Kafka message timestamp (which is either set by your producer, or the time at which it was received by the broker)
Tag(s) are taken from the tags field in the message. This field must be a map type - see below
Value fields are taken from the rest of the message, and must be numeric or boolean
Measurement name can be specified as a field of the message, or hardcoded in the connector config.

Jan 21, 2020

Monitoring Sonos with ksqlDB, InfluxDB, and Grafana

I’m quite a fan of Sonos audio equipment but recently had some trouble with some of the devices glitching and even cutting out whilst playing. Under the covers Sonos stuff is running Linux (of course) and exposes some diagnostics through a rudimentary frontend that you can access at http://<sonos player IP>:1400/support/review: Whilst this gives you the current state, you can’t get historical data on it. It felt like the problems were happening "all the time", but were they actually?

May 12, 2016

Monitoring Logstash Ingest Rates with InfluxDB and Grafana

In this article I’m going to show you how to easily monitor the rate at which Logstash is ingesting data, as well as in future articles the rate at which Elasticsearch is indexing it. It’s a nice little touch to add to any project involving Logstash, and it’s easy to do. Logstash is powerful tool for data ingest, processing, and distribution. It originated as simply the pipe to slurp at log files and put them into Elasticsearch, but has evolved into a whole bunch more.

Feb 27, 2016

Streaming data to InfluxDB from any bash command

InfluxDB is a great time series database, that’s recently been rebranded as part of the “TICK” stack, including data collectors, visualisation, and ETL/Alerting. I’ve yet to really look at the other components, but InfluxDB alone works just great with my favourite visualisation/analysis tool for time series metrics, Grafana. Getting data into InfluxDB is easy, with many tools supporting the native InfluxDB line input protocol, and those that don’t often supporting the carbon protocol (from Graphite), which InfluxDB also supports (along with others).