DISCLAIMER: I am not a MongoDB person (even if it is Web Scale X-D) - below instructions may work for you, they may not. Use with care! For some work I’ve been doing I wanted to access the data in Ubiquiti’s Unifi controller which it stores in MongoDB. Because I didn’t want to risk my actual Unifi device by changing local settings to enable remote access, and also because the version of MongoDB on it is older than ideal, I wanted to clone the data elsewhere.

Debezium is a CDC tool that can stream changes from MySQL, MongoDB, and PostgreSQL into Kafka, using Kafka Connect. In this article we’ll see how to set it up and examine the format of the data. A subsequent article will show using this realtime stream of data from a RDBMS and join it to data originating from other sources, using KSQL. The software versions used here are: Confluent Platform 4.0 Debezium 0.

io.confluent.ksql.exception.KafkaTopicException: Topic 'KSQL_NOTIFY' does not conform to the requirements Partitions:1 v 4. Replication: 1 v 1 Why? Because the topic KSQL creates to underpin a CREATE STREAM AS SELECT or CREATE TABLE AS SELECT already exists, and doesn’t match what it expects. By default it will create partitions & replicas based on the same values of the input topic. Options: Use a different topic, via the WITH (KAFKA_TOPIC='FOO') syntax, e.g.

This article is part of a series exploring Streaming ETL in practice. You can read about setting up the ingest of realtime events from a standard Oracle platform, and building streaming ETL using KSQL. This post shows how we take data streaming in from an Oracle transactional system into Kafka, and simply stream it onwards into Elasticsearch. This is a common pattern, for enabling rapid search or analytics against data held in systems elsewhere.

System: rmoff@proxmox01:~$ uname -a Linux proxmox01 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 GNU/Linux rmoff@proxmox01:~$ head -n1 /etc/os-release PRETTY_NAME="Debian GNU/Linux 8 (jessie)" rmoff@proxmox01:~$ python --version Python 2.7.9 Following: https://www.confluent.io/blog/introduction-to-apache-kafka-for-python-programmers/ https://github.com/confluentinc/confluent-kafka-python Install librdkafka, which is a pre-req for the Python library: wget -qO - https://packages.confluent.io/deb/4.0/archive.key | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://packages.confluent.io/deb/4.0 stable main" sudo apt-get install librdkafka-dev python-dev Setup virtualenv: sudo apt-get install virtualenv virtualenv kafka_push_notify source .

(This is an expanded version of the intro to an article I posted over on the Confluent blog. Here I get to be as verbose as I like ;)) My first job from university was building a datawarehouse for a retailer in the UK. Back then, it was writing COBOL jobs to load tables in DB2. We waited for all the shops to close and do their end of day system processing, and send their data back to the central mainframe.

This is the detailed step-by-step if you want to recreate the process I describe in the Confluent blog here I used Oracle’s Oracle Developer Days VM, which comes preinstalled with Oracle 12cR2. You can see the notes on how to do this here. These notes take you through installing and configuring: Swingbench, to create a sample “Order Entry” schema and simulate events on the Oracle database Oracle GoldenGate (OGG, forthwith) and Oracle GoldenGate for Big Data (OGG-BD, forthwith) I’m using Oracle GoldenGate 12.

See also Kafka Listeners - Explained A short post to help Googlers. On a single-node sandbox Apache Kafka / Confluent Platform installation, I was getting this error from Schema Registry, Connect, etc: WARN [AdminClient clientId=adminclient-3] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) KSQL was throwing a similar error: KSQL cannot initialize AdminCLient. I had correctly set the machine’s hostname in my Kafka server.

Some notes that I made on installing and configuring Oracle GoldenGate with Confluent Platform. Excuse the brevity, but hopefully useful to share! I used the Oracle Developer Days VM for this - it’s preinstalled with Oracle 12cR2. Big Data Lite is nice but currently has an older version of GoldenGate. Login to the VM (oracle/oracle) and then install some useful things: sudo rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm sudo yum install -y screen htop collectl rlwrap p7zip unzip sysstat perf iotop sudo su - cd /etc/yum.

Here’s where I’ll be! If you use Google Calendar you can click on individual entries above and select copy to my calendar - which of course you’ll want to do for all the ones I’ve marked as [SPEAKING] :-) Here’s a list of all the Apache Kafka talks at OpenWorld and JavaOne, most of which I’ll be trying to get to.