tl;dr: specify the --user root argument:

This is a short summary discussing what the options are for integrating Oracle RDBMS into Kafka, as of December 2018 (refreshed June 2020). For a more detailed background to why and how at a broader level for all databases (not just Oracle) see this blog and this talk. What techniques & tools are there? Franck Pachot has written up an excellent analysis of the options available here. As of June 2020, this is what the line-up looks like:

The Kafka Connect JDBC Connector by default does not cope so well with: NUMBER columns with no defined precision/scale. You may end up with apparent junk (bytes) in the output, or just errors. TIMESTAMP WITH LOCAL TIME ZONE. Throws JDBC type -102 not currently supported warning in the log. Read more about NUMBER data type in the Oracle docs. tl;dr : How do I make it work? There are several options:

This article is part of a series exploring Streaming ETL in practice. You can read about setting up the ingest of realtime events from a standard Oracle platform, and building streaming ETL using KSQL. This post shows how we take data streaming in from an Oracle transactional system into Kafka, and simply stream it onwards into Elasticsearch. This is a common pattern, for enabling rapid search or analytics against data held in systems elsewhere.

This is the detailed step-by-step if you want to recreate the process I describe in the Confluent blog here I used Oracle’s Oracle Developer Days VM, which comes preinstalled with Oracle 12cR2. You can see the notes on how to do this here. These notes take you through installing and configuring: Swingbench, to create a sample “Order Entry” schema and simulate events on the Oracle database Oracle GoldenGate (OGG, forthwith) and Oracle GoldenGate for Big Data (OGG-BD, forthwith) I’m using Oracle GoldenGate 12.

Some notes that I made on installing and configuring Oracle GoldenGate with Confluent Platform. Excuse the brevity, but hopefully useful to share! I used the Oracle Developer Days VM for this - it’s preinstalled with Oracle 12cR2. Big Data Lite is nice but currently has an older version of GoldenGate. Login to the VM (oracle/oracle) and then install some useful things: sudo rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm sudo yum install -y screen htop collectl rlwrap p7zip unzip sysstat perf iotop sudo su - cd /etc/yum.

Here’s where I’ll be! If you use Google Calendar you can click on individual entries above and select copy to my calendar - which of course you’ll want to do for all the ones I’ve marked as [SPEAKING] :-) Here’s a list of all the Apache Kafka talks at OpenWorld and JavaOne, most of which I’ll be trying to get to.

The Replicat was kapput: GGSCI (localhost.localdomain) 3> info rkconnoe REPLICAT RKCONNOE Last Started 2017-09-12 17:06 Status ABENDED Checkpoint Lag 00:00:00 (updated 00:46:34 ago) Log Read Checkpoint File /u01/app/ogg/dirdat/oe000000 First Record RBA 0 So checking the OGG error log ggserr.log showed 2017-09-12T17:06:17.572-0400 ERROR OGG-15051 Oracle GoldenGate Delivery, rkconnoe.prm: Java or JNI exception: oracle.goldengate.util.GGException: Error detected handling operation added event. 2017-09-12T17:06:17.572-0400 ERROR OGG-01668 Oracle GoldenGate Delivery, rkconnoe.prm: PROCESS ABENDING. So checking the replicat log dirrpt/RKCONNOE_info_log4j.

There are various reasons for this error, but the one I hit was that the table name is case sensitive, and returned from Oracle by the JDBC driver in uppercase. If you specify the tablename in your connecter config in lowercase, it won’t be matched, and this error is thrown. You can validate this by setting debug logging (edit etc/kafka/connect-log4j.properties to set log4j.rootLogger=DEBUG, stdout), and observe: (I’ve truncated some of the output for legibility)