Streaming data to Kafka from IBM i

There are several approaches to streaming data from Db2 transactions to Kafka, including but not limited to:

  1. Db2 triggers and Apache Camel: stream events in real-time

  2. Kafka Connect JDBC Source connector: Simple, polling-based technique for importing Db2 data into Kafka

  3. InfoSphere Data Replication and the CDC Replication Engine for Kafka (external link): a CDC-based approach that may be a good options for current IBM CDC customers

  4. Native ILE Kafka client (unsupported) (external link): call Kafka functions directly from ILE programs.

Deploying Kafka on IBM i

These steps walk you through installing Kafka 2.6.0 (built with Scala 2.13) and deploying on an IBM i system using OpenJDK Early Access Builds.

This assumes you are using an SSH terminal and that you have your PATH set up properly, for instance:

export PATH

and you are hopefully, but optionally, using bash

Also, note that this deploys with the default “out of the box” settings for Zookeeper and Kafka. Please refer to the Zookeeper and Kafka documentation to learn about customizing these appropriately for a production deployment as needed.

1. Download requisite software

yum install wget ca-certificates-mozilla gzip tar-gnu openjdk-11 coreutils-gnu sed-gnu grep-gnu

2. Change to your installation directory

cd /home/myusr/mydir

3. Download kafka

wget https://dlcdn.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz

(you may need to update the version number on this and subsequent steps based on the latest version.

4. extract Kafka

tar xzvf kafka_2.13-3.0.0.tgz

5. Set up environment to use OpenJDK

export JAVA_HOME
export PATH

6. Start a Zookeeper server

cd kafka_2.13-3.0.0/config
../bin/zookeeper-server-start.sh zookeeper.properties

7. Open a new session and change to your installation directory from earlier

cd /home/myusr/mydir

8. Set up environment to use OpenJDK and start a Kafka server

export JAVA_HOME
export PATH
cd kafka_2.13-2.6.0/config
../bin/kafka-server-start.sh server.properties

Starting a Kafka visualizer

The Kafka visualizer used for the interactive workshop is an open source, Java-based visualizer that is accessed through a web browser. The project page has more information. It can be installed with docker via

docker pull kbhargava/kafka-visuals

and run like this (substitute host names and port numbers as appropriate).

docker run -p 8080:8080 --rm kbhargava/kafka-visuals idevphp.idevcloud.com:2181 idevphp.idevcloud.com:9092 DEV

If you want to run this visualizer on IBM i, install via yum:

yum install ca-certificates-mozilla
yum install https://github.com/ThePrez/kafka-visualizer/releases/download/v1.0.0/kafka-visualizer-1.0.0-0.ibmi7.3.ppc64.rpm

… and run using the following command and navigate to localhost:8080 on your browser:


You can opt to use any kafka visualizer you’d like. Kafka even comes with a single-topic visualizer that can run in your SSH terminal, for instance

export JAVA_HOME
export PATH
cd kafka_2.13-3.0.0/config
../bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mytopic