Kafka for Developers — Part 1

I wrote this blog mainly as a way for me to organise my understanding of Kafka in a logical step-by-step way. It explains how you can install a local Kafka broker for application development, and how to use Kafka in Spring Boot applications. If I don’t run out of enthusiasm, I plan to cover over several articles:

• Installing Kafka on your development machine (this article)
• Spring Boot and Embedded Kafka
• Producers, partitions, consumers
• Serialization and deserialisation
• Authentication and encryption with TLS

In this first article we’ll install Kafka, and we’ll run a single test to verify that it is installed and that we can produce and receive events.

These articles will not teach you how to run a Kafka cluster suitable for production. They also don’t go into detail about whether Kafka is right for your application. Although message queues and Kafka event streams have some overlapping functionality they are not the same. Some use cases are better served by a message queue, and some are better served by an event streamer by Kafka. There’s a good explanation of the differences between message queues and event streams here: RabbitMQ vs Kafka.

Prerequisites

To run the examples here you’ll need:

• git
• Maven
• Java 8
• Docker (Linux) or Docker Desktop (Windows or MacOS) or Podman (Redhat Linux 8.0 or later)

To install Docker Desktop on Windows 10 Home edition, you will need to enable Hyper-V and install Windows Subsystem for Linux 2 (WSL 2) before you can install Docker Desktop.

Running Kafka Locally

A production installation of Kafka will include a minimum of three message brokers and an Apache zookeeper. But for developing and testing an application on your local machine, you can get by with just Zookeeper and a broker.

Rather than installing directly to an OS, you can use Docker to pull down container images and run them. Running Docker images simplifies installation and makes cleaning up much easier — just delete the containers to uninstall everything. It also means the instructions here are the same whether you are using Windows, MacOS or Linux.

Once you have Docker installed, you can run Kafka using the bitnami/kafka image. Clone github repo https://github.com/cyberycon/kafka-howto to get all the example code for these articles.

The 00-docker directory contains a docker-compose.yml file, adapted from one in the bitnami-docker-kafka github repo. This file will start a single broker and zookeeper instance. Treat this as a starting point to customise the installation for your needs. The file looks like this:

docker-compose.yaml

To start your local Kafka instance running, run the following command in the same directory as docker-compose.yml:

docker-compose up -d

The -d option starts the containers detached from the command terminal. You won't see any output, warnings or errors when running containers detached. You can either omit the -d - in which case all the output from the containers' stdout and stderr is sent straight to your console - or you can use the command:

docker logs bitnami-kafka_kafka_1

to display the logs from the Kafka container on the console.

The settings in this docker-compose file make it as simple as possible to start developing code with Kafka:

• Topics are auto-created as soon as referenced by a producer
• All communications are plain text (no encryption)
• Broker and Zookeeper are exposed on your localhost with their normal default ports (9092 and 2181 respectively)
• All data is held on the Docker container local volumes — stop the containers and the data is gone.

These settings are good for development, but not for production. Data and messages are insecure in this setup, and for production the events in Kafka partitions should be backed up or mirrored.

Verify the installation is working

Build the code from github in folder 01-installation-test:

mvn package

This project doesn’t have an application, but it has a single test which attempts to send and receive some messages using the broker we just installed. If you see a Build Success message, everything is working.

The verification project

I generated the verification project using the Spring Initializr. I set the Java level to 8.0, and the only dependency I added was for Kafka. The dependencies element in pom.xml looks like this:

Dependencies section of POM file

The source code for the test is below.

This class contains a single test and some helper methods. The test does the following:

1. Creates a KafkaTemplate (a Spring Framework abstraction for producing messages).
2. Sends two messages to the default topic, “topic-1”. This has the effect of creating the topic if it doesn’t already exist (that is how we configured the Kafka broker in the previous section).
3. Creates a CountDownLatch with a count of 2.
4. Defines a listener on “topic-1”. The onMessage() method is called each time the listener receives a new message, and invokes countDown() on the latch.
5. Defines a listener container, adds the listener to it, and starts the container. Starting the container means the listener connects to the topic and starts waiting for messages.
6. The latch.await(60, TimeUnit.SECONDS) inside the assert will wait until the latch has either counted down to zero, or till 60 seconds have passed. If the latch hasn't counted down in this time, the await method returns false and the test fails.

This code looks very simple, but the producer (KafkaTemplate) and consumer (MessageListener) are running in different threads. This is why there is a Thread.Sleep() before starting the message listener, and why we use a CountDownLatch to count the messages as they are received.

In the next article, we’ll look at how Spring Boot lets us get rid of boilerplate code, and how we can used EmbeddedKafka to write better tests.

Edit: 12 Jan 2021 — Added property ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, “earliest” to consumer props — otherwise test fails first time it is run, and changed teaser for the next article.