Apache Kafka, the event streaming bus that started life at LinkedIn, has become very popular over the last few years. Kafka has lots of strengths — but is it always the right choice?

Kafka can scale to handle very high throughputs — much higher than most message queueing systems. It’s robust, and can be configured to store events persistently for any length of time.

The persistent storage of events within the topic is a powerful feature of Kafka, but it also adds some complexity — you can configure consumers to only read events from after when they joined the topic…

This is second in a series of articles for developers about working with Kafka. We are covering the practicalities of running Kafka on your development machine, how to write code that uses it, and how to write reliable tests. In the previous article (Kafka for Developers — Part 1) we started a local Kafka broker using Docker containers, and wrote a simple JUnit test to show it was working.

All the example code for these articles is at https://github.com/cyberycon/kafka-howto. This repo consists of numbered directories each of which has a separate project. The code for this article is in folder…

I wrote this blog mainly as a way for me to organise my understanding of Kafka in a logical step-by-step way. It explains how you can install a local Kafka broker for application development, and how to use Kafka in Spring Boot applications. If I don’t run out of enthusiasm, I plan to cover over several articles:

  • Installing Kafka on your development machine (this article)
  • Spring Boot and Embedded Kafka
  • Producers, partitions, consumers
  • Serialization and deserialisation
  • Authentication and encryption with TLS

In this first article we’ll install Kafka, and we’ll run a single test to verify that it is installed…

Paul Kelly

Senior Solutions Architect at VMware Tanzu Labs.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store