Consumer group offset - and introduction What is a consumer group offset? It's a way for a consumer to keep track of where it is in the topic. For each topic and partition the consumer group will have one number…
Sometimes your Kafka producer code is doing things that need to be properly validated and of course, we developers resort to writing a test. If the functionality we want to test is nicely encapsulated we can do that using a…
It's been a long waiting but it's finally here: starting with Apache Kafka 2.7 it is now possible to use TLS certificates in PEM format with brokers and java clients. You might wonder - why does it matter? PEM is…
Full guide on working with Protobuf in Apache Kafka
Kafkacat is an awesome tool and today I want to show you how easy it is to use it and what are some of the cool things you can do with it. All the features explained below are available in…
Or, how to produce and consume Kafka records using Avro serialization in Java. So far we've seen how to produce and consume simple String records using Java and console tools. In this post, I would like to show you how…
Avro is an open-source binary serialization format. But why use it with Kafka? Why not send JSON or XML messages? What benefits does it give us? This is what we'll be exploring today. Serialization It's important to understand that records…
When sending messages in a distributed setup using a messaging system, you typically have two scenarios you want to achieve. Either you want to: send a message to a targeted group of consumers (which might be just one consumer) orbroadcast…
Every now and then I get a request from my colleagues who would like to delete some or all the records from a Kafka topic. The request usually comes after someone has produced the wrong data in a test topic…
In this post, I'll show you how to consume Kafka records in Java. We'll read data from a topic called java_topic. To test how our consumer is working, we'll produce data using the Kafka CLI tool. And all this in…