Kafka Streams Series — 0

I recently got a chance to work with Kafka Streams library which is a very powerful way to develop streaming applications on top of Kafka topics. It provides a fluent API which makes it easy to chain different operations on the data in Kafka topic. Kafka Streams library provides support for Vertical scalability (Scale Up with more threads on a Single Node) or Horizontal scalability (Scale Out with more Nodes).

Mapping data from format to another, Filtering data based on some conditions are some examples which can be done using Kafka Streams library.

Before we start let’s take look at the main classes available in this library.

1. StreamsBuilder — This class has builder methods to create KStream, KTable and GlobalKTable instances.
2. KStream — This class provides all the methods to perform operations such as map, filter etc. on top of a Kafka topic in a streaming manner. A key is not mandatory for KStream. A KStream is stateless meaning it can not used to store and retain calculations, aggregations etc.
3. KTable — This class provides all the methods to perform operations such as map, filter etc. on top of a Kafka topic in tabular key/value manner. A key is mandatory for KTable. A KTable is stateful meaning it can be used to store and retain calculations, aggregations etc.
4. GlobalKTable — This class is similar to KTable but supports to retain state globally across multiple nodes in a distributed horizontally scaled deployment.
5. Topology — This class helps to represent a Directed Acyclic Graph (DAG) of topic sources, processors and sinks which defines how data flows. The StreamsBuilder is responsible for building this.
6. KafkaStreams — This is class that is responsible for creating a running Kafka Streams application using Topology and Properties (Kafka Broker, Number of threads for tasks etc). This class supports starting and stopping a Kafka Stream application.

We will looking more into each of these classes and their functionalities in the upcoming articles. So stay tuned and follow me for exciting articles about Kafka Streams.